Robot vision closed-loop pouring method and device based on weakly supervised liquid perception

Abstract

The present application relates to a kind of robot vision closed loop pouring method and device based on weakly supervised liquid perception, the method includes the following steps: semi-automatically collect image-level labeled water pouring scene image;Weakly supervised liquid perception model is constructed, to generate class activation map;Class activation map generated by using trained weakly supervised liquid perception model, extract two-dimensional liquid region;Source container and target container pose and size estimation are carried out;Based on two-dimensional liquid region, and pose and size estimation, reconstruct three-dimensional liquid trajectory;According to the pose of target container and three-dimensional liquid trajectory, feedback is formed, controls robot to adjust source container position and pour water into target container, realize vision closed loop control.Compared with prior art, the present application can accurately control liquid not to spill out of target container during pouring.

Description

Technical Field

[0001] This invention relates to the field of robot grasping and manipulation technology, and in particular to a robot vision closed-loop tilting method and device based on weakly supervised liquid perception. Background Technology

[0002] In recent years, robots have made significant progress in their ability to grasp and manipulate rigid objects. However, manipulating non-rigid objects (such as liquids, fabrics, and ropes) remains a major challenge because the inherent flexibility of these objects results in a lack of fixed geometry, making them difficult to perceive. Perceiving liquids from images is particularly challenging because liquid perception relies on light refraction as the primary visual cue, and depth sensors struggle to measure their depth. Therefore, improving robots' ability to perceive liquids, enabling them to better perform service tasks such as cooking, beverage service, and plant watering, has become especially important.

[0003] Pouring water is a highly relevant task in the field of liquid manipulation, which involves many significant challenges. These challenges include: (1) the need for large amounts of pixel-annotated data to facilitate effective training; (2) the lack of prominent visual cues in the images; and (3) the lack of reliable depth measurement systems to achieve accurate liquid pouring.

[0004] To address these challenges, researchers explored using additional sensors to generate realistic labels, including:

[0005] 1) Using thermal imagers and hot water to obtain pixel annotations of real liquid areas; however, this method is very time-consuming and relies on additional equipment.

[0006] 2) Relying on audio rather than visual signals to assist the robot in tipping limits its effectiveness in noisy environments. Furthermore, existing methods calculate the initial tipping point by estimating the container's pose and size, without any liquid sensing as feedback, thus failing to guarantee that the liquid is accurately poured into the target container.

[0007] 3) Self-supervised methods can convert colored liquids into transparent liquids to detect liquid areas without manual labeling. However, this method is limited by the requirement to use colored liquids and statically placed transparent containers, which restricts its applicability in more general environments. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art by providing a robot vision closed-loop pouring method and device based on weakly supervised liquid perception, which can accurately control the liquid from spilling out of the target container during the pouring process.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] According to a first aspect of the present invention, a robot vision closed-loop tipping method based on weakly supervised liquid perception is provided, the method comprising:

[0011] Semi-automatic collection of images of water-pouring scenes with image-level tags;

[0012] A weakly supervised liquid perception model is constructed to generate a class activation map; the two-dimensional liquid region is extracted using the class activation map generated by the trained weakly supervised liquid perception model.

[0013] Perform pose and size estimation for the source and target containers;

[0014] Based on the two-dimensional liquid region, as well as pose and size estimation, a three-dimensional liquid trajectory is reconstructed;

[0015] Based on the pose of the target container and the three-dimensional liquid trajectory, feedback is generated, and the robot is manipulated to adjust the position of the source container to pour water into the target container, thus achieving visual closed-loop control.

[0016] Preferably, the water-pouring scene image includes positive sample data and negative sample data, and the collection processes are as follows:

[0017] Positive sample data: By performing a pouring action, the liquid is poured from the source container to the target container, and the RGB color image frame at the moment the liquid flows out is saved. At the same time, the RGB color image frame of the scene where the faucet on the washbasin is turned on and water is flowing is also saved.

[0018] Negative sample data: The collection process is similar to that of positive sample data, but only source containers without liquid are used, and RGB color image frames of the scene where the sink tap is off and there is no water flow are also saved.

[0019] Preferably, the weakly supervised liquid perception model includes multiple Transformer modules used as the backbone network for feature extraction and a Multilayer Perceptron (MLP) layer for outputting image categories; the Transformer modules include multi-head attention blocks for outputting multiple attention maps.

[0020] Preferably, the liquid monitoring process of the weakly supervised liquid sensing model includes:

[0021] 1) Divide the collected water-pouring scene data into multiple image blocks of a set size, flatten the image blocks and linearly project them into multiple markers;

[0022] 2) The labels are fed into the Transformer module to obtain the corresponding attention map, the expression of which is:

[0023]

[0024] Among them, query key Sum It is formed by projecting multi-head attention blocks, i = 1, 2, ..., n, where n is the number of attention maps, d l and d v are the feature dimensions of the key and value, respectively, and h and w are the height and width of the image;

[0025] 3) Concatenate all attention maps and feed them into the feedforward layer to obtain feature maps;

[0026] 4) Determine the final category based on the weight ratio of each feature in the feature map. The final class activation graph A is obtained by weighted generation. c ; where j is the number in the feature map.

[0027] Preferably, the weakly supervised liquid perception model is optimized and trained using the backpropagation algorithm and the gradient descent algorithm; its loss function includes a classification loss and a feature contrast loss function, wherein the feature contrast loss function includes a positive contrast loss and a contrast loss, which are used to cluster features of similar regions and separate foreground and background features, respectively; the positive contrast loss L pos Contrast loss L neg The expressions are as follows:

[0028]

[0029]

[0030] In the formula, This represents the local foreground feature corresponding to pixel position i. Let be the local background feature corresponding to pixel position j, and m and n be the number of local features in the foreground and background, respectively.

[0031] Preferably, the step of extracting the two-dimensional liquid region using the category activation map generated by the trained weakly supervised liquid perception model specifically involves: selecting a threshold to binarize the activation map based on the category activation map generated by the trained weakly supervised liquid perception model to obtain the potential two-dimensional liquid region, and extracting the refined two-dimensional liquid region from the potential two-dimensional liquid region mask M through a morphological skeleton.

[0032] Preferably, the step of estimating the pose and size of the source container and the target container specifically involves:

[0033] Given a source container, a target container, and an RGB-D image, the mask of the container in the image is first detected by a 2D detection network. Then, it is fed into a class-level object pose and size estimation network to estimate the pose and size, thus obtaining the container's three-degree-of-freedom position, three-degree-of-freedom rotation, and three-dimensional size.

[0034] Preferably, the reconstructing of the three-dimensional liquid trajectory specifically includes:

[0035] Given the pose of the source container {R} x |R y |R z ;t}, the equation of the plane aligned with the container opening and the direction of gravity:

[0036] (pt) T ·R z⊥g =0,R z =R z⊥g +R z||g

[0037] Where p is a point on the plane, R z⊥g and R z||g They are R z Components perpendicular to and parallel to the gravity vector g;

[0038] Given the camera's intrinsic parameters K, the ray projected back from the camera's center is denoted as:

[0039] r(u,v)=o+κd,d=K -1 ·[u,v,1] T

[0040] In the formula, o is the starting point of the ray; κ is a scalar defined as any position along the ray; d is the direction vector of the ray.

[0041] The final 3D intersection point of the ray and the plane is:

[0042]

[0043] After obtaining the pixel coordinates (u,v) of the potential two-dimensional liquid region mask M, the corresponding three-dimensional liquid point cloud R = {p(u,v)|u,v∈M} is estimated.

[0044] Preferably, the visual closed-loop control based on the pose of the target container and the calculation error at the end of the three-dimensional liquid trajectory specifically involves:

[0045] First, based on the pose and size of the source and target containers, the initial point for pouring the liquid is calculated. After the liquid flows out, the three-dimensional trajectory of the liquid is obtained. Then, the error is calculated based on the pose of the target container and the end of the three-dimensional trajectory. This error is sent to the controller to calculate the corresponding motion of the robot, completing real-time closed-loop tracking and ensuring that the liquid flows into the target container without spilling.

[0046] According to a second aspect of the present invention, a robot vision closed-loop tipping device based on weakly supervised liquid perception is provided, employing any of the methods described above, the device comprising:

[0047] The water-pouring scene data collection unit is used for semi-automatic collection of water-pouring scene images with image-level tags;

[0048] A weakly supervised liquid sensing unit is used to sense potential two-dimensional liquid regions from images of a water-pouring scene;

[0049] The object pose and size estimation unit is used to estimate the six-DOF pose and three-dimensional dimensions of the source container and the target container;

[0050] The 3D liquid trajectory reconstruction unit is used to combine the pose and size of the detected 2D liquid region to reconstruct a 3D point cloud;

[0051] The vision-based closed-loop control unit generates feedback based on the pose of the 3D liquid and the target container, and controls the robot to adjust the position of the source container to pour water into the target container.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] 1) This invention proposes a weakly supervised liquid perception method for the first time. Even when the object's pose is not measurable in terms of liquid depth, the trajectory of the liquid can be perceived in real time in three dimensions. The robot is guided to complete the task of accurately pouring various liquids in a visual closed-loop manner, ensuring that the liquid does not spill out of the target container during the pouring process.

[0054] 2) Semi-automatic collection of water-pouring scene data with image-level labels. Only image-level labels are needed, reducing the tedious and time-consuming data labeling process when users train the liquid perception network.

[0055] 3) Train a weakly supervised liquid perception model to generate class activation maps to guide the network to focus its attention on the visual differences between the two types of collected data, i.e. whether there is flowing liquid, so as to capture and learn the visual lines related to liquid in the image with higher accuracy.

[0056] 4) Employ a feature contrast strategy to improve the quality of the weakly supervised liquid perception model, thereby outputting an activation map that completely and closely covers the actual liquid region;

[0057] 5) This invention can be applied to the field of home service robots, such as cooking, serving tea and water, watering plants, and preparing biological reagents in medical scenarios, thereby improving the robot's operational capabilities in different service scenarios. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method of the present invention;

[0059] Figure 2 This is a flowchart of the method in the embodiment;

[0060] Figure 3This is a network structure diagram of the robot vision closed-loop tilting method in the embodiment;

[0061] Figure 4 This is a schematic diagram of the three-dimensional liquid reconstruction method in the embodiment;

[0062] Figure 5 This is a schematic diagram of the robot vision closed-loop tilting method in the embodiment. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0064] Example

[0065] like Figure 2 As shown in the figure, this embodiment presents a robot vision closed-loop tipping method based on weakly supervised liquid perception. The method includes the following steps:

[0066] Step S1: Semi-automatically collect water-pouring scene data with image-level labels. Specific steps are as follows:

[0067] (1) Positive sample data: by performing a pouring action by a robot or human, the liquid is poured from the source container to the target container, and the RGB color image frame at the moment the liquid flows out is saved. At the same time, the RGB color image frame of the scene where the faucet on the washbasin is turned on and water flows is also saved.

[0068] (2) Negative sample data was collected in a similar manner to (1), but only source containers without liquid were used. RGB color image frames of a scene where the faucet on the sink was turned off and there was no water flow were also saved. Finally, to balance the number of positive and negative samples, the data volume of both samples was kept consistent.

[0069] Step S2: Design a weakly supervised liquid sensing model. For example... Figure 3 As shown, the specific steps are as follows:

[0070] The liquid perception framework uses a Transformer as the backbone network to extract features and a Multilayer Perceptron (MLP) layer to output the image category. First, the RGB image... The image is divided into multiple 4×4 blocks, where H is the image height and W is the image width. These blocks are then flattened and linearly projected into multiple tokens, which are fed into the Transformer module. The Transformer module includes multi-head self-attention blocks, thus outputting multiple attention maps. For each self-attention head, these tokens are projected into a query. Key Sum of values ​​(Value) i = 1, 2, ..., n, where n is the number of attention maps, and h and w are the height and width of the image, respectively. Then, the attention maps are obtained. in It is a scale factor, d k and d v These represent the feature dimensions of the keys and values, respectively. Finally, all attention maps are concatenated and fed into a feedforward layer to obtain the feature maps. The model consists of four Transformer blocks, each outputting a corresponding feature map F. (i) i = 1, 2, 3, 4. The sizes of these feature maps are respectively the original image size.

[0071] Step S3: Generate the category activation map. (e.g.) Figure 3 As shown, the specific steps are as follows:

[0072] The classification network classifies each feature in the final feature map. Weighting of the final category determination The weighted generation yields the final Class Activation Map (CAM). c In binary classification tasks, it can be simplified as follows: Where w j These are the weight matrix elements in the fully connected layer. ReLU is used to remove negative activation regions. Finally, the activation map is normalized to the (0,1) interval.

[0073] Step S4: Construct the loss function, including classification loss and feature contrast loss function. The specific steps are as follows:

[0074] Step S4.1: Construct the classification loss function. Since the model is a binary classification model, the binary cross-entropy loss function is used as the classification loss L. cls .

[0075] Step S4.2: Construct the feature contrast loss function. After obtaining the activation map A in step S3, the corresponding pixel position i is obtained according to A ≥ ∈ (∈ = 0.7). Then, the same size feature map F (4) In the diagram, the local foreground feature corresponding to pixel position i is: Where d is the feature map F (4) The dimension. Similarly, based on A < ∈ (∈ = 0.7), the corresponding pixel position j is obtained, and the corresponding local background feature is obtained as To reduce the distance between local features in the foreground and increase the distance between foreground and background features, the cosine similarity metric sim(·,·) is used to measure the distance between local features.

[0076] For positive contrast loss, the goal is to cluster features from similar regions using a loss function. For negative contrast loss, the goal is to separate foreground and background features to obtain a more complete region belonging to the actual flowing liquid, using a loss function. m and n are the number of local features in the foreground and background, respectively. Finally, the network's loss function L = L cls +L pos +L neg .

[0077] Step S5: Train the network model. The specific steps are as follows:

[0078] The network was trained using the backpropagation algorithm with gradient descent optimization. Specifically, the AdamW optimizer was used to optimize the network, with an initial learning rate of 6 × 10⁻⁶ for the parameters of the Transformer backbone network. -5 For the parameters of the fully connected layer, the optimizer's initial learning rate is set to 6 × 10. -4 The training process involved approximately 14,000 iterations until convergence.

[0079] Step S6: Two-dimensional liquid region extraction. This includes classification loss and feature contrast loss functions. The specific steps are as follows:

[0080] Based on the category activation map A obtained in step S3, the activation map is binarized by selecting a threshold A≥σ (σ=0.5) to obtain the potential two-dimensional liquid region M, and the liquid region is refined by morphological skeleton extraction of the mask M.

[0081] Step S7: Pose and size estimation of the source and target containers. The specific steps are as follows:

[0082] Given a source container, a target container, and an RGB-D image, the mask of the container in the image is first detected by a 2D detection network (such as Mask-RCNN), and then fed into a class-level object pose and size estimation network (such as SAR-Net[5]) for pose and size estimation. The three-degree-of-freedom position, three-degree-of-freedom rotation, and three-dimensional size of the container are obtained.

[0083] Step S8: Reconstruct the 3D liquid trajectory by combining the pose of the source container. For example... Figure 4 As shown, the specific steps are as follows:

[0084] Given the pose of the source container {R} x |R y |R z The equation of the plane aligned with the container opening and the direction of gravity, denoted as t}, can be written as: (pt) T ·R z⊥g =0,R z =R z⊥g +R z||g Where p is a point on the plane, R z⊥g and R z||g They are R z The components perpendicular to and parallel to the gravity vector g. Given the camera intrinsic parameters K, the ray projected back from the camera center can be denoted as: r(u,v)=o+kd,d=K -1 ·[u,v,1] T In the formula, o is the starting point of the ray; κ is a scalar defined as any position along the ray; and d is the direction vector of the ray. The final 3D intersection point of the ray and the plane is... Therefore, after obtaining the pixel coordinates (u,v) of the potential liquid mask M in the image, the three-dimensional point cloud of the liquid in the region, P = {p(u,v)|u,v∈M}, can be approximately estimated according to the above formula.

[0085] Step S9, visual closed-loop control. (e.g.) Figure 3 and Figure 5 As shown, the specific steps are as follows:

[0086] First, based on the pose and size of the source and target containers obtained in step S7, the initial point for liquid pouring is calculated to pour the liquid. After the liquid flows out, the three-dimensional trajectory of the liquid is obtained according to steps S6-8. Then, the error is calculated based on the pose of the target container and the end of the three-dimensional trajectory. This error is sent to the PID controller to calculate the corresponding motion of the robot, completing real-time closed-loop tracking and ensuring that the liquid can flow into the target container without spilling.

[0087] Furthermore, for greater convenience in practical use, the two-dimensional liquid detection network trained through steps S1 to S5 can be packaged into a two-dimensional liquid detection unit. This two-dimensional liquid detection unit, together with the scene image acquisition unit, pose and size estimation unit, liquid 3D reconstruction unit, and visual closed-loop control unit, constitutes a robot visual closed-loop pouring device based on weakly supervised liquid perception (e.g., a computer storing executable programs for the two-dimensional liquid detection unit, scene image acquisition unit, pose estimation unit, liquid 3D reconstruction unit, and visual closed-loop control unit). After the scene image acquisition unit acquires the scene image, the two-dimensional liquid detection unit, pose estimation unit, and liquid 3D reconstruction unit ultimately reconstruct the detected liquid, and the error between the 3D liquid and the target container is sent to the visual closed-loop control unit to complete the precise pouring of the liquid.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot vision-based closed-loop tipping method based on weakly supervised liquid perception, characterized in that, The method includes: Semi-automatic collection of images of water-pouring scenes with image-level tags; A weakly supervised liquid perception model is constructed to generate a class activation map; the two-dimensional liquid region is extracted using the class activation map generated by the trained weakly supervised liquid perception model. Perform pose and size estimation for the source and target containers; Based on the two-dimensional liquid region, as well as pose and size estimation, a three-dimensional liquid trajectory is reconstructed; Based on the pose of the target container and the three-dimensional liquid trajectory, feedback is generated to control the robot to adjust the position of the source container and pour water into the target container, thus achieving visual closed-loop control. The weakly supervised liquid perception model includes multiple Transformer modules used as the backbone network for feature extraction and a multilayer perceptron layer for outputting image categories; the Transformer modules include multi-head attention blocks for outputting multiple attention maps. The liquid monitoring process of the weakly supervised liquid sensing model includes: 1) Divide the collected water-pouring scene data into multiple image blocks of a set size, flatten the image blocks and linearly project them into multiple markers; 2) The labels are fed into the Transformer module to obtain the corresponding attention map, the expression of which is: Among them, query ,key Sum It is formed by projecting multi-head attention blocks. , For the number of attention maps, and These are the feature dimensions for keys and values, respectively. The height and width of the image; 3) Concatenate all attention maps and feed them into the feedforward layer to obtain feature maps; 4) Determine the final category based on the weight ratio of each feature in the feature map. The final class activation graph is obtained by weighted generation. ;in, This refers to the number in the feature map.

2. The robot vision closed-loop tipping method based on weakly supervised liquid perception according to claim 1, characterized in that, The water-pouring scene images include positive sample data and negative sample data, and the collection process is as follows: Positive sample data: By performing a pouring action, the liquid is poured from the source container to the target container, and the RGB color image frame at the moment the liquid flows out is saved. At the same time, the RGB color image frame of the scene where the faucet on the washbasin is turned on and water is flowing is also saved. Negative sample data: Only source containers without liquid are used, while also saving RGB color image frames of a scene where the sink tap is off and there is no water flow.

3. The robot vision closed-loop tipping method based on weakly supervised liquid perception according to claim 1, characterized in that, The weakly supervised liquid sensing model is optimized and trained using the backpropagation algorithm and the gradient descent algorithm. Its loss function includes a classification loss and a feature contrast loss function. The feature contrast loss function includes positive contrast loss and contrast loss, which are used to cluster features of similar regions and separate foreground and background features, respectively. Comparison with negative loss The expressions are as follows: In the formula, pixel position Corresponding local foreground features pixel position Corresponding local background features and These represent the number of local features in the foreground and background, respectively.

4. The robot vision closed-loop tipping method based on weakly supervised liquid perception according to claim 1, characterized in that, The extraction of two-dimensional liquid regions using the category activation map generated by the trained weakly supervised liquid perception model specifically involves: selecting a threshold to binarize the activation map based on the category activation map generated by the trained weakly supervised liquid perception model to obtain the potential two-dimensional liquid region; and then masking the potential two-dimensional liquid region using a morphological skeleton. Extract the refined two-dimensional liquid region.

5. The robot vision closed-loop tipping method based on weakly supervised liquid perception according to claim 1, characterized in that, The estimation of the pose and size of the source and target containers specifically involves: Given a source container, a target container, and an RGB-D image, the mask of the container in the image is first detected by a 2D detection network. Then, it is fed into a class-level object pose and size estimation network to estimate the pose and size, thus obtaining the container's three-degree-of-freedom position, three-degree-of-freedom rotation, and three-dimensional size.

6. The robot vision closed-loop tipping method based on weakly supervised liquid perception according to claim 1, characterized in that, The reconstructed three-dimensional liquid trajectory is specifically as follows: Given the pose of the source container The equation of the plane aligned with the container opening and the direction of gravity: in, It is a point on a plane. and They are Perpendicular to and parallel to the gravity vector The amount; Given camera intrinsic parameters The ray projected back from the center of the camera is denoted as: In the formula, The origin of the ray; is a scalar, defined as any position along the ray; It is the direction vector of the ray; The final 3D intersection point of the ray and the plane is: Obtain a mask for a potential two-dimensional liquid region. pixel coordinates Then, the corresponding liquid 3D point cloud is estimated. .

7. The robot vision closed-loop tipping method based on weakly supervised liquid perception according to claim 1, characterized in that, Based on the calculation error of the target container's pose and the end of the 3D liquid trajectory, visual closed-loop control is performed, specifically as follows: First, based on the pose and size of the source and target containers, the initial point for pouring the liquid is calculated. After the liquid flows out, the three-dimensional trajectory of the liquid is obtained. Then, the error is calculated based on the pose of the target container and the end of the three-dimensional trajectory. This error is sent to the controller to calculate the corresponding motion of the robot, completing real-time closed-loop tracking and ensuring that the liquid flows into the target container without spilling.

8. A robot vision closed-loop tilting device based on weakly supervised liquid perception, characterized in that, The apparatus comprising the method according to any one of claims 1 to 7, wherein the apparatus includes: The water-pouring scene data collection unit is used for semi-automatic collection of water-pouring scene images with image-level tags; A weakly supervised liquid sensing unit is used to sense potential two-dimensional liquid regions from images of a water-pouring scene; The object pose and size estimation unit is used to estimate the six-DOF pose and three-dimensional dimensions of the source container and the target container; The 3D liquid trajectory reconstruction unit is used to combine the pose and size of the detected 2D liquid region to reconstruct a 3D point cloud; The vision-based closed-loop control unit generates feedback based on the pose of the 3D liquid and the target container, and controls the robot to adjust the position of the source container to pour water into the target container.

Images