A Visual Grasping Method for Robotic Arms Based on Sim2Real Transfer under Space Constraints

Abstract

This invention provides a robotic arm visual grasping method based on Sim2Real transfer under space-constrained conditions, relating to the fields of robotic arm visual grasping and Sim2Real technology. The method first constructs a task scenario, selects randomization parameters for the visual domain of the simulation environment, and conducts grasping simulation experiments within this scenario to determine the randomization parameters, thereby creating various different simulation environments. Then, a first-stage grasping model is trained in the generated randomized simulation environment, enabling the robotic arm to grasp all objects. Next, a second-stage grasping model is trained in a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from this environment. Finally, the second-stage trained grasping model is used to test target grasping in a space-constrained scenario. This method considers the reality gap between the simulation environment and the real world, exposing the grasping model to various environmental conditions, further improving the model's robustness.

Description

Technical Field

[0001] This invention relates to the fields of robotic arm visual grasping and Sim2Real technology, and in particular to a robotic arm visual grasping method based on Sim2Real migration under space-constrained conditions. Background Technology

[0002] With the continuous development of robotics technology, robot vision grasping technology, as one of the core functions of robot operation, has become an important research direction in fields such as industrial automation, intelligent logistics, and service robots. However, in practical applications, robot vision grasping technology still faces many challenges, especially in crowded and cluttered scenarios, where accurately and efficiently grasping target objects remains an urgent problem to be solved.

[0003] The problem of target grasping in crowded and cluttered real-world scenarios presents several challenges: First, target objects are often occluded or surrounded by other objects, making direct grasping impossible; second, target objects vary in color and shape, requiring accurate identification and stable grasping; and third, the gap between simulation and real-world environments can lead to lower grasping performance in the real world compared to the simulation environment. Current research often employs a "push-grab combination" strategy in a single simulation environment, aiming to disperse all objects through a "push" motion to create sufficient grasping space for the target object. However, these methods are suitable for target grasping tasks in unconstrained workspaces. In confined workspaces, the "push" motion is ineffective because the target object is often tightly surrounded or occluded by surrounding clutter. Furthermore, a single environment cannot meet the demands of real-world environmental changes, resulting in insufficient robustness of grasping models trained in simulation environments. Therefore, exploring new grasping strategies to achieve efficient and accurate target grasping in confined and crowded real-world scenarios is a pressing technical challenge that needs to be addressed.

[0004] Sim2Real transfer is a significant challenge in the fields of robotics and computer vision, involving the transfer of control strategies or models learned in a simulation environment to real-world applications.

[0005] Sim2Real transfer, or the transfer from simulation to reality, refers to the process by which a model, algorithm, or control strategy trained or optimized in a simulation environment can be effectively implemented in the real world, either directly or with minor adjustments. Due to the complexity and uncertainty of the real-world environment, direct training or optimization in the real world is often costly and inefficient; therefore, Sim2Real transfer has become an important research direction.

[0006] Sim2Real transfer has broad application prospects in fields such as robotics, autonomous driving, and augmented reality. For example, in robotics, Sim2Real transfer can be used to train robots capable of performing complex tasks in the real world; in the field of autonomous driving, simulation environments can be used to test and optimize the safety and reliability of autonomous driving algorithms; and in the field of augmented reality, Sim2Real transfer can be used to seamlessly integrate virtual objects with the real world.

[0007] Visual domain randomization is a key technique in Sim2Real transfer learning, aiming to improve the model's generalization ability and robustness by introducing diverse visual features into the simulation environment. The core idea of ​​this method is to randomize the visual parameters in the simulation environment, enabling the model to handle unpredictable changes in the real world during training, such as lighting variations, camera viewpoint, object shape and position, and image noise. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a robotic arm visual grasping method based on Sim2Real migration under space-constrained conditions, thereby realizing the visual grasping of the robotic arm based on Sim2Real migration.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a robotic arm vision grasping method based on Sim2Real migration under space-constrained conditions, comprising:

[0010] A task scenario is constructed, and randomization parameters of the visual domain of the simulation environment are selected. A grasping simulation experiment is conducted in this task scenario to determine the randomization parameters. The randomization parameters include camera pose; brightness, saturation, and contrast of the RGB image captured by the camera; and Gaussian noise in the image.

[0011] By setting randomization ranges for each randomization parameter, various different simulation environments can be created.

[0012] The first-stage grasping model is trained in a generated random simulation environment to enable the robotic arm to grasp all objects. The first-stage grasping model includes a first perception module and a grasping module. The first perception module extracts image features. The grasping module predicts the Q value of each pixel in the image.

[0013] The second-stage grasping model is trained in a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from the environment. The second-stage grasping model includes a second perception module and a grasping module. The second perception module extracts image features, and the grasping module predicts the Q value of each pixel in the image.

[0014] The grasping model trained in the second stage was used to test target grasping in a space-constrained scenario.

[0015] Furthermore, the process of constructing the task scenario involves selecting visual domain randomization parameters for the simulation environment and conducting a grasping simulation experiment within that scenario to determine the randomization parameters, including:

[0016] Multiple objects of different colors and shapes are randomly placed in a workspace, and the task is to grab all of them.

[0017] Based on the camera poses that may occur in the real environment, the three-dimensional position and rotation angle of the camera are randomized to simulate different perspective changes and train a network model deployed on the robotic arm to guide the robotic arm to perform grasping tasks.

[0018] Based on different lighting conditions in the real environment, the visual features of the RGB images captured by the camera are adjusted to enhance the images before training the network model.

[0019] To mitigate the impact of sensor noise in real-world environments, Gaussian noise is added to the RGB images captured by the camera during network model training.

[0020] Furthermore, the randomization range is set for each randomization parameter to create various different simulation environments, including:

[0021] Set the camera position distribution range and camera angle distribution range;

[0022] Set the range of visual feature variations in the RGB images captured by the camera;

[0023] Set the Gaussian noise added during image enhancement;

[0024] Each round of training randomly samples from the above-defined randomized parameter distribution range to generate different parameter configurations that constitute simulation environments with different conditions, so that each round of training of the network model is conducted in a simulation environment with different conditions.

[0025] Furthermore, the camera position distribution range is set to (±0.05, ±0.025, ±0.05)m, that is, the camera randomly selects three-dimensional position coordinates within a cuboid with a length of 0.1m, a width of 0.05m, and a height of 0.1m; the camera angle distribution range is set to (0, ±0.1, ±0.1)rad, that is, the randomization range of the pitch angle and yaw angle is ±0.1rad;

[0026] The range of brightness, saturation, and contrast of the RGB image captured by the camera is set to [0.7, 1.3], that is, it varies between 70% and 130% of the brightness, contrast, and saturation of the original image.

[0027] The Gaussian noise added to the RGB image captured by the camera is set to have a mean μ = 0 and a variance σ = 25.

[0028] Furthermore, the training of the first-stage grasping model in the generated random simulation environment, enabling the robotic arm to grasp all objects, includes:

[0029] Acquire RGB-D images of the scene from a depth camera randomly placed above the workspace;

[0030] Randomly change the brightness, saturation, and contrast of an RGB-D image;

[0031] Add random Gaussian noise to the modified RGB-D image;

[0032] An RGB-D image with added random Gaussian noise is fed into a first perception module to extract image features. Then, a grasping module is used to predict the Q value of each pixel. The Q value represents the expected future reward obtained by grasping at that pixel. The first perception module uses a DenseNet network to extract image features. The grasping module consists of convolution, normalization, and activation layers, and is connected to the perception module through upsampling and skip connection operations to predict the Q value of the pixel.

[0033] Perform a grabbing action at the pixel corresponding to the maximum Q value. If the grabbing is successful, a corresponding reward will be obtained.

[0034] If the set number of grasping actions has been performed, the first stage of grasping model training is stopped, and the trained first stage grasping model is used as the initialization model for the robotic arm grasping task; otherwise, the RGB-D image of the scene is re-acquired from the depth camera randomly placed above the workspace.

[0035] Furthermore, the second-stage training of the grasping model from a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from the cluttered and crowded simulation environment, includes:

[0036] S1: Randomly place M objects of different colors and shapes into a workspace. The task is to grab all objects of one color. Randomly select the color of the target objects to be grabbed and set the number of target objects to be grabbed. Initialize the number of target objects already grabbed to 0.

[0037] S2: Acquire an RGB-D image of a random scene, change its brightness, contrast, saturation, and add random Gaussian noise; send the RGB-D image to the second perception module to extract image features, and then use the grasping module to predict the Q value of each pixel; the second perception module uses the SE-Densenet network to extract image features.

[0038] S3: Obtain a binary mask image containing all target objects, and then use the connected component function to obtain a binary mask image containing only a single target object and its center position.

[0039] S4: Calculate the edge binary mask and edge height map corresponding to the binary mask map of the target object, and then calculate the boundary occupancy rate of the target object;

[0040] S5: Select the target object with the smallest boundary occupancy rate as the current target to be captured;

[0041] S6: If the boundary occupancy rate of the current target to be grabbed is lower than the set threshold γ, then the target will be grabbed; otherwise, the first search will be performed with the target as the center to find grabbable obstacles within the set range around it.

[0042] S7: If there is an obstacle within the set range with a boundary occupancy rate lower than the set threshold γ, then the obstacle will be captured; otherwise, the obstacle with the lowest boundary occupancy rate will be used as the center, and a second search will be performed to capture the obstacle with the lowest boundary occupancy rate within the set range around it.

[0043] S8: If the capture is successful, the corresponding reward will be obtained;

[0044] S9: Increment the number of captured target objects by 1;

[0045] S10: If the set number of grasping actions has been performed, stop training and save the second-stage grasping model; otherwise, determine whether there is still a target object in the workspace: if a target object is detected, repeat steps S2 to S10.

[0046] If no target object is detected and the number of target objects already captured is less than the total number of target objects to be captured, it means that the target object is completely blocked by other obstacles and thus not detected. In this case, all objects in the workspace with a height exceeding the preset threshold are considered as potential objects to be captured. The object corresponding to the maximum Q value is captured until the target object enters the detection range, and the cycle from step S2 to step S10 is started.

[0047] If no target object is detected and the number of target objects already captured is equal to the number of target objects to be captured, then repeat step S1 to start a new round of training.

[0048] Furthermore, the SE-Densenet network is obtained by connecting the DenseNet-121 network and the attention mechanism SENet network: First, the DenseNet-121 network extracts global features of the image, and then global average pooling is used to compress the global features output by the DenseNet-121 network; then, the number of channels in the feature map is reduced to one-quarter of the original number through the first fully connected layer and activated using ReLU, and then the number of channels is restored through the second fully connected layer to model the correlation between channels; then, the importance weights of each channel are normalized by the Sigmoid function to ensure that the weight values ​​are between 0 and 1; finally, the normalized weights are applied to each channel of the feature map, assigning different importance to each channel, so that the SE-Densenet network can focus on key features during prediction.

[0049] On the other hand, the present invention also provides a robotic arm vision grasping system based on Sim2Real migration under space-constrained conditions, including: a simulation environment construction module: setting randomization ranges for various randomization parameters to form a variety of different simulation environments;

[0050] First training module: Train the first stage of the grasping model in a generated random simulation environment to enable the robotic arm to grasp all objects;

[0051] The second training module trains the second-stage grasping model in a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from the cluttered and crowded simulation environment.

[0052] Test module: Use the grasping model trained in the second stage to test target grasping in space-constrained scenarios.

[0053] Thirdly, this application proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the Sim2Real migration-based robotic arm vision grasping method under space-constrained conditions.

[0054] Fourthly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the described space-constrained robotic arm vision grasping method based on Sim2Real migration.

[0055] The beneficial effects of adopting the above technical solution are as follows: The present invention provides a robotic arm visual grasping method based on Sim2Real transfer under space-constrained conditions. (1) The present invention adopts a phased training strategy. The first phase focuses on enabling the robotic arm to have a certain success rate in grasping objects of different shapes and sizes, and to initially master the basic grasping skills. The second phase focuses on coordinating the grasping order of the target and obstacles to improve the efficiency of the action and the success rate of grasping the target, ensuring that the robotic arm can accurately grasp the target object in a complex environment. This phased training strategy reduces the training difficulty of the task and improves the grasping performance of the grasping model.

[0056] (2) The obstacle removal-based target grasping framework provided by this invention selects to grasp obstacles around the target when the target object cannot be directly grasped, thereby reducing the complexity of the workspace and providing sufficient grasping space for the target. This not only improves the success rate of target grasping, but also improves the efficiency of the action.

[0057] (3) In the second stage of training, the present invention proposes a new perceptual network module SE-DenseNet, which embeds the SENet network into the DenseNet module, enhancing the nonlinear modeling ability and feature representation ability of the network. The SENet network enhances the network's response to channels related to object color, shape, position information and object edge contour information by adaptively learning channel attention weights, thereby improving the sensitivity and perception ability of key feature information and further improving the success rate of grasping.

[0058] (4) This invention takes into account the real gap between the simulation environment and the real world. When training the grasping model in the simulation environment, a visual domain randomization method based on expert prior knowledge is used to generate a variety of different random environments, so that the grasping model can be exposed to a variety of different environmental conditions, thereby further improving the robustness of the model. Attached Figure Description

[0059] Figure 1 A framework diagram of a robotic arm vision grasping method based on Sim2Real migration under space-constrained conditions provided in an embodiment of the present invention;

[0060] Figure 2 This is a diagram of the SE-DenseNet network structure for extracting image features provided in an embodiment of the present invention.

[0061] Figure 3 This is a schematic diagram of the search for obstacles provided in an embodiment of the present invention, wherein (a) is the case where the boundary occupancy rate of the target object is lower than the threshold, and (b) is the case where the boundary occupancy rate of the target object is higher than or equal to the threshold;

[0062] Figure 4The diagram illustrates test scenarios with different space sizes provided in the embodiments of the present invention, wherein (a) is a normal working space and (b) is a restricted working space;

[0063] Figure 5 This is a comparison chart of the capture success rate before and after introducing the SENet network, provided for an embodiment of the present invention. Detailed Implementation

[0064] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0065] This invention provides a method applicable to robotic arm vision grasping training, aiming to improve the target grasping performance of robotic arms in real-world space-constrained scenarios. The system framework diagram of this invention is shown below. Figure 1 As shown, the RGB-D height map of the workspace is first rotated 16 times with a step size of 22.5° and then fed into the network. The network outputs 16 Q-value pixel maps with different rotation angles. The Q-value of each pixel represents the expected future reward obtained by performing a grab at that pixel. The rotation angle of the height map corresponds to the angle of rotation of the end effector around the z-axis.

[0066] The entire training process of this invention is carried out in four steps: the first step is to determine the environmental parameters that affect the model performance; the second step is to set a reasonable distribution range for each environmental parameter, in order to ensure the diversity of the simulation environment while avoiding the influence of invalid or extreme parameter configurations; the third step is to train the robotic arm to grasp all objects from a relatively dispersed working environment, in order to initially master the grasping skills of objects of different shapes and sizes; the fourth step is to train the robotic arm to grasp target objects in a more chaotic working environment, in order to improve the efficiency of the action and the success rate of grasping the target.

[0067] In this embodiment, the robotic arm vision grasping method based on Sim2Real migration under space-constrained conditions includes the following steps:

[0068] Step 1: Build a task scenario, select visual domain randomization parameters for the simulation environment based on certain prior knowledge and experience, and conduct a grasping simulation experiment in the task scenario to determine the randomization parameters; the randomization parameters include camera pose; brightness, saturation and contrast of the RGB image captured by the camera; and Gaussian noise in the image;

[0069] Step 1-1: Randomly place 7 objects of different colors and shapes into a workspace. The task is to grab all the objects.

[0070] Steps 1-2: Based on the camera poses that may occur in the real environment, randomize the 3D position and rotation angle of the camera to simulate different viewpoint changes and train a network model deployed on the robotic arm to guide the robotic arm to perform grasping tasks. Compare the network model with a network model that has not been trained with randomized camera poses to verify the effectiveness of randomized camera poses.

[0071] Steps 1-3: Adjust the brightness, saturation, and contrast of the RGB images captured by the camera according to different lighting conditions in the real environment to enhance the image, and then train the network model. Compare the network model with the network model that has not been trained to verify the effectiveness of image enhancement.

[0072] Steps 1-4: Based on the influence of sensor noise in the real environment, Gaussian noise is added to the RGB images captured by the camera during network model training. The training model is then compared with the model trained without adding image noise to verify the effectiveness of adding random noise.

[0073] Step 2: Set reasonable randomization ranges for each randomization parameter to create various different simulation environments;

[0074] Step 2-1: Set the camera position distribution range and camera angle distribution range;

[0075] To reflect possible camera pose changes in the real environment as much as possible, ensure coverage of potential camera position errors in reality, and avoid generating invalid training data due to excessive randomization range, the camera position distribution range is set to (±0.05, ±0.025, ±0.05) m, meaning the camera randomly selects its three-dimensional position coordinates within a cuboid with a length of 0.1 m, a width of 0.05 m, and a height of 0.1 m; the camera angle distribution range is set to (0, ±0.1, ±0.1) rad, meaning the randomization range for pitch and yaw angles is ±0.1 rad.

[0076] Step 2-2: Set the range of visual feature variations in the RGB image captured by the camera;

[0077] The range of brightness, saturation, and contrast of the RGB image captured by the camera is set to [0.7, 1.3], which means that the brightness, contrast, and saturation of the original image vary between 70% and 130%.

[0078] Steps 2-3: Set the Gaussian noise added during image enhancement;

[0079] The Gaussian noise added to the RGB image captured by the camera is set to mean μ = 0 and variance σ = 25.

[0080] Steps 2-4: Each round of training randomly samples from the above-defined randomized parameter distribution range to generate different parameter configurations to form simulation environments with different conditions. This ensures that each round of training of the network model is conducted in a simulation environment with different conditions, allowing the network model to be exposed to a wide variety of scenarios.

[0081] Step 3: Train the first-stage grasping model in the random simulation environment generated in Step 2, so that the robotic arm can grasp all objects; the first-stage grasping model includes a first perception module and a grasping module; the first perception module extracts image features; the grasping module predicts the Q value of each pixel in the image;

[0082] Step 3-1: Acquire an RGB-D image of the scene from a depth camera randomly placed above the workspace;

[0083] Step 3-2: Randomly change the brightness, saturation, and contrast of the RGB-D image;

[0084] Step 3-3: Add random Gaussian noise to the modified RGB-D image;

[0085] Steps 3-4: The RGB-D image with added random Gaussian noise is fed into the perception module network to extract image features. Then, the grasping module is used to predict the Q value of each pixel. The Q value represents the expected future reward obtained by performing grasping at that pixel. The perception module uses the DenseNet network to extract image features. The grasping module consists of convolution, normalization, and activation layers, and is connected to the perception module through upsampling and skip connection operations to predict the Q value of the pixel.

[0086] Steps 3-5: Perform a grabbing action at the pixel corresponding to the maximum Q value. If the grabbing is successful, a corresponding reward will be obtained.

[0087] In this embodiment, the reward function for successfully capturing an item and receiving a reward is set as follows:

[0088]

[0089] Among them, R g The reward function value;

[0090] Step 3-6: If 3000 grasping actions have been performed, stop the training of the first stage grasping model and use the trained grasping model as the initialization model for the robotic arm grasping task; otherwise, go to step 3-1.

[0091] Step 4: Train the second-stage grasping model in a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from the cluttered and crowded simulation environment; the second-stage grasping model includes a second perception module and a grasping module; the second perception module uses the SE-Densenet network to extract image features; the grasping module predicts the Q value of each pixel in the image;

[0092] Step 4-1: Randomly place M objects of different colors and shapes into a workspace. The task is to grab all objects of one color. Randomly select the color of the target objects to be grabbed and set the number of target objects to be grabbed. Initialize the number of already grabbed target objects to 0.

[0093] Step 4-2: Obtain an RGB-D image of a random scene, change its brightness, contrast, saturation, and add random Gaussian noise; send the RGB-D image into the second perception module network to extract image features, and then use the grasping module to predict the Q value of each pixel;

[0094] The SE-DenseNet network is obtained by connecting the DenseNet-121 network and the attention-based SENet network, such as... Figure 2 As shown: First, the DenseNet-121 network extracts global features from the image. Then, global average pooling is used to compress the global features output by the DenseNet-121 network for more efficient feature extraction. Next, the number of channels in the feature map is reduced to one-quarter of its original size using a first fully connected layer and activated with ReLU. Then, the number of channels is restored using a second fully connected layer to model the correlation between channels. After that, the importance weights of each channel are normalized using the Sigmoid function to ensure that the weight values ​​are between 0 and 1, which facilitates subsequent weighting operations. Finally, the normalized weights are applied to each channel of the feature map, assigning different importance to each channel, so that the SE-DenseNet network can focus more on key features during prediction.

[0095] Step 4-3: Obtain a binary mask image containing all target objects, and then use the connected component function to obtain a binary mask image containing only a single target object and its center position;

[0096] Step 4-4: Calculate the edge binary mask and edge height map corresponding to the binary mask of the target object, and then calculate the boundary occupancy rate of the target object;

[0097] The edge binary mask, edge depth-height map, and boundary occupancy rate corresponding to the binary mask of the target object are shown in the following formulas:

[0098]

[0099] Where m represents the binary mask image of the target object's edge, m depth A depth-height map representing the edge of a target object; m obj A binary mask representing the target object, m dilation Binary mask representing the target object Figure 2 The binary mask image after value dilation, H depth This represents the depth-height map, where v represents the number of pixels with a value of 1 in the edge binary mask image m. o The depth-to-height map at the edge is represented by m. depth The number of pixels with a median value that is not zero; r is the boundary occupancy rate of the target object.

[0100] Steps 4-5: Select the target object with the smallest boundary occupancy as the current target to be grabbed;

[0101] Steps 4-6: If the boundary occupancy rate of the current target to be grabbed is lower than the set threshold γ, then grab the target; otherwise, perform the first search with the target as the center to find grabbable obstacles within 10cm around it.

[0102] Steps 4-7: If an obstacle with a boundary occupancy rate lower than the set threshold γ exists within 10cm, then that obstacle is captured; otherwise, using the obstacle with the lowest boundary occupancy rate as the center, a second search is performed to capture the obstacle with the lowest boundary occupancy rate within 10cm around it, such as... Figure 3 As shown;

[0103] Steps 4-8: If the capture is successful, the corresponding reward will be obtained;

[0104] In this embodiment, the reward function for a successful capture is set as follows:

[0105]

[0106] Among them, R g ′ represents the reward function value for a successful capture;

[0107] Steps 4-9: Increment the number of captured target objects by 1;

[0108] Steps 4-10: If 3500 grasping actions have been performed, stop training and save the second-stage grasping model; otherwise, determine whether the target object still exists in the workspace.

[0109] Case 1: If the target object is detected, repeat steps 4-2 to 4-10.

[0110] Case 2: If no target object is detected and the number of target objects already captured is less than the total number of target objects to be captured, it means that the target object is completely blocked by other obstacles and thus not detected. In this case, all objects in the workspace with a height exceeding the preset threshold are considered as potential objects to be captured. The object corresponding to the maximum Q value is captured until the target object enters the detection range, and the loop from step 4-2 to step 4-10 is started.

[0111] Case 3: If no target object is detected and the number of target objects already captured is equal to the number of target objects to be captured, then proceed to step 4-1 to start a new round of training;

[0112] Step 5: Use the second-stage grasping model trained in Step 4 to conduct target grasping test experiments in space-constrained scenarios;

[0113] Step 5-1: Set up two different test scenarios: one is the same as the training scenario in Step 4, and the other is a space-constrained work scenario;

[0114] Step 5-2: Use the grasping model trained in Step 4 to perform grasping tests on the robotic arm in two different test scenarios;

[0115] Step 5-3: Compare the grasping results of Step 5-2 with the baseline scheme of push-grab combination to verify the advantages of the grasping model trained by this invention in target grasping in space-constrained scenarios.

[0116] Step 5-5: Compare the capture results from Step 5-2 with the capture scheme that does not use the Sim2Real migration method to verify the advantages of the visual domain randomization scheme used in this invention.

[0117] To verify the effectiveness of the method of the present invention, this embodiment conducts tests and comparative experiments on the grasping model in both normal and restricted workspaces. The test scenarios are as follows: Figure 4 As shown in Tables 1-2, the comparison results demonstrate that the target grasping success rate and motion efficiency of the method described in this invention are superior to the baseline method, especially in confined workspaces. This is because, in space-constrained scenarios, the "push" action in the baseline method often proves ineffective, as the complexity of the workspace remains consistently high, resulting in insufficient grasping space for the target and thus poor performance of the baseline solution. In contrast, the grasping scheme of this invention effectively reduces the complexity of the workspace by continuously removing obstacles through the "grabbing" action. This not only provides more ample grasping space for the target object but also significantly improves the target grasping success rate and overall motion efficiency.

[0118] Table 1 Normal working space

[0119] method Target capture success rate Action efficiency Task completion rate BPG 79.2％ 67.4％ 100％ Method of the present invention 86.3％ 85.8％ 100％

[0120] Table 2 Constrained Workspace

[0121] method Target capture success rate Action efficiency Task completion rate BPG 70.5％ 40.2％ 96.7％ Method of the present invention 83.5％ 82.6％ 100％

[0122] Secondly, to verify the effectiveness of the perceptual network module proposed in this invention, the model was trained using the network structures before and after adding the SENet network module. As the number of training iterations increased, the model's grasping performance curve was as follows: Figure 5 As shown in the figure, the success rate of grasping at each step represents the success rate of performing grasping actions within the previous 200 steps. The figure shows that the grasping success rate of the model improved by nearly 10% after introducing the attention mechanism SENet, indicating that the network structure proposed in this invention can effectively improve grasping performance.

[0123] Finally, to verify the effectiveness of the proposed visual domain randomization method, the grasping performance with and without the method was compared in a normal simulation workspace. The comparison results are shown in Table 3. The results show that after using the visual domain randomization method, the average grasping success rate was improved accordingly, and the average number of actions was reduced accordingly, indicating that the proposed visual domain randomization method can effectively improve the robustness of the model.

[0124] Table 3 Comparison of grasping performance with and without visual domain randomization in normal simulation workspace

[0125] method Average crawling success rate Average number of moves Task completion rate SE-Densenet 93.1％ 7.60 100％ SE-Densenet+sim2real 95.5％ 7.37 100％

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A robotic arm vision grasping method based on Sim2Real transfer under space-constrained conditions, characterized in that: include: A task scenario is constructed, and the visual domain randomization parameters of the simulation environment are selected. A grasping simulation experiment is conducted in this task scenario to determine the randomization parameters. The randomization parameters include camera pose; brightness, saturation, and contrast of the RGB image captured by the camera; and Gaussian noise in the image. Randomization ranges are set for each randomization parameter to create a variety of different simulation environments. Each round of training randomly samples from the set randomization parameter distribution range to generate different parameter configurations to form simulation environments with different conditions, so that each round of training of the network model is conducted in a simulation environment with different conditions. The first-stage grasping model is trained in a generated random simulation environment to enable the robotic arm to grasp all objects; the first-stage grasping model includes a first perception module and a grasping module; the first perception module extracts image features; The capture module predicts the Q value of each pixel in the image; The second-stage grasping model is trained in a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from the environment. The second-stage grasping model includes a second perception module and a grasping module. The second perception module extracts image features. The capture module predicts the Q value of each pixel in the image; The second-stage training of the grasping model from a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from the cluttered and crowded simulation environment, includes: S1: Randomly place M objects of different colors and shapes into a workspace. The task is to grab all objects of one color. Randomly select the color of the target objects to be grabbed and set the number of target objects to be grabbed. Initialize the number of target objects already grabbed to 0. S2: Acquire an RGB-D image of a random scene, change its brightness, contrast, saturation, and add random Gaussian noise; send the RGB-D image to the second perception module to extract image features, and then use the grasping module to predict the Q value of each pixel; the second perception module uses the SE-Densenet network to extract image features. S3: Obtain a binary mask image containing all target objects, and then use the connected component function to obtain a binary mask image containing only a single target object and its center position. S4: Calculate the edge binary mask and edge height map corresponding to the binary mask map of the target object, and then calculate the boundary occupancy rate of the target object; S5: Select the target object with the smallest boundary occupancy rate as the current target to be captured; S6: If the boundary occupancy rate of the current target to be captured is lower than the set threshold. If the target is found to be a target, then the target will be grabbed; otherwise, the first search will be performed with the target as the center to find grabbable obstacles within a set range around it. S7: If the boundary occupancy rate within the set range is lower than the set threshold. If an obstacle is found to be an obstacle, then that obstacle will be captured; otherwise, a second search will be performed with the obstacle corresponding to the minimum boundary occupancy rate as the center, and the obstacle with the minimum boundary occupancy rate within the set range around it will be captured. S8: If the capture is successful, the corresponding reward will be obtained; S9: Increment the number of captured target objects by 1; S10: If the set number of grasping actions has been performed, stop training and save the second-stage grasping model; otherwise, determine whether there is still a target object in the workspace: if a target object is detected, repeat steps S2 to S10. If no target object is detected and the number of target objects already captured is less than the total number of target objects to be captured, it means that the target object is completely blocked by other obstacles and thus not detected. In this case, all objects in the workspace with a height exceeding the preset threshold are considered as potential objects to be captured. The object corresponding to the maximum Q value is captured until the target object enters the detection range, and the cycle from step S2 to step S10 is started. If no target object is detected and the number of target objects already captured is equal to the number of target objects to be captured, then repeat step S1 to start a new round of training. The grasping model trained in the second stage was used to test target grasping in a space-constrained scenario.

2. The robotic arm vision grasping method based on Sim2Real transfer under space-constrained conditions according to claim 1, characterized in that: The task scenario is constructed by selecting the visual domain randomization parameters of the simulation environment and conducting a grasping simulation experiment in this task scenario to determine the randomization parameters, including: Multiple objects of different colors and shapes are randomly placed in a workspace, and the task is to grab all of them. Based on the camera poses that may occur in the real environment, the three-dimensional position and rotation angle of the camera are randomized to simulate different perspective changes and train a network model deployed on the robotic arm to guide the robotic arm to perform grasping tasks. Based on different lighting conditions in the real environment, the visual features of the RGB images captured by the camera are adjusted to enhance the images before training the network model. To mitigate the impact of sensor noise in real-world environments, Gaussian noise is added to the RGB images captured by the camera during network model training.

3. The robotic arm vision grasping method based on Sim2Real transfer under space-constrained conditions according to claim 2, characterized in that: The randomization range is set for each randomization parameter to create various different simulation environments, including: Set the camera position distribution range and camera angle distribution range; Set the range of visual feature variations in the RGB images captured by the camera; Set the Gaussian noise added during image enhancement.

4. The robotic arm vision grasping method based on Sim2Real transfer under space-constrained conditions according to claim 3, characterized in that: The camera position distribution range is set to (±0.05, ±0.025, ±0.05) m, meaning the camera randomly selects its three-dimensional position coordinates within a cuboid with a length of 0.1m, a width of 0.05m, and a height of 0.1m; the camera angle distribution range is set to (0, ±0.1, ±0.1) rad, meaning the randomization range of the pitch and yaw angles is ±0.1 rad. The range of brightness, saturation, and contrast of the RGB image captured by the camera is set to [0.7, 1.3], that is, it varies between 70% and 130% of the brightness, contrast, and saturation of the original image. The Gaussian noise added to the RGB image captured by the camera is set to have a mean μ=0 and a variance σ=25.

5. The robotic arm vision grasping method based on Sim2Real transfer under space-constrained conditions according to claim 4, characterized in that: The first stage of training the grasping model in the generated random simulation environment enables the robotic arm to grasp all objects, including: Acquire RGB-D images of the scene from a depth camera randomly placed above the workspace; Randomly change the brightness, saturation, and contrast of an RGB-D image; Add random Gaussian noise to the modified RGB-D image; An RGB-D image with added random Gaussian noise is fed into a first perception module to extract image features. Then, a grasping module is used to predict the Q value of each pixel. The Q value represents the expected future reward obtained by grasping at that pixel. The first perception module uses a DenseNet network to extract image features. The grasping module consists of convolution, normalization, and activation layers, and is connected to the perception module through upsampling and skip connection operations to predict the Q value of the pixel. Perform a grabbing action at the pixel corresponding to the maximum Q value. If the grabbing is successful, a corresponding reward will be obtained. If the set number of grasping actions has been performed, the first stage of grasping model training is stopped, and the trained first stage grasping model is used as the initialization model for the robotic arm grasping task; otherwise, the RGB-D image of the scene is re-acquired from the depth camera randomly placed above the workspace.

6. The robotic arm vision grasping method based on Sim2Real transfer under space-constrained conditions according to claim 5, characterized in that: The SE-Densenet network is obtained by connecting the DenseNet-121 network and the attention mechanism SENet network: First, the DenseNet-121 network extracts global features of the image, and then the global features output by the DenseNet-121 network are compressed using global average pooling. Next, the number of channels of the feature map is reduced to one-quarter of the original number through the first fully connected layer and ReLU is used for activation. Then, the number of channels is restored through the second fully connected layer to model the correlation between channels. Then, the importance weights of each channel are normalized using the Sigmoid function to ensure that the weight values ​​are between 0 and 1. Finally, the normalized weights are applied to each channel of the feature map, assigning different importance to each channel, so that the SE-Densenet network can focus on key features during prediction.

7. A robotic arm vision grasping system based on Sim2Real migration under space-constrained conditions, implemented based on the method described in claim 1, characterized in that: include: Simulation environment construction module: Sets the randomization range for each randomization parameter to create a variety of different simulation environments; First training module: Train the first stage of the grasping model in a generated random simulation environment to enable the robotic arm to grasp all objects; The second training module trains the second-stage grasping model in a cluttered and crowded simulation environment, enabling the robotic arm to sequentially grasp target objects from the cluttered and crowded simulation environment. Test module: Use the grasping model trained in the second stage to test target grasping in space-constrained scenarios.

8. An electronic device, comprising: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the Sim2Real migration-based robotic arm vision grasping method under space-constrained conditions as described in claims 1-6.

9. A computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the Sim2Real migration-based robotic arm vision grasping method according to claims 1-6 under space-constrained conditions.

Images