Robotic arm grasping model domain randomization training method, system, device and medium

Abstract

The present application relates to the technical field of intelligent grasping, and particularly relates to a mechanical arm grasping model domain randomization training method, system, device and medium, the method comprising: rendering and generating multi-view virtual point cloud data of a target object in a simulation environment, applying physical characteristic domain randomization processing to the virtual point cloud data, randomizing at least two dimensions of depth noise distribution, point cloud density distribution, occlusion mode and surface reflection characteristics of the virtual point cloud; automatically generating a grasping label for each virtual point cloud sample in the simulation environment, the grasping label being determined by collision detection of the object state after the virtual gripper performs a grasping action through a simulation physics engine, the randomized virtual point cloud data and the corresponding grasping label being combined to form a training data set; training a grasping pose generation model, the grasping pose generation model taking the randomized virtual point cloud as input and taking grasping pose parameters as output; and deploying the trained grasping pose generation model to a real mechanical arm system.

Description

Technical Field

[0001] This invention relates to the field of intelligent grasping technology, and in particular to a method, system, device and medium for randomized training of a robotic arm grasping model domain. Background Technology

[0002] Deep learning-based robotic arm grasping posture generation models require a large amount of labeled training data to achieve usable accuracy. In real-world scenarios, collecting and labeling point cloud grasping data faces challenges such as high cost, low efficiency, and poor labeling consistency. A single grasping experiment requires robotic arm movement, depth camera acquisition, and manual labeling of success / failure states. Data collection typically consumes a significant amount of time, and the judgment criteria for manual labeling vary from person to person, easily introducing label noise. Simulation environments can automatically generate labeled data on a large scale, becoming an important way to overcome the data bottleneck.

[0003] However, existing technologies suffer from the following problem: there is a significant "domain gap" between the virtual point cloud rendered in the simulation environment and the point cloud acquired by a real depth camera. This domain gap mainly stems from the mismatch in the physical properties of the point cloud, causing the performance of the simulation-trained grasping model to degrade significantly when deployed to a real robotic arm and faced with real point cloud inputs with different noise, density, occlusion, and reflection characteristics, resulting in a low zero-sample transfer success rate.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] This invention provides a method, system, device, and medium for randomized training of a robotic arm grasping model domain, thereby effectively solving the problems in the background art.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a randomized training method for a robotic arm grasping model domain, comprising the following steps:

[0007] Multi-view virtual point cloud data of the target object is generated in a simulation environment. Physical property domain randomization processing is applied to the virtual point cloud data. The physical property domain randomization processing includes randomizing at least two dimensions of the virtual point cloud's depth noise distribution, point cloud density distribution, occlusion mode, and surface reflection characteristics.

[0008] In the simulation environment, a grasping label is automatically generated for each virtual point cloud sample. The grasping label is determined by collision detection of the object state after the virtual gripper performs the grasping action through the simulation physics engine. The randomized virtual point cloud data and the corresponding grasping label are combined to form a training dataset.

[0009] The grasping pose generation model is trained using the training dataset. The grasping pose generation model takes randomized virtual point cloud as input and grasping pose parameters as output.

[0010] The trained grasping posture generation model is deployed to a real robotic arm system to achieve zero-sample grasping posture generation of point clouds in real scenes.

[0011] Furthermore, the simulation environment includes a physical simulation engine and a virtual depth camera model;

[0012] The intrinsic parameters of the virtual depth camera model are consistent with those of the real depth camera, and the physical simulation engine is used to simulate rigid body dynamics and collision detection during the gripper's grasping of an object.

[0013] Furthermore, the randomization of the depth noise distribution employs a hybrid model of Gaussian noise and impulse noise, the expression of which is:

[0014] ;

[0015] in This represents Gaussian noise with a standard deviation of σ, where σ ranges from a preset range. sampling; Let A represent impulse noise with amplitude A that occurs with probability p, where both p and A are randomly sampled from their respective preset ranges.

[0016] Furthermore, the randomization of the point cloud density distribution is achieved by simulating the point cloud sparsity effect under different measurement distances and incident angles:

[0017] For each point in the virtual point cloud, based on the distance d from the point to the virtual depth camera and the angle θ between the normal vector of the surface where the point is located and the camera's optical axis, the point is randomly deleted according to the following formula:

[0018] ;

[0019] Where α is the distance attenuation coefficient, from the preset range sampling, The probability of retaining this point is given when the generated random number is less than 1. Keep the point if it is valid, otherwise delete the point.

[0020] Furthermore, the randomization of the occlusion pattern includes:

[0021] Several occlusion objects are randomly placed in a virtual scene. The size, shape, position, and orientation of the occlusion objects are randomly sampled from a preset distribution, so that the target object generates random self-occlusion and mutual occlusion areas, simulating the situation in a real scene where an object is occluded by other objects or grippers.

[0022] Furthermore, the randomization of the surface reflection characteristics is achieved by randomly assigning bidirectional reflection distribution function parameters to each surface of the target object;

[0023] The bidirectional reflectance distribution function parameters include diffuse reflectance, specular reflectance, and surface roughness. Each parameter is sampled independently from its own preset distribution, enabling the virtual point cloud to cover a variety of material representations from matte to highly reflective.

[0024] Furthermore, the automatic generation of the crawling tags includes:

[0025] In the simulation environment, the virtual gripper is controlled to close according to the candidate gripping posture. If the object is stably gripped after closing and does not fall within the preset lifting height, the candidate gripping posture is marked as a success.

[0026] If an object bounces off, slips off, or is not gripped during the clamping process, it is marked as a failure.

[0027] Furthermore, the physical property domain randomization process independently samples each randomization parameter each time a virtual point cloud sample is generated, so that the combination of physical property parameters for each sample in the training dataset is different from each other, forming an enhanced dataset that covers the range of physical property changes in the real scene.

[0028] Furthermore, the method also includes: simultaneously introducing virtual point clouds of multi-object scenes of the target object under different stacking numbers, different stacking densities and different background clutter levels into the training dataset, so that the grasping posture generation model can adapt to different scene complexities from single objects to dense stacking.

[0029] Furthermore, the method can be applied to the handling of messy parts in industrial sorting scenarios, the handling of mixed boxes and depalletized boxes in logistics warehousing scenarios, or the handling of everyday items in service robot scenarios.

[0030] The present invention also includes a randomized training device for a robotic arm grasping model domain, which applies the method described above. Further, the device includes:

[0031] The simulation point cloud generation module is used to render and generate multi-view virtual point cloud data of target objects in a simulation environment.

[0032] The physical property domain randomization module is used to apply randomization processing to the virtual point cloud data in at least two dimensions of depth noise distribution, point cloud density distribution, occlusion mode and surface reflection characteristics.

[0033] The automatic annotation and dataset construction module is used to automatically generate capture labels for each virtual point cloud sample in the simulation environment, and to combine the randomized virtual point cloud data and the corresponding capture labels into a training dataset.

[0034] The model training module is used to train the grasping pose generation model using the training dataset;

[0035] The deployment migration module is used to deploy the trained grasping posture generation model to a real robotic arm system to achieve zero-sample grasping posture generation.

[0036] Furthermore, the simulated point cloud generation module includes a physical simulation engine unit and a virtual depth camera rendering unit. The physical simulation engine unit is used to simulate rigid body dynamics and collision detection, and the virtual depth camera rendering unit is used to generate a virtual point cloud that is consistent with the intrinsic parameters of a real depth camera.

[0037] Furthermore, the device also includes a real robotic arm execution module, which includes a multi-degree-of-freedom robotic arm, a parallel gripper-type end effector, and a depth camera, for acquiring point clouds in a real scene and executing the grasping posture output by the grasping posture generation model.

[0038] The present invention also includes a computer device comprising a processor and a memory coupled to the processor and the memory, the memory storing program instructions which, when executed by the processor, implement the method described above.

[0039] The present invention also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0040] The present invention also includes a computer program product comprising a computer program that, when run on a computer, causes the computer to perform the methods described above.

[0041] The beneficial effects of this invention are as follows: By applying four-dimensional physical property domain randomization to the simulated virtual point cloud, the simulated training data covers the range of point cloud physical property variations under various conditions from a real depth camera. Since the grasping model has already encountered various combinations of noise, density, occlusion, and material during the training phase, it can achieve zero-sample transfer without any real data fine-tuning when deployed to a real scene. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0044] Figure 2 This is a schematic diagram of the device in Embodiment 1 of the present invention;

[0045] Figure 3 This is an overall flowchart of the randomized training method for the robotic arm grasping model domain in Embodiment 2 of the present invention;

[0046] Figure 4 This is a schematic diagram of the randomization processing of the physical characteristic domain in Embodiment 2 of the present invention;

[0047] Figure 5 This is a schematic diagram of the automatic annotation and model training process in Embodiment 2 of the present invention;

[0048] Figure 6 This is a comparison chart of the zero-sample migration capture success rate under different experimental conditions in Example 2 of the present invention;

[0049] Figure 7 This is a schematic diagram of the structure of the randomized training device for the robotic arm grasping model domain in Embodiment 6 of the present invention;

[0050] Figure 8 This is a schematic diagram of the migration architecture from simulation training to real deployment in Embodiment 6 of the present invention.

[0051] Figure 9 This is a schematic diagram of the computer device in Embodiment 6 of the present invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0053] Example 1:

[0054] like Figure 1 As shown: A method for randomized training of a robotic arm grasping model domain includes the following steps:

[0055] Multi-view virtual point cloud data of the target object is generated in a simulation environment. Physical property domain randomization is applied to the virtual point cloud data. The physical property domain randomization includes randomizing at least two dimensions of the virtual point cloud's depth noise distribution, point cloud density distribution, occlusion mode, and surface reflection characteristics.

[0056] In the simulation environment, a grasping label is automatically generated for each virtual point cloud sample. The grasping label is determined by collision detection of the object state after the virtual gripper performs the grasping action through the simulation physics engine. The randomized virtual point cloud data and the corresponding grasping label are combined to form a training dataset.

[0057] The grasping pose generation model is trained using a training dataset. The grasping pose generation model takes randomized virtual point cloud as input and grasping pose parameters as output.

[0058] The trained grasping posture generation model is deployed to a real robotic arm system to achieve zero-sample grasping posture generation of point clouds in real scenes.

[0059] By applying four-dimensional physical property domain randomization to the simulated virtual point cloud, the simulated training data covers the range of point cloud physical property variations under various conditions from a real depth camera. Since the grasping model has already encountered various combinations of noise, density, occlusion, and material during the training phase, it can achieve zero-sample transfer learning without any real data fine-tuning when deployed to a real scene.

[0060] The simulation environment includes a physical simulation engine and a virtual depth camera model;

[0061] The intrinsic parameters of the virtual depth camera model are consistent with those of the real depth camera. The physics simulation engine is used to simulate the rigid body dynamics and collision detection during the gripper's grasping of an object.

[0062] In this embodiment, the randomization of the depth noise distribution adopts a mixture model of Gaussian noise and impulse noise. The expression of the mixture model is as follows:

[0063] ;

[0064] in This represents Gaussian noise with a standard deviation of σ, where σ ranges from a preset range. sampling; Let A represent impulse noise with amplitude A that occurs with probability p, where both p and A are randomly sampled from their respective preset ranges.

[0065] As a preferred embodiment of the above, the randomization of the point cloud density distribution is achieved by simulating the point cloud sparsity effect under different measurement distances and incident angles:

[0066] For each point in the virtual point cloud, based on the distance d from the point to the virtual depth camera and the angle θ between the normal vector of the surface where the point is located and the camera's optical axis, the point is randomly deleted according to the following formula:

[0067] ;

[0068] Where α is the distance attenuation coefficient, from the preset range sampling, The probability of retaining this point is given when the generated random number is less than 1. Keep the point if it is valid, otherwise delete the point.

[0069] In this embodiment, the randomization of the occlusion pattern includes:

[0070] Several occlusion objects are randomly placed in the virtual scene. The size, shape, position and orientation of the occlusion objects are randomly sampled from a preset distribution, so that the target object generates random self-occlusion and mutual occlusion areas, simulating the situation in the real scene where the object is occluded by other objects or grippers.

[0071] The randomization of surface reflection characteristics is achieved by randomly assigning bidirectional reflection distribution function parameters to each surface of the target object;

[0072] The bidirectional reflectance distribution function parameters include diffuse reflectance, specular reflectance, and surface roughness. Each parameter is sampled independently from its own preset distribution, enabling the virtual point cloud to cover a variety of material representations from matte to highly reflective.

[0073] In this embodiment, the automatic generation of capture tags includes:

[0074] In the simulation environment, the virtual gripper is controlled to close according to the candidate gripping posture. If the object is stably gripped after closing and does not fall within the preset lifting height, the candidate gripping posture is marked as a success.

[0075] If an object bounces off, slips off, or is not gripped during the clamping process, it is marked as a failure.

[0076] Among them, the physical property domain randomization process independently samples each randomization parameter when generating virtual point cloud samples, so that the combination of physical property parameters of each sample in the training dataset is different, forming an enhanced dataset that covers the range of physical property changes in the real scene.

[0077] As a preferred embodiment of the above, the method further includes: simultaneously introducing virtual point clouds of multi-object scenes with different stacking numbers, different stacking densities and different background clutter levels into the training dataset, so that the grasping posture generation model can adapt to different scene complexities from single objects to dense stacking.

[0078] As a preferred embodiment of the above, the method is applied to the picking of messy parts in industrial sorting scenarios, the picking of mixed boxes and depalletizing in logistics warehousing scenarios, or the picking of everyday items in service robot scenarios.

[0079] like Figure 2 As shown, this embodiment also includes a randomized training device for a robotic arm grasping model domain, which applies the method described above. As a preferred embodiment, the device includes:

[0080] The simulation point cloud generation module is used to render and generate multi-view virtual point cloud data of target objects in a simulation environment.

[0081] The physical property domain randomization module is used to apply randomization processing to virtual point cloud data in at least two dimensions of depth noise distribution, point cloud density distribution, occlusion mode and surface reflection characteristics.

[0082] The automatic annotation and dataset construction module is used to automatically generate capture labels for each virtual point cloud sample in the simulation environment, and to combine the randomized virtual point cloud data and the corresponding capture labels into a training dataset.

[0083] The model training module is used to train the grasping pose generation model using the training dataset;

[0084] The deployment migration module is used to deploy the trained grasping posture generation model to a real robotic arm system to achieve zero-shot grasping posture generation.

[0085] In this embodiment, the simulated point cloud generation module includes a physical simulation engine unit and a virtual depth camera rendering unit. The physical simulation engine unit is used to simulate rigid body dynamics and collision detection, and the virtual depth camera rendering unit is used to generate a virtual point cloud that is consistent with the intrinsic parameters of the real depth camera.

[0086] As a preferred embodiment of the above, the device further includes a real robotic arm execution module, which includes a multi-degree-of-freedom robotic arm, a parallel gripper-type end effector, and a depth camera, for collecting point clouds in a real scene and executing a grasping posture to generate a grasping posture output by the model.

[0087] Example 2:

[0088] This embodiment uses an industrial sorting scenario as an example. The robotic arm needs to grasp and sort various industrial parts (screws, gears, connectors, gaskets) that are randomly stacked in a bin. (Refer to...) Figure 3 The training method provided by this invention includes the following steps:

[0089] Step 1: Simulation environment configuration and virtual point cloud generation.

[0090] A virtual work scene was built using the PyBullet physics simulation engine. Eight 3D mesh models of industrial parts (from a CAD model library) were imported into the scene, and physical properties such as mass, friction coefficient, and center of mass position were configured for each part to match those of real parts. A virtual depth camera model was configured with intrinsic parameters (focal length, resolution, field of view) identical to the Intel RealSense D435i depth camera used in the actual deployment: resolution 640×480, field of view 87°×58°, focal length... Pixels. A virtual camera captures multiple frames of virtual depth images in the scene from a top-down (approximately 30° tilt) and side-view perspectives, converting them into virtual point cloud data. Each part is randomly positioned in 10 different poses in the scene, resulting in 80 poses for 8 different parts. Each pose captures 5 viewpoints, generating a total of 400 frames of virtual point cloud data. Each frame of virtual point cloud contains approximately 50,000 to 70,000 points, covering the main areas of the part's surface.

[0091] Step 2: Randomization of the physical property domain.

[0092] Reference Figure 4 Each frame of virtual point cloud is independently randomized in four dimensions of physical property domain. The randomization methods, parameter ranges and physical meanings of the four dimensions are shown in Table 1.

[0093] Dimension Randomization method Parameter range Physical meaning Depth noise Gaussian + Impulse Hybrid Model (Equation 1) Measurement noise of simulated TOF camera + multipath flying points Point cloud density Distance-incident angle sparsification (Equation 2) Simulated point density decrease in long-range and grazing angle regions Obscuring mode Randomly place obstructions Number of obstructions size Simulate objects occlusion in a cluttered scene Surface reflection BRDF parameter randomization Diffuse reflectance Specular reflectance roughness Matte finish → High-reflective metallic material

[0094] Each time a virtual point cloud sample is generated, it is independently and uniformly sampled from the aforementioned parameter ranges, ensuring that the combination of physical characteristic parameters for each sample in the training dataset is unique, thus forming an augmented dataset that covers the range of physical characteristic variations in the real scene. Taking depth noise as an example... The sample simulates high-quality close-range depth measurement. The sample simulates low-quality measurements over long distances; The sample simulations were performed in a scenario without multipath interference. The sample simulates a scenario with severe multipath effects. Taking point cloud density as an example, The sparsity effect is relatively weak at this time. Probability of point retention at a distance of 0.5m Probability of retention at a distance of 2m This allows for the simulation of significant distance sparsity. By covering a broad range of parameters, it ensures that the model has seen all possible real-world conditions.

[0095] Taking occlusion mode as an example, the placement of occluders follows these rules: the number of occluders is uniformly sampled from [0,5], the size is uniformly sampled from [2,15] cm, and the position is randomly selected within a sphere with a radius of 30 cm centered on the target object, with the orientation randomly rotated. Simultaneously, occluders can simulate mutual occlusion of the target part by other parts in the scene, as well as dynamic occlusion of the target part during the approach of the gripper. Taking surface reflection as an example, a set of BRDF parameters is independently sampled for each triangular facet of the part's mesh model. Diffuse reflectivity controls the object's intrinsic brightness, specular reflectivity controls the intensity of the specular highlight, and roughness controls the sharpness of the specular highlight. When the specular reflectivity is higher than 0.4 and the roughness is lower than 0.3, holes and flying spots appear in the corresponding surface area during depth rendering, thus simulating the depth loss phenomenon caused by the high reflectivity of real metal parts.

[0096] Step 3: Automatic annotation and training dataset construction.

[0097] Reference Figure 5 In the simulation environment, grasping labels are automatically generated for each virtual point cloud sample. For each frame of virtual point cloud, a sampling strategy is used to generate 100 candidate grasping postures: grasping points are randomly selected on the object surface, and approach directions and opening / closing widths are randomly assigned. For each candidate grasping posture, the virtual gripper (a URDF model with parameters consistent with the real Robotiq 2F-85) is controlled to close according to that posture, and the grasping process is simulated using the PyBullet physics engine.

[0098] If the object is stably held after the grippers close, and is lifted to the preset lifting height... If the object does not fall during the process, the candidate grasping posture is marked as a success. );

[0099] If the object springs away, slips off, or fails to be held after the grippers close, it is marked as a failure. );

[0100] Each virtual point cloud sample generates 100 candidate grasping pose labels, resulting in 40,000 sets of label data across 400 frames. The entire automatic annotation process runs in simulation acceleration mode, completing all annotations in approximately 8 minutes (about 200 sets per second), far superior to the annotation speed in real-world scenes (about 50 sets per hour). Furthermore, the success / failure determination is objectively made by the rigid body dynamics and collision detection of the physics engine, eliminating the subjective bias of manual annotation.

[0101] The training dataset consists of 400 randomized virtual point clouds and 40,000 captured labels, of which approximately 14,000 labels were successful (35%) and approximately 26,000 labels were unsuccessful (65%).

[0102] Step 4: Training the pose generation model.

[0103] PointNet++ is used as the backbone network for the grasping pose generation model. It takes randomized virtual point clouds as input and outputs the predicted grasping success probability and grasping pose parameters (position, orientation, and opening / closing width) for each candidate grasping pose. The PointNet++ network consists of four ensemble abstraction layers (SA layers), with the number of sampling points and feature dimensions of each layer being (512, 64), (256, 128), (128, 256), and (64, 512), respectively. The features of the last layer are subjected to global max pooling and then connected to two output heads: a classification head outputting the grasping success probability (sigmoid activation), and a regression head outputting 6-DOF pose parameters (3D position + 3D orientation quaternion free parameters) and opening / closing width. Training employs a joint loss function of binary cross-entropy loss (for grasping success / failure classification) and smoothed L1 regression loss (for pose parameter regression), with a total loss of... Using the Adam optimizer with an initial learning rate of 0.001, and employing cosine annealing for learning rate scheduling, The training process lasted 200 epochs. Training was run on a single NVIDIA RTX 3090 GPU, with a total training time of approximately 4 hours.

[0104] Step 5: Zero-sample migration deployment.

[0105] Export the trained grasping posture generation model as an ONNX format and deploy it directly to the real robotic arm system. This achieves zero-sample transfer without using any real-world data for fine-tuning. Real robotic arm systems achieve this through... A depth camera captures point clouds of a real scene, which are then fed into a deployed grasping posture generation model. The model selects the posture with the highest probability of successful grasping from all candidate postures as the output, controlling the robotic arm to perform the grasping action. Grasping tests were conducted on eight types of industrial parts in a real sorting scenario, with 50 grasping attempts for each part, for a total of 400 grasping attempts.

[0106] Example 2 Performance Summary: Zero-sample transfer average capture success rate was 88% (baseline without domain randomization was 52%, an improvement of 36 percentage points); among them, the success rate of screws (small parts + slender shapes) increased from 38% to 82%, and gears (including hole structures) increased from 55% to 90%; in high-noise scenarios (long-distance acquisition) The success rate was 84% ​​(baseline 31%) under normal conditions; the success rate was 85% (baseline 28%) under cluttered stacking scenarios (5+ overlapping objects); training data generation and annotation took 8 minutes, and model training took 4 hours. The comparison results for each condition are as follows: Figure 6 As shown.

[0107] By applying four-dimensional physical property domain randomization (depth noise + point density + occlusion + surface reflection) to the simulated virtual point cloud, the simulated training data covers the range of point cloud physical property variations under various conditions from a real depth camera. The grasping model, having already encountered various combinations of noise, density, occlusion, and material during training, can achieve zero-shot transfer computing without any real-data fine-tuning when deployed to a real scene. In this embodiment, the zero-shot transfer grasping success rate increased from 52% without domain randomization to 88%, an improvement of 36 percentage points.

[0108] By employing a Gaussian + impulse hybrid noise model, both the high-frequency measurement noise (Gaussian component) from a real depth camera and the impulse noise (impulse component) caused by multipath effects / edge flying points are simulated simultaneously. This model more closely approximates the noise statistical characteristics of real point clouds than a single noise model, enabling the model to operate under high-noise conditions (long-distance acquisition). The success rate of crawling increased from a baseline of 31% to 84%.

[0109] By using a distance-incident angle dependent point cloud sparsification model, the physical phenomenon of point density decrease in distant and grazing angle regions in a real depth camera is accurately simulated, making the model robust to changes in point density. Moreover, the model only needs to sample a single parameter α to simultaneously control the influence of two physical factors, distance and angle, making it simple to implement and computationally inefficient.

[0110] Through an automatic collision detection and labeling mechanism, high-quality crawling labels can be generated on a large scale without manual labeling. The labeling speed is about 200 sets per second (simulation acceleration). Training 100,000 sets of data only takes about 8 minutes. Moreover, the determination of successful / failed labels is based on the rigid body dynamics simulation of the physics engine. The determination criteria are objective and uniform, avoiding the inconsistency problem of manual labeling.

[0111] Example 3:

[0112] This embodiment uses a mixed-box depalletizing scenario in logistics warehousing as an example to verify the transferability of the method of the present invention under different object types and scenario complexity. The objects to be grasped are four types of packaging boxes (small square boxes, rectangular boxes, cylindrical cans, and flat square boxes), which are placed on a standard logistics pallet in a stacked manner.

[0113] The simulation environment uses PyBullet to import four types of packaging box CAD models, and the domain randomization parameter range is consistent with that of Example 2. Additionally, multi-object scene clutter randomization is introduced, randomly placing 3 to 15 objects in each virtual scene, with stacking density varying randomly from loose arrangement to dense stacking, simulating the different complexities from the top layer (loose) to the bottom layer (dense) in a real destacking scenario.

[0114] The training dataset contains 500 frames of virtual point clouds (4 types of packaging boxes × 125 poses) with a total of 50,000 labels. After training the model for 200 epochs, it was deployed to a real UR5e robotic arm for zero-shot testing. Experimental results show that in real depalletizing scenarios, the average success rate of zero-shot transfer is 86% (48% for the domain-free randomization baseline). The success rate is 92% for the top layer (loose stacking) and 79% for the bottom layer (dense stacking), with the bottom-layer dense stacking scenario showing a 31 percentage point improvement over the baseline (48%). This embodiment verifies the transferability of the method of the present invention in logistics scenarios and under different scenario complexities, and also verifies the technical effect of clutter randomization features in multi-object scenarios.

[0115] Example 4:

[0116] This embodiment uses a service robot grasping everyday household items as an example to verify the effectiveness of the method of the present invention on objects made of challenging materials such as transparent and highly reflective materials. The objects to be grasped are seven types of everyday items: ceramic water cup, glass cup, stainless steel plate, plush toy, remote control, aluminum can, and tissue box. The glass cup is transparent, the stainless steel plate is highly reflective, and the ceramic water cup is semi-matte, covering the most challenging material categories in depth camera point cloud acquisition.

[0117] The simulation environment setup is consistent with Example 2, but the randomization parameter range for surface reflection characteristics is expanded to accommodate material diversity: diffuse reflectance ∈ [0.05, 0.95], specular reflectance ∈ [0.0, 0.9], and roughness ∈ [0.05, 1.0], allowing the virtual point cloud to cover the complete range from deeply absorbing matte materials (diffuse reflectance 0.05) to specularly dominant materials (spectral reflectance 0.9) to deeply penetrating transparent materials (transmittance 0.8, simulated by reducing surface opacity). The randomization parameters for the other three dimensions are consistent with Example 2.

[0118] The training dataset contains 420 frames of virtual point clouds (7 types of objects × 60 poses), with a total of 42,000 labels. After training the model for 200 epochs, it was deployed to a real service robot arm (AUBO i5 + Robotiq 2F-85 + RealSense L515) for zero-shot testing. Experimental results show that the overall zero-shot transfer grasping success rate is 87% (45% for the domain-free randomization baseline); the success rate for matte objects such as plush toys and tissue boxes is 92%, for moderately reflective objects such as remote controls and soda cans it is 88%, for ceramic water cups (semi-matte) it is 85%, for glass cups (transparent) it is 74% (baseline only 11%), and for stainless steel plates (highly reflective) it is 78% (baseline only 9%). The significant improvement for transparent and highly reflective objects directly verifies the technical effect of surface reflection characteristic randomization. The model has seen highly reflective / transparent simulated point clouds with holes and flying points during the training phase, thus learning to infer grasping postures from the incomplete point cloud.

[0119] Example 5:

[0120] This embodiment verifies the necessity and synergistic effect of randomizing the "at least two dimensions" physical property domain through ablation experiments. The experimental conditions are the same as in Embodiment 2, only the combination of randomized dimensions is changed.

[0121] experimental group Randomized dimensional combinations Zero-sample crawling success rate relative baseline improvement control group No-field randomization 52% — One-dimensional noise Depth noise only 63% +11pp One-dimensional - reflection Surface reflection only 59% +7pp Two dimensions Depth noise + point cloud density 71% +19pp Three dimensions Noise + Density + Obstruction 78% +26pp Four Dimensions (This Invention) Noise + Density + Obstruction + Reflection 88% +36pp

[0122] As shown in Table 2, single-dimensional randomization has limited improvement on zero-sample transfer (maximum 63%), and the effects of different single dimensions (noise vs. reflection) vary significantly, indicating that the domain gap in real-world scenarios is the result of the superposition of multiple physical factors, which a single dimension cannot cover. When the randomization dimension increases to two or more, the success rate improvement increases significantly—two dimensions improve by 8 percentage points compared to one dimension, and four dimensions improve by 10 percentage points compared to three dimensions, indicating synergistic gains among the dimensions. This ablation experiment verifies the necessity of the "at least two dimensions" technical feature and also proves that multi-dimensional physical property synergistic randomization produces synergistic technical effects that cannot be achieved by simply superimposing a single dimension.

[0123] Example 6:

[0124] Reference Figure 7 This invention provides a robotic arm grasping model domain randomization training device based on simulated point cloud data augmentation, including a simulated point cloud generation module, a physical characteristic domain randomization module, an automatic annotation and dataset construction module, a model training module, a deployment and migration module, and a real robotic arm execution module.

[0125] The simulated point cloud generation module comprises a PyBullet physics simulation engine unit and a virtual depth camera rendering unit. The physics simulation engine unit is responsible for scene construction (importing object CAD models, setting physical parameters such as friction coefficient and gravity), rigid body dynamics simulation, and collision detection. The virtual depth camera rendering unit renders virtual depth images and converts them into point clouds according to the intrinsic parameters of a real RealSense D435i. This module runs on an industrial PC equipped with an NVIDIA RTX 3090 GPU, and the generation time for a single frame of virtual point cloud is approximately 20 ms. The data output of the simulated point cloud generation module is connected to the data input of the physical property domain randomization module.

[0126] The Physical Properties Domain Randomization module implements four dimensions of processing: depth noise randomization (Equation 1), point density sparsity reduction (Equation 2), random occlusion placement, and BRDF parameter randomization. This module independently samples parameters for each dimension each time a virtual point cloud is generated, ensuring the diversity of physical properties in the dataset. The four-dimensional randomization processing for a single frame takes approximately 5 ms. The data output of the Physical Properties Domain Randomization module is connected to the data input of the automatic annotation and dataset construction module.

[0127] The automatic labeling and dataset construction module controls the virtual gripper to execute candidate grasping postures in the simulation environment. It automatically determines successful / failed labels through collision detection and packages the randomized virtual point cloud and grasped labels into a training dataset. In simulation acceleration mode, it generates labels at a rate of approximately 200 sets per second, completing 100,000 sets of data in about 8 minutes. The data output of the automatic labeling and dataset construction module is connected to the data input of the model training module.

[0128] The model training module uses the PointNet++ backbone network and is trained on an RTX 3090 GPU with the Adam optimizer. Training takes 4 hours and 200 epochs, outputting the trained grasping pose generation model parameters. The data output of the model training module is connected to the data input of the deployment and migration module.

[0129] The deployment and migration module exports the trained model to ONNX format and deploys it to the industrial control computer of the real robotic arm for inference. No fine-tuning with real data is required. The data output of the deployment and migration module is connected to the data input of the real robotic arm's execution module.

[0130] The real-world robotic arm execution module includes the UR5e six-DOF robotic arm, a Robotiq 2F-85 parallel gripper, and a RealSense D435i depth camera. This module acquires point clouds in a real-world scene, inputs the model to obtain the grasping posture, and controls the robotic arm to perform the grasping action. A single end-to-end inference operation takes approximately 120 ms.

[0131] Reference Figure 8The complete workflow of the above device is as follows: the simulation point cloud generation module generates a virtual point cloud in the simulation domain, which is then enhanced by the physical property domain randomization module and labeled by the automatic annotation and dataset construction module. The model training module completes model training in the simulation domain. Subsequently, the deployment transfer module transfers the model parameters to the real domain, and the real robotic arm execution module collects real point clouds in the real domain and performs grasping, thus forming a complete closed loop of "simulation domain training - model transfer - real domain execution".

[0132] The above-described end-to-end process has been verified in Examples 2, 3, and 4, demonstrating the complete functionality from simulation training to real deployment.

[0133] Please see Figure 9 The diagram shows a structural schematic of a computer device provided in an embodiment of this application. An embodiment of this application provides a computer device 400, including a processor 410 and a memory 420. The memory 420 stores a computer program executable by the processor 410. When the computer program is executed by the processor 410, it performs the method described above.

[0134] This application embodiment also provides a storage medium 430, on which a computer program is stored, and the computer program is executed by a processor 410 to perform the above method.

[0135] The storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0136] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0137] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0138] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0139] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0140] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0141] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0142] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0143] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A randomized training method for a robotic arm grasping model domain, characterized in that, Includes the following steps: Multi-view virtual point cloud data of the target object is generated in a simulation environment. Physical property domain randomization processing is applied to the virtual point cloud data. The physical property domain randomization processing includes randomizing at least two dimensions of the virtual point cloud's depth noise distribution, point cloud density distribution, occlusion mode, and surface reflection characteristics. In the simulation environment, a grasping label is automatically generated for each virtual point cloud sample. The grasping label is determined by collision detection of the object state after the virtual gripper performs the grasping action through the simulation physics engine. The randomized virtual point cloud data and the corresponding grasping label are combined to form a training dataset. The grasping pose generation model is trained using the training dataset. The grasping pose generation model takes randomized virtual point cloud as input and grasping pose parameters as output. The trained grasping posture generation model is deployed to a real robotic arm system to achieve zero-sample grasping posture generation of point clouds in real scenes.

2. The method according to claim 1, characterized in that, The simulation environment includes a physical simulation engine and a virtual depth camera model; The intrinsic parameters of the virtual depth camera model are consistent with those of the real depth camera, and the physical simulation engine is used to simulate rigid body dynamics and collision detection during the gripper's grasping of an object.

3. The method according to claim 1, characterized in that, The randomization of the depth noise distribution employs a mixture model of Gaussian noise and impulse noise, and the expression for the mixture model is as follows: ； in This represents Gaussian noise with a standard deviation of σ, where σ ranges from a preset range. sampling; Let A represent impulse noise with amplitude A that occurs with probability p, where both p and A are randomly sampled from their respective preset ranges.

4. The method according to claim 3, characterized in that, The randomization of the point cloud density distribution is achieved by simulating the point cloud sparsity effect under different measurement distances and incident angles: For each point in the virtual point cloud, based on the distance d from the point to the virtual depth camera and the angle θ between the normal vector of the surface where the point is located and the camera's optical axis, the point is randomly deleted according to the following formula: ； Where α is the distance attenuation coefficient, from the preset range sampling, The probability of retaining this point is given when the generated random number is less than 1. Keep the point if it is valid, otherwise delete the point.

5. The method according to claim 1, characterized in that, The randomization of the occlusion pattern includes: Several occlusion objects are randomly placed in a virtual scene. The size, shape, position, and orientation of the occlusion objects are randomly sampled from a preset distribution, so that the target object generates random self-occlusion and mutual occlusion areas, simulating the situation in a real scene where an object is occluded by other objects or grippers.

6. The method according to claim 5, characterized in that, The randomization of the surface reflection characteristics is achieved by randomly assigning bidirectional reflection distribution function parameters to each surface of the target object; The bidirectional reflectance distribution function parameters include diffuse reflectance, specular reflectance, and surface roughness. Each parameter is sampled independently from its own preset distribution, enabling the virtual point cloud to cover a variety of material representations from matte to highly reflective.

7. The method according to claim 1, characterized in that, The automatic generation of the crawling tags includes: In the simulation environment, the virtual gripper is controlled to close according to the candidate gripping posture. If the object is stably gripped after closing and does not fall within the preset lifting height, the candidate gripping posture is marked as a success. If an object bounces off, slips off, or is not gripped during the clamping process, it is marked as a failure.

8. The method according to claim 1, characterized in that, The physical property domain randomization process independently samples each randomization parameter each time a virtual point cloud sample is generated, so that the combination of physical property parameters of each sample in the training dataset is different, forming an enhanced dataset that covers the range of physical property changes in the real scene.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: simultaneously introducing virtual point clouds of multi-object scenes of the target object under different stacking numbers, different stacking densities and different background clutter levels into the training dataset, so that the grasping posture generation model can adapt to different scene complexities from single objects to dense stacking.

10. The method according to any one of claims 1 to 8, characterized in that, The method is applied to the handling of messy parts in industrial sorting scenarios, the handling of mixed boxes and depalletized boxes in logistics warehousing scenarios, or the handling of everyday items in service robot scenarios.

11. A randomized training device for a robotic arm grasping model domain, using the method as described in any one of claims 1 to 10, characterized in that, The device includes: The simulation point cloud generation module is used to render and generate multi-view virtual point cloud data of target objects in a simulation environment. The physical property domain randomization module is used to apply randomization processing to the virtual point cloud data in at least two dimensions of depth noise distribution, point cloud density distribution, occlusion mode and surface reflection characteristics. The automatic annotation and dataset construction module is used to automatically generate capture labels for each virtual point cloud sample in the simulation environment, and to combine the randomized virtual point cloud data and the corresponding capture labels into a training dataset. The model training module is used to train the grasping pose generation model using the training dataset; The deployment migration module is used to deploy the trained grasping posture generation model to a real robotic arm system to achieve zero-sample grasping posture generation.

12. The apparatus according to claim 11, characterized in that, The simulated point cloud generation module includes a physical simulation engine unit and a virtual depth camera rendering unit. The physical simulation engine unit is used to simulate rigid body dynamics and collision detection, and the virtual depth camera rendering unit is used to generate a virtual point cloud that is consistent with the intrinsic parameters of a real depth camera.

13. The apparatus according to claim 11, characterized in that, The device also includes a real robotic arm execution module, which includes a multi-degree-of-freedom robotic arm, a parallel gripper-type end effector, and a depth camera, used to collect point clouds in a real scene and execute the grasping posture output by the grasping posture generation model.

14. A computer device, characterized in that, The method includes a processor and a memory, the processor and the memory being coupled together, the memory storing program instructions, which, when executed by the processor, implement the method of any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method of any one of claims 1 to 10.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when run on a computer, causes the computer to perform the method of any one of claims 1 to 10.

Images