A twin scene-based visual algorithm simulation dataset generation and verification method

Abstract

The application discloses a kind of based on twin scene visual algorithm simulation dataset generation and verification method, belong to computer vision field, specifically include: first, the simulation scene of equal proportion industrial manipulator is grabbed target is constructed, physical property is added to object, and scene configuration is optimized;And construct virtual scene management tree, with model as unit control scene information. Then, through random configuration produces random scene, so that scene random motion around collection target, and the random change of position and posture of target on work surface realizes scene enhancement and data enhancement. Then, through the image data of rendering collection, obtain RGB-D graph, target frame and position label, segmentation graph, depth chart and the three-dimensional posture information of object are labeled. Finally, using computer vision algorithm training dataset, drive verification detection unit completes operation task, realizes the cross validation of virtual scene.The application automatically and quickly generates large quantities of training data, and the annotation accuracy is high.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a method for generating and verifying a visual algorithm simulation data set based on twin scenes. Background Art

[0002] With the development of deep learning neural networks, computer vision is becoming increasingly intertwined with industry, with object detection, semantic segmentation, instance segmentation, and pose estimation becoming increasingly integrated with industrial production. The success of visual neural networks relies on high-capacity models, powerful computing power, and large, comprehensive datasets. However, data collection in industry is difficult, and objects are not portable. Traditionally, cameras are used to capture data, and datasets are obtained through manual annotation of crowdsourced data. Manually annotating datasets is labor-intensive, cumbersome, and prone to errors and omissions. Furthermore, some dataset generation tools have low resolution and cannot provide high-quality images. Therefore, methods for generating highly simulated data for twin scenes with automatic annotation capabilities are particularly important.

[0003] Most existing datasets cater to specific needs. For objects to be annotated in a scene, their focus is primarily on detection, segmentation, or pose estimation. This makes it difficult to meet diverse functional needs and results in a low level of richness. Building multi-format datasets for these targets can help meet the diverse application needs of computer vision algorithms. It can also assist in cross-validation and comparison of algorithms, enabling the synergy of multiple functions.

[0004] Traditional data augmentation methods, such as rotation, translation, random cropping, and color space transformation, are limited in effectiveness when sampling angles are restricted. While they can simulate and identify objects in different scenes, they cannot restore detailed features such as shadows. However, random augmentation during data collection can best simulate the state of objects under different conditions, providing broader and more comprehensive data coverage, which helps improve the real-world application of vision algorithms.

[0005] Constructing a digital model of the virtual scene and conducting simulation testing and verification on it can not only simulate the operation of equipment and the changes caused by parameter adjustment during production, but also save manpower and money and avoid destructive waste. Summary of the Invention

[0006] To address the above problems, the present invention proposes a method for generating and verifying a visual algorithm simulation dataset based on twin scenes. Through the virtual reality method, a highly restored industrial assembly twin scene is constructed, which can quickly generate a high-resolution dataset that meets the various requirements of the visual algorithm and can be verified in the same scene.

[0007] The method for generating and verifying a visual algorithm simulation dataset based on random enhancement of twin scenes has the following specific steps:

[0008] Step 1: For the scenario where an industrial robot arm grasps a target, collect objects in the scene, classify them by category, and construct a proportional twin scene;

[0009] The objects include the target object to be collected, the background object that can be changed, and other objects that remain unchanged;

[0010] The scaled twin scene includes an environment model, a data acquisition model, and a verification and detection unit model.

[0011] The environmental model includes:

[0012] Target audience: The target of the data required by the user.

[0013] Working environment: collect the working environment of the target object;

[0014] Weather environment and lighting scenes: including different lighting conditions such as morning, noon and evening, scenes under different indoor lighting, etc.

[0015] Background objects: other objects in the scene that interact with the target object.

[0016] Data acquisition model: The acquisition tool is a simulation camera. The intrinsic and extrinsic parameters, position, and distortion coefficient of the simulation camera are set to be consistent with the real scene, and a camera coordinate system independent of the world coordinate system is constructed.

[0017] The main form of the verification and detection unit model is: robotic arm grasping in a simulation scene.

[0018] Step 2: Perform geometric modeling on the structure of each object in the isometric twin scene and add physical properties to all objects in the scene;

[0019] Physical properties include intrinsic properties such as gravity, collision edge, reaction force during collision, and elastic force;

[0020] Taking the maximum outer contour of each object as the collision edge, add feature maps including color map, normal map and height map to each object, and adjust parameters such as roughness according to the actual characteristics of each object;

[0021] Step 3: According to the functions and positions of objects in the real scene, use the target object, background object and other objects to build a scene management binary tree;

[0022] The root nodes of the scene management binary tree are attributes and positions. The child nodes of attributes include target objects and background objects; the child nodes of positions include acquisition planes and background object pools.

[0023] Among them, the molecular nodes of the acquisition plane include the reachable area of ​​the robotic arm and other areas; the background object pool refers to the scene location where background objects can be placed.

[0024] Step 4: Construct random scenes through random configuration of the scene management tree, and implement random enhancement of scenes and data respectively;

[0025] When building a scene, first access the attribute node, then access the target object node and background object node under the attribute node, randomly select the molecular nodes of the target object and background object subnodes, and call the corresponding prefab.

[0026] Access the position node in the same way to initialize the position of the prefab, where the target object must be located on the acquisition plane.

[0027] Scene random enhancement means: using the central axis of the acquisition area as the rotation axis, making all background objects in the scene move at the same speed around the rotation axis;

[0028] Data random enhancement refers to the random x, y position changes and random posture changes of the target object on the acquisition plane.

[0029] Step 5: Keep the acquisition camera in place, place the target object on the acquisition plane, and set the target object's attitude angle. Make it randomly sampled within (0, 360).

[0030] Step 6: Render the randomly sampled images, capture the RGB-D images taken by the acquisition camera in time sequence, and process them as a dataset.

[0031] Use the acquisition camera to capture the RGB image of the current scene at the current moment; automatically obtain the location and type annotation information of each object in the RGB image and save it as an XML format file;

[0032] In the RGB image, based on the maximum outer contour of the image, a rectangular target box is constructed in the W (width) and H (height) directions. The center pixel position of the target box and the length and width are recorded in the JSON file as position labels.

[0033] A segmentation map is constructed using the maximum outer contour of the target object in the acquisition plane. Different objects are distinguished by different colors. A color coding value is set for each target object. Target objects of the same category are stored with the same color coding value, and the background color is set to black.

[0034] Construct a cube bounding box based on the maximum outer contour of each target object, and record the coordinate positions of the four corners and save them in a json file;

[0035] Starting from the acquisition camera coordinates, the depth information of all objects within the camera's field of view is collected according to the camera parameters and saved as a depth map.

[0036] Step 7: Use the computer vision algorithm to train the dataset; in the twin scene, use the training results to drive the verification detection unit to complete the operation task, realizing cross-validation of the virtual scene;

[0037] Specifically:

[0038] Train the dataset with a computer vision algorithm, add an external control link component, and establish communication between the control end and the virtual scene; call the motion planning service, take a screenshot of the scene as input, instantiate a new RGBA sensor image with the defined size, and send a new vision algorithm service request to the control end.

[0039] The control end uses an algorithm to predict the corresponding position and make a plan, controlling the virtual scene grasping robot arm to move to the target position to grasp the target object for verification.

[0040] The advantages of the present invention are:

[0041] 1) A method for generating and verifying a visual algorithm simulation dataset based on twin scenarios. This method collects target data through simulation software, avoiding the use of equipment such as depth cameras. Compared with traditional methods, it has lower hardware costs.

[0042] 2) A method for generating and verifying a visual algorithm simulation dataset based on twin scenarios. Considering the complex problem of dataset annotation, this invention uses automatic annotation to rely on computers to automatically and quickly generate large quantities of data for neural network training, avoiding manual annotation and achieving high annotation accuracy.

[0043] 3) A method for generating and verifying a visual algorithm simulation dataset based on twin scenarios. By freely generating target data under different angles, backgrounds, and lighting conditions, the generated dataset is of great inspiration to industrial production and can serve as a substitute for real data when real data is insufficient or difficult to obtain. It has a wide range of applications.

[0044] 4) A method for generating and verifying a visual algorithm simulation dataset based on twin scenarios. The generated dataset is trained through the YOLOv5 network, with a final test accuracy of 98%. When used for detecting real objects, the accuracy reaches 97%, which is of great significance to the development of computer vision and virtual-to-reality applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of a method for generating and verifying a visual algorithm simulation data set based on a twin scene in the present invention;

[0046] Figure 2 A binary tree for scene management constructed for the present invention;

[0047] Figure 3 This is a flow chart of a simulation data set generation method and verification device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is described in further detail and in depth with reference to the accompanying drawings.

[0049] The method for generating and verifying a visual algorithm simulation dataset based on random enhancement of twin scenes includes the following parts:

[0050] S1: Data preparation: Based on the industrial robot arm grasping scene, collect objects in the scene to build a proportional simulation scene.

[0051] S2: Twin Scenario Preparation: After obtaining real-world scene information for industrial assembly, geometric modeling of the corresponding structures is performed. This model is imported into simulation software, and a virtual scene is constructed based on the real scene. Physical properties are added to objects, and the scene configuration is optimized. A virtual scene management tree is then constructed to manage scene information on a model-by-model basis.

[0052] S3: Data random enhancement: This includes scene random enhancement and target random enhancement. Random scenes are generated by randomly configuring the scene management tree, while the scene is randomly moved around the target. This combination provides diverse background information for random scene enhancement. Target data enhancement is primarily achieved by randomly changing the target's position and posture on the work surface.

[0053] S4: Data acquisition and annotation: By rendering the image data collected from the camera, RGB-D images, target boxes and position labels, segmentation maps and 3D pose information of the object are obtained.

[0054] S5: Repeat the above S2-S4 process until a specified number of simulation data sets are generated.

[0055] S6: Use computer vision algorithms to train the above dataset;

[0056] S7: In the twin scene, use the above training results to drive the verification detection unit to complete the operation task and realize cross-validation of the virtual scene.

[0057] like Figure 1 As shown, the specific steps are:

[0058] Step 1: For the scenario where an industrial robot arm grasps a target, collect the modeled objects in the scene and classify them by category to construct a proportional twin scene;

[0059] The objects include the target object to be collected, the background object that can be changed, and other objects that remain unchanged;

[0060] The scaled twin scene includes an environment model, a data acquisition model, and a verification and detection unit model.

[0061] The environmental model includes:

[0062] Target audience: The target of the data required by the user.

[0063] Work environment: Collect the target object's work environment, mainly including factory buildings and other movable objects in the factory buildings;

[0064] Weather environment and lighting scenes: including different lighting conditions such as morning, noon, and evening, and scenes under different indoor lighting, etc.

[0065] Background objects: other objects in the scene that interact with the target object.

[0066] Data acquisition model: includes acquisition objects and acquisition tools. The acquisition tool is a simulated camera. According to the image acquisition requirements of industrial scenarios, the camera's focal length, distortion, brightness adjustment, gamma adjustment, white balance adjustment, color space, depth of field, and high dynamic range (HDR) color tone are set to be consistent with the real scene. The camera's installation position and angle always remain stationary relative to the working plane, and a camera coordinate system independent of the world coordinate system is constructed.

[0067] The main form of the verification and detection unit model is: robotic arm grasping in a simulation scene.

[0068] Step 2: Perform proportional simulation geometric modeling on the structure of each object in the proportional twin scene, and add physical properties to all objects in the scene;

[0069] Physical properties include gravity, collision edge, reaction force during collision, elastic force and other intrinsic properties; the gravity proportional coefficient is 9.8N / kg;

[0070] Taking the maximum outer contour of each object as the collision edge, add feature maps including color map, normal map and height map to each object, and adjust parameters such as roughness according to the actual characteristics of each object;

[0071] Step 3: According to the functions and positions of objects in the real scene, use the acquisition targets, background objects and other objects to build a scene management binary tree;

[0072] like Figure 2 As shown, the root nodes of the scene management binary tree are attributes and positions, and the child nodes of attributes include acquisition targets and background objects; the child nodes of positions include acquisition planes and background object pools;

[0073] Among them, the molecular nodes of the acquisition plane include the reachable area of ​​the robotic arm and other areas; the background object pool refers to the scene location where background objects can be placed.

[0074] Step 4: Construct random scenes through random configuration of the scene management tree, and implement random enhancement of scenes and data respectively;

[0075] When building a scene, first access the attribute node, then access the acquisition target node and background object node under the attribute node, randomly select the molecular nodes of the acquisition target and background object subnodes, and call the corresponding prefab.

[0076] Access the position node in the same way to initialize the position of the prefab, where the acquisition target must be located on the acquisition plane.

[0077] Scene random enhancement means: using the central axis of the acquisition area as the rotation axis, making all background objects in the scene move at the same speed around the rotation axis;

[0078] Data random enhancement refers to the random changes in the position and posture of the acquisition target on the acquisition plane (random x, y position).

[0079] Specifically: import the target's 3D model from 3D software, including but not limited to 3ds Max, SolidWorks, Unity, etc.; adjust the scale of the model in the scene according to its actual size, add the gravity acceleration g in the dynamics simulation engine, and generate a refined grid based on the outer contour of the object as the collision edge; place the target object to be collected on the work plane, and set the model's posture parameters, including the three-dimensional space position coordinates x, y, z and posture angle Set the constraint area and randomize the trajectory and posture parameters of each part within the area.

[0080] Step 5: Keep the acquisition camera in place, place the target object on the acquisition plane, and set the target object's attitude angle. Make it randomly sampled within (0, 360).

[0081] Keep the acquisition camera still and change the posture of the target object to enhance the acquisition target data; similarly, the background pool randomly calls and changes the background to achieve data enhancement of a background.

[0082] Step 6: Use the built-in renderer to render the randomly sampled images, capture the RGB-D images taken by the acquisition camera in time sequence, and process them as a dataset.

[0083] First, add three lighting types: natural light, blue light, and pink light, control them to switch on and off at a fixed frequency, and randomly set the light source position and light intensity for rendering;

[0084] Then, the focal length, specific position, and distortion coefficient of the simulated camera are set; and a pixel mapping relationship is constructed. The spatial coordinates of the vertices of the target minimum three-dimensional rectangular box are calculated through coordinate transformation, and the two-dimensional projection pixel coordinates on the target image are used to complete the labeling of the target minimum three-dimensional rectangular box.

[0085] Through the acquisition camera, the RGB image of the captured object in the current scene is obtained; the location and type annotation information of the captured object in the RGB image is automatically obtained and saved as an XML format file;

[0086] In the RGB image, a rectangular target box is constructed in the W (width) and H (height) directions based on the maximum outer contour of the image. The center pixel position of the target box in the RGB image and the length and width are recorded in a JSON file as a position label.

[0087] Construct a cube bounding box based on the maximum outer contour of each object, and record the coordinate positions of the four corners and save them in a json file;

[0088] Next, set the color coding value of the target object to be collected. Target objects of the same category are stored with the same color coding value, and the background color is set to black.

[0089] A segmentation map is constructed using the maximum outer contour of the target object within the acquisition plane. Different target objects are distinguished by different colors. A color coding value is set for each target object. Target objects of the same type are stored with the same color coding value, and the background color is set to black. A segmentation map of the acquired object under the perspective of the virtual camera is automatically obtained.

[0090] Finally, starting from the acquisition camera coordinates, the depth information of all objects within the camera's field of view is collected according to the camera parameters and saved as a depth map.

[0091] Step 7: Use the computer vision algorithm to train the dataset; in the twin scene, use the training results to drive the verification detection unit to complete the operation task, realizing cross-validation of the virtual scene:

[0092] Specifically:

[0093] Train the dataset with a computer vision algorithm, add an external control link component, and establish communication between the control end and the virtual scene; call the motion planning service, take a screenshot of the scene as input, instantiate a new RGBA sensor image with the defined size, and send a new vision algorithm service request to the control end.

[0094] The control end uses the algorithm to predict the corresponding position and make a plan, controlling the virtual scene grasping robot arm to move to the target position to grasp the object for verification.

[0095] Computer vision algorithms include but are not limited to object detection algorithms such as YOLO, semantic segmentation algorithms such as FCN, and pose estimation algorithms such as PointRCNN;

[0096] The control end includes but is not limited to ROS, MATLAB, etc.

[0097] This example describes the technical process in detail by taking the generation of simulation data sets in industrial scenarios and verifying the use of virtual robotic arms to grasp targets.

[0098] like Figure 3 The specific process is as follows:

[0099] 1. Establish object model and work scenario

[0100] Before generating simulation training data, create a 1:1 scale 3D model of the industrial robotic arm and the grasping target using modeling software, including but not limited to 3ds Max, SolidWorks, CAD, and Unity. Add textures, textures, and material information to the grasping target to form a target model library. 3D model file formats include but are not limited to obj, stl, and fbx. Build a highly realistic simulated factory based on the actual industrial assembly scene and set its size and position. Import the model and scene into Unity and adjust the size, gravity, and collision edges.

[0101] 2. Data Processing and Collection

[0102] Step S21: Create a simulated camera as a perception camera, align the camera with the working plane, use the center of the working plane as the origin of the world coordinate system, and record the relative position of the perception camera and the origin;

[0103] Step S22: Using the reachable range of a robotic arm commonly used in industrial production as the movement range of the grasped object, and randomly generating its position and rotation angle;

[0104] Step S23: Place three point light sources at appropriate locations above the working plane, and switch the light sources on and off and their colors at a frequency of 10 Hz;

[0105] Step S24: Set the rendering parameters. To generate multimodal simulation data and annotation information, this embodiment sets the parameters of the simulated camera in the rendering engine based on the parameters of the actual camera. The camera parameters include but are not limited to: the resolution W×H of the image generated by the camera, the focal length f of the camera, the sensor size of the camera, the camera's position, and the camera's projection angle type. Select the target camera view as the rendering object, customize the frame number range and image format, such as rendering images from 0 to 3000 frames and saving them as 1040×680 JPG files.

[0106] 3. Generate annotation results

[0107] Use a rendering engine to generate the RGB image, detection bounding box, segmentation map, and pose information of the grasped object from the perspective of the virtual camera. Used rendering engines include but are not limited to: Blender, Renderman, Maxwell, and Render.

[0108] In order to obtain the RGB image of the simulated scene, based on the parameters of the simulated light source in the rendering engine, the material, texture, color and other properties of the object, the rendering engine finally generates the RGB image of the scene from the camera perspective.

[0109] Using the type and position annotation information of the captured object, a rectangular detection frame is marked on the maximum edge of the object in the RGB image, and its type, position and size of the detection frame are recorded. The marked information is stored in a file in ".xml" format.

[0110] Using the parameters of the simulated camera, a pixel mapping relationship is constructed, and the two-dimensional projection pixel coordinates of the vertex spatial coordinates of the target minimum three-dimensional rectangular box on the target image are calculated through coordinate transformation to complete the labeling of the target minimum three-dimensional rectangular box.

[0111] The colors of the labeled objects are set with different coding values, and then a segmentation map of the scene is generated by the rendering engine. That is, the pixels belonging to the same object in the segmentation map store the same color coding value.

[0112] The pose and segmentation information generated above are stored in a file in the format of a "json" file.

[0113] 4. Training Dataset

[0114] Adjust the algorithm parameters to train the dataset, change the parameter name to the block name, the image size to the image size obtained by the virtual camera, and modify the position information. Use machine learning algorithms to train the block posture, including but not limited to object detection algorithms such as YOLO, semantic segmentation algorithms such as FCN, and posture estimation algorithms such as PointRCNN.

[0115] 5. Virtual Robot Arm Grasping Verification

[0116] Step S51: Add ROS link component to establish communication between ROS and Unity

[0117] Step S52: Call the motion planning service, create a ROS service function, take the screenshot of the scene as input, and instantiate a new RGBA sensor image with the defined size, and send a new pose estimation service request to ROS.

[0118] Step S53: Manage ROS service responses. When a pose estimation service response arrives, the incoming pose is automatically converted to the Unity Engine type and the UI elements are updated accordingly. After conversion, the estimated pose information is used as input to the trajectory planning service.

[0119] Step S54: Use the Moveit trajectory planning service to control the robot arm and its end effector to move to the position of the grasping object and grasp the target, completing the grasping verification of the robot arm.

[0120] Based on the research, we found that the existing technologies have the following problems: most of the data annotation information needs to be generated by manual labeling; the generated data sets have problems such as relatively single object types, low scene complexity, and single data modality; the generated data has poor displayability.

[0121] Therefore, in this invention, it is proposed to automatically generate large quantities of simulation data close to the real world through automatic labeling, obtain large-scale RGB-D multimodal data of building blocks; and intuitively display them through robotic arm grasping.

[0122] This method utilizes automatic annotation to perform dynamic simulation and scene rendering using a 3D object model as input, generating large quantities of multimodal simulation data and annotation information for training computer vision models. Compared to manual annotation, this method significantly reduces labor and time costs. Furthermore, by interactively displaying dataset training results in a virtual environment in real time, it provides valuable guidance for real-world applications.

Claims

1. A method for generating and verifying a visual algorithm simulation dataset based on twin scenes, characterized in that: The specific steps are: Step 1: For the scenario where an industrial robot arm grasps a target, collect objects in the scene, classify them by category, and construct a proportional twin scene; The objects include the target object to be collected, the background object that can be changed, and other objects that remain unchanged; Step 2: Perform geometric modeling on the structure of each object in the isometric twin scene and add physical properties to all objects in the scene; Step 3: According to the functions and positions of objects in the real scene, use the target object, background object and other objects to build a scene management binary tree; The root nodes of the scene management binary tree are attributes and positions. The child nodes of attributes include target objects and background objects; the child nodes of positions include acquisition planes and background object pools. The molecular nodes of the acquisition plane include the reachable area of ​​the robot arm and other areas; the background object pool refers to the scene locations where background objects can be placed; Step 4: Construct random scenes through random configuration of the scene management tree, and implement random enhancement of scene and target data respectively; When building a scene, first access the attribute node, then access the target object node and background object node under the attribute node, randomly select the molecular nodes of the target object and background object subnodes, and call the corresponding prefab; Use the same method to access the position node and initialize the position of the prefab. The target object must be located on the acquisition plane. Data random enhancement: including scene random enhancement and target random enhancement; Scene random enhancement means: using the central axis of the acquisition area as the rotation axis, making all background objects in the scene move at the same speed around the rotation axis; Target random enhancement means: achieving this through random x, y position changes and random posture changes of the target object on the acquisition plane; Step 5: The acquisition camera remains stationary, the target object is placed on the acquisition plane, and the attitude angles φ, ψ, and θ of the target object are set so that they are randomly sampled within (0, 360); Step 6: Render the randomly sampled images, capture the RGB-D images taken by the acquisition camera in time sequence, and process them as a data set; Step 7: Use the computer vision algorithm to train the data set; in the twin scene, use the training results to drive the verification detection unit to complete the operation task and realize cross-validation of the virtual scene.

2. The method for generating and verifying a visual algorithm simulation data set based on a twin scene according to claim 1, characterized in that: In step 1, the scaled twin scene includes an environment model, a data acquisition model, and a verification and detection unit model; The environmental model includes: Target audience: the target of the data required by the user; Working environment: collect the working environment of the target object; Weather environment and lighting scenes: including scenes under different lighting conditions and different indoor lighting; Background objects: other objects in the scene that interact with the target object; Data acquisition model: The acquisition tool is a simulated camera. The intrinsic and extrinsic parameters, position, and distortion coefficient of the simulated camera are set to be consistent with the real scene, and a camera coordinate system independent of the world coordinate system is constructed; The main form of the verification and detection unit model is: robotic arm grasping in a simulation scene.

3. The method for generating and verifying a visual algorithm simulation data set based on a twin scene according to claim 1, wherein: In step 2, the physical properties include gravity, collision edge, reaction force during collision, and elastic intrinsic properties; Taking the maximum outer contour of each object as the collision edge, a feature map including color map, normal map and height map is added to each object, and the roughness is adjusted according to the actual characteristics of each object.

4. The method for generating and verifying a visual algorithm simulation data set based on a twin scene according to claim 1, wherein: The step six is ​​specifically as follows: Use the acquisition camera to capture the RGB-D image of the current scene at the current moment; automatically obtain the location and type annotation information of each object in the RGB-D image and save it as an XML format file; In the RGB-D image, based on the maximum outer contour of the image, a rectangular target box is constructed in the width and height directions. The center pixel position of the target box and the length and width are recorded in the JSON file as the position label; A segmentation map is constructed using the maximum outer contour of the target object in the acquisition plane. Different target objects are distinguished by different colors. A color coding value is set for each target object. Target objects of the same type are stored with the same color coding value, and the background color is set to black. Construct a cube bounding box based on the maximum outer contour of each object, and record the coordinate positions of the four corners and save them in a json file; Finally, starting from the acquisition camera coordinates, the depth information of all objects within the camera's field of view is collected according to the camera parameters and saved as a depth map.

5. The method for generating and verifying a visual algorithm simulation data set based on a twin scene according to claim 1, wherein: The step seven is specifically as follows: Train the dataset with a computer vision algorithm, add an external control link component, and establish communication between the control end and the virtual scene. Invoke the motion planning service, take a screenshot of the scene as input, instantiate a new RGBA sensor image with the defined dimensions, and send a new vision algorithm service request to the control end. The control end uses an algorithm to predict the corresponding position and make a plan, controlling the virtual scene grasping robot arm to move to the target position to grasp the target object for verification.

Images