Data generation, model training, pose recognition, robot control method and device

Abstract

This application belongs to the field of robot control technology, specifically relating to a data generation, model training, pose recognition, and robot control method and apparatus. The data generation method includes: acquiring simulation data and real-world data for robot pose control; inputting the simulation data into a cross-domain alignment model to perform cross-domain alignment from the simulation domain to the real domain, obtaining cross-domain aligned data; the cross-domain alignment model is trained using a recurrent generative adversarial network (RGAN) on a first training dataset, which includes simulated images and pose labels of the robot pose, as well as corresponding real images; merging the cross-domain aligned data and the real-world data to obtain a second training dataset for training a pose estimation model. This application aligns simulation data to real-world data, thereby generating a high-quality fused dataset that combines the diversity of simulation data with the fidelity of real-world data, significantly improving the transfer performance of robot models from simulation to reality.

Description

Technical Field

[0001] This application belongs to the field of robot control technology, specifically relating to a data generation, model training, pose recognition, robot control method and device. Background Technology

[0002] The development of embodied intelligent robots relies heavily on large-scale, high-quality training data. However, obtaining effective training data faces challenges in specialized applications such as the nuclear industry. Existing technologies primarily acquire data through the following two methods: 1. Real-world data acquisition: This method involves collecting data on robot interactions with the environment in real-world settings through manual remote operation or teaching. While this approach offers high data fidelity, its drawbacks are also significant. High cost and significant risk: Data acquisition in hazardous environments such as those with nuclear radiation or high-voltage electrical systems poses a threat to personnel safety and incurs extremely high equipment wear and tear and time costs. Limited data scale and diversity: It is difficult to cover all possible operating conditions and abnormal scenarios, resulting in insufficient generalization ability of the trained model, making it prone to failure when encountering unfamiliar scenarios. Difficult annotation: Obtaining accurate annotation information such as physical state and 3D pose is extremely difficult and time-consuming.

[0003] 2. Virtual Data Generation in Simulation Environments: This method involves constructing a digital twin environment within simulation software to generate large-scale, automated training data with perfect annotations. It is safe, efficient, and low-cost. However, its core bottleneck lies in the differences between the simulation environment and the physical world in terms of physical laws, lighting rendering, material textures, and sensor noise. This leads to a sharp performance drop, or even complete failure, when models trained on purely simulation data are directly deployed to real robots.

[0004] In an effort to combine the advantages of both, existing technologies attempt to train models by mixing simulated and real data, but this does not fundamentally solve the model performance problems caused by differences in data distribution. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for data generation, model training, pose recognition, and robot control, so as to solve the model performance problem caused by the difference in data distribution between simulation data and real data in the prior art.

[0006] The technical solution to achieve the purpose of this application is as follows: The first aspect of this application provides a data generation method, the method comprising: Acquire simulation data and real-world data for robot pose control; The simulation data is input into a pre-trained cross-domain alignment model to perform cross-domain alignment from the simulation domain to the real domain, resulting in cross-domain aligned data. The cross-domain alignment model is trained using a first training dataset based on a recurrent generative adversarial network. The first training dataset includes simulation images and pose labels of the robot pose, as well as the corresponding real images. The cross-domain aligned data and the real collected data are combined to obtain a second training dataset for training the pose estimation model.

[0007] Optionally, the coordinate systems and timestamps of the simulated image and the real image are aligned.

[0008] Optionally, the cross-domain alignment model assigns higher weights to object edges and contact points in the image compared to other regions during training.

[0009] Optionally, after obtaining the cross-domain aligned data, the process further includes: Based on the cross-domain aligned data, other simulated data are generated using a structure-texture decoupled generative adversarial network. Therefore, the second training dataset also includes the other simulation data.

[0010] A second aspect of this application provides a data generation apparatus, the apparatus comprising: The first acquisition module is used to acquire simulation data and real-world data for robot pose control; The data alignment module is used to input the simulation data into a pre-trained cross-domain alignment model for cross-domain alignment to obtain cross-domain aligned data. The cross-domain alignment model is trained based on a recurrent generative adversarial network, and the corresponding first training dataset includes simulation images and pose labels of robot poses and corresponding real images. The data merging module is used to merge the cross-domain aligned data and the real collected data to obtain a second training dataset for training the pose estimation model.

[0011] A third aspect of this application provides a model training method, the method comprising: Obtain a second training dataset; the second training dataset is obtained using any one of the data generation methods provided in the second aspect of the embodiments of this application. The pose estimation model is trained using the second training dataset.

[0012] A fourth aspect of this application provides a model training apparatus, the apparatus comprising: The second acquisition module is used to acquire a second training dataset; the second training dataset is obtained using any one of the data generation methods provided in the second aspect of the embodiments of this application. The model training module is used to train the pose estimation model using the second training dataset.

[0013] A fifth aspect of this application provides a pose recognition method, the method comprising: Obtain an image of the robot's current position; The current pose of the robot is determined using a pre-trained pose estimation model; the pose estimation model is obtained according to any one of the model training methods provided in the third aspect of the embodiments of this application.

[0014] A sixth aspect of this application provides a pose recognition device, the device comprising: The third acquisition module is used to acquire an image of the robot's current position. The pose determination module is used to determine the current pose of the robot using a pre-trained pose estimation model; the pose estimation model is obtained according to any one of the model training methods provided in the third aspect of the embodiments of this application.

[0015] A seventh aspect of this application provides a robot control method, the method comprising: The current pose of the robot can be obtained using any of the pose recognition methods provided in the fifth aspect of the embodiments of this application; The robot is controlled based on the current pose and the desired pose.

[0016] An eighth aspect of this application provides a robot control device, the device comprising: The fourth acquisition module is used to acquire the current pose of the robot using any one of the pose recognition methods provided in the fifth aspect of the embodiments of this application; The pose control module is used to control the robot based on the current pose and the desired pose.

[0017] A ninth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, it implements any one of the data generation methods provided in the first aspect of this application; or, any one of the model training methods provided in the third aspect of this application; or, any one of the pose recognition methods provided in the fifth aspect of this application; or, any one of the pose recognition methods provided in the seventh aspect of this application.

[0018] A tenth aspect of this application provides a controller, including a memory and a processor; the memory stores a computer program; when the controller executes the computer program, it implements any one of the data generation methods provided in the first aspect of this application; or, implements any one of the model training methods provided in the third aspect of this application; or, implements any one of the pose recognition methods provided in the fifth aspect of this application; or, implements any one of the pose recognition methods provided in the seventh aspect of this application.

[0019] The beneficial technical effects of this application are as follows: This application provides a data generation, model training, pose recognition, and robot control method and apparatus. The data generation method includes: acquiring simulation data and real-world data for robot pose control; inputting the simulation data into a pre-trained cross-domain alignment model to perform cross-domain alignment from the simulation domain to the real domain, obtaining cross-domain aligned data; the cross-domain alignment model is trained using a recurrent generative adversarial network (RGAN) on a first training dataset, which includes simulation images and pose labels of the robot pose, as well as corresponding real images; and merging the cross-domain aligned data and the real-world data to obtain a second training dataset for training a pose estimation model. This method utilizes RGAN technology to align large-scale, easily accessible simulation data to real-world data in terms of visual style and data distribution, thereby generating a high-quality fused dataset that combines the diversity of simulation data with the fidelity of real-world data, significantly improving the transfer performance of robot models from simulation to reality. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a data generation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a data generation device provided in an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only a part of the embodiments of this application, and not all of them. Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The inventors of this application have discovered in their research that how to effectively utilize the scale advantage of simulation data while bridging the gap with real data is a key technical challenge that urgently needs to be addressed in the field of embodied intelligence.

[0023] To this end, embodiments of this application provide a data generation, model training, pose recognition, and robot control method and apparatus. By aligning large-scale simulation data with small-scale real data in terms of style and distribution, a high-quality fusion dataset with both diversity and realism is generated, aiming to significantly improve the transfer performance of robot models from simulation to reality.

[0024] Based on the above, in order to clearly and in detail illustrate the advantages of this application, the specific embodiments of this application will be described below in conjunction with the accompanying drawings.

[0025] See Figure 1 The figure is a flowchart illustrating a data generation method provided in an embodiment of this application.

[0026] This application provides a data generation method, including: Step S101: Acquire simulation data and real-world data for robot pose control.

[0027] In practical implementation, simulation data can be generated on a large scale and parametrically to construct a digital twin environment highly similar to the target physical scene, including visual images, sensor readings, robot joint states, and precise ground truth labels. Parametric generation refers to randomizing lighting, object poses, textures, and other parameters in the scene to increase data diversity. Real-world data can be acquired in a real physical environment through teleoperation devices (such as force feedback gloves or joysticks) or expert teaching, collecting small-scale, representative real-world interactive data. The acquired real-world data is consistent with the simulation data but requires no complex annotation. Simulation data and real-world data can be generated in parallel.

[0028] Step S102: Input the simulation data into the pre-trained cross-domain alignment model to perform cross-domain alignment from the simulation domain to the real domain, and obtain the cross-domain aligned data.

[0029] In this embodiment of the application, the cross-domain alignment model is trained using a first training dataset based on a recurrent generative adversarial network. The first training dataset includes simulated images of robot poses, pose labels, and corresponding real images.

[0030] It is understandable that the methods for acquiring simulated images and real images can be the same as those for acquiring simulated data and real collected data; this will not be limited or elaborated upon here.

[0031] In one example, the coordinate systems and timestamps of the simulated image and the real image are aligned.

[0032] It should be noted that by establishing a spatiotemporal correlation between the simulation environment and the physical environment, the comparability of simulated and real images in spatial pose and temporal sequence is ensured, providing a foundation for subsequent cross-domain alignment. In specific implementation, alignment can be performed based on coordinate system alignment algorithms of transformation trees and timestamp synchronization mechanisms such as network time protocols; no specific limitations are imposed here.

[0033] It should be noted that the cross-domain alignment model receives both simulated and real images as input, and uses a deep generative model to transfer the domain of the simulated image to the domain of the real image. An unpaired image-to-image translation network with a recurrent generative adversarial network (CycleGAN) as its core is employed. The cross-domain alignment model learns the mapping function from the simulated image domain to the real image domain, enabling it to convert the rendering style (such as lighting and texture) of the simulated image into a realistic physical world style without altering the core content of the image (such as robot pose and object position), thus achieving cross-domain image alignment. In practical applications, the effectiveness of cross-domain alignment can be objectively evaluated and optimized by minimizing the domain difference index.

[0034] In another example, to address the image artifacts or loss of critical details that may occur during the transfer process in standard CycleGAN, the cross-domain alignment model assigns higher weights to object edges and contact points in the image compared to other regions during training. This ensures the clarity and fidelity of the generated cross-domain aligned data in critical operational areas, better protecting key visual details related to robot operation. In practical implementation, this can be achieved by introducing a boundary-weighted loss function.

[0035] Step S103: Merge the cross-domain aligned data and the real collected data to obtain a second training dataset for training the pose estimation model.

[0036] This application combines the scale advantages of simulation data generation with the realism of physical data acquisition. Using unpaired image translation networks such as CycleGAN as the core, it translates simulated data into the real domain to obtain cross-domain aligned data. Through cross-domain alignment, the generated fused dataset (i.e., the second training dataset) more closely resembles the real world in data distribution, forming the final large-scale, high-quality, virtual-real fusion training dataset.

[0037] Tests have shown that the FID (Fréchet Inception Distance) between simulation and real-world images can be reduced by more than 35%, thereby significantly improving the robot's task success rate in real-world environments. This application's embodiments only require collecting a small amount of real-world data as "style samples" to utilize massive amounts of simulation data, greatly reducing reliance on expensive and dangerous real-world data acquisition. The generated second training dataset combines the diversity of simulation data with the fidelity of real-world data, covering various extreme and rare conditions. The trained robot model is more robust to environmental changes and has stronger generalization capabilities.

[0038] In practical implementation, a continuously optimized data loop can be formed. After discovering scenarios in real-world environments where the model trained on the second training dataset performs poorly, more such data can be generated in the simulation in a targeted manner, and then aligned and fused using the methods provided in the embodiments of this application, thereby rapidly iteratively optimizing model performance.

[0039] In some possible implementations of the embodiments of this application, the step of obtaining the cross-domain aligned data may further include: Based on the cross-domain aligned data, other simulated data are generated using a structure-texture decoupled generative adversarial network. Therefore, the second training dataset also includes the other simulation data.

[0040] Understandably, self-supervised dense correspondence learning networks and structure-texture decoupled generative adversarial networks (CoordGAN) can be used to generate more diverse samples on cross-domain aligned data, further enhancing the richness of the dataset.

[0041] The following describes in detail, with a specific example, a data generation method provided by the embodiments of this application.

[0042] This application provides a data generation method that constructs a 3D scene including a robot model, a workbench, and a wrench. Through automated scripts, the wrench's position and pose, the direction and intensity of ambient lighting, and the workbench texture are randomized in each of 10,000 iterations. In each iteration, the system records the robot's RGB camera image, depth image, and the wrench's precise 6D pose as labels, generating 10,000 sets of simulation data. An operator then remotely controls the robot to grasp the wrench 20 times on a real production line workbench. During this process, the system records the actual video stream from the robot's camera. This real data is only used for subsequent style transfer and does not require manual labeling of the wrench pose.

[0043] If digital twin-level alignment is required, ensure that the coordinate systems of the simulated robot and the real robot are aligned, and that the timestamps of the data streams are aligned. This step can be simplified for style transfer.

[0044] 10,000 simulated images and 20 real-world capture videos (approximately several hundred images) were fed into the CycleGAN model for training. One generator in the model, G_S→R, learned to convert the simulated images to a real style, while another generator, G_R→S, learned the reverse conversion.

[0045] During training, in addition to standard adversarial loss and cycle consistency loss, a boundary-weighted loss function can be used to calculate image gradients, identify the contours of wrenches and robot grippers, and apply a larger loss penalty to these regions, forcing the generator to maintain the sharpness and integrity of these key areas during style transfer. This boundary-weighted loss function first extracts the image's gradient map using edge detection algorithms such as the Sobel operator to identify the contours of the tools and robot grippers. Then, when calculating the pixel-level reconstruction loss, a weight coefficient greater than 1 is applied to the pixel errors of these contour regions, forcing the generator G_S→R to reconstruct the details of these key regions more accurately during style transfer.

[0046] After training, the CycleGAN model can transform any simulated grasping image into an image that looks like it was taken on a real production line, while keeping the relative poses of the wrench and the robot unchanged, resulting in a cross-domain alignment model. 10,000 simulated images that have undergone cross-domain alignment and their original precise pose labels are merged with a small number of original, unlabeled real images to form the final second training dataset. This high-quality fused dataset (i.e., the second training dataset) is used to train an object pose estimation algorithm. Because the second training dataset is visually highly consistent with the real world and contains a wide variety of poses and lighting conditions, the trained model can be directly deployed on real robots, exhibiting high accuracy and robust grasping capabilities.

[0047] The embodiments of this application can effectively utilize simulation data and successfully solve the data bottleneck problem in robot training through an innovative cross-domain alignment method, providing key technical support for the application of embodied intelligence in complex industrial scenarios.

[0048] Based on the data generation method provided in the above embodiments, this application also provides a data generation apparatus.

[0049] See Figure 2 The figure is a schematic diagram of the structure of a data generation device provided in an embodiment of this application.

[0050] This application provides a data generation apparatus, comprising: The first acquisition module 100 is used to acquire simulation data and real-world data for robot pose control. The data alignment module 200 is used to input the simulation data into a pre-trained cross-domain alignment model for cross-domain alignment to obtain cross-domain aligned data; the cross-domain alignment model is trained based on a recurrent generative adversarial network, and the corresponding first training dataset includes simulation images and pose labels of robot poses and corresponding real images; The data merging module 300 is used to merge the cross-domain aligned data and the real collected data to obtain a second training dataset for training the pose estimation model.

[0051] In one example, the coordinate systems and timestamps of the simulated image and the real image are aligned.

[0052] In another example, the cross-domain alignment model assigns higher weights to object edges and contact points in the image compared to other regions during training.

[0053] In some possible implementations of the embodiments of this application, the apparatus further includes: The data generation module is used to generate other simulated data based on the cross-domain aligned data using a structure-texture decoupled generative adversarial network. Therefore, the second training dataset also includes the other simulation data.

[0054] This application utilizes recurrent generative adversarial network technology to align large-scale, easily accessible simulation data with real-world data in terms of visual style and data distribution, thereby generating a high-quality fused dataset that combines the diversity of simulation data with the fidelity of real data, significantly improving the transfer performance of robot models from simulation to reality.

[0055] Based on the data generation method and apparatus provided in the above embodiments, this application also provides a model training method.

[0056] This application provides a model training method, including: Obtain a second training dataset; the second training dataset is obtained using any one of the data generation methods provided in the above embodiments. The pose estimation model is trained using the second training dataset.

[0057] Based on the data generation method and apparatus and model training method provided in the above embodiments, this application also provides a model training apparatus.

[0058] This application provides a model training device, comprising: The second acquisition module is used to acquire a second training dataset; the second training dataset is obtained using any one of the data generation methods provided in the above embodiments. The model training module is used to train the pose estimation model using the second training dataset.

[0059] Based on the data generation method and apparatus, and model training method and apparatus provided in the above embodiments, this application also provides a pose recognition method.

[0060] This application provides a pose recognition method, including: Obtain an image of the robot's current position; The current pose of the robot is determined using a pre-trained pose estimation model; the pose estimation model is obtained according to any one of the model training methods provided in the above embodiments.

[0061] Based on the data generation method and apparatus, model training method and apparatus, and pose recognition method provided in the above embodiments, this application also provides a pose recognition apparatus.

[0062] This application provides a pose recognition device, comprising: The third acquisition module is used to acquire an image of the robot's current position. The pose determination module is used to determine the current pose of the robot using a pre-trained pose estimation model; the pose estimation model is obtained according to any one of the model training methods provided in the above embodiments.

[0063] Based on the data generation method and apparatus, model training method and apparatus, and pose recognition method and apparatus provided in the above embodiments, this application also provides a robot control method.

[0064] This application provides a robot control method, including: The current pose of the robot can be obtained using any of the pose recognition methods provided in the above embodiments; The robot is controlled based on the current pose and the desired pose.

[0065] Based on the data generation method and apparatus, model training method and apparatus, pose recognition method and apparatus, and robot control method provided in the above embodiments, this application also provides a robot control apparatus.

[0066] This application provides a robot control device, comprising: The fourth acquisition module is used to acquire the current pose of the robot using any one of the pose recognition methods provided in the above embodiments; The pose control module is used to control the robot based on the current pose and the desired pose.

[0067] Based on the data generation method and apparatus, model training method and apparatus, pose recognition method and apparatus, and robot control method and apparatus provided in the above embodiments, this application also provides a computer-readable storage medium storing a computer program. When the computer program is executed, it implements any one of the data generation methods provided in the above embodiments; or, any one of the model training methods provided in the above embodiments; or, any one of the pose recognition methods provided in the above embodiments; or, any one of the pose recognition methods provided in the above embodiments.

[0068] Based on the data generation method and apparatus, model training method and apparatus, pose recognition method and apparatus, and robot control method and apparatus provided in the above embodiments, this application also provides a controller, including a memory and a processor; the memory stores a computer program; when the controller executes the computer program, it implements any one of the data generation methods provided in the above embodiments; or, implements any one of the model training methods provided in the above embodiments; or, implements any one of the pose recognition methods provided in the above embodiments; or, implements any one of the pose recognition methods provided in the above embodiments.

[0069] The present application has been described in detail above with reference to the accompanying drawings and embodiments. However, the present application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present application. All content not described in detail in this application can be derived from existing technology.

Claims

1. A data generation method, characterized in that, The method includes: Acquire simulation data and real-world data for robot pose control; The simulation data is input into a pre-trained cross-domain alignment model to perform cross-domain alignment from the simulation domain to the real domain, resulting in cross-domain aligned data. The cross-domain alignment model is trained using a first training dataset based on a recurrent generative adversarial network. The first training dataset includes simulation images and pose labels of the robot pose, as well as the corresponding real images. The cross-domain aligned data and the real collected data are combined to obtain a second training dataset for training the pose estimation model.

2. The data generation method according to claim 1, characterized in that, The coordinate systems and timestamps of the simulated image and the real image are aligned.

3. The data generation method according to claim 1, characterized in that, The cross-domain alignment model assigns higher weights to object edges and contact points in the image compared to other regions during training.

4. The data generation method according to claim 1, characterized in that, After obtaining the cross-domain aligned data, the process further includes: Based on the cross-domain aligned data, other simulated data are generated using a structure-texture decoupled generative adversarial network. Therefore, the second training dataset also includes the other simulation data.

5. A data generation device, characterized in that, The device includes: The first acquisition module is used to acquire simulation data and real-world data for robot pose control; The data alignment module is used to input the simulation data into a pre-trained cross-domain alignment model for cross-domain alignment to obtain cross-domain aligned data. The cross-domain alignment model is trained based on a recurrent generative adversarial network, and the corresponding first training dataset includes simulation images and pose labels of robot poses and corresponding real images. The data merging module is used to merge the cross-domain aligned data and the real collected data to obtain a second training dataset for training the pose estimation model.

6. A model training method, characterized in that, The method includes: Obtain a second training dataset; the second training dataset is obtained using the data generation method according to any one of claims 1-4; The pose estimation model is trained using the second training dataset.

7. A model training device, characterized in that, The device includes: The second acquisition module is used to acquire a second training dataset; the second training dataset is obtained using the data generation method according to any one of claims 1-4. The model training module is used to train the pose estimation model using the second training dataset.

8. A pose recognition method, characterized in that, The method includes: Obtain an image of the robot's current position; The current pose of the robot is determined using a pre-trained pose estimation model; the pose estimation model is obtained according to the model training method described in claim 6.

9. A pose recognition device, characterized in that, The device includes: The third acquisition module is used to acquire an image of the robot's current position. The pose determination module is used to determine the current pose of the robot using a pre-trained pose estimation model; the pose estimation model is obtained according to the model training method described in claim 6.

10. A robot control method, characterized in that, The method includes: The current pose of the robot is obtained using the pose recognition method described in claim 8; The robot is controlled based on the current pose and the desired pose.

11. A robot control device, characterized in that, The device includes: The fourth acquisition module is used to acquire the current pose of the robot using the pose recognition method described in claim 8; The pose control module is used to control the robot based on the current pose and the desired pose.

12. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed, implements the data generation method as described in any one of claims 1-4; or, implements the model training method as described in claim 6; or, implements the pose recognition method as described in claim 8; or, implements the robot control method as described in claim 10.

13. A controller, characterized in that, It includes a memory and a processor; the memory stores a computer program; when the controller executes the computer program, it implements the data generation method as described in any one of claims 1-4; or, implements the model training method as described in claim 6; or, implements the pose recognition method as described in claim 8; or, implements the robot control method as described in claim 10.

Images