Robot data generation method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of robot training, and discloses a robot data generation method, device and equipment and a storage medium, and the method comprises the steps: obtaining a multi-view video of a robot executing a preset action task, carrying out the track extraction of the multi-view video based on a track extraction algorithm, and obtaining a multi-view sample track; encoding the multi-view sample track to obtain a multi-view sample track feature; acquiring an analog video of the robot action, and preprocessing the analog video of the robot action to obtain a plurality of frames of action analog images; encoding the multiple frames of action simulation images to obtain simulation image features; splicing and fusing the multi-view sample track features and the analog image features to obtain fused data; and inputting the fused data as a pre-trained attention mechanism neural network model, and outputting a multi-view sample video by the attention mechanism neural network model. According to the invention, the collection difficulty of the sample data is reduced, and the difference between the sample data and the real environment can be avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot training, and in particular relates to a robot data generation method, device, equipment and storage medium. Background Art

[0002] In the control of dual-arm robots, algorithms based on imitation learning need to collect a large amount of demonstration data for the motion training of dual-arm robots. However, the existing data collection methods are time-consuming and labor-intensive, which seriously affects the development and progress of dual-arm robots. The existing methods can be mainly divided into the following categories: Data collection method based on teach pendant: This method requires building a completely identical teach pendant model and ensuring that the teach pendant's robotic arm control can be accurately fed back to the robot end; the data collection personnel need to operate the teach pendant to complete the corresponding actions, such as folding clothes, pouring coffee, etc. While the robot end receives the action control of the teach pendant, cameras with different perspectives complete the image data collection.

[0003] This method has the following disadvantages: This data collection method is relatively intuitive, but the hardware cost is high, a teaching pendant needs to be built, and the collection process is time-consuming and labor-intensive.

[0004] Simulator-based data generation method: This method requires building a corresponding scene in the simulator and modeling the target object; since the target object and the robotic arm can be precisely controlled in the simulation scene, the corresponding data can be generated through a preset trajectory.

[0005] This method has the following disadvantages: it takes a lot of effort to model different targets, and there is a certain gap between the data generated by simulation and the data of the real scene, which often leads to poor training results for real robotic arms.

[0006] Data collection method based on VR (virtual reality) / AR (augmented reality): This method builds collection equipment based on common devices with gyroscopes in daily life, such as VR helmets and mobile phones, and installs them on the corresponding body parts of data collectors. Data collectors collect real-time image data and gyroscope data from these devices while completing actions.

[0007] This method has the following disadvantages: it requires additional hardware equipment, and the data synchronization between different devices has a great impact on the data quality. At the same time, the collection process is time-consuming.

[0008] In summary, the existing data generation methods have at least the following problems: 1. Data collection is time-consuming and labor-intensive, resulting in low data collection efficiency; 2. The accuracy of the data generated by simulation is low, and there is a certain gap between it and the data of the real scene, resulting in poor training results for the real robotic arm. Summary of the invention

[0009] The purpose of the present invention is to provide a robot data generation method, device, equipment and storage medium to solve the problems of low data collection efficiency and low data accuracy in existing data generation methods.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for generating robot data, the method comprising: Obtain a multi-view video of the robot performing a preset action task, extract trajectories from the multi-view video based on a preset trajectory extraction algorithm, and obtain multi-view sample trajectories; Encode the multi-view sample trajectories to obtain multi-view sample trajectory features; Acquire a simulated video of the robot's motion, pre-process the simulated video of the robot's motion, and obtain a multi-frame motion simulation image; Encode multiple frames of action simulation images to obtain simulation image features; The multi-view sample trajectory features are spliced ​​and fused with the simulated image features to obtain fused data; The fused data is input into a pre-trained attention mechanism neural network model, which outputs multi-view sample videos.

[0011] Preferably, the multi-perspective video includes: a head perspective video, a chest perspective video, a left wrist perspective video and a right wrist perspective video.

[0012] Preferably, the head perspective video is collected by a first image acquisition device deployed on the head of the robot; The chest perspective video is collected by a second image acquisition device deployed on the chest of the robot; The left wrist perspective video is collected by a third image acquisition device deployed on the left wrist of the robot; The right wrist perspective video is collected by a fourth image acquisition device deployed on the right wrist of the robot.

[0013] Preferably, the multi-viewpoint sample videos include: a head viewpoint sample video, a chest viewpoint sample video, a left wrist viewpoint sample video and a right wrist viewpoint sample video.

[0014] Preferably, the simulated video of the robot's action includes at least: a video captured by a third-party device when the robot performs a preset action task, a robot action video on the Internet, and a demonstration video of a grasping task.

[0015] Preferably, the preset action task includes: the robot grabs the target object at a preset initial position to a preset target position.

[0016] Preferably, the method further comprises: constructing text constraints of the fused data, wherein the text constraints are used to expand the types of target objects that the robot can grasp; Construct text constraints for fused data, including: Create text prompts for the target object based on the large language model; Encode the text prompt words based on the encoder in this paper to obtain text features; Generate constrained image features based on text features; Add constrained image features to the fused data.

[0017] In a second aspect, the present invention provides a robot data generating device, the device is used to implement the above-mentioned robot data generating method, and is characterized in that the device comprises: The trajectory extraction module is used to obtain a multi-view video of the robot performing a preset action task, and extract trajectories from the multi-view video based on a preset trajectory extraction algorithm to obtain multi-view sample trajectories; A trajectory encoding module is used to encode multi-view sample trajectories to obtain multi-view sample trajectory features; An image simulation module is used to obtain a simulated video of the robot's motion, pre-process the simulated video of the robot's motion, and obtain a multi-frame motion simulation image; An image encoding module is used to encode multiple frames of action simulation images to obtain simulation image features; A data fusion module is used to merge the multi-view sample trajectory features with the simulated image features to obtain fused data; The sample output module is used to input the fused data into a pre-trained attention mechanism neural network model, and the attention mechanism neural network model outputs a multi-view sample video.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned robot data generation method when executing the computer program.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned robot data generation method when executed by a processor.

[0020] Beneficial effects: 1. The data collection process of the present invention is simple and convenient, does not require additional hardware and specific hardware support, and the collection process can be achieved without any sense of collection; 2. The present invention only needs to collect a small amount of multi-view videos of robots performing preset action tasks, and then combine them with a large number of simulated videos of robot actions, and splice and fuse the multi-view sample trajectory features with the simulated image features, and then use a deep learning algorithm to complete a variety of required perspective transformations, which can generate a large amount of sample data (i.e., multi-view sample videos) that can be directly used for robot imitation learning, reducing the difficulty of collecting sample data; 3. The present invention collects a large number of simulated videos, so the generated sample data is closer to the real data in the real environment, and the difference between the sample data and the real environment can be effectively avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 is a flow chart of a robot data generation method provided by one embodiment of the present invention; Figure 2 It is a block diagram of a robot data generating device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0023] Embodiment 1 Figure 1 FIG. 1 is a flow chart of a method for generating robot data provided by an embodiment of the present invention. Figure 1 As shown, this embodiment provides a robot data generation method, the method comprising: Step S10: obtaining a multi-view video of the robot performing a preset action task, extracting trajectories from the multi-view video based on a preset trajectory extraction algorithm, and obtaining multi-view sample trajectories.

[0024] In this embodiment, the preset action task includes: the robot grabs the target object at a preset initial position to a preset target position, such as pouring coffee, or folding clothes; then installing cameras and other monitoring equipment on the robot to collect multi-view videos of the robot performing the preset action task.

[0025] The multi-view video includes: videos from four different angles, namely, head perspective video, chest perspective video, left wrist perspective video and right wrist perspective video. The videos from the four angles can reflect the trajectories under four specific perspectives.

[0026] The head perspective video is collected by a first image acquisition device deployed on the head of the robot; the chest perspective video is collected by a second image acquisition device deployed on the chest of the robot; the left wrist perspective video is collected by a third image acquisition device deployed on the left wrist of the robot; and the right wrist perspective video is collected by a fourth image acquisition device deployed on the right wrist of the robot. The first image acquisition device, the second image acquisition device, the third image acquisition device, and the fourth image acquisition device all use high-definition cameras.

[0027] In this embodiment, the preset trajectory extraction algorithm can adopt a deep learning-based method. For example, target detection algorithms such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), Fast R-CNN, and FasterR-CNN can be used to extract trajectories. Video image point tracking technology can also be used, such as CoTracker. CoTracker is a transformer-based model that can track dense point trajectories of video sequence images.

[0028] In this embodiment, the multi-perspective video of the robot performing a preset action task can be directly used to train the robot, but such data is relatively scarce, and a large number of scenes need to be established to collect relevant multi-perspective videos, which is time-consuming and labor-intensive. In order to solve the problem of scarce data, this embodiment extracts trajectories from the multi-perspective videos collected under a small number of preset action tasks to obtain multi-perspective sample trajectories, and then executes steps S20 to S60 to obtain more multi-perspective sample videos, and use the multi-perspective sample videos as sample data for robot training.

[0029] Step S20: Encode the multi-view sample trajectory to obtain multi-view sample trajectory features; In this embodiment, a convolutional neural network is used to construct a trajectory encoder, and the trajectory encoder is used to encode the multi-view sample trajectory, and the trajectory encoder outputs the multi-view sample trajectory features.

[0030] Step S30: obtaining a simulated video of the robot's motion, and preprocessing the simulated video of the robot's motion to obtain a plurality of frames of motion simulation images; In this embodiment, the simulated video of the robot's action includes at least: a video captured by a third-party device when the robot performs a preset action task, a robot action video on the Internet, and a demonstration video of the grasping task.

[0031] Among them, the video captured by a third-party device when the robot performs a preset action task, the third-party device is a camera installed on the head of the robot, through which the third-party perspective video of the robot when performing the preset action task is captured; or the camera is installed in other positions to capture the third-party perspective video of the robot when performing the preset action task.

[0032] Among them, the robot action videos on the Internet are videos of robot execution-related actions that are searched on the Internet. There are sufficient videos on the Internet, which can provide a large amount of sample data.

[0033] Among them, the demonstration video of the grasping task is a video of a demonstrator completing an action, and then the demonstrator's arm in the demonstration video is replaced by the robot's mechanical arm.

[0034] Step S40: encoding multiple frames of action simulation images to obtain simulation image features; In this embodiment, a convolutional neural network is also used to construct an image encoder, and the image encoder is used to encode multiple frames of various simulated images to obtain simulated image features.

[0035] Step S50: splicing and fusing the multi-view sample trajectory features with the simulated image features to obtain fused data; In this embodiment, the stitching and fusion process can adopt a deep learning algorithm, such as a multi-channel data stitching algorithm. The multi-channel data stitching algorithm combines data from different channels or sources to form a more complete or richer data set. The stitching logic can adopt time alignment, spatial alignment and feature alignment.

[0036] Step S60: The fused data is used as an input to a pre-trained attention mechanism neural network model, and the attention mechanism neural network model outputs a multi-view sample video.

[0037] In this embodiment, the pre-trained attention mechanism neural network model includes: an attention layer and a decoding layer. During the training process, the attention mechanism neural network model continuously obtains an output of a specific dimension through encoding, convolution operations, upsampling, decoding and other operations. The output is compared with the real samples in the training data set, and is continuously iterated through gradient descent to finally obtain a trained model.

[0038] Among them, the decoding layer includes: a head decoder, a chest decoder, a left wrist decoder and a right wrist decoder; after the fusion data is encoded based on the attention mechanism of the attention layer, it is decoded by the decoding layer to output a multi-view sample video, and the multi-view sample video includes: a head perspective sample video, a chest perspective sample video, a left wrist perspective sample video and a right wrist perspective sample video.

[0039] In this embodiment, the decoding operation is performed by the head decoder, and the head decoder outputs a head perspective sample video; the decoding operation is performed by the chest decoder, and the chest decoder outputs a chest perspective sample video; the decoding operation is performed by the left wrist decoder, and the left wrist decoder outputs a left wrist perspective sample video; the decoding operation is performed by the right wrist decoder, and the right wrist decoder outputs a right wrist perspective sample video.

[0040] As a further optimization of this embodiment, the method further includes: constructing text constraints of the fused data, the text constraints being used to expand the types of target objects that the robot can grasp; Among them, the text constraints of the fused data are constructed, including: Step a10: Create text prompt words for the target object based on the large language model; Step a20: Encode the text prompt words based on the encoder in this article to obtain text features; Step a30: generating constrained image features based on text features; Step a40: adding the constrained image features to the fused data.

[0041] In this embodiment, after constructing the text constraint, the target object can be edited through the text and replaced with other objects, such as replacing the action of pouring coffee with the action of pouring water, or replacing it with the action of serving rice, or replacing the action of folding clothes with the action of folding quilts, thereby achieving the effect of data augmentation.

[0042] Therefore, the present invention does not rely on the data set for imitation learning. It is based on the action video data from the third-party perspective, and completes a variety of required perspective transformations by adding a deep learning algorithm with specific trajectory data and perspective data, thereby generating data that can be directly used for robot imitation learning; and, it makes full use of the large language model as a base, adds text constraints in the process of data generation, and combines the text-based graph model or the text-based video model to realize the image / video editing of the target object, thereby achieving the effect of data augmentation; and, training an independent camera (a camera from a third-party perspective) can more finely control the perspective required for generating the video, and the generated video data is more realistic and reliable; secondly, data acquisition is easier, and there is no need to build a teaching pendant to collect data, and a large amount of data from a third-party perspective can be used for transformation and video generation.

[0043] Embodiment 2 Figure 2 FIG. 1 is a block diagram of a robot data generating device provided by one embodiment of the present invention. Figure 2 As shown, this embodiment provides a robot data generating device, which is used to implement the robot data generating method of embodiment 1, and is characterized in that the device includes: The trajectory extraction module is used to obtain a multi-view video of the robot performing a preset action task, and extract trajectories from the multi-view video based on a preset trajectory extraction algorithm to obtain multi-view sample trajectories; A trajectory encoding module is used to encode multi-view sample trajectories to obtain multi-view sample trajectory features; An image simulation module is used to obtain a simulated video of the robot's motion, pre-process the simulated video of the robot's motion, and obtain a multi-frame motion simulation image; An image encoding module is used to encode multiple frames of action simulation images to obtain simulation image features; A data fusion module is used to merge the multi-view sample trajectory features with the simulated image features to obtain fused data; The sample output module is used to input the fused data into a pre-trained attention mechanism neural network model, and the attention mechanism neural network model outputs a multi-view sample video.

[0044] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the robot data generating method of the first embodiment is implemented when the processor executes the computer program.

[0045] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the robot data generating method of the first embodiment is implemented.

[0046] The data collection process of the present invention is simple and convenient, does not require additional hardware and specific hardware support, and the collection process can be achieved without any sense of collection; moreover, the present invention only needs to collect a small amount of multi-perspective videos of robots performing preset action tasks, and then combine them with a large amount of simulated videos of robot actions, and splice and fuse the multi-perspective sample trajectory features with the simulated image features, and then use a deep learning algorithm to complete a variety of required perspective transformations, which can generate a large amount of sample data (i.e., multi-perspective sample videos) that can be directly used for robot imitation learning, reducing the difficulty of collecting sample data; secondly, the present invention collects a large amount of simulated videos, so the generated sample data is closer to the real data in the real environment, and the difference between the sample data and the real environment can be effectively avoided.

[0047] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0048] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0049] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A robot data generation method, characterized in that: The method comprises: Obtain a multi-view video of the robot performing a preset action task, extract trajectories from the multi-view video based on a preset trajectory extraction algorithm, and obtain a multi-view sample trajectory; Encode the multi-view sample trajectories to obtain multi-view sample trajectory features; Acquire a simulated video of the robot's motion, pre-process the simulated video of the robot's motion, and obtain a multi-frame motion simulation image; Encode multiple frames of action simulation images to obtain simulation image features; The multi-view sample trajectory features are spliced ​​and fused with the simulated image features to obtain fused data; The fused data is input into a pre-trained attention mechanism neural network model, which outputs multi-view sample videos.

2. The robot data generation method according to claim 1, characterized in that: The multi-view video includes: a head view video, a chest view video, a left wrist view video and a right wrist view video.

3. The robot data generation method according to claim 2, characterized in that: The head perspective video is collected by a first image acquisition device deployed on the head of the robot; The chest perspective video is collected by a second image acquisition device deployed on the chest of the robot; The left wrist perspective video is collected by a third image acquisition device deployed on the left wrist of the robot; The right wrist perspective video is collected by a fourth image acquisition device deployed on the right wrist of the robot.

4. The robot data generation method according to claim 2 or 3, characterized in that: The multi-view sample videos include: a head view sample video, a chest view sample video, a left wrist view sample video and a right wrist view sample video.

5. The robot data generation method according to claim 1, characterized in that: The simulated video of the robot's action includes at least: a video captured by a third-party device when the robot performs a preset action task, a robot action video on the Internet, and a demonstration video of a grasping task.

6. The robot data generation method according to claim 1, characterized in that: The preset action task includes: the robot grabs the target object at a preset initial position to a preset target position.

7. The robot data generation method according to claim 6, characterized in that: The method further includes: constructing text constraints of the fused data, wherein the text constraints are used to expand the types of target objects that the robot can grasp; Construct text constraints for fused data, including: Create text prompts for the target object based on the large language model; Encode the text prompt words based on the encoder in this paper to obtain text features; Generate constrained image features based on text features; Add constrained image features to the fused data.

8. A robot data generating device, the device being used to implement the robot data generating method according to any one of claims 1 to 7, characterized in that: The device comprises: The trajectory extraction module is used to obtain a multi-view video of the robot performing a preset action task, and extract trajectories from the multi-view video based on a preset trajectory extraction algorithm to obtain multi-view sample trajectories; A trajectory encoding module is used to encode multi-view sample trajectories to obtain multi-view sample trajectory features; An image simulation module is used to obtain a simulated video of the robot's motion, pre-process the simulated video of the robot's motion, and obtain a multi-frame motion simulation image; An image encoding module is used to encode multiple frames of action simulation images to obtain simulation image features; A data fusion module is used to merge the multi-view sample trajectory features with the simulated image features to obtain fused data; The sample output module is used to input the fused data into a pre-trained attention mechanism neural network model, and the attention mechanism neural network model outputs a multi-view sample video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the robot data generating method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the robot data generating method according to any one of claims 1 to 7 is implemented.

Images