Robot multi-modal data generation method and system based on potential diffusion model

Abstract

The invention discloses a robot multi-modal data generation method and system based on a potential diffusion model, and the method comprises the steps: obtaining the multi-modal data needed by the perception of a robot, carrying out the preprocessing and data enhancement operation of the multi-modal data, and generating a multi-modal data set; constructing a multi-modal data generation model based on the potential diffusion model, wherein the model comprises a variational auto-encoder, a U-Net network and a conditional embedding encoder; determining training configuration, dynamically adjusting the fusion weight of each mode, and training the model by using a multi-mode data set; the condition information is used for guiding, and controllable multi-modal data generation is carried out through the model. The problems of difficulty in data acquisition and high cost in the field of robot sensing are effectively solved. By introducing the potential diffusion model, high-quality large-scale multi-modal data can be generated, and support is provided for various downstream tasks, so that the sensing ability of the robot is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a robot multimodal data generation method and system based on a potential diffusion model. Background Art

[0002] In recent years, robotics technology has developed rapidly and has been gradually applied to a variety of scenarios such as industrial manufacturing, medical rehabilitation, and the service industry. With the expansion of application scenarios, the demand for robots in terms of autonomy and intelligence has increased significantly. Especially at the perception level, robots need to accurately identify and understand various information in the external environment in order to make real-time decisions. The key to achieving this goal is multimodal perception, that is, robots understand the environment through multiple information sources such as vision, touch, and hearing.

[0003] In the process of robot perception training, the sufficiency and quality of multimodal data directly affect its recognition and response capabilities. High-quality multimodal data is not only expensive to obtain, but also has the problem of data missing, especially in scenarios with complex environments and multiple tasks. Therefore, how to obtain sufficient high-quality multimodal data at a low cost has become a key requirement for the development of robot perception technology.

[0004] In response to this problem, a variety of generative models have emerged in recent years to generate simulated training data, such as generative adversarial networks and autoencoders. However, these methods still face certain limitations in generating multimodal data. For example, the training of generative adversarial networks in data generation is prone to instability and it is difficult to balance the correlation between the modalities of the data; autoencoders are prone to losing detailed features in the data. In addition, most generative models can only generate unimodal data, which limits their scope of application and reduces generation efficiency. Summary of the invention

[0005] In view of this, the present invention proposes a robot multimodal data generation method and system based on a potential diffusion model.

[0006] One aspect of the present invention includes a method for generating multimodal data of a robot based on a potential diffusion model, comprising the following steps: S1, obtains the multimodal data required for robot perception, and performs preprocessing and data enhancement operations on it to generate a multimodal dataset; S2, according to the actual application, select the specific model architecture according to the data type, and build a multimodal data generation model based on the potential diffusion model. The multimodal data generation model based on the potential diffusion model includes a variational autoencoder, a U-Net network, and a conditional embedding encoder; among them, the variational autoencoder is used for data compression and data reconstruction; the U-Net network is responsible for predicting the noise in the diffusion process and continuously optimizing the generation process; the conditional embedding encoder is used to continuously inject conditional information while the U-Net network predicts the noise; S3, determine the training configuration, dynamically adjust the fusion weight of each modality, and use the multimodal data set to train the multimodal data generation model based on the latent diffusion model; S4, guided by conditional information, controllable multimodal data generation is performed through a multimodal data generation model based on a latent diffusion model.

[0007] Furthermore, the multimodal data in step S1 includes multimodal information of different uses, materials and object types. These data can be defined as mode 1, mode 2 and mode 3, and specifically include the following contents: 1) Application: texture recognition, object recognition, grasping state recognition; 2) Material: metal, wood, rubber; 3) Object type: kitchen supplies, toys, food.

[0008] Furthermore, step S2 includes the following sub-steps: S21, build a variational autoencoder for compressing and reconstructing multimodal data. The variational autoencoder is a generative model based on the encoder-decoder architecture; the encoder can convert the input data into low-dimensional latent features and use them as the input of U-Net, and the decoder can reconstruct the low-dimensional latent features into data similar to the training data; S22, construct a U-Net network for predicting noise. The U-Net network adopts a symmetrical structure of encoder-decoder. The encoder part extracts multi-scale features through multi-layer convolution and downsampling operations, while the decoder part gradually reconstructs the data through upsampling. The intermediate layer passes the low-level features in the encoder part to the decoder part through skip connections to retain detail information and enhance the generated spatial consistency; S23, constructs a conditional embedding encoder for guiding model generation, and guides the data generation process in a classifier-free manner to achieve a balance between sample quality and diversity; the conditional embedding encoder extracts features from the input conditional information and encodes the conditional information into an embedding vector; the embedding vector is mapped to the U-Net network through a cross-attention mechanism, thereby providing guidance information for the generation process.

[0009] Furthermore, in step S3, training the multimodal data generation model based on the potential diffusion model includes training the variational autoencoder and joint training of the U-Net network and the conditional embedding encoder. The training process is as follows: S31, training multiple variational autoencoders, compressing the corresponding modal data into the latent space, and weightedly fusing the distribution parameters; S32, jointly train the U-Net network and the conditional embedding encoder in the latent space of the variational autoencoder; the U-Net network, as the core architecture of the latent diffusion model, gradually denoises the latent representation through the diffusion process to learn to restore the noise to a clear latent representation; the conditional embedding encoder encodes the conditional information as the guiding information into an embedding vector and maps it to the U-Net network through the cross-attention mechanism.

[0010] Furthermore, in step S4, the multimodal data generation step is equivalent to the reverse reasoning process of the multimodal data generation model based on the potential diffusion model, including: The conditional information is encoded into a latent conditional vector through the conditional embedding encoder, which is then input into the U-Net network of the diffusion model together with the sampled noise latent variable; U-Net gradually removes noise in the diffusion step and updates the latent variables in reverse iteration to gradually approach the target distribution. In each iteration, the U-Net network generates residual predictions based on the current time step and conditional information to infer the denoising latent variables for the next step. After completing all diffusion steps, the denoised latent variables are restored to the data space through the decoder of the trained variational autoencoder to finally generate the required data.

[0011] Furthermore, the training process of the variational autoencoder adopts a combination of L1 reconstruction loss and perceptual loss as the loss function, and introduces a patch-based adversarial training strategy to improve the detail quality and authenticity of the generated samples; in order to control the variance of the latent space and prevent the model from overfitting, the KL regularization term is added during the training process. The complete loss function of the variational autoencoder is expressed as: , in, represents L1 reconstruction loss and perceptual loss, represents the discriminator loss of PachGAN, represents the KL regularization loss, For the encoder, For the decoder.

[0012] Furthermore, the loss function of the U-Net network is: , in, is a randomly sampled Gaussian noise, For the U-Net network, is the latent variable at time step t, is a conditional encoder.

[0013] Furthermore, the joint training of the U-Net network and the conditional embedding encoder involves the forward diffusion process of the diffusion denoising model, which transforms the original data into pure Gaussian noise by gradually adding noise to the data distribution. This process is from the real data distribution At the beginning, the data is gradually perturbed through a series of Markov steps to achieve gradual randomization of the data; given a data sample , a small amount of noise is introduced at each diffusion time step t, and the data sample at any time step t It can be expressed as: , in, , , is a noise intensity parameter defined at each time step t that gradually increases.

[0014] Furthermore, in the reverse reasoning process, the sample It is obtained by the following calculation steps: S51, for each time step, the U-Net network predicts the conditional and unconditional noise and ; S52, using the classifier-free bootstrap formula, combines conditional and unconditional noise to obtain the final noise prediction: , where w is the guiding strength; S53, using prediction noise and the current sample , calculate the previous step sample: ， in, is the standard deviation of the noise, which is adjusted according to different sampling strategies.

[0015] The present invention also discloses a robot multimodal data generation system based on a potential diffusion model, comprising: The data acquisition module is used to obtain the multimodal data required for robot perception, and perform preprocessing and data enhancement operations on it to generate a multimodal data set; The model design module is used to select a specific model architecture according to the data type based on the actual application situation, and to build a multimodal data generation model based on the potential diffusion model. The multimodal data generation model based on the potential diffusion model includes a variational autoencoder, a U-Net network, and a conditional embedding encoder; wherein the variational autoencoder is used for data compression and data reconstruction; the U-Net network is responsible for predicting the noise in the diffusion process and continuously optimizing the generation process; the conditional embedding encoder is used to continuously inject conditional information while the U-Net network predicts the noise; The model training module is used to determine the training configuration, dynamically adjust the fusion weights of each modality, and use the multimodal data set to train the multimodal data generation model based on the latent diffusion model; The data generation module is used to generate controllable multimodal data guided by conditional information through a multimodal data generation model based on a latent diffusion model.

[0016] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: This invention will provide an efficient and low-cost solution for robot perception training, which can effectively improve the generation quality and diversity of multimodal data, thereby further enhancing the robot's perception ability in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a robot multimodal data generation method based on a potential diffusion model according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a multimodal data generation model based on a potential diffusion model according to an embodiment of the present invention; Figure 3 A schematic diagram of data generation according to an embodiment of the present invention; Figure 4 It is a schematic diagram of forward diffusion of a diffusion model according to an embodiment of the present invention; Figure 5 It is a reverse reasoning schematic diagram of the diffusion model of an embodiment of the present invention; Figure 6 This is a structural diagram of a robot multimodal data generation system based on a potential diffusion model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0019] like Figure 1As shown, the present invention discloses a robot multimodal data generation method based on a potential diffusion model, including the following steps S1 to S4.

[0020] S1, obtains multimodal data for robot perception, and performs preprocessing and data enhancement operations on it to generate a multimodal data set. In practical applications, multimodal data contains multimodal data of different uses, different materials, and different objects. These data can be defined as mode 1, mode 2, mode 3, etc. For example, the visual data of a water cup used in object recognition is stored in the "datasets / object_recog-nition / mugs / vision" folder. Regarding training data preprocessing and data enhancement operations, for time series data, preprocessing includes time window segmentation and normalization, and then a sliding window strategy is used to generate more samples to increase the amount of data; for image data, preprocessing includes image format and size adjustment and normalization, and data enhancement uses horizontal or vertical flipping and rotation of a certain angle to simulate images from different perspectives.

[0021] S2. According to the actual application situation and the data type, a specific model architecture is selected to build a multimodal data generation model based on the latent diffusion model. The multimodal data generation model based on the latent diffusion model includes a variational autoencoder, a U-Net network, and a conditional embedding encoder. Among them, the variational autoencoder is used for data compression and data reconstruction; the U-Net network is responsible for predicting the noise in the diffusion process and continuously optimizing the generation process; the conditional embedding encoder is used to continuously inject conditional information while the U-Net network predicts the noise.

[0022] S3, determine the training configuration, dynamically adjust the fusion weights of each modality, and use the multimodal data set to train the multimodal data generation model based on the latent diffusion model. The training of the multimodal data generation model based on the latent diffusion model includes the training of the variational autoencoder and the joint training of the U-Net network and the conditional embedding encoder. First, determine the training configuration, including the noise level adjustment strategy configuration, model structure parameter configuration, training data configuration, etc.; then train multiple variational autoencoders, and weightedly fuse the distribution parameters to compress the data into the latent space; finally, jointly train the U-Net network and the conditional embedding encoder in the latent space of the variational autoencoder. As the core architecture of the latent diffusion model, the U-Net network gradually denoises the latent representation through the diffusion process to learn to restore the noise to a clear latent representation. The conditional embedding encoder encodes the conditional information as the guide information into an embedding vector, which is mapped to the U-Net network through the cross attention mechanism.

[0023] S4, guided by the conditional information, controllable multimodal data generation is performed through the above-mentioned generation model.

[0024] like Figure 2 As shown, it is a model architecture diagram of an embodiment of the present invention, including: 1) Variational autoencoder for compressing and reconstructing multimodal data. Variational autoencoder is a generative model based on encoder-decoder architecture, which generates and reconstructs data by learning the probability distribution of data. The encoder can convert input data into low-dimensional latent features and use them as input to the U-Net network; the decoder can reconstruct low-dimensional latent features into data similar to the training data.

[0025] When entering a size of The encoder module encodes the data into a The low-dimensional latent features of is the downsampling rate of the variational autoencoder. Conversely, the decoder module has the same upsampling rate to reconstruct the low-dimensional latent features into data.

[0026] Specifically, the variational autoencoder has three basic components: GSC component: GroupNorm+Swish+Conv; Downsample component: Padding+Conv; Upsample component: Interpolate+Conv. There are also two core modules: ResNetBlock module and Self-Attention module.

[0027] Specifically, the encoder part of the variational autoencoder contains three DownBlock modules, one ResNetBlock module, and one MidBlock module, which compress the input data into the latent space and convert it into a Gaussian distribution. The decoder part is just the opposite. It inputs the latent space features and reconstructs them into data as output. It contains three UpBlock modules, one ResNetBlock module, and one MidBlock module. The DownBlock module includes the ResNetBlock module and the Downsample component; the MidBlock module includes the ResNetBlock module and the Self-Attention module; the UpBlock module includes the Upsample component and the ResNetBlock module; and the ResNetBlock module includes the ResNet structure and the GSC component.

[0028] The loss function in the variational autoencoder training process is designed to minimize the reconstruction error and the distribution deviation in the latent space. Specifically, the variational autoencoder training uses a combination of L1 reconstruction loss and perceptual loss as the loss function, and uses a patch-based adversarial training strategy. In addition, in order to control the variance of the latent space and avoid overfitting, a regularization technique, KL (Kullback-Leibler) regularization, is introduced in the training process. The complete loss function of the variational autoencoder is: , in, represents L1 reconstruction loss and perceptual loss, represents the discriminator loss of PachGAN, represents the KL regularization loss, For the encoder, For the decoder.

[0029] In practical applications, there are many forms of variational autoencoders to adapt to data of different modalities. For example, in image data, the variational autoencoder will use a two-dimensional convolutional neural network to construct the encoder, while time series data will sample a one-dimensional convolutional neural network or a recurrent neural network to construct the encoder. Finally, multiple variational autoencoders are trained to obtain potential distribution parameters for weighted fusion to obtain a potential vector.

[0030] 2) Construct a U-Net network for predicting noise. The U-Net network adopts a symmetrical structure of encoder-decoder. The encoder part extracts multi-scale features through multi-layer convolution and downsampling operations, while the decoder part gradually reconstructs the data through upsampling. The intermediate layer passes the low-level features in the encoder part to the decoder part through jump connections to retain detail information and enhance the spatial consistency of generation. In the potential diffusion model, the U-Net network adds temporal conditions to process the time step information in the diffusion process, so that the model can more accurately remove the noise at each time step.

[0031] During the inference process, the generative model repeatedly calls the U-Net network to remove the predicted noise residuals from the original noise matrix, obtains the latent features after gradual denoising, and then reconstructs the latent features into data form through the decoder of the variational autoencoder.

[0032] Specifically, U-Net is mainly composed of three modules: CrossAttnDownBlock, CrossAttnUpBlock and CrossAttnMidBlock. Among them, the CrossAttnDownBlock module is composed of the ResNetBlock module, the DownSample module and the Spatial Transformer module; the CrossAttnUpBlock module is composed of the ResNetBlock module, the UpSample module and the Spatial Transformer module; and the CrossAttnMidBlock module is composed of the ResNetBlock module and the Spatial Transformer module.

[0033] Specifically, the training process of the U-Net network without classifier guidance includes: (1) Data sampling. Sample a real sample from the data distribution , sample the corresponding condition c from the conditional distribution; (2) Conditional random dropout: Replace condition c with an empty condition with a certain probability to achieve unconditional training; (3) According to the forward diffusion process, noise is added to In, generate : , in, , , is a noise intensity parameter defined at each time step t that gradually increases.

[0034] (4) Use U-Net network to To predict noise; (5) Calculate the loss and optimize the model parameters. The loss function of the U-Net network is: , in, is a randomly sampled Gaussian noise, For the U-Net network, is the latent variable at time step t, It is a conditional encoder. Use an optimization algorithm to minimize the loss function and update the model parameters.

[0035] 3) Construct a conditional embedding encoder for guiding model generation. Use a classifier-free guided approach to guide data generation to achieve a trade-off between sample quality and diversity. The conditional embedding encoder extracts features from the input conditional information, encodes the conditional information as guiding information into an embedding vector, and maps it to the U-Net network through a cross-attention mechanism.

[0036] Specifically, the conditional embedding encoder structure is relatively simple, consisting of an embedding layer and a linear layer.

[0037] like Figure 3 As shown, it is a schematic diagram of data generation of the present invention. First, the conditional information is encoded into a latent conditional vector through a conditional embedding encoder, and a random Gaussian noise matrix is ​​initialized at the same time; the conditional vector and the random Gaussian noise are sent to the U-Net network through a cross-attention mechanism; then, the U-Net network gradually removes the noise in the diffusion step, and updates the latent variables in reverse iteration to gradually approach the target distribution. In each iteration, the U-Net network generates a residual prediction based on the current time step and conditional information to infer the denoised latent variables for the next step. After completing all diffusion steps, the denoised latent variables are restored to the data space through the decoder of the trained variational autoencoder, and finally the required multimodal data is generated.

[0038] like Figure 4 , which is a schematic diagram of the forward diffusion of the diffusion model of an embodiment of the present invention. The joint training of the U-Net network and the conditional embedding encoder involves the forward diffusion process of the diffusion denoising model. The forward diffusion process converts the original data into pure Gaussian noise by gradually adding noise to the data distribution.

[0039] Specifically, for the initial data, the number of diffusion steps is set to T steps, and a certain amount of noise is added in each step. If T is set large enough, the initial data can be converted into a random noise matrix. This process starts from the latent variable First, the data is gradually perturbed through a series of Markov steps to achieve gradual randomization of the data. Given a data sample , a small amount of noise is introduced at each diffusion time step t, and the forward diffusion process can be described as: , According to this formula, the samples are Convert to (Gaussian noise) After T time steps, the noise added in each step gradually increases, and finally the data presents the form of a standard Gaussian distribution.

[0040] Furthermore, by recursion, the data samples at any time step t Expressed as the original sample And the combination of standard Gaussian noise: , Through this formula, the forward diffusion process can be described as a closed expression, directly sampling at any time step t .

[0041] like Figure 5 FIG. 1 is a schematic diagram of reverse reasoning of a diffusion model according to an embodiment of the present invention. Reverse reasoning aims to gradually remove noise from pure noise and reconstruct latent variables. This process starts with a Gaussian noise sample and gradually removes the noise through a series of reverse steps to obtain the final latent variable.

[0042] Specifically, the reverse process is defined as Initially, we predict at each time step t in the form of a Markov chain , until t=0, and finally get the final latent variable . Samples in the reverse reasoning process It is obtained by the following calculation steps: 1) For each time step, the U-Net network predicts the conditional and unconditional noise and ; 2) Using the classifier-free bootstrap formula, we combine conditional and unconditional noise to get the final noise prediction: , where w is the guiding strength; 3) Using prediction noise and the current sample , calculate the previous step sample: ， in, is the standard deviation of the noise, which is adjusted according to different sampling strategies.

[0043] like Figure 6 As shown, the present invention discloses a robot multimodal data generation system based on a potential diffusion model, comprising: The data acquisition module 601 is used to acquire the multimodal data required for robot perception, and perform preprocessing and data enhancement operations on the multimodal data to generate a multimodal data set; The model design module 602 is used to select a specific model architecture according to the data type according to the actual application situation to build a multimodal data generation model based on the potential diffusion model. The multimodal data generation model based on the potential diffusion model includes a variational autoencoder, a U-Net network and a conditional embedding encoder; wherein the variational autoencoder is used for data compression and data reconstruction; the U-Net network is responsible for predicting the noise in the diffusion process and continuously optimizing the generation process; the conditional embedding encoder is used to continuously inject conditional information while the U-Net network predicts the noise; A model training module 603 is used to determine the training configuration, dynamically adjust the fusion weight of each modality, and use the multimodal data set to train a multimodal data generation model based on a potential diffusion model; The data generation module 604 is used to generate controllable multimodal data through a multimodal data generation model based on a potential diffusion model under the guidance of conditional information.

[0044] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot multimodal data generation method based on a potential diffusion model, characterized in that: The following steps are involved: S1, obtains the multimodal data required for robot perception, and performs preprocessing and data enhancement operations on it to generate a multimodal dataset; S2, according to the actual application, select the specific model architecture according to the data type, and build a multimodal data generation model based on the potential diffusion model. The multimodal data generation model based on the potential diffusion model includes a variational autoencoder, a U-Net network, and a conditional embedding encoder; among them, the variational autoencoder is used for data compression and data reconstruction; the U-Net network is responsible for predicting the noise in the diffusion process and continuously optimizing the generation process; the conditional embedding encoder is used to continuously inject conditional information while the U-Net network predicts the noise; S3, determine the training configuration, dynamically adjust the fusion weight of each modality, and use the multimodal data set to train the multimodal data generation model based on the latent diffusion model; S4, guided by conditional information, controllable multimodal data generation is performed through a multimodal data generation model based on a latent diffusion model.

2. According to claim 1, a robot multimodal data generation method based on a potential diffusion model is characterized in that: The multimodal data in step S1 covers multimodal information of different uses, materials and object types. These data can be defined as mode 1, mode 2 and mode 3, and specifically include the following contents: 1) Application: texture recognition, object recognition, grasping state recognition; 2) Material: metal, wood, rubber; 3) Object type: kitchen supplies, toys, food.

3. The method for generating robot multimodal data based on a potential diffusion model according to claim 1, characterized in that: Step S2 includes the following sub-steps: S21, build a variational autoencoder for compressing and reconstructing multimodal data. The variational autoencoder is a generative model based on the encoder-decoder architecture; the encoder can convert the input data into low-dimensional latent features and use them as the input of U-Net, and the decoder can reconstruct the low-dimensional latent features into data similar to the training data; S22, construct a U-Net network for predicting noise. The U-Net network adopts a symmetrical structure of encoder-decoder. The encoder part extracts multi-scale features through multi-layer convolution and downsampling operations, while the decoder part gradually reconstructs the data through upsampling. The intermediate layer passes the low-level features in the encoder part to the decoder part through skip connections to retain detail information and enhance the generated spatial consistency; S23, constructs a conditional embedding encoder for guiding model generation, and guides the data generation process in a classifier-free way to achieve a balance between sample quality and diversity; The conditional embedding encoder extracts features from the input conditional information and encodes the conditional information into an embedding vector; the embedding vector is mapped to the U-Net network through the cross-attention mechanism to provide guidance information for the generation process.

4. The method for generating robot multimodal data based on a potential diffusion model according to claim 1, characterized in that: In step S3, training the multimodal data generation model based on the potential diffusion model includes training the variational autoencoder and joint training of the U-Net network and the conditional embedding encoder. The training process is as follows: S31, training multiple variational autoencoders, compressing the corresponding modal data into the latent space, and weightedly fusing the distribution parameters; S32, jointly train the U-Net network and the conditional embedding encoder in the latent space of the variational autoencoder; the U-Net network, as the core architecture of the latent diffusion model, gradually denoises the latent representation through the diffusion process to learn to restore the noise to a clear latent representation; the conditional embedding encoder encodes the conditional information as the guiding information into an embedding vector and maps it to the U-Net network through the cross-attention mechanism.

5. The method for generating robot multimodal data based on a potential diffusion model according to claim 1, characterized in that: In step S4, the multimodal data generation step is equivalent to the reverse reasoning process of the multimodal data generation model based on the potential diffusion model, including: The conditional information is encoded into a latent conditional vector through the conditional embedding encoder, which is then input into the U-Net network of the diffusion model together with the sampled noise latent variable; U-Net gradually removes noise in the diffusion step and updates the latent variables in reverse iteration to gradually approach the target distribution. In each iteration, the U-Net network generates residual predictions based on the current time step and conditional information to infer the denoising latent variables for the next step. After completing all diffusion steps, the denoised latent variables are restored to the data space through the decoder of the trained variational autoencoder to finally generate the required data.

6. The method for generating robot multimodal data based on a potential diffusion model according to claim 4, characterized in that: The training process of the variational autoencoder uses a combination of L1 reconstruction loss and perceptual loss as the loss function, and introduces a patch-based adversarial training strategy to improve the detail quality and authenticity of the generated samples; in order to control the variance of the latent space and prevent the model from overfitting, the KL regularization term is added during the training process. The complete loss function of the variational autoencoder is expressed as: ， in, represents L1 reconstruction loss and perceptual loss, represents the discriminator loss of PachGAN, represents the KL regularization loss, For the encoder, For the decoder.

7. The method for generating robot multimodal data based on a potential diffusion model according to claim 4, characterized in that: The loss function of the U-Net network is: ， in, is a randomly sampled Gaussian noise, For the U-Net network, is the latent variable at time step t, is a conditional encoder.

8. The method for generating robot multimodal data based on a potential diffusion model according to claim 4, characterized in that: The joint training of the U-Net network and the conditional embedding encoder involves the forward diffusion process of the diffusion denoising model, which transforms the original data into pure Gaussian noise by gradually adding noise to the data distribution. This process is from the real data distribution At the beginning, the data is gradually perturbed through a series of Markov steps to achieve gradual randomization of the data; given a data sample , a small amount of noise is introduced at each diffusion time step t, and the data sample at any time step t It can be expressed as: ， in, , , is a noise intensity parameter defined at each time step t that gradually increases.

9. The method for generating robot multimodal data based on a potential diffusion model according to claim 5, characterized in that: Samples in the reverse reasoning process It is obtained by the following calculation steps: S51, for each time step, the U-Net network predicts the conditional and unconditional noise and ; S52, using the classifier-free bootstrap formula, combines conditional and unconditional noise to obtain the final noise prediction: , where w is the guiding strength; S53, using predicted noise and the current sample , calculate the previous step sample: ， in, is the standard deviation of the noise, which is adjusted according to different sampling strategies.

10. A robot multimodal data generation system based on a potential diffusion model, characterized in that: include: The data acquisition module is used to acquire the multimodal data required for robot perception, and perform preprocessing and data enhancement operations on it to generate a multimodal data set; The model design module is used to select a specific model architecture according to the data type based on the actual application situation, and to build a multimodal data generation model based on the potential diffusion model. The multimodal data generation model based on the potential diffusion model includes a variational autoencoder, a U-Net network, and a conditional embedding encoder; wherein the variational autoencoder is used for data compression and data reconstruction; the U-Net network is responsible for predicting the noise in the diffusion process and continuously optimizing the generation process; the conditional embedding encoder is used to continuously inject conditional information while the U-Net network predicts the noise; The model training module is used to determine the training configuration, dynamically adjust the fusion weights of each modality, and use the multimodal data set to train the multimodal data generation model based on the latent diffusion model; The data generation module is used to generate controllable multimodal data guided by conditional information through a multimodal data generation model based on a latent diffusion model.

Images