Reinforcement learning multi-mode body-equipped agent data generation method

Abstract

The invention provides a reinforcement learning multi-mode body-equipped agent data generation method. According to the method, interaction between an intelligent agent and an environment and data collection are realized by establishing a data pre-acquisition and preprocessing module; learning similar features among different modes through joint representation learning; then on-line interaction between the intelligent agent and the environment is carried out based on reinforcement learning so as to optimize the current strategy; and finally, generating a new action sequence through the strategy network, and inputting the new action sequence into a conditional variable molecular encoder to generate corresponding multi-modal data. In addition, a modal discriminator is introduced for adversarial training, so that the quality of the generated data is further optimized. According to the method, high-quality and diversified multi-modal data can be generated, and powerful support is provided for research and application of the intelligent field.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal data generation, and particularly relates to a method for generating data of a reinforcement learning multimodal embodied agent. Background Art

[0002] In the fields of artificial intelligence and robotics, embodied agents (Embodied AI) have gradually become the core technology for achieving complex tasks (such as autonomous navigation and object manipulation) through real-time interaction with physical or virtual environments. And embodied agents need to rely on a large amount of multimodal data for model training and algorithm optimization. Traditional methods rely on predefined rules or static datasets to drive the behavior of agents. In recent years, reinforcement learning (RL) has been widely used in the training and optimization of embodied agents due to its autonomous decision-making ability in dynamic environments. At the same time, multimodal perception technologies (such as the fusion of vision, touch, and motion signals) provide agents with a more comprehensive environmental understanding ability, further promoting the development of embodied intelligence.

[0003] In an embodied intelligence system, a robot or an agent needs to process visual, auditory, tactile, and even motion data simultaneously, which requires the system not only to perceive various types of information in the environment but also to achieve efficient fusion of multimodal information during the decision-making process. With the help of reinforcement learning technology, an agent can learn and optimize the strategy of interacting with the environment through continuous trial and error in a virtual or semi-virtual environment, thereby generating multimodal data highly similar to the real situation. This method enables the agent to reproduce complex scenarios in the real world in a simulated environment by designing a reasonable reward function and state transition mechanism, greatly improving the diversity and authenticity of the data.

[0004] Currently, existing multimodal data generation methods still have some challenges and problems, mainly manifested as follows: 1) There are certain technical barriers to the synchronous generation of multimodal data, and it is difficult to ensure the consistency and relevance between modalities; 2) The model has high training complexity and instability, and the training process may require fine-tuning of hyperparameters and is difficult to be truly implemented in real scenarios; 3) The generation of simulation data in embodied intelligence scenarios is not accurate enough, and high-quality data cannot be generated by interacting with the environment in real scenarios. Static data cannot meet the real-time interaction needs of embodied agents in dynamic environments.

[0005] Paper 1 (Zizhao Hu, Mohammad Rostami. Learning Multilingual Cognitively Inspired Cross-Modal Data Generation Using Diffusion Models) proposed a cross-modal data generation method inspired by human cognition, aiming to jointly generate multi-modal data through diffusion models. A channel-level image conditioning method was designed to learn the associations between different modalities during the training phase to better simulate the learning process of the brain. Experimental results showed that this method could generate data under the conditions of relevant modalities. However, this method introduced a module for channel-level image conditioning, increasing the complexity of the model. The training and inference processes required more computing resources, and its generality in different downstream sub-scenarios was unstable, requiring a certain amount of expert data as additional supplementary guidance. Paper 2 (Cong Fei, Bin Wang, Yuzheng Zhuang, Zongzhang Zhang, Jianye Hao, Hongbo Zhang, Xuewu Ji, Wulong Liu. Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets) proposed a multi-modal imitation learning framework called Triple-GAIL, which combines generative adversarial networks and can jointly learn skill selection and imitation from expert demonstrations and continuously generated experiences. By introducing an auxiliary skill selector, the framework achieved data augmentation and was verified on real driving trajectory and real-time strategy game datasets. However, this method still has some drawbacks: 1) Dependence on expert data: This method requires a large amount of high-quality expert demonstration data, and obtaining such data may be challenging in some application scenarios; 2) Model training complexity: Introducing an auxiliary skill selector and generative adversarial networks may increase the training complexity and instability of the model, and the training process may require fine-tuning of hyperparameters. Summary of the Invention

[0006] The object of the present invention is to solve the problem that existing methods cannot accurately provide multi-modal data for the field of embodied intelligence, and a method for generating data for a reinforcement learning multi-modal embodied intelligent agent is proposed. Through the real-time interaction between the intelligent agent and the physical / virtual environment, combined with multi-modal perception signals and task objectives, high-quality and diverse training data is dynamically generated.

[0007] Step 1: Establish a data pre-acquisition and preprocessing module to enable the intelligent agent to interact with the environment and collect raw multi-modal data;

[0008] Step 2: Perform joint representation learning to minimize the feature distance and reconstruction loss between modalities, and train to obtain optimized model parameters for input to the downstream policy network.

[0009] Step 3: Conduct online interaction based on reinforcement learning. The agent executes corresponding actions according to the current policy, receives environmental feedback, and stores the interaction data in the experience replay pool.

[0010] Step 4: Update the agent's policy. Sample a batch of data, calculate the advantage function, and update the Actor network parameters and Critic network parameters based on the calculation results of each round.

[0011] Step 5: Input the action sequence generated by the policy network into the conditional variational autoencoder to generate corresponding multi-modal data, and iteratively optimize the generator and discriminator through adversarial training of the modality discriminator.

[0012] The beneficial effects of the present invention are as follows: The present invention proposes a method for generating data of a reinforcement learning multi-modal embodied agent. First, it realizes the interaction and data transfer between the embodied agent and the environment, and first learns the similar features between different modalities through joint representation learning. Then, through reinforcement learning, it conducts online interaction between the agent and the environment to optimize the current policy. Finally, a corresponding new action sequence is generated through the policy network, thereby generating multi-modal data. In addition, a generative adversarial network is added to further optimize the data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the interaction between the agent and the environment through reinforcement learning;

[0014] Figure 2 It is a schematic diagram of the conditional variational autoencoder model. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] The following further elaborates on the present application in conjunction with the attached Figure 1-2 , and makes a more detailed description of the present application.

[0016] Detailed Embodiment 1: The present embodiment describes a method for generating data of a reinforcement learning multi-modal embodied agent, which is specifically implemented through the following steps:

[0017] Step 1: Establish an interaction module between the embodied agent and the environment to implement a physical or virtual interface, supporting real-time communication with, for example, a UR5 robotic arm or a virtual simulation platform. The visual module uses an RGB-D camera to collect three-dimensional point cloud and texture information. The tactile module uses an ATI Mini40 as a force sensor to obtain contact force distribution data. Before model training, a certain amount of multi-modal data is collected. , and perform preprocessing steps such as data cleaning, denoising, and normalization. The key is to remove sensor noise, such as the Kalman filter smoothing of tactile signals. The gain calculation formula and state update formula are as follows;

[0018]

[0019]

[0020] where is the Kalman gain, which determines the influence of the new measurement value on the final state estimate. is the prediction error covariance, representing the uncertainty of the current estimate, is the measurement matrix, is the measurement noise covariance, used to measure the magnitude of the sensor noise. is the final estimated value of the tactile signal, where is the predicted state, represents the measurement residual, that is, the deviation between the measurement value and the predicted value.

[0021] Step 2: Construct a multi-modal encoder. Use the Transformer-based Cross-model Encoder as the baseline coding network structure, which is generally divided into two parts. The first part is the modality-specific coding sub-network, which consists of encoders for each different modality. In the visual branch, a 3D-CNN model is used to process point cloud data and output the visual feature vector . In the tactile branch, a multi-layer perceptron is used to understand the haptic signal and output the tactile feature . The second part is the cross-modal alignment layer based on the multi-head attention mechanism, which jointly learns to generate the latent representation

[0022]

[0023] where is the learnable fusion weight matrix;

[0024] Step 3: In the environment such as Figure 1 , the agent performs online interactive learning through reinforcement learning. Use the policy-based Actor-Critic algorithm to randomly initialize the relevant parameters and the agent's policy for the Actor network and the Critic network , that is, execute the action according to the policy and the feature data. After receiving the agent's action, the environment returns a new state and a reward . The interactive data at this stage will be stored in the experience replay pool;

[0025] Step 4: After a certain amount of interaction, sample batch interaction data and calculate the Advantage Function

[0026]

[0027] Reward function Consists of three parts, ensuring the exploration of different action spaces and the balance of choosing the optimal solution

[0028]

[0029] Among them is the task completion reward (for example, in the intelligent robot grasping task, +1 for success and -1 for failure), is the KL divergence of the agent's action trajectory, encouraging the policy to explore new action patterns, constraining some special behaviors of the agent (such as the joint torque of the robotic arm not exceeding a certain threshold);

[0030] Step 5: After completing the training and learning of the policy network, input the latent representation and the action into the Conditional Variational Autoencoder (CVAE) as Figure 2 . The CVAE can generate multi-modal data highly related to the target according to the input conditions (representation and action), learn the data distribution through variational inference, generate diverse samples, and avoid generating repetitive data. And the decoder of the CVAE is designed as a multi-branch structure, so as to output visual, tactile, and action modal data simultaneously, and constrain the consistency between modalities through the shared latent space

[0031] Specific implementation method 2: Different from the specific implementation method 1, in step 5, additional adversarial training is introduced, and the discriminator model is used to further guide the generator to approximate the real data distribution. For visual data, based on the PatchGAN model, the image is segmented into local blocks, and the authenticity of each block is judged separately to enhance the detail generation ability. For tactile data, a one-dimensional convolutional network is used to process the force sense time series to capture the dynamic contact pattern. The loss function of the generator is , is the weight coefficient, balancing the adversarial loss and the reconstruction loss

[0032]

[0033] Specific Embodiment 3: The difference between this embodiment and Specific Embodiment 1 is that when updating the agent policy network in Step 4, the Elastic Weight Consolidation (EWC) technique is used for the update. In a new task, the loss function is calculated as follows, where is the Fisher information (importance weight) in the old task, is the regularization strength coefficient, is the parameter value after training in the old task:

[0034]

[0035] EWC can alleviate the catastrophic forgetting problem of neural networks in continuous task learning, help the agent retain knowledge of historical environments and tasks when adapting to new environments or tasks, and improve the diversity and robustness of the generated data.

[0036] The present invention provides a method for generating data of a reinforcement learning multi-modal embodied agent. In a dynamic interaction environment, high-quality, cross-modal consistent multi-modal training data is generated through the real-time feedback between the agent and the physical / virtual environment. In terms of the model architecture, first, a task-driven reinforcement learning policy network is designed, and a multi-modal encoder (Transformer-based Cross-modal Encoder) is combined to map multi-modal signals such as vision and touch to a unified latent space to solve the problem of modal heterogeneity; secondly, a dynamic data generator (based on CVAE and adversarial training) is introduced, and through conditional generation and optimization of the multi-modal discriminator, the diversity and physical rationality of the generated data are ensured; finally, continuous learning is achieved through the Elastic Weight Consolidation (EWC) technique to balance the knowledge of old and new tasks and avoid catastrophic forgetting.

[0037] The beneficial effects of the present invention are as follows: 1) The quality and diversity of the generated data are improved. Through the task-driven mechanism of reinforcement learning and CVAE probability modeling, the generated data covers complex scenarios (such as dynamic lighting and object deformation), and is highly relevant to the goals of the embodied agent, reducing invalid samples; 2) Cross-modal spatio-temporal consistency is guaranteed. Based on joint latent space encoding and cross-modal attention mechanisms, accurate alignment of signals such as vision and touch is achieved; 3) It has strong adaptability to dynamic environments. The online reinforcement learning strategy updates the generation rules in real time, combined with the EWC continuous learning technique, to adapt to environmental mutations (such as target displacement and sensor noise), improving the robustness of the system; 4) The simulation-to-reality transfer learning is enhanced. Through the collaborative optimization of adversarial training and the physical engine, the difference between the generated data and the real sensor signals is reduced, and the training cost of real robots is lowered.

[0038] The above numerical examples of the present invention are only for explaining in detail the calculation model and calculation process of the present invention, rather than limiting the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation manners here. Any obvious changes or modifications derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A method for generating data of a reinforcement learning multi-modal embodied intelligent agent, characterized in that, It includes the following steps: Step 1: Establish a data pre-collection and pre-processing module, enable the agent to interact with the environment, and collect original multi-modal data; Step 2: Conduct joint representation learning, train to obtain the optimized model parameters by minimizing the feature distance and reconstruction loss between modalities for input to the downstream policy network; Step 3: Perform online interaction based on reinforcement learning. The agent executes corresponding actions according to the current policy, receives environmental feedback, and stores the interaction data in the experience replay pool; Step 4: Update the agent's policy, sample batch data, calculate the advantage function, and update the Actor network parameters and Critic network parameters according to the calculation results of each round; Step 5: Input the action sequence generated by the policy network into the conditional variational autoencoder to generate the corresponding multi-modal data, and iteratively optimize the generator and discriminator through the adversarial training of the modality discriminator.

2. The method for generating data of a reinforcement learning multi-modal embodied intelligent agent according to claim 1, wherein In the above Step 1, the establishment of the data pre-collection and pre-processing module includes the setting of the agent's perception device and the construction of the environmental interaction interface to ensure the effective collection of original multi-modal data.

3. The method for generating data of a reinforcement learning multi-modal embodied intelligent agent according to claim 1, wherein In the above Step 2, during the joint representation learning process, a specific distance metric method is used to calculate the feature distance between modalities, and a corresponding reconstruction loss function is used to optimize the model parameters.

4. The method for generating data of a reinforcement learning multi-modal embodied intelligent agent according to claim 1, wherein In the above Step 3, the policy for the agent to execute actions is determined by the policy network constructed based on the reinforcement learning algorithm, and the environmental feedback includes reward signals and state information.

5. The method for generating data of a reinforcement learning multi-modal embodied intelligent agent according to claim 1, wherein In the above Step 4, the specific algorithm used to calculate the advantage function and the optimization algorithms for updating the Actor network parameters and Critic network parameters are specific and clear.

6. The method for generating data of a reinforcement learning multi-modal embodied intelligent agent according to claim 1, wherein, In the above Step 5, the network structures and parameter settings of the conditional variational autoencoder and the modality discriminator are optimized for the multi-modal data generation task.

Images