Body-equipped intelligent agent training method and device, storage medium and electronic equipment

Abstract

The application provides a body-equipped intelligent agent training method and device, a storage medium and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining a task requirement; setting initial parameters of a body-equipped intelligent agent and constructing a virtual scene based on the task requirement; controlling the body-equipped intelligent agent to perform skill operation and obtaining a virtual-real data set, wherein the virtual-real data set comprises virtual data collected by the body-equipped intelligent agent in a virtual scene and real data collected in a real scene; taking minimizing the feature distribution difference between the virtual scene and the real scene as a target, correcting the virtual data in the virtual-real data set based on the real data to obtain a training data set; and performing skill training on the body-equipped intelligent agent based on the training data set. The feature level alignment of the virtual scene and the real scene can be realized, the virtual training skill can be directly migrated to the real scene, and the virtual-real gap is broken. The method is suitable for various body-equipped intelligent agents and multi-field application scenes, and does not need to reconstruct existing equipment, so the deployment cost is low.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to an embodied agent training method, an embodied agent training device, a machine-readable storage medium, and an electronic device. Background Technology

[0002] Embodied intelligent agents are physical-intelligent carriers that enable interaction between artificial intelligence and the physical world. Their core lies in forming a closed loop of behavior involving environmental perception, autonomous learning, and physical operation through multimodal perception channels (such as vision, touch, force, and inertia). This allows them to replace humans in completing complex, dangerous, or repetitive physical operation tasks, thereby driving various industries towards intelligent and unmanned transformation.

[0003] Currently, skill training systems for embodied intelligent agents are mainly divided into two categories: pure physical training and pure virtual training. Pure virtual training has the advantages of low sample loss and high iteration efficiency, but there are perceptual domain shifts (such as lighting, texture, and sensor noise), interaction domain shifts (such as contact feedback, friction, and object physical properties), and environmental parameter shifts (such as object size and spatial layout) between the virtual environment and the real physical environment. This leads to a significant performance drop when skills learned in the virtual environment are transferred to real-world scenarios, a phenomenon known as the "virtual-real gap." Essentially, the virtual-real gap stems from the difference in data feature distribution between virtual and real-world scenarios, and traditional fixed mapping methods struggle to effectively align the two types of feature distributions. Pure physical training can guarantee the authenticity of training data, but it suffers from low sample collection efficiency, high training costs, and significant equipment wear and tear. Furthermore, it is difficult to cover the training needs of extreme scenarios such as high temperature, high pressure, and strong radiation, resulting in significant shortcomings in training efficiency and security.

[0004] It is evident that existing purely virtual training and purely physical training methods cannot bridge the gap between the virtual and the real world, thus limiting the application scope of embodied intelligent agents. Summary of the Invention

[0005] The purpose of this invention is to provide an embodied agent training method, an embodied agent training device, a machine-readable storage medium, and an electronic device. This embodied agent training method can achieve feature-level alignment between virtual and real scenes, ensuring that virtual training skills can be directly transferred to real scenes, thus bridging the gap between the virtual and real worlds.

[0006] To achieve the above objectives, the first aspect of this application provides an embodied agent training method, comprising: Obtain task requirements; Based on the task requirements, set the initial parameters of the embodied intelligent agent and construct a virtual scene; The embodied intelligent agent is controlled to perform skill operations and acquire virtual and real datasets, which include virtual data collected by the embodied intelligent agent in the virtual scene and real data collected in the real scene; With the goal of minimizing the feature distribution difference between virtual and real scenes, the virtual data in the virtual and real dataset is corrected based on the real data to obtain the training dataset; Based on the training dataset, the embodied agent is trained in skills.

[0007] In this embodiment of the application, based on the task requirements, initial parameters of the embodied intelligent agent are set and a virtual scene is constructed, including: The task requirements are semantically parsed to obtain the task feature vector; Based on the task feature vector, a task model is obtained; Based on the task model, the initial parameters of the embodied agent are set and a virtual scene is constructed.

[0008] In this embodiment of the application, the step of correcting the virtual data in the virtual-real dataset based on the real data, with the goal of minimizing the difference in feature distribution between virtual and real scenes, to obtain a training dataset includes: A virtual-real feature mapping model is used to map the real data to a virtual feature space to obtain the mapped virtual data. The virtual-real feature mapping model is trained with the goal of minimizing the maximum mean difference between the mapped virtual data and the real data, resulting in a trained virtual-real feature mapping model. Based on the real data, the virtual data in the virtual-real dataset is corrected using the trained virtual-real feature mapping model to obtain the training dataset.

[0009] In this embodiment of the application, based on the real data, the virtual data in the virtual-real dataset is corrected using the trained virtual-real feature mapping model to obtain a training dataset, including: The trained virtual-real feature mapping model is used to map the real data to the virtual feature space to obtain the trained mapped virtual data. Based on the trained mapped virtual data, the virtual data in the virtual and real datasets are corrected to obtain the training dataset.

[0010] In this embodiment of the application, the real data includes real perception data, real feedback data and real environmental parameters, and the virtual-real feature mapping model includes a perception mapping model, an interaction mapping model and an environment mapping model; The process of mapping the real data to a virtual feature space using a virtual-real feature mapping model to obtain the mapped virtual data includes: The real perceptual data is mapped to a virtual feature space using the aforementioned perceptual mapping model to obtain the mapped perceptual data. The interactive mapping model is used to map the real feedback data to the virtual feature space to obtain the mapped feedback data. The real environment parameters are mapped to a virtual feature space using the environment mapping model to obtain the mapped environment parameters; Based on the mapped perception data, the mapped feedback data, and the mapped environmental parameters, the mapped virtual data is obtained.

[0011] In this embodiment of the application, it also includes: The real and virtual features in the training dataset are weighted and fused to obtain the fused features; Extract the domain-invariant features from the fusion features; Using the domain-invariant features, the embodied agent is trained to obtain first skill parameters; Obtain real-world scene samples; Based on the real-world scene samples, the first skill parameters are adjusted using a model-independent meta-learning algorithm to obtain the second skill parameters.

[0012] In this embodiment of the application, it also includes: Obtain environmental change data and task change information in the real-world scenario; Based on the environmental change data and the task change information, determine whether the embodied intelligent agent needs to perform skill generalization. When it is determined that the embodied agent needs to perform skill generalization, a skill generalization strategy is generated using the PPO-GAE algorithm based on the skill training dataset, the environmental change data, and the task change information. Based on the skill generalization strategy, the skill execution parameters and operation trajectory of the embodied intelligent agent are adjusted.

[0013] In this embodiment of the application, it also includes: Acquire execution data, which is obtained by the embodied intelligent agent performing generalized skills in a new real-world scenario; Based on the execution data and the preset generalization evaluation index, the parameters of the skill generalization strategy are adjusted to obtain a new skill generalization strategy. Based on the new skill generalization strategy, the skill execution parameters and operation trajectory of the embodied agent are adjusted.

[0014] In this embodiment of the application, it also includes: Obtain the generalized skill parameters and execution data; The training dataset is updated based on the generalized skill parameters and execution data.

[0015] A second aspect of this application provides an embodied intelligent agent training device, comprising: The acquisition module is used to acquire task requirements; An initialization module is used to set the initial parameters of the embodied intelligent agent and construct a virtual scene based on the task requirements. The acquisition module is used to control the embodied intelligent agent to perform skill operations and acquire virtual and real datasets, which include virtual data collected by the embodied intelligent agent in the virtual scene and real data collected in the real scene; The correction module is used to correct the virtual data in the virtual-real dataset based on the real data, with the goal of minimizing the difference in feature distribution between the virtual and real scenes, to obtain a training dataset. The training module is used to train the embodied agent's skills based on the training dataset.

[0016] A third aspect of this application provides an electronic device, the electronic device comprising: At least one processor; A memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the above-described embodied agent training method by executing the instructions stored in the memory.

[0017] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned embodied agent training method.

[0018] The above technical solution involves: acquiring task requirements; setting initial parameters for the embodied agent and constructing a virtual scene based on these requirements; controlling the embodied agent to perform skill operations and acquiring a virtual-real dataset, which includes virtual data collected by the embodied agent in the virtual scene and real data collected in the real scene; aiming to minimize the feature distribution difference between the virtual and real scenes, correcting the virtual data in the virtual-real dataset based on the real data to obtain a training dataset; and training the embodied agent on skills based on the training dataset. By correcting the virtual data with the goal of minimizing the feature distribution difference between the virtual and real scenes, feature-level alignment between the virtual and real scenes can be achieved, ensuring that virtual training skills can be directly transferred to real scenes, bridging the virtual-real divide. By replacing most physical training with virtual training, physical losses are reduced by more than 80%, and the training cycle is significantly shortened, solving the industry pain points of high training costs and long training cycles for embodied agents. This method is compatible with various embodied intelligent agents (service robots, industrial robots, medical robots, autonomous driving vehicles) and multiple application scenarios (industrial manufacturing, smart homes, healthcare, intelligent transportation). It supports flexible adjustment of system parameters, can seamlessly connect with existing embodied intelligent agent execution systems, requires no reconstruction of existing equipment, and has low deployment costs.

[0019] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 The illustration shows a flowchart of an embodied agent training method according to an embodiment of this application; Figure 2 This illustration schematically shows the overall architecture of a virtual-real fusion transfer learning and skill generalization system for embodied intelligent agents according to an embodiment of this application; Figure 3 This illustration schematically shows the three-layer dynamic mapping structure and data interaction diagram of the virtual-real fusion mapping layer according to an embodiment of this application; Figure 4 This illustration schematically shows a workflow diagram of multimodal cross-domain transfer learning and skill generalization according to embodiments of this application; Figure 5 This illustration schematically shows a multimodal sensing data synchronous acquisition and feature fusion according to an embodiment of this application; Figure 6 The illustration shows a comparison diagram of skill transfer and generalization effects according to embodiments of this application; Figure 7 This illustration shows a comparison of skill transfer accuracy and generalization ability according to embodiments of this application. Figure 8 This illustration schematically shows a structural diagram of an embodied intelligent agent training device according to an embodiment of this application; Figure 9 The diagram illustrates the internal structure of a computer device according to an embodiment of this application.

[0021] Explanation of reference numerals in the attached figures 410 - Acquisition Module; 420 - Initialization Module; 430 - Acquisition Module; 440 - Calibration Module; 450 - Training Module; A01 - Processor; A02 - Network Interface; A03 - Internal Memory; A04 - Display Screen; A05 - Input Device; A06 - Non-volatile Storage Medium; B01 - Operating System; B02 - Computer Program. Detailed Implementation

[0022] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0023] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0024] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0025] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0026] Please refer to Figure 1 , Figure 1 The illustration shows a flowchart of an embodied agent training method according to an embodiment of this application. This embodiment provides an embodied agent training method, including the following steps: Step 210: Obtain task requirements; In this embodiment, the aforementioned task requirement refers to the task that the embodied intelligent agent needs to complete, which can be natural language or instructions input by the user.

[0027] Step 220: Based on the task requirements, set the initial parameters of the embodied intelligent agent and construct a virtual scene; In this embodiment, based on the task requirements, the parameters required for the embodied intelligent agent to complete the task can be determined, that is, the initial parameters can be determined and a virtual scene can be established. The virtual scene is used to simulate the real scene of the task.

[0028] In some embodiments, based on the task requirements, initial parameters of the embodied agent are set and a virtual scene is constructed, including: First, the task requirements are semantically parsed to obtain the task feature vector; In this embodiment, the language parsing described above can be performed using a large language model, or it can be obtained by using a semantic analysis algorithm. For example, a pre-trained model based on Bidirectional Encoder Representations from Transformers (BERT) can be used to perform semantic parsing on the user's input natural language / structured instructions (task requirements) and extract task feature vectors. The above semantic parsing process is existing technology and will not be described in detail here.

[0029] Then, based on the task feature vector, the task model is obtained; In this embodiment, there can be multiple task feature vectors, which are combined to generate a formalized task model. For example, task feature vectors include skill types. (Grab, assemble, inspect, etc.) Task objectives Scene parameters (Lighting, friction, object size, etc.), execution accuracy requirements Constraints (Execution time, equipment wear and tear) can then generate a formalized task model. : .

[0030] Then, based on the task model, the initial parameters of the embodied agent are set and a virtual scene is constructed.

[0031] In this embodiment, based on the task model It allows configuration of sensor parameters (e.g., sampling frequency) in the multimodal perception layer of the embodied intelligent agent. Perception range (etc.), initialize the mapping parameters of the virtual-real fusion mapping layer. Algorithm parameters of the transfer learning layer Generalization evaluation indicators for the skill generalization layer Ensure that the parameters of each module match the task requirements. Setting initial parameters can specifically involve setting the task model. The quantitative requirements are broken down and mapped into executable parameter indicators for each functional layer. This ensures that the initial parameters fully match the core elements in M, such as skill type, scene parameters, and execution accuracy, guaranteeing that the execution direction of each layer's functions aligns with the task objective. Specifically: mapping parameters. Matching scene parameters of M +Execution accuracy requirements Initial weights can be set according to the physical attributes of the scene, and fine-tuned according to accuracy requirements to achieve scene and accuracy adaptation with virtual and real feature alignment; the algorithm parameters of the transfer learning layer Matching M's skill type +Execution accuracy requirements +Constraints The learning rate step size is determined by skill complexity, and transfer accuracy is guaranteed by precision while balancing transfer efficiency by constraints (such as timeliness); the generalization evaluation index β matches the task objective of M. +Execution accuracy requirements +Constraints The evaluation weights are allocated according to the core requirements of the task, the generalization accuracy is determined by precision, and the exploration and stability of the strategy are balanced by constraints.

[0032] Initial values ​​can be assigned to each parameter using the system's built-in resource library, pre-trained models, and empirical thresholds. This initial parameter is then written into the corresponding module's hardware / software framework, such as sensor drivers, PyTorch, and the Proximal Policy Optimization with Generalized Advantage Estimation for Generalization (PPO-GAE-Gen) algorithm. Parameters with calibration mismatches can be quickly corrected using a parameter fine-tuning module. After matching, all parameters are fixed, generating a parameter configuration table as the benchmark for subsequent system execution. Simultaneously, NeRF can be used to construct a high-fidelity virtual simulation scene to obtain the virtual environment.

[0033] It should be noted that the system can generate a task requirement analysis report, including a formalized task model and parameter configuration, which is then fed back to the user for verification. If the user requests modifications, the parameters at each level are updated through the parameter fine-tuning module until the requirement analysis conforms to the actual application scenario. The final confirmed task model can be recorded as follows. .

[0034] By performing semantic parsing on the task requirements, and based on the task feature vector obtained from the semantic parsing, the initial parameters of the embodied agent can be set quickly and accurately, and a virtual scene can be constructed.

[0035] Step 230: Control the embodied intelligent agent to perform skill operations and acquire virtual and real datasets, the virtual and real datasets including virtual data collected by the embodied intelligent agent in the virtual scene and real data collected in the real scene; In this embodiment, after setting the initial parameters, the embodied intelligent agent performs task operations. In the virtual scene, virtual multimodal data (virtual data) can be collected through the simulation engine, which can be represented as: ,in, Representing images in a virtual scene, Represents the force value in a virtual scene. Indicates haptic feedback in virtual scenes, This represents pose data in a virtual scene; in a real scene, real multimodal data (real data) can be collected through a multi-sensor group and can be represented as: ,in, Representing images in real-world scenes Represents the force value in a real-world scenario. Represents tactile feedback in real-world scenarios, This represents pose data in a real-world scenario.

[0036] It should be noted that, for the differences in sampling frequency of different sensors, a linear interpolation + timestamp alignment algorithm can be used to synchronize virtual and real data, thereby ensuring that the synchronization error is ≤200ms.

[0037] In some embodiments, the collected data can also be preprocessed, including cleaning, standardization, and feature extraction. Cleaning may involve using the 3σ criterion to remove outlier data and using median filtering to remove noise. Standardization may involve Z-score standardization of numerical data (force, posture, haptic feedback), using the following formula: in The original data, The mean, The standard deviation is used; the image data is normalized (pixel values ​​are mapped to [0,1]); feature extraction can be performed by using ResNet50 to extract visual features from the image data. Temporal features were extracted from force / tactile / posture data using a multilayer perceptron (MLP). Generate standardized multimodal sensing datasets ,in For virtual preprocessed data, This is for real preprocessed data.

[0038] Step 240: With the goal of minimizing the difference in feature distribution between virtual and real scenes, the virtual data in the virtual and real dataset is corrected based on the real data to obtain the training dataset; In this embodiment, by correcting the virtual data, the difference in feature distribution between the virtual and real scenes is minimized, thereby aligning the features of the virtual scene with those of the real scene. This ensures that virtual training skills can be directly transferred to the real scene, fundamentally solving the problem of the virtual-real divide.

[0039] In some embodiments, the step of correcting the virtual data in the virtual-real dataset based on the real data, with the goal of minimizing the feature distribution difference between the virtual and real scenes, to obtain a training dataset includes: First, a virtual-real feature mapping model is used to map the real data to a virtual feature space to obtain the mapped virtual data. In this embodiment, the virtual-real feature mapping model described above can be constructed using a deep learning model to convert real data into data in a virtual scene.

[0040] In some embodiments, the real data includes real perception data, real feedback data, and real environmental parameters, and the virtual-real feature mapping model includes a perception mapping model, an interaction mapping model, and an environment mapping model. In this embodiment, real-world perception data can include real-world perception features. Real-world perception features refer to high-dimensional feature vectors representing the state of the real physical environment, generated by collecting raw perception data from multimodal sensors deployed on the embodied agent and processing it through a pre-defined feature extraction process. For example, for the visual modality, a pre-trained ResNet-50 convolutional neural network can encode the real RGB image captured by a physical camera, and the feature map output from the last convolutional layer can be used as the real-world perception feature. Real-world feedback data can include real-world feedback features, which can refer to high-dimensional feature vectors extracted from force, touch, and other feedback signals collected during the embodied agent's operation execution. Real-world environmental parameters can refer to a set of quantified parameters measured or estimated from the embodied agent's working scene to describe the non-contact physical context, including illumination, friction, object physical properties, and spatial layout. In other words, real-world perception data refers to the sensor's observation of the environment, real-world feedback data refers to the sensor's perception of physical interactions, and real-world environmental parameters refer to measurements of the inherent properties and layout of the physical world. The aforementioned perception mapping model can employ a CNN (Convolutional Neural Network) + Transformer model to map real-world perception features to a virtual perception feature space, enabling real-time updates of virtual scene perception data. The aforementioned interaction mapping model can employ an MLP model to calibrate the differences in operational feedback (force and touch) between virtual and real scenes, ensuring that the feature distribution of virtual operational feedback is consistent with that of real operational feedback. The aforementioned environment mapping model can employ a GNN (Graph Neural Network) model to dynamically synchronize environmental parameters between virtual and real scenes, supporting real-time adjustments to virtual scene parameters based on changes in the real scene.

[0041] Accordingly, the step of using a virtual-real feature mapping model to map the real data to a virtual feature space to obtain the mapped virtual data includes: The first step is to use the perception mapping model to map the real perception data to the virtual feature space to obtain the mapped perception data. In this embodiment, the above mapping process can be represented by a mapping function, wherein the mapping function is: ,in, For the parameters of the perception mapping model, For the perception mapping function, The mapped sensory data, For truly perceptible data.

[0042] The second step is to use the interactive mapping model to map the real feedback data to the virtual feature space to obtain the mapped feedback data. In this embodiment, the above mapping process can be represented by a mapping function, wherein the mapping function is: ,in, For the parameters of the interaction mapping model, For interactive mapping functions, The feedback data after mapping This is real feedback data.

[0043] The third step is to use the environment mapping model to map the real environment parameters to the virtual feature space to obtain the mapped environment parameters. In this embodiment, the above mapping process can be represented by a mapping function, wherein the mapping function is: ,in, For the parameters of the environment mapping model, For environment mapping functions, These are the mapped environment parameters. These are real-world environmental parameters.

[0044] The fourth step is to obtain the mapped virtual data based on the mapped perception data, the mapped feedback data, and the mapped environmental parameters.

[0045] In this embodiment, , This refers to the mapped virtual features (virtual data). , For real features (real data).

[0046] By setting up a perception mapping model, an interaction mapping model, and an environment mapping model respectively, a three-layer dynamic mapping structure of perception mapping + interaction mapping + environment mapping can be constructed. Each layer can realize feature mapping from different angles, which helps to obtain a more accurate virtual scene.

[0047] Then, with the goal of minimizing the maximum mean difference between the mapped virtual data and the real data, the virtual-real feature mapping model is trained to obtain the trained virtual-real feature mapping model. In this embodiment, minimizing the maximum mean difference between the mapped virtual data and the real data can be expressed as: , in, For Gaussian kernel function, , These represent the number of virtual feature samples and the number of real feature samples, respectively.

[0048] It can use embodied intelligent agents to perform diverse operational tasks in multiple real-world scenarios and collect multiple sets of data. As a true feature sample set { }, i=1~ Following the steps above, using the virtual-real feature mapping model with the current parameters, the corresponding mapped virtual feature sample sets are obtained respectively. For j=1~n, calculate the maximum mean difference between the virtual feature distribution and the real feature distribution after mapping. Take the parameters of the virtual-real feature mapping model when the maximum mean difference between the virtual feature distribution and the real feature distribution is minimized. That is, the training objective is to jointly optimize the parameters of the three-layer mapping model. This minimizes the MMD, resulting in a well-trained virtual-real feature mapping model.

[0049] Finally, based on the real data, the trained virtual-real feature mapping model is used to correct the virtual data in the virtual-real dataset to obtain the training dataset.

[0050] In this embodiment, the above correction process may involve using a trained virtual-real feature mapping model to map real data to a virtual feature space, so that the mapped virtual feature data is aligned with the real data, and then replacing the original virtual data in the virtual-real dataset with this data to obtain the training dataset.

[0051] In some embodiments, based on the real data, the virtual data in the virtual-real dataset is corrected using the trained virtual-real feature mapping model to obtain a training dataset, including: The first step is to use the trained virtual-real feature mapping model to map the real data to the virtual feature space, thereby obtaining the trained mapped virtual data. The second step involves correcting the virtual data in the virtual-real dataset based on the trained virtual mapping data to obtain the training dataset.

[0052] In this embodiment, the calibration process relies on the real features collected by the embodied agent as the mapping and calibration benchmark. The calibrated data can directly serve the model training for the embodied agent's operational skills, forming a complete closed loop from real collection, mapping calibration, simulation training, and physical deployment.

[0053] By employing a virtual-real feature mapping model, the real data is mapped to a virtual feature space, resulting in mapped virtual data. The virtual-real feature mapping model is trained with the objective of minimizing the maximum mean difference between the mapped virtual data and the real data, resulting in a trained virtual-real feature mapping model. Based on the real data, the trained virtual-real feature mapping model is used to correct the virtual data in the virtual-real dataset, resulting in a training dataset. Training the virtual-real feature mapping model aligns the virtual data with the real data, facilitating better training of the embodied agent.

[0054] Using the minimization of MMD (Multi-Dimensional Mapping) as the objective function, it calibrates perceptual differences, interaction differences, and environmental differences in real time, supporting dynamic adjustment of scene parameters. Experimental results show that the mapping error is ≤3%, the virtual-real data synchronization delay is ≤200ms, and the similarity of virtual-real feature distributions is ≥97%, thus effectively solving the problem of the virtual-real divide.

[0055] Step 250: Based on the training dataset, perform skill training on the embodied agent.

[0056] In this embodiment, an improved Proximal Policy Optimization (PPO) algorithm can be used for embodied agent skill training. During training, multimodal fusion features (features extracted from the training dataset) can be used as input to the policy network, with task completion and execution accuracy as reward functions to optimize the skill execution policy. The policy network can adopt a hybrid Transformer+MLP structure, with multimodal fusion features as input. The output is the skill execution action. The objective function for training is: ,in, For policy network parameters, For trajectory, For trajectory rewards, For the entropy of the strategy, This is the entropy coefficient (to ensure exploratory learning). Skill parameters can be recorded during training. Information such as sensory data and operational feedback is used to form a skills training dataset. Evaluation metrics based on task models can be used through virtual simulation testing platforms. Evaluate indicators such as skill execution accuracy, stability, and task completion rate; if the requirements are not met, adjust the training parameters. The mapping parameters ω of the virtual-real feature mapping model are retrained until the skill training effect meets the requirements.

[0057] In the above implementation process, the task requirements are obtained; based on the task requirements, the initial parameters of the embodied agent are set and a virtual scene is constructed; the embodied agent is controlled to perform skill operations, and a virtual-real dataset is obtained, which includes virtual data collected by the embodied agent in the virtual scene and real data collected in the real scene; with the goal of minimizing the feature distribution difference between the virtual and real scenes, the virtual data in the virtual-real dataset is corrected based on the real data to obtain a training dataset; and the embodied agent is trained on the training dataset. By correcting the virtual data with the goal of minimizing the feature distribution difference between the virtual and real scenes, feature-level alignment between the virtual and real scenes can be achieved, ensuring that virtual training skills can be directly transferred to real scenes, bridging the virtual-real divide. By replacing most physical training with virtual training, physical losses are reduced by more than 80%, and the training cycle is significantly shortened, solving the industry pain points of high training costs and long training cycles for embodied agents. This method is compatible with various embodied intelligent agents (service robots, industrial robots, medical robots, autonomous driving vehicles) and multiple application scenarios (industrial manufacturing, smart homes, healthcare, intelligent transportation). It supports flexible adjustment of system parameters, can seamlessly connect with existing embodied intelligent agent execution systems, requires no reconstruction of existing equipment, and has low deployment costs.

[0058] In some embodiments, it also includes: First, the real and virtual features in the training dataset are weighted and fused to obtain the fused features; In this embodiment, please refer to Figure 5 , Figure 5 This diagram illustrates the synchronous acquisition and feature fusion of multimodal sensing data according to an embodiment of this application. A Transformer multi-head self-attention mechanism can be used to weightedly fuse virtual and real multimodal features, highlighting the weights of key features. The fusion function is: , in, , These are query, key, and value vectors, respectively. For learnable parameters, For the number of attention heads.

[0059] Then, the domain-invariant features are extracted from the fusion features; In this embodiment, based on domain adaptive learning, virtual / real features can be distinguished by a domain discriminator and domain-invariant features can be extracted by a feature encoder, based on the fused features. This forms an adversarial training framework. The objective function is: ,in, For feature encoder, As a domain discriminator, adversarial training is used to make the features extracted by the encoder indistinguishable by the discriminator, thus achieving domain invariance.

[0060] Then, using the domain-invariant features, the embodied agent is trained to obtain the first skill parameters; Then, obtain real-world scene samples; Finally, based on the real-world scene samples, the first skill parameters are adjusted using a model-independent meta-learning algorithm to obtain the second skill parameters.

[0061] In this embodiment, the Model-Independent Meta-Learning (MAML) algorithm can be used to process the skill parameters obtained from virtual training. As the initial parameters (i.e., the first skill parameters), they are rapidly fine-tuned using a small number of real-world scene samples to adapt the skill parameters to the real-world scenario, thus obtaining the second skill parameters. The core goal of MAML is to learn an initial parameter. The objective function is to minimize the loss in the new domain (real-world scenario) after a small number of gradient updates. ,in Task distribution (real-world scenario tasks). The meta-learning rate, This represents the task loss in a real-world scenario. This allows for the transfer of skill parameters.

[0062] In practical implementation, a multi-dimensional migration effect evaluation index system can also be constructed, including execution accuracy. Operational stability Environmental adaptability Task completion rate The evaluation formula is: in This is the weighting coefficient. If... (Execution accuracy requirements), adjust migration algorithm parameters and skill parameters Then, the migration process is repeated until the skills are adapted to real-world scenarios.

[0063] By integrating the Transformer multi-head self-attention mechanism with MAML meta-learning, a MAML-Trans multimodal feature fusion transfer algorithm is proposed. This algorithm can automatically extract skill domain-invariant features from different scenarios and tasks, supporting multi-skill collaborative transfer. Experimental results demonstrate that the transfer efficiency is improved by more than 70% compared to existing technologies, the cross-domain skill reuse rate is ≥85%, and the real-world skill transfer accuracy is ≥90%.

[0064] In some embodiments, it also includes: First, acquire environmental change data and task change information in the real-world scenario; In this embodiment, environmental change data of real-world scenes can be collected through a multimodal perception layer. (Changes in object placement, ambient lighting, and object type) and task change information .

[0065] Then, based on the environmental change data and the task change information, it is determined whether the embodied intelligent agent needs to perform skill generalization. In this embodiment, a scene recognition model can be constructed using CNN+MLP to identify elements based on environmental change data and task change information, in order to determine whether skill generalization is necessary, with a recognition accuracy of ≥95%. The above-mentioned scene recognition model constructed using CNN+MLP can be implemented using existing technologies, and will not be elaborated further here.

[0066] Then, when it is determined that the embodied agent needs to perform skill generalization, a skill generalization strategy is generated using the PPO-GAE algorithm based on the skill training dataset, the environmental change data, and the task change information. Finally, based on the skill generalization strategy, the skill execution parameters and operation trajectory of the embodied agent are adjusted.

[0067] In this embodiment, changes in the scene can be considered. Task changes and skill training dataset The algorithm employs an improved proximal policy optimization-generalized advantage estimation (PPO-GAE) algorithm to autonomously generate skill generalization policies. The core of the algorithm lies in constructing a system that adapts to changing scenarios. Task changes and skill training dataset For the joint input conditional policy network and conditional value network, scene and task change information is injected into the middle layer of the network through the FiLM layer to achieve adaptive modulation of behavior distribution. Simultaneously, a hybrid experience replay buffer is maintained to perform hierarchical sampling of trajectory data for different scene-task combinations. Gaussian Advantage (GAE) is used to calculate advantage estimation, and PPO is used to truncate the objective function to ensure training stability. The finally trained policy network can infer the optimal operation command adapted to the current conditions online based on the embodied agent's real-time perception of scene and task dynamic changes, forming a closed-loop generalized control of perception, mapping, reasoning, and execution. The aforementioned skill generalization strategy refers to a neural network control model trained by the improved PPO-GAE algorithm, internalizing the ability to adapt to scene and task changes. This model can infer the optimal operation command online based on the real-time perceived environmental and task changes when the embodied agent performs a task.

[0068] The improved PPO-GAE algorithm described above adds a generalization constraint term to the traditional PPO algorithm to ensure the policy's generalization ability in new scenarios. The objective function is: , in, For probability ratios, This is the generalized advantage estimate (GAE). This is the cutting factor. To generalize the constraint loss (minimize the policy difference between the new scene and the training scene). The generalization constraint weights are used to optimize the policy network parameters through this objective function. Adjust skill execution parameters and operation trajectories to adapt to new scenarios and tasks.

[0069] Based on the environmental change data and task change information, it is determined whether the embodied agent needs skill generalization. If skill generalization is required, a skill generalization strategy is generated using the PPO-GAE algorithm based on the skill training dataset, the environmental change data, and the task change information. Based on this strategy, the skill execution parameters and operation trajectory of the embodied agent are adjusted. This achieves policy-level autonomous skill generalization, allowing the embodied agent to autonomously adjust its execution strategy according to new scenarios and tasks. The generalization response time is ≤500ms, eliminating the need for manual retraining and significantly improving environmental adaptability and task suitability.

[0070] In some embodiments, it also includes: First, acquire execution data, which is obtained by the embodied intelligent agent executing generalized skills in a new real-world scenario; In this embodiment, the embodied agent can be controlled to execute generalized skills in a new scenario. Specifically, the execution control layer can receive skill execution instructions output by the skill generalization layer. The system employs a force-position hybrid control algorithm to control the actuators (robotic arm, mobile chassis) of the embodied intelligent agent to perform skill operations, and then collects execution data in real time. (Operational error, execution efficiency, force feedback) and environmental feedback data , and obtain the execution data.

[0071] The core formula for force-position hybrid control is: ,in For joint torque, For Jacobian matrices, For the desired / actual pose, For expected / actual torque, For proportional / derivative / torque coefficients.

[0072] Then, based on the execution data and the preset generalization evaluation index, the parameters of the skill generalization strategy are adjusted to obtain a new skill generalization strategy; In this embodiment, the preset generalization evaluation metrics include scenario adaptability, execution accuracy, stability, and response time, used to evaluate the generalization effect. The corresponding generalization evaluation metric values ​​can be matched based on the execution data, and then a weighted comprehensive generalization evaluation score is synthesized. When the comprehensive generalization evaluation score is lower than a preset threshold, the parameters of the generalization strategy can be automatically adjusted according to the deviation of each sub-metric. This includes adjusting the PPO cutoff threshold to control the policy update magnitude, adjusting the discount factor to change the long-term reward weight, and adjusting the entropy regularization coefficient to adjust the exploration intensity. Based on the adjusted parameters, the improved PPO-GAE policy optimization process is re-executed, and the policy network is retrained while maintaining existing transfer knowledge to generate an updated generalization strategy. The strategy is then redeployed to the physical robot for iterative verification until the comprehensive evaluation score reaches the preset requirement, thus forming a closed-loop adaptive generalization optimization mechanism of deployment, evaluation, parameter tuning, re-optimization, and redeployment.

[0073] Finally, based on the new skill generalization strategy, the skill execution parameters and operation trajectory of the embodied agent are adjusted.

[0074] By introducing multi-dimensional generalization evaluation metrics (scenario adaptability, execution accuracy, stability, and response time), and incorporating generalization constraints into the objective function, policy-level dynamic generalization of skills is achieved. Experimental results show that the generalization response time is ≤500ms, the skill adaptability accuracy in new scenarios is ≥90%, and the generalization success rate for untrained tasks is ≥85%.

[0075] In some embodiments, it also includes: First, obtain the generalized skill parameters and execution data; Then, the training dataset is updated based on the generalized skill parameters and execution data.

[0076] In this embodiment, the generalized skill parameters are... Execution data The domain-invariant feature library and training dataset for virtual scenes are updated to the transfer learning layer, enabling synchronous optimization of virtual scene skill training. This forms a closed loop of virtual training, real-world transfer, generalization optimization, and synchronous updates, continuously improving the skill generalization ability and environmental adaptability of the embodied agent.

[0077] After the skill is executed, the skill execution data can be transmitted via the ROS2 / MQTT encrypted transmission protocol. and environmental feedback data Real-time data is transmitted to the task requirement layer and the skill generalization layer, providing real-time data support for skill generalization optimization and transfer learning adjustments. The skill generalization layer then adjusts the data based on the feedback. and The skill execution strategy is further optimized using gradient descent. ; Adjusting transfer learning algorithm parameters Optimize the ability to extract domain-invariant features; the virtual-real fusion mapping layer uses environmental feedback data. Dynamic calibration mapping parameters This enables end-to-end continuous optimization of skills, ensuring that embodied agents can execute skills stably and accurately in real-world scenarios.

[0078] It should be noted that model pruning and quantization techniques can be used to lightweight the deep learning model in the system, reducing the model size by more than 70% and increasing the inference speed by more than 80%. This allows it to be deployed on edge computing devices (such as NVIDIA Jetson AGX Orin) to meet the real-time requirements of embodied intelligent agents.

[0079] The following examples illustrate this; please refer to them. Figures 2-7 And Table 1, Figure 2 This diagram schematically illustrates the overall architecture of a virtual-real fusion transfer learning and skill generalization system for embodied intelligent agents according to an embodiment of this application. Figure 2 This illustration schematically shows the overall architecture of a virtual-real fusion transfer learning and skill generalization system for embodied intelligent agents according to an embodiment of this application; Figure 3 This illustration schematically shows the three-layer dynamic mapping structure and data interaction diagram of the virtual-real fusion mapping layer according to an embodiment of this application; Figure 4 This illustration schematically shows a workflow diagram of multimodal cross-domain transfer learning and skill generalization according to embodiments of this application; Figure 5 This illustration schematically shows a multimodal sensing data synchronous acquisition and feature fusion according to an embodiment of this application; Figure 6 The illustration shows a comparison diagram of skill transfer and generalization effects according to embodiments of this application; Figure 7 The illustration shows a comparison of skill transfer accuracy and generalization ability according to an embodiment of this application; Table 1 is a functional technology table for each layer.

[0080]

[0081] The task requirement layer issues task objectives and evaluation indicators to each layer → the multimodal perception layer collects and preprocesses virtual and real multimodal data → the virtual-real fusion mapping layer achieves virtual-real feature-level alignment → the transfer learning layer extracts domain-invariant features and completes cross-domain transfer of skill parameters → the skill generalization layer autonomously optimizes strategies based on environmental changes → the execution control layer executes skills and collects feedback data → the feedback data is fed back to the task requirement layer and the skill generalization layer → each layer dynamically adjusts parameters and strategies based on feedback, achieving end-to-end closed-loop optimization. Specific steps include: Step 1: Task Requirements Analysis and System Initialization Task semantic parsing: A BERT pre-trained model is used to perform semantic parsing of the user's natural language / structured instructions, extracting task feature vectors, including skill type. (Grab, assemble, inspect, etc.) Task objectives Scene parameters (Lighting, friction, object size, etc.), execution accuracy requirements Constraints (Execution time, equipment wear and tear), generate a formalized task model:

[0082] System parameter initialization: based on the task model Configure the sensor parameters (sampling frequency) of the multimodal sensing layer. Perception range A high-fidelity virtual simulation scene is constructed using NeRF, and the mapping parameters of the virtual-real fusion mapping layer are initialized. Algorithm parameters of the transfer learning layer Generalization evaluation indicators for the skill generalization layer Ensure that the parameters of each module match the task requirements.

[0083] Requirements Verification and Revision: The system generates a task requirements analysis report, including a formalized task model and parameter configuration, and provides it to the user for verification. If the user requests revisions, the parameters at each level are updated through the parameter fine-tuning module until the requirements analysis conforms to the actual application scenario. The final confirmed task model is then recorded as [the correct model]. .

[0084] Step 2: Multimodal sensing data acquisition and preprocessing Multimodal data synchronous acquisition: In a virtual scene, virtual multimodal data is acquired through a simulation engine. (Image, force, haptic feedback, posture data); In real-world scenarios, real multimodal data is collected through a multi-sensor array. To address the differences in sampling frequencies among different sensors, a linear interpolation + timestamp alignment algorithm is used to synchronize virtual and real data, with a synchronization error ≤200ms.

[0085] Data preprocessing: Cleaning: Outlier data is removed using the 3σ criterion, and noisy data is removed using median filtering; Standardization: Perform Z-score standardization on numerical data (force, posture, haptic feedback), using the following formula: in The original data, The mean, The standard deviation is used to normalize the image data (mapping pixel values ​​to [0,1]). Feature extraction: Visual features were extracted from the image data using ResNet50. MLP is used to extract temporal features from force / tactile / posture data. Generate standardized multimodal sensing datasets ,in For virtual preprocessed data, This is for real preprocessed data.

[0086] Step 3: Dynamic mapping of virtual and real worlds and virtual skills training Three-layer dynamic mapping model construction: Construct a three-layer dynamic mapping structure of "perception mapping + interaction mapping + environment mapping". Each layer uses a deep learning model to achieve feature-level mapping. The core is to minimize the difference in feature distribution between virtual and real scenes. Perceptual mapping: Using a CNN+Transformer model, real perceptual features are mapped... Mapping to the virtual perception feature space enables real-time updates of virtual scene perception data. The mapping function is: ,in For the parameters of the perception mapping model; Interaction Mapping: An MLP model is used to calibrate the differences in operational feedback (force and touch) between virtual and real scenes. The mapping function is as follows: To ensure that the feature distribution of virtual operation feedback is consistent with that of real operation feedback; Environment mapping: A GNN (Graph Neural Network) model is used to dynamically synchronize environmental parameters between virtual and real scenes. (Lighting, friction, physical properties of objects, spatial layout), the mapping function is: It supports real-time adjustment of virtual scene parameters based on changes in the real scene.

[0087] The overall objective function of the three-layer mapping is to minimize the maximum mean difference (MMD) between virtual and real features, thereby achieving feature distribution alignment. Virtual Skills Training: Based on Standardized Multimodal Aware Datasets In a dynamically mapped virtual scene, an improved PPO algorithm is used to train the embodied agent's skills. Multimodal fusion features are used as input to the policy network, and task completion and execution accuracy are used as reward functions to optimize the skill execution policy. The policy network adopts a hybrid Transformer+MLP structure, with multimodal fusion features as input. The output is the skill execution action. The objective function for training is: in For policy network parameters, For trajectory, For trajectory rewards, For the entropy of the strategy, This is the entropy coefficient (to ensure exploratory nature). Skill parameters are recorded during training. Information such as sensory data and operational feedback is used to form a skills training dataset. .

[0088] Training effectiveness verification: Based on the task model, a virtual simulation testing platform was used for testing. Evaluation indicators Evaluate indicators such as skill execution accuracy, stability, and task completion rate; if the requirements are not met, adjust the training parameters. and mapping parameters Then, retrain until the skills training results meet the requirements.

[0089] Step 4: Multimodal cross-domain transfer learning and skill adaptation The core of this step is to extract domain-invariant features from virtual and real-world scenarios, and to achieve rapid cross-domain transfer of skill parameters through meta-learning. The MAML-Trans multimodal cross-domain transfer learning algorithm is adopted, and the specific process is as follows: Multimodal feature fusion: A Transformer multi-head self-attention mechanism is employed to weight and fuse virtual and real multimodal features, highlighting the weights of key features. The fusion function is as follows: .

[0090] Domain-invariant feature extraction: Based on domain adaptive learning, and after feature fusion, a domain discriminator distinguishes between virtual and real features, and a feature encoder extracts domain-invariant features. This forms an adversarial training framework with the objective function being: .

[0091] Meta-learning-driven skill parameter transfer: Employing the Model-Independent Meta-Learning (MAML) algorithm, skill parameters obtained from virtual training are transferred... As initial parameters, they are rapidly fine-tuned using a small number of real-world scenario samples to adapt skill parameters to real-world scenarios. The core objective of MAML is to learn these initial parameters. The objective function is to minimize the loss in the new domain (real-world scenario) after a small number of gradient updates. .

[0092] Migration effectiveness evaluation: Construct a multi-dimensional migration effectiveness evaluation index system, including execution accuracy. Operational stability Environmental adaptability Task completion rate The evaluation formula is: in This is the weighting coefficient. If... (Execution accuracy requirements), adjust migration algorithm parameters and skill parameters Then, the migration process is repeated until the skills are adapted to real-world scenarios.

[0093] Step 5: Autonomous Skill Generalization and Continuous Optimization The core of this step is to achieve policy-level autonomous generalization of the embodied agent. This is achieved by introducing a skill generalization evaluation module and an improved PPO-GAE algorithm, incorporating generalization ability into the objective function of policy optimization. The specific process is as follows: Generalized scene recognition: Collects environmental change data of real-world scenes through a multimodal perception layer. (Changes in object placement, ambient lighting, and object type) and task change information A scene recognition model is constructed using CNN+MLP to determine whether skill generalization is needed, with a recognition accuracy of ≥95%.

[0094] Generalization strategy generation: Based on the improved PPO-GAE algorithm, combined with scene changes. Task changes and skill training dataset The system autonomously generates a skill generalization strategy. Based on traditional PPO (Professional Skill Object Generation), a generalization constraint is added to ensure the strategy's generalization ability in new scenarios. The objective function is: .

[0095] The network parameters of the strategy are optimized using this objective function. Adjust skill execution parameters and operation trajectories to adapt to new scenarios and tasks.

[0096] Generalization effect verification: Control the embodied agent to execute the generalized skill in a new scenario and collect execution data. The generalization effect is evaluated based on generalization assessment metrics (scenario adaptability, execution accuracy, stability, and response time); if the generalization effect does not meet the requirements, the parameters of the generalization strategy are adjusted. , , Then, perform generalization optimization again.

[0097] Continuous optimization of the closed loop: generalized skill parameters Execution data The domain-invariant feature library and the training dataset of the virtual scene are updated to the transfer learning layer to achieve synchronous optimization of virtual scene skill training, forming a closed loop of "virtual training - real transfer - generalization optimization - synchronous update", which continuously improves the skill generalization ability and environmental adaptability of the embodied agent.

[0098] Step 6: Skill Execution and Feedback Loop Precise skill execution: The execution control layer receives skill execution instructions output by the skill generalization layer. The force-position hybrid control algorithm is used to control the actuators (robotic arm, mobile chassis) of the embodied intelligent agent to complete skill operations, and the execution data is collected in real time. (Operational error, execution efficiency, force feedback) and environmental feedback data The core formula for force-position hybrid control is: .

[0099] Real-time data feedback: Skill execution data is transmitted via ROS2 / MQTT encrypted transmission protocol. and environmental feedback data Real-time data transmission to the task requirement layer and skill generalization layer provides real-time data support for skill generalization optimization and transfer learning adjustments.

[0100] End-to-end closed-loop optimization: The skill generalization layer optimizes based on feedback data. and The skill execution strategy is further optimized using gradient descent. ; Adjusting transfer learning algorithm parameters Optimize the ability to extract domain-invariant features; the virtual-real fusion mapping layer uses environmental feedback data. Dynamic calibration mapping parameters This enables end-to-end continuous optimization of skills, ensuring that embodied agents can execute skills stably and accurately in real-world scenarios.

[0101] A hybrid deep learning model combining CNN, Transformer, and GNN is used to construct a virtual-real scene mapping model. Minimizing the virtual-real difference (MMD) is the objective function, and perceptual, interactive, and environmental differences are calibrated in real time, supporting dynamic adjustment of scene parameters. Experiments have verified that the mapping error is ≤3%, the virtual-real data synchronization delay is ≤200ms, and the similarity of virtual-real feature distributions is ≥97%, fundamentally solving the "virtual-real gap" problem.

[0102] Multimodal cross-domain transfer learning algorithm design: Integrating the Transformer multi-head self-attention mechanism and MAML meta-learning, a MAML-Trans multimodal feature fusion transfer algorithm is proposed. This algorithm can automatically extract skill domain-invariant features from different scenarios and tasks, supporting multi-skill collaborative transfer. Experimental results show that the transfer efficiency is improved by more than 70% compared to existing technologies, the cross-domain skill reuse rate is ≥85%, and the real-world skill transfer accuracy is ≥90%.

[0103] Autonomous Skill Generalization Algorithm Optimization: An improved deep reinforcement learning algorithm, PPO-GAE-Gen, was designed, introducing multi-dimensional generalization evaluation metrics (scene adaptability, execution accuracy, stability, and response time). Generalization constraints were incorporated into the objective function to achieve policy-level dynamic generalization of skills. Experimental results show that the generalization response time is ≤500ms, the skill adaptation accuracy in new scenes is ≥90%, and the generalization success rate on untrained tasks is ≥85%.

[0104] Multimodal perception fusion and data security assurance: Multimodal data synchronous acquisition is achieved by using timestamp alignment and linear interpolation. Deep fusion of multimodal features is achieved through a hybrid model of Transformer+CNN+MLP, with a fusion accuracy of ≥98% for perception features. Data interaction is achieved using the MQTT / ROS2 encrypted transmission protocol. A hash value verification mechanism is introduced to prevent abnormal data access. A sensor fault detection model (CNN+LSTM) is designed, with a sensor fault identification accuracy of ≥99%, ensuring the stable operation of the perception system and data security.

[0105] The following section uses industrial robot assembly and home service robot grasping as examples to provide a detailed explanation of this method, including experimental scenario modeling, experimental environment configuration, experimental steps, experimental results and analysis, to illustrate the adaptability, effectiveness and superiority of this method in different embodied intelligent agent scenarios.

[0106] I. Virtual-Real Integration, Transfer, and Generalization of Industrial Robot Assembly Skills Specifically applied in the field of industrial manufacturing, this scenario involves the assembly of precision mechanical parts. The embodied intelligent agent is a six-axis industrial assembly robot, and the task objective is to achieve the precision assembly of multiple types of mechanical parts. The formal scenario model is described as follows: Skill Type Bolt assembly, parts splicing; scene parameters Assembly ambient lighting (500-2000 lux), parts placement error (±0.5 mm), parts friction coefficient (0.15-0.3); execution accuracy requirements. Assembly positioning accuracy ±0.03mm, assembly pass rate ≥99%; constraints The assembly time for a single part is ≤5s, and the equipment wear rate is ≤0.1% / thousandth of times. The core technical challenges of this scenario are: multiple part models (3 different specifications), high assembly accuracy requirements, high real-world training costs, and dynamic changes in scenario parameters. Traditional pure physical training methods suffer from low sample efficiency and high equipment wear, while pure virtual training methods suffer from the virtual-real gap and low transfer accuracy.

[0107] Hardware environment: Server: Intel Xeon Platinum 8470C (40 cores, 80 threads), 256GB RAM, 4TB SSD, responsible for system computation and model training; Industrial assembly robot: Six-axis robotic arm (repeat positioning accuracy ±0.02mm), equipped with vision sensor (1920×1080, 30fps), force sensor (0-500N, 100Hz), and tactile sensor (1024-point array), responsible for physical operations; Virtual simulation server: NVIDIA A100 (80GB) GPU, supporting NeRF high-fidelity virtual scene rendering, responsible for virtual scene construction and virtual training.

[0108] Software environment: Operating System: Ubuntu 22.04 LTS, ROS2 Humble (robot operating system); Programming Languages ​​and Frameworks: Python 3.10, PyTorch 2.1 (deep learning), Unity 2023 (virtual simulation), OpenAI Gym (reinforcement learning environment); Core Algorithms: MAML-Trans (transfer learning), PPO-GAE-Gen (generalization learning), ResNet50 / Transformer (feature extraction / fusion); Data Transmission and Monitoring: MQTT encrypted protocol, Prometheus+Grafana (real-time monitoring), TensorBoard (model training visualization).

[0109] Specific implementation steps: 1. Use the BERT model to analyze task requirements and generate a formal task model. Initialize sensor parameters (vision 30fps, force 100Hz), construct a NeRF high-fidelity industrial assembly virtual scene, and set the MAML meta-learning rate. PPO cutting factor .

[0110] 2. Simultaneously collect virtual and real multimodal data, remove outliers using the 3σ criterion, standardize numerical data using Z-score, extract visual features using ResNet50, extract temporal features using MLP, and achieve a timestamp alignment synchronization error of 150ms to generate a standardized multimodal perception dataset. .

[0111] 3. A three-layer dynamic mapping model was constructed, and the mapping parameters were trained using MMD as the objective function, with a mapping error of 2.5%. The PPO-GAE algorithm was used to train the assembly skills of three types of parts in a virtual scene. After 10,000 training iterations, the virtual assembly accuracy reached ±0.025mm, forming a skill training dataset. .

[0112] 4. Employ the MAML-Trans algorithm to extract domain-invariant features and integrate virtual skill parameters. As initial parameters, they are rapidly fine-tuned using 500 real-world scenario samples to achieve skill parameter transfer; the transfer effect is evaluated. The assembly accuracy in real-world scenarios reaches ±0.03mm, and the migration cycle is shortened by 75% compared to existing technologies.

[0113] 5. When new specifications of parts are introduced into the production line, the scene recognition model identifies the new task requirements (accuracy of 98%), uses the PPO-GAE-Gen algorithm to generate a generalization strategy, with a generalization response time of 450ms and an assembly accuracy of ±0.035mm for the new parts; the generalized skill parameters are updated to the feature library and virtual scene to achieve synchronous optimization.

[0114] 6. The robot is controlled by a force-position hybrid control algorithm to perform assembly skills. The execution data is collected in real time by ROS2, which improves the assembly efficiency by 65% ​​and the assembly qualification rate by 99%. Feedback data is sent back to each layer to achieve end-to-end closed-loop optimization and reduce equipment loss by 80%.

[0115] The experimental results are shown in Table 2 below. Table 2 is a comparison table of the core experimental results:

[0116] The experimental results show that the assembly accuracy achieved by this method is ±0.03mm, which meets the requirements of industrial precision assembly. It is comparable to pure physical training and far superior to pure virtual training + fine-tuning, proving that the three-layer dynamic mapping mechanism effectively solves the problem of the virtual-real gap. The migration cycle is only 6 hours, which is more than 75% shorter than existing technologies, and the cross-domain reuse rate reaches 92%, proving that the MAML-Trans algorithm can effectively extract domain-invariant features and improve cross-domain migration efficiency. With a generalization response time of only 450ms and an assembly accuracy of ±0.035mm for new parts, the PPO-GAE-Gen algorithm demonstrates that it can achieve policy-level autonomous generalization and quickly adapt to new task requirements. Assembly efficiency is increased by 65%, assembly qualification rate is 99%, and equipment wear is reduced by 80%, proving that the present invention has significant industrial value in improving production efficiency and reducing costs.

[0117] II. Virtual-Real Fusion Transfer and Generalization of Grasping Skills in Home Service Robots This scenario involves a home service robot in the smart home field, where the embodied intelligent agent is a wheeled mobile service robot equipped with a robotic arm and multimodal sensors. The task objective is to accurately grasp various types of objects in the complex environment of a home. The formalized scenario model is as follows: Skill type Object grasping (rigid / flexible, different shapes / weights); Scene parameters : Home environment lighting (100-1500 lux), clutter of objects (0.6-0.9), weight of objects (0.1-5 kg); Execution accuracy requirements Grabbing accuracy ≥90%, grabbing stability ≥85%; Constraints No professional trainers were involved, and the training period was ≤72 hours.

[0118] The core technical challenges of this scenario are: a wide variety of object types (cups, books, fruits, remote controls, toys, etc.), a complex environment (changing lighting, and objects being placed haphazardly), and a lack of professional trainers. Traditional training methods cannot meet the needs for low cost and high generalization.

[0119] Hardware environment: Home service robot (wheeled mobile chassis + four-axis robotic arm), equipped with vision sensor (1280×720, 30fps), tactile sensor, inertial sensor; edge computing device (NVIDIA Jetson AGX Orin, 32GB); virtual simulation server (NVIDIA RTX 4090, 24GB).

[0120] Software environment: Ubuntu 20.04 LTS, ROS2 Foxy, Python 3.9, PyTorch 2.0, Unity 2023 (home scene simulation), MAML-Trans and PPO-GAE-Gen algorithms (lightweight version).

[0121] This method enables rapid virtual training, accurate transfer to real-world scenarios, and autonomous generalization of grasping skills for home service robots. The core experimental results are as follows: The virtual training cycle is only 12 hours, which is 80% shorter than physical training, and no professional trainers are required. The accuracy of capturing real-world home scenarios reached 92%, and the capturing stability was improved by 70%. The generalization success rate for grasping untrained objects (remote control, toy) reached 88%, with a generalization response time of 400ms; The multimodal perception fusion accuracy reaches 98%, and the grasping success rate remains above 85% even in complex environments with changing lighting and cluttered objects.

[0122] The lightweight algorithm version of this method can be effectively deployed on edge computing devices to meet the real-time requirements of home service robots; the autonomous generalization mechanism allows the robot to quickly adapt to various types of objects in the home environment, solving the problem of "fixed grasping skills and weak generalization ability" in traditional service robots; the low-cost and easy-to-operate training method makes the large-scale application of home service robots possible and has significant industrial value.

[0123] Please refer to Figure 8 , Figure 8 This illustration schematically shows a structural diagram of an embodied agent training device according to an embodiment of the present application. This embodiment provides an embodied agent training device, including an acquisition module 410, an initialization module 420, a data collection module 430, a correction module 440, and a training module 450, wherein: Module 410 is used to obtain task requirements; The initialization module 420 is used to set the initial parameters of the embodied intelligent agent and construct a virtual scene based on the task requirements. The acquisition module 430 is used to acquire virtual and real datasets, which include virtual data collected by the embodied intelligent agent in the virtual scene and real data collected in the real scene. The correction module 440 is used to correct the virtual data in the virtual-real dataset based on the real data with the goal of minimizing the feature distribution difference between the virtual and real scenes, so as to obtain a training dataset. The training module 450 is used to train the embodied agent's skills based on the training dataset.

[0124] The embodied intelligent agent training device includes a processor and a memory. The acquisition module 410, initialization module 420, acquisition module 430, correction module 440 and training module 450 are all stored as program units in the memory. The processor executes the program units stored in the memory to realize the corresponding functions.

[0125] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the embodied agent can be trained by adjusting the kernel parameters.

[0126] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0127] This invention provides a machine-readable storage medium storing a program that, when executed by a processor, implements the embodied agent training method.

[0128] This invention provides a processor for running a program, wherein the program executes the embodied agent training method during runtime.

[0129] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements an embodied agent training method. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0130] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0131] In one embodiment, the embodied intelligent agent training device provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 9 The computer device shown is running the program. The computer device's memory can store the various program modules that make up the embodied intelligent agent training device, for example, Figure 8 The diagram shows an acquisition module 410, an initialization module 420, a data acquisition module 430, a correction module 440, and a training module 450. The computer program comprised of these modules causes the processor to execute the steps in the embodied intelligent agent training methods of the various embodiments of this application described in this specification.

[0132] Figure 9 The computer equipment shown can be used as follows Figure 8 The acquisition module 410 in the embodied intelligent agent training device shown executes step 210. The computer device can execute step 220 via the initialization module 420. The computer device can execute step 230 via the acquisition module 430. The computer device can execute step 240 via the calibration module 440. The computer device can execute step 250 via the training module 450.

[0133] This application provides an electronic device comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the aforementioned embodied agent training method by executing the instructions stored in the memory. When the processor executes the instructions, it performs the following steps: Obtain task requirements; Based on the task requirements, set the initial parameters of the embodied intelligent agent and construct a virtual scene; The embodied intelligent agent is controlled to perform skill operations and acquire virtual and real datasets, which include virtual data collected by the embodied intelligent agent in the virtual scene and real data collected in the real scene; With the goal of minimizing the feature distribution difference between virtual and real scenes, the virtual data in the virtual and real dataset is corrected based on the real data to obtain the training dataset; Based on the training dataset, the embodied agent is trained in skills.

[0134] In one embodiment, based on the task requirements, initial parameters of the embodied agent are set and a virtual scene is constructed, including: The task requirements are semantically parsed to obtain the task feature vector; Based on the task feature vector, a task model is obtained; Based on the task model, the initial parameters of the embodied agent are set and a virtual scene is constructed.

[0135] In one embodiment, the step of correcting the virtual data in the virtual-real dataset based on the real data, with the goal of minimizing the feature distribution difference between the virtual and real scenes, to obtain a training dataset includes: A virtual-real feature mapping model is used to map the real data to a virtual feature space to obtain the mapped virtual data. The virtual-real feature mapping model is trained with the goal of minimizing the maximum mean difference between the mapped virtual data and the real data, resulting in a trained virtual-real feature mapping model. Based on the real data, the virtual data in the virtual-real dataset is corrected using the trained virtual-real feature mapping model to obtain the training dataset.

[0136] In one embodiment, based on the real data, the virtual data in the virtual-real dataset is corrected using the trained virtual-real feature mapping model to obtain a training dataset, including: The trained virtual-real feature mapping model is used to map the real data to the virtual feature space to obtain the trained mapped virtual data. Based on the trained mapped virtual data, the virtual data in the virtual and real datasets are corrected to obtain the training dataset.

[0137] In one embodiment, the real data includes real perception data, real feedback data, and real environmental parameters, and the virtual-real feature mapping model includes a perception mapping model, an interaction mapping model, and an environment mapping model. The process of mapping the real data to a virtual feature space using a virtual-real feature mapping model to obtain the mapped virtual data includes: The real perceptual data is mapped to a virtual feature space using the aforementioned perceptual mapping model to obtain the mapped perceptual data. The interactive mapping model is used to map the real feedback data to the virtual feature space to obtain the mapped feedback data. The real environment parameters are mapped to a virtual feature space using the environment mapping model to obtain the mapped environment parameters; Based on the mapped perception data, the mapped feedback data, and the mapped environmental parameters, the mapped virtual data is obtained.

[0138] In one embodiment, it also includes: The real and virtual features in the training dataset are weighted and fused to obtain the fused features; Extract the domain-invariant features from the fusion features; Using the domain-invariant features, the embodied agent is trained to obtain first skill parameters; Obtain real-world scene samples; Based on the real-world scene samples, the first skill parameters are adjusted using a model-independent meta-learning algorithm to obtain the second skill parameters.

[0139] In one embodiment, it also includes: Obtain environmental change data and task change information in the real-world scenario; Based on the environmental change data and the task change information, determine whether the embodied intelligent agent needs to perform skill generalization. When it is determined that the embodied agent needs to perform skill generalization, a skill generalization strategy is generated using the PPO-GAE algorithm based on the skill training dataset, the environmental change data, and the task change information. Based on the skill generalization strategy, the skill execution parameters and operation trajectory of the embodied intelligent agent are adjusted.

[0140] In one embodiment, it also includes: Acquire execution data, which is obtained by the embodied intelligent agent performing generalized skills in a new real-world scenario; Based on the execution data and the preset generalization evaluation index, the parameters of the skill generalization strategy are adjusted to obtain a new skill generalization strategy. Based on the new skill generalization strategy, the skill execution parameters and operation trajectory of the embodied agent are adjusted.

[0141] In one embodiment, it also includes: Obtain the generalized skill parameters and execution data; The training dataset is updated based on the generalized skill parameters and execution data.

[0142] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0147] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0148] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0150] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for training an embodied intelligent agent, characterized in that, include: Obtain task requirements; Based on the task requirements, set the initial parameters of the embodied intelligent agent and construct a virtual scene; The embodied intelligent agent is controlled to perform skill operations and acquire virtual and real datasets, which include virtual data collected by the embodied intelligent agent in the virtual scene and real data collected in the real scene; With the goal of minimizing the feature distribution difference between virtual and real scenes, the virtual data in the virtual and real dataset is corrected based on the real data to obtain the training dataset; Based on the training dataset, the embodied agent is trained in skills.

2. The embodied intelligent agent training method according to claim 1, characterized in that, Based on the aforementioned task requirements, initial parameters for the embodied intelligent agent are set, and a virtual scene is constructed, including: The task requirements are semantically parsed to obtain the task feature vector; Based on the task feature vector, a task model is obtained; Based on the task model, the initial parameters of the embodied agent are set and a virtual scene is constructed.

3. The embodied intelligent agent training method according to claim 1, characterized in that, The goal is to minimize the difference in feature distribution between virtual and real scenes. Based on the real data, the virtual data in the virtual-real dataset is corrected to obtain a training dataset, including: A virtual-real feature mapping model is used to map the real data to a virtual feature space to obtain the mapped virtual data. The virtual-real feature mapping model is trained with the goal of minimizing the maximum mean difference between the mapped virtual data and the real data, resulting in a trained virtual-real feature mapping model. Based on the real data, the virtual data in the virtual-real dataset is corrected using the trained virtual-real feature mapping model to obtain the training dataset.

4. The embodied intelligent agent training method according to claim 3, characterized in that, Based on the real data, the virtual data in the virtual-real dataset is corrected using the trained virtual-real feature mapping model to obtain a training dataset, including: The trained virtual-real feature mapping model is used to map the real data to the virtual feature space to obtain the trained mapped virtual data. Based on the trained mapped virtual data, the virtual data in the virtual and real datasets are corrected to obtain the training dataset.

5. The embodied intelligent agent training method according to claim 3, characterized in that, The real data includes real perception data, real feedback data, and real environmental parameters, and the virtual-real feature mapping model includes a perception mapping model, an interaction mapping model, and an environment mapping model. The process of mapping the real data to a virtual feature space using a virtual-real feature mapping model to obtain the mapped virtual data includes: The real perceptual data is mapped to a virtual feature space using the aforementioned perceptual mapping model to obtain the mapped perceptual data. The interactive mapping model is used to map the real feedback data to the virtual feature space to obtain the mapped feedback data. The real environment parameters are mapped to a virtual feature space using the environment mapping model to obtain the mapped environment parameters; Based on the mapped perception data, the mapped feedback data, and the mapped environmental parameters, the mapped virtual data is obtained.

6. The embodied intelligent agent training method according to claim 1, characterized in that, Also includes: The real and virtual features in the training dataset are weighted and fused to obtain the fused features; Extract the domain-invariant features from the fusion features; Using the domain-invariant features, the embodied agent is trained to obtain first skill parameters; Obtain real-world scene samples; Based on the real-world scene samples, the first skill parameters are adjusted using a model-independent meta-learning algorithm to obtain the second skill parameters.

7. The embodied intelligent agent training method according to claim 1, characterized in that, Also includes: Obtain environmental change data and task change information in the real-world scenario; Based on the environmental change data and the task change information, determine whether the embodied intelligent agent needs to perform skill generalization. When it is determined that the embodied agent needs to perform skill generalization, a skill generalization strategy is generated using the PPO-GAE algorithm based on the skill training dataset, the environmental change data, and the task change information. Based on the skill generalization strategy, the skill execution parameters and operation trajectory of the embodied intelligent agent are adjusted.

8. The embodied intelligent agent training method according to claim 7, characterized in that, Also includes: Acquire execution data, which is obtained by the embodied intelligent agent performing generalized skills in a new real-world scenario; Based on the execution data and the preset generalization evaluation index, the parameters of the skill generalization strategy are adjusted to obtain a new skill generalization strategy. Based on the new skill generalization strategy, the skill execution parameters and operation trajectory of the embodied agent are adjusted.

9. The embodied intelligent agent training method according to claim 7, characterized in that, Also includes: Obtain the generalized skill parameters and execution data; The training dataset is updated based on the generalized skill parameters and execution data.

10. A training device for an embodied intelligent agent, characterized in that, include: The acquisition module is used to acquire task requirements; An initialization module is used to set the initial parameters of the embodied intelligent agent and construct a virtual scene based on the task requirements. The acquisition module is used to control the embodied intelligent agent to perform skill operations and acquire virtual and real datasets, which include virtual data collected by the embodied intelligent agent in the virtual scene and real data collected in the real scene; The correction module is used to correct the virtual data in the virtual-real dataset based on the real data, with the goal of minimizing the difference in feature distribution between the virtual and real scenes, to obtain a training dataset. The training module is used to train the embodied agent's skills based on the training dataset.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; A memory connected to the at least one processor; The memory stores instructions executable by the at least one processor, which implements the method of any one of claims 1 to 9 by executing the instructions stored in the memory.

12. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the method according to any one of claims 1 to 9.

Images