An agent method and system for continuous evolution of robotic job skills

Abstract

The application discloses an agent method and system for continuous evolution of robot operation skills, comprising a skill sharing semantic rendering module, a skill sharing representation distillation module and a skill specific evolution planner. In view of long-term challenges such as 3D scene representation and human task learning faced by robots when adapting to new sequence tasks in actual scenes, the application can continuously learn new 3D scene semantics and robot operation skills from skill sharing and skill specific attributes. The skill sharing semantic rendering module and the skill sharing representation distillation module effectively learn 3D scene semantics by means of neural radiance fields and visual basic models, solve the problem of 3D scene representation neglect, and use the skill specific evolution planner to decouple skill knowledge through latent and low-rank space, and continuously embed new skill specific knowledge. The application also designs a robot operation benchmark, and experiments show that the method significantly improves the performance in robot operation tasks.

Description

Technical Field

[0001] This invention relates to the field of robot learning, and more particularly to an agent method and system for the continuous evolution of robot operation skills. Background Technology

[0002] In recent years, robot learning has attracted increasing attention for integrating machine learning with robot control systems to solve various robotic tasks, such as manipulation, navigation, mapping, and localization. Behavioral cloning agents can be effectively trained and implemented directly on real robots with only a small number of demonstrations. However, most existing language-conditional behavioral cloning methods assume that the skills mastered by the robot remain unchanged in real-world 2D scenes and focus on training policy models on a fixed set of manipulation tasks. These methods are resource-constrained when dealing with new skills and new objects with complex structures. To address this issue, robots should be able to learn new and challenging manipulation skills in a continuous learning manner, much like humans.

[0003] In recent years, robot learning has attracted much attention for integrating machine learning with robot control systems to solve various robot tasks, such as manipulation, navigation, mapping, and localization. Behavioral cloning agents can be effectively trained with a small number of demonstrations and implemented directly on real robots. However, most current language-based behavioral cloning methods have significant limitations. Existing methods use PerAct to encode language targets and RGB-D voxel observations to generate discrete robot actions (Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In CoRL, pages 785–799. PMLR, 2023.), but these methods generally assume that the skills the robot possesses in real-world 2D scenes are constant and focus only on training policy models on a fixed set of maneuvers. When faced with new skills and new objects with complex structures, resource constraints become apparent. For example, when handling maneuvers of irregularly shaped and complex objects, the lack of an effective learning mechanism for the semantics and skills of the new scene makes it difficult for the robot to accurately perform the task. To overcome these limitations, robots should possess the ability to continuously learn and acquire new and challenging operational skills, just like humans, in order to adapt to ever-changing task requirements and complex environments. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, this invention proposes an intelligent agent method and system for the continuous evolution of robot operation skills. By employing a skill-sharing semantic rendering module, a skill-sharing representation distillation module, and a skill-specific evolution planner, it solves the problems of ignoring 3D scene representations and forgetting skills.

[0005] The present invention is achieved by at least one of the following technical solutions.

[0006] An agent-based method for the continuous evolution of robot operational skills includes the following steps:

[0007] S1. Transfer 2D semantic information from the environment to 3D space to obtain 3D scene semantic features; the 2D semantic information includes RGB-D images and language instructions describing the task.

[0008] S2. Input the semantic features of the 3D scene into the skill-sharing representation distillation module and output the aligned voxel representation and keyframe prediction intermediate results.

[0009] S3. Based on the aligned voxel representation and keyframe prediction intermediate results output in step 2, skill-specific knowledge is learned through a skill-specific evolutionary planner, and skill-specific latent features and adapted voxel features are output.

[0010] S4. The voxel representation aligned in step S2, the skill-specific latent features output in step S3, and the adapted voxel features are spliced ​​together, and the spliced ​​features are fused to obtain the comprehensive features.

[0011] S5. Input the comprehensive features into the Q function header, combine the semantic embedding vector to predict the robot's discrete translation, rotation, gripper opening and closing and collision avoidance action parameters, and output the action prediction results;

[0012] S6. Based on the motion prediction results of step S5, control the robot to perform moving, grasping, and placing actions to complete the specified task.

[0013] Further, step S1 includes the following steps:

[0014] S11. The voxel encoder converts the RGB-D image into a 3D voxel representation, inputs the 3D voxel representation into the neural radiation field model, and generates the RGB color and semantic features of the new view based on the 3D voxel features and camera parameters.

[0015] S12. The language instructions describing the task are encoded into semantic embedding vectors through the encoding operation of the language encoder.

[0016] S13. Input the semantic embedding vector and the noisy image into the visual base model to extract visual language semantic features;

[0017] S14. Calculate the difference between the semantic features generated by the neural radiation field model and the semantic features of visual language through a loss function to supervise the accuracy of the semantic features of 3D scene.

[0018] Further, step S2 includes the following steps:

[0019] First, the current multimodal keyframe is input into the student model. The encoder of the student model processes the semantic features of the 3D scene to obtain the 3D voxel representation of the current task. Then, the Q function header is called to generate intermediate keyframe prediction results. The intermediate keyframe prediction results include the probability distribution of discrete translation, rotation, gripper opening and closing, and collision avoidance. The perceptron model of the current task is used as the student model.

[0020] Next, the same current multimodal keyframes are input into the teacher model to extract the soft labels output by the teacher model, where the perceptron model trained on the historical task is used as the teacher model.

[0021] Subsequently, a temperature parameter is introduced to perform probability calibration on the intermediate keyframe prediction results of the soft label and the student model using the softmax function, and the KL divergence of the intermediate keyframe prediction results of the soft label and the student model is calculated as the distillation loss. The softmax function ensures that optimization is only performed on historical task keyframes, and forces the voxel representation of the student model to align with the teacher model in the semantic space.

[0022] Finally, the distillation loss, cross-entropy loss, and neural radiation field model loss are combined to form the total loss to update the student model parameters. After convergence, the aligned voxel representation and the optimized keyframe prediction intermediate results are output.

[0023] Furthermore, step S3 specifically includes:

[0024] First, the current skill encoding is determined through an adaptive language knowledge base. Based on the skill encoding, latent features are extracted in a dynamic skill-specific latent space. In the low-rank space, the voxel representation of step S2 is adapted to skill-specific parameters through a LoRA layer to decouple skill sharing and skill-specific knowledge, and the skill-specific latent features and adapted voxel features are output.

[0025] Furthermore, the current skill code is determined through an adaptive language knowledge base. The specific process is as follows: calculate the cosine similarity between the semantic embedding vector and the vector in the knowledge base; if the maximum similarity is greater than the threshold, update the corresponding vector and determine the skill code; if it is less than the threshold, expand the knowledge base and initialize a new skill code.

[0026] The system implementing the aforementioned agent method for the continuous evolution of robot operational skills includes the NBAgent model, which comprises:

[0027] The skill-sharing semantic rendering module includes a voxel encoder, a neural radiation field model, a visual base model, and a language encoder. The skill-sharing semantic rendering module receives RGB-D scenes and language instructions as input. The voxel encoder converts RGB-D images into 3D voxel representations. The neural radiation field model is used to generate RGB color and semantic features of new views. The diffusion module of the visual base model is used to extract and diffuse semantic information from the 3D voxel representations. The language encoder converts language instructions describing the task into semantic embeddings. Finally, the semantic embeddings are combined with the 3D voxel representations to output the semantic representation of the 3D scene.

[0028] The skill-sharing representation distillation module includes a channel attention block, a spatial attention block, and a perceptual model. The channel attention block includes a channel attention mechanism, and the spatial attention block includes a spatial attention mechanism. The channel attention mechanism is used to extract and align the channel information of keyframes, and the spatial attention mechanism is used to extract and align the spatial information of keyframes. The spatial channel information of keyframes is processed by the perceptual model to output the aligned voxel representation.

[0029] The skill-specific evolution planner includes an adaptive language knowledge base and a low-rank adaptation layer. The skill-specific evolution planner module receives language instructions and voxel representations, and outputs skill-specific latent features and adapted voxel features through the adaptive language knowledge base and the low-rank adaptation layer.

[0030] Cross-attention blocks and self-attention blocks are used to fuse the aligned voxel representation, the skill-specific latent features output in step S3, and the adapted voxel features to obtain comprehensive features; based on the comprehensive features, the robot's discrete translation, rotation, gripper opening and closing, and collision avoidance action parameters are predicted, and the action prediction results are output.

[0031] A memory buffer is used to dynamically store key data samples from historical tasks to provide data support.

[0032] Furthermore, it also includes a continuous robot operation benchmark module, which receives a detailed description of the robot operation task and scenario information as input, processes them through a task sequence and evaluation module, and outputs the robot's success rate and performance indicators in different operation tasks.

[0033] Furthermore, the evaluation module's evaluation process includes setting three types of indicators: semantic alignment, action prediction, and overall task. During task execution, each indicator is calculated, and the calculation results are output in the form of quantitative reports and visual charts to provide feedback and guide the iterative optimization of the NBAgent model and task sequence.

[0034] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0035] The present invention provides a computer-readable storage medium storing a computer program, wherein the method described by the processor is executed by the computer program.

[0036] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0037] 1. Continuous Learning Capability: This intelligent agent can continuously learn new 3D scene semantics and robot operation skills through the skill-sharing semantic rendering module and the skill-sharing representation distillation module. This continuous learning capability enables the robot to face new tasks without retraining the entire model, thereby significantly improving learning efficiency and adaptability.

[0038] 2. Reducing Skill Forgetting: Through the skill-sharing representation distillation module and the skill-specific evolutionary planner, the agent can effectively reduce the problem of skill forgetting. The skill-sharing representation distillation module aligns the voxel representations between the old and new models through knowledge distillation technology, ensuring that the new model does not forget old skills while learning new skills. The skill-specific evolutionary planner module further reduces forgetting by embedding new skill-specific knowledge in the low-rank space through a low-rank adaptation layer.

[0039] 3. Multimodal Data Processing: This agent can process multimodal data, including RGB-D images and language commands. Through a skill-sharing semantic rendering module and a visual base model, the agent can transfer skill-sharing semantics from 2D space to 3D space, enhancing its understanding of 3D scenes. This multimodal data processing capability enables the agent to perform better in complex scene operations. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of an intelligent agent method for the continuous evolution of robot operation skills according to an embodiment of the present invention.

[0041] Figure 2 This is a flowchart of an intelligent agent method for the continuous evolution of robot operation skills according to an embodiment of the present invention.

[0042] Figure 3 This is a visualization of the prediction results of the method and different models in various operational skills according to the embodiments of the present invention. Detailed Implementation

[0043] The method of the present invention will be further explained in detail below with reference to the accompanying drawings and specific embodiments. The continuous behavioral clone agent for robot operation proposed in this invention aims to solve the core challenge of robot skill and knowledge learning in 3D scenes, enabling continuous learning of new operational skills and avoiding catastrophic forgetting.

[0044] This embodiment provides an intelligent agent system for the continuous evolution of robot operation skills, including the NBAgent model. The NBAgent model includes a skill-sharing semantic rendering module (SSR), a skill-sharing representation distillation module (SRD), a skill-specific evolution planner (SEP), and a continuous robot operation baseline module.

[0045] like Figure 1 As shown, when a new skill learning task arrives, the voxel encoder first encodes the input RGB-D image into a 3D voxel representation, while the language encoder converts the language instructions into language feature representations. Subsequently, the patches of the 3D voxel representation and the language features are processed sequentially by the perception model and the Q-function head, outputting the corresponding action sequence to achieve the robot's operational skills. During this process, the agent needs to simultaneously learn both skill-shared knowledge and skill-specific knowledge.

[0046] At the skill-sharing level: The skill-sharing semantic rendering module renders 3D voxels based on the NeRF model, generating rendered images and their semantic information. Subsequently, a visual-linguistic model is used to extract visual-linguistic semantic features, and a loss function is used to constrain the consistency between rendered semantics and visual-linguistic semantics, thereby achieving skill knowledge sharing between different tasks.

[0047] Skill-specific layer: A specific skill planner is embedded into the spatial attention block of the perception model. It first filters out skill-specific elements related to the current task in the latent space and low-rank space, and then embeds these elements into the spatial attention network layer to process the input 3D voxel patches and language features more specifically, thereby achieving skill-specific knowledge learning between tasks.

[0048] The Skill-Sharing Semantic Rendering (SSR) module primarily transfers skill-sharing semantics from 2D space to 3D space to address the issue of neglecting 3D scene representation. This module introduces a Neural Radiation Field (NeRF) model and a Visual Foundation Model to supervise the semantic transfer process. These two models work collaboratively. The NeRF model is responsible for generating RGB colors and semantic features of the new view based on 3D voxel features and camera parameters, constructing an implicit representation of the 3D scene. The Visual Foundation Model, leveraging its pre-trained visual-linguistic semantic extraction capabilities, provides semantic supervision for the rendering results of the NeRF model. Together, they ensure the accuracy of the 2D-to-3D semantic transfer.

[0049] As one embodiment, the skill-sharing semantic rendering module includes a voxel encoder, a neural radiation field model, a visual base model, and a language encoder. The skill-sharing semantic rendering module receives RGB-D scenes and language instructions as input, processes them through the voxel encoder, neural radiation field model, visual base model, and language encoder, and outputs a semantic representation of the 3D scene.

[0050] The voxel encoder converts RGB-D images into 3D voxel representations. A neural radiation field model generates RGB color and semantic features for the new view. To address insufficient semantic capture of 3D scenes, a pre-trained visual foundation model (Stable Diffusion model) is introduced, providing robust visual-linguistic semantic supervision. The diffusion module of the visual foundation model is responsible for semantic extraction and diffusion. The language encoder converts the language instructions describing the task into semantic embedding vectors. In the actual process, the diffusion module receives the semantic embedding vectors output by the language encoder and the noisy image related to the 3D voxel representations generated by the voxel encoder, extracting the ground truth semantics of the visual-linguistic fusion. Then, combined with the 3D semantic features generated by the neural radiation field model. Alignment is used to supervise the accuracy of semantic features in 3D scenes, ultimately achieving a reliable transfer of 2D semantics to 3D space. Semantic information is extracted and diffused from 3D voxel representations. The language encoder converts language instructions describing the task into semantic embedding vectors, and finally combines these vectors with the 3D voxel representations to output a semantic representation of the 3D scene.

[0051] The Skill Shared Representation Distillation (SRD) module includes a channel attention block, a spatial attention block, and a perceptual model. The channel attention block includes a channel attention mechanism, and the spatial attention block includes a spatial attention mechanism. The channel attention mechanism is used to extract channel information of multimodal keyframes and aligned keyframes, and the spatial attention mechanism is used to extract spatial information of multimodal keyframes and aligned keyframes. The spatial information of the keyframes is processed by the perceptual model to output aligned voxel representations.

[0052] Specifically, the channel attention mechanism assigns weights to each channel of the voxel features (such as feature channels corresponding to different semantic categories) and the semantic channels of the language embedding, extracts and aligns channel information that is strongly related to language instructions in the keyframe, and ensures that the voxel features focus on task-related semantics in the channel dimension.

[0053] In the spatial self-attention block, the spatial attention mechanism learns weights for each channel of the voxel feature itself, strengthening the spatially significant channel information in the keyframe, and realizing the extraction and alignment of information within the keyframe channels. This allows voxel features to more accurately reflect the semantic structure of the 3D scene. The purpose of the skill-sharing representation distillation module is to align the voxel representations between the old and new models, reducing skill forgetting. The old model is the perceptron model trained on the historical task, i.e., the... The model trained for each task stores the voxel representations and keyframe prediction capabilities formed during past skill learning and serves as the teacher model in this module. The new model is the perceptron model to be trained for the current task. It is initialized from the old model and needs to retain the skill-sharing knowledge in the old model through distillation while learning the new skills for the current task.

[0054] The Skill-Specific Evolutionary Planner (SEP) is used for skill-specific knowledge learning, continuously embedding new skill-specific knowledge. This knowledge, which is the multimodal information and rules related to task execution required by the robot in the process of learning operational skills, covers 3D scene semantic understanding (such as object position and scene structure), operational logic (such as action sequence and object interaction rules), and the relationship between language instructions and visual observation, and is decoupled into skill-shared knowledge and skill-specific knowledge.

[0055] As one embodiment, the Skill-Specific Evolutionary Planner (SEP) includes an adaptive language knowledge base and a low-rank adaptation layer. The SEP module receives language instructions and aligned voxel representations, processes them through the adaptive language knowledge base and the low-rank adaptation layer, and outputs a skill-specific knowledge representation.

[0056] Skill-specific knowledge refers to unique information specific to a particular skill, such as the specific stacking order of bottles when stacking bottles, the precise rotation angle of the gripper, and the contact angle between the dust shovel and the ground when sweeping dust.

[0057] A memory buffer is used to dynamically store key data samples from historical tasks, providing data support for "inheriting shared knowledge of old skills and calibrating specific knowledge of new skills" during continuous learning. By adjusting the size of the memory buffer, it was verified that increasing the size of the memory buffer can significantly reduce catastrophic forgetting, but it also increases the memory load.

[0058] As one embodiment, the system also includes a continuous robot operation benchmark module, which includes an evaluation module. This module receives a detailed description of the robot's operation tasks and scene information as input. Through processing by the task sequence and evaluation module, it outputs the robot's success rate and other performance indicators in different operation tasks. Regarding the task sequence, each individual task is first coded with skills based on an adaptive language knowledge base. Then, a semantic index is generated based on the task's logical dependencies, and initialization parameters are bound to complete the construction. During execution, tasks are processed sequentially, using a perceptron model, SSR module, and SEP module to collaboratively process multimodal keyframes and other data to complete the actions. After execution, the sequence is dynamically updated based on evaluation feedback. The evaluation module's evaluation process includes setting three types of indicators: semantic alignment, action prediction, and overall task performance. During task execution, the system collects the output of the NBAgent model, real-world feedback, and intermediate process data in real time to calculate each indicator. Specifically, the semantic alignment indicator uses the Euclidean distance algorithm to calculate the error between the 3D scene semantic features and the visual-linguistic semantic ground truth. The result reflects the degree of difference between the semantic features and the ground truth. Simultaneously, a classification accuracy algorithm is used to calculate the semantic feature classification accuracy, statistically analyzing the proportion of correctly identified object categories. The result reflects the accuracy of semantic extraction.

[0059] The motion prediction metric uses the mean squared error algorithm to calculate the error between the predicted motion parameters and the actual parameters in keyframes, and the result measures the accuracy of motion prediction. In addition, by statistically analyzing the proportion of prediction results that match the actual state of the next keyframe, the accuracy of the Q-function head in predicting the state of the next keyframe is obtained, which reflects the motion prediction capability.

[0060] The overall task metrics utilize statistical counting algorithms to calculate the success rate of individual tasks, which is the ratio of the number of successfully executed tasks to the total number of attempts. The total completion time of the task sequence is calculated using timestamp differences, yielding the total time from the start of the first task to the end of the last. These two results respectively reflect the efficiency and stability of task execution. Finally, the results are output in the form of quantitative reports and visual charts to provide feedback and guide the iterative optimization of the NBAgent model and task sequences.

[0061] like Figure 1 and Figure 2 This embodiment of an agent method for the continuous evolution of robot operation skills includes the following steps:

[0062] S1. Obtain 2D semantic information from the environment, including RGB-D images and language instructions describing the task, and transfer it to 3D space to obtain 3D scene semantic features, including the following steps:

[0063] S11. The voxel encoder converts the RGB-D image into a 3D voxel representation, inputs the 3D voxel representation into the Neural Radiation Field Model (NeRF), and generates the RGB color and semantic features of the new view based on the 3D voxel features and camera parameters.

[0064] The neural radiation field model is a neural rendering function constructed using a multilayer perceptron. :

[0065] ;

[0066] in For 3D spatial points, As the perspective direction, It is a 3D voxel feature. The density of points in 3D space. RGB colors These are semantic features.

[0067] During the process of synthesizing a new view, through the loss function To supervise the reconstruction process, when the Skill-Sharing Semantic Rendering (SSR) module transfers 2D semantic information (RGB-D images, language instructions) to 3D space and generates the RGB colors and semantic features of the new view using the Neural Radiation Field (NeRF) model, a loss function is needed to constrain the accuracy of the new view reconstruction. This ensures that the semantic features of the 3D scene not only conform to the real-world environment but also inherit the skill-sharing semantics of the historical task. It avoids semantic distortion in subsequent skill learning or the forgetting of old skills due to deviations in the new view reconstruction. The formula is as follows:

[0068]

[0069] in, It is the RGB color reconstruction loss, which measures the difference between the RGB colors of the synthesized image and the real image; It is a semantic reconstruction loss, ensuring that the rendered semantic features are consistent with the real semantics; and It is a weighting coefficient used to balance the importance of the two types of losses.

[0070] S12. Language instructions for describing the task The semantic embedding vector is obtained through the encoding operation of the language encoder.

[0071] Real RGB image VAE encoder input to pre-trained visual base model VAE encoder Real RGB image After being transferred to the latent semantic space, a noisy image is generated using the following formula:

[0072] ;

[0073] in For noisy images, For noise scheduling parameters, It is Gaussian noise. For camera rays, Represents a true RGB image. It is the VAE (Variational Autoencoder) encoder built into pre-trained visual base models (such as the Stable Diffusion model).

[0074] S13. Supervised extraction of visual language semantics using a pre-trained visual base model, specifically:

[0075] semantic embedding vectors with noisy images A common input visual base model is used to extract visual language semantic features. .

[0076] S14. Calculate the semantic features generated by the NeRF neural radiation field model. With visual language semantic features Based on the differences, construct the loss function:

[0077]

[0078] in Indicates from 3D voxel features and camera rays Starting from the neural radiation field (NeRF) rendering, the 3D scene semantic features are the expression of scene semantics in 3D space from a specific perspective; This indicates that the pre-trained visual base model (such as Stable Diffusion) receives ray from the camera. The corresponding 2D perspective, combined with embedded language commands The extracted visual-linguistic semantic ground truth is used as a reference standard to supervise the accuracy of 3D semantic features; The square of the L2 norm is used to quantify the Euclidean distance between two semantic feature vectors. The specific calculation is as follows: and The degree of difference between them.

[0079] This represents the set of camera rays, which is obtained by iterating over and summing all possible camera ray directions. The semantic feature alignment loss function (NeRF) monitors the accuracy of 3D scene semantic features by calculating the difference between the 3D scene semantic features and the ground truth visual-linguistic semantics, ensuring that the semantic features generated from 3D voxels match the visual-linguistic semantics of the real scene. Backpropagation optimizes the NeRF parameters to achieve accurate 2D to 3D semantic transfer.

[0080] This invention transfers shared skill semantics from 2D space to 3D space using a Skill-Shared Semantic Rendering (SSR) module, solving the problem of neglecting 3D scene representation. Skill-shared semantics refers to the common basic semantic information between different operational skills and the basic semantic information related to 3D scene understanding. Specifically, it includes the visual features of common objects (such as the shape, color, and structure of drawers and cups, which are consistent across skills), scene spatial relationships (such as the common spatial logic of objects being placed on a table or drawers being located under cabinets), and the basic association between language instructions and the visual scene (such as the semantics of openable objects corresponding to the "open" action and the semantics of small, holdable objects corresponding to the "grab" action). For example, in the skills of opening a drawer and retrieving items from a drawer, the 3D outline of the drawer and the relative position of the drawer to the cabinet are examples of skill-shared semantics. In specific implementation, the loss function of the SSR module can also be used. Optimization training is performed to enable the model to accurately transfer 2D semantic information into the 3D scene, outputting optimized 3D scene semantic features. This feature transfer does not end the process; rather, it serves as the core input data for the next step, providing a 3D semantic foundation for subsequent voxel representation alignment.

[0081] S2. Input the 3D scene semantic features into the Skill Shared Representation Distillation (SRD) module and output the aligned voxel representation and keyframe prediction intermediate results. This includes the following steps:

[0082] First, the current multimodal keyframe Input student model , Indicates the first The first task The robot motion parameters corresponding to each multimodal keyframe, such as the quantified values ​​of the robot's translation, rotation, gripper opening and closing actions; Indicates the first The first task The RGB-D image data corresponding to each multimodal keyframe contains the color (RGB) and depth (D) information of the scene, and serves as the visual input for the robot to perceive the environment. Indicates the first The language instruction embedding vector corresponding to each task is a vector representation obtained by converting the natural language instruction describing the task (such as putting the cup on the table) through a language encoder, used to convey the semantic information of the task. The encoder of the student model processes the semantic features of the 3D scene to obtain the 3D voxel representation of the current task. and call the Q function header. Generate intermediate keyframe prediction results, which include discrete translations. Rotation , handle opening and closing and collision avoidance The probability distribution is obtained; at the same time, the 3D scene semantic features output from step S1 and the semantic embedding vector are input into the perceptron model of the current task to obtain the voxel representation and keyframe prediction output.

[0083] The multimodal keyframes include various types of information such as RGB-D images and language commands.

[0084] Next, the same current multimodal keyframes are input into the teacher model. Extract soft labels from the teacher model output. The perceptron model trained on the historical task was among them. As a teacher model, the perceptron model for the current task As a student model;

[0085] Subsequently, a temperature parameter was introduced to perform probability calibration on the intermediate keyframe prediction results of the soft label and the student model using the softmax function, and the KL divergence between the two was calculated as the distillation loss. The softmax function ensures optimization only on keyframes of historical tasks, forcing the student model's voxel representation to align with the teacher model in semantic space.

[0086] Finally, the distillation loss With cross-entropy loss SSR module loss The total loss is used to update the student model parameters, and after convergence, the aligned voxel representation and the optimized keyframe prediction intermediate results are output.

[0087] Perceptron Model It is the core model for realizing continuous learning of robot operation skills, mainly responsible for processing multimodal inputs, extracting 3D voxel representations and predicting keyframe actions, and introducing cross-entropy loss. The formula for supervising an agent to learn skills and knowledge is:

[0088] ;

[0089] in, It is a label for the actual state of the next keyframe. It is the probability distribution predicted by the perceptron model. It refers to the number of samples.

[0090] However, since old skill data—that is, multimodal data related to the skills already mastered, accumulated by the robot during learning from previous tasks (i.e., tasks m-1 and earlier)—can lead to semantic drift in voxel representations, a knowledge distillation method is used, employing the perceptron model from the previous task. As a teacher model, soft labels are extracted, and Kullback-Leibler divergence is applied. To align the outputs of the old and new models, the formula is as follows:

[0091] ;

[0092] in, It is the sample size. It is the probability distribution output by the teacher model. This is the probability distribution output by the student model. The final distillation loss. for:

[0093] ;

[0094] in, It is a coefficient that controls the weight of knowledge distillation loss.

[0095] During training, the parameters of the new model are continuously adjusted based on the distillation loss to ensure that the new model retains the memory of old skills when learning new skills. Finally, the aligned voxel representation and keyframe prediction intermediate results are output. These results will serve as the key input for the next step, providing an aligned voxel basis for skill-specific knowledge learning.

[0096] S3. Based on the aligned voxel representation output in step 2, skill-specific knowledge is learned through a skill-specific evolutionary planner (SEP), outputting skill-specific latent features and adapted voxel features, specifically including:

[0097] First, the current skill code is determined through an adaptive language knowledge base. The specific process is as follows: calculate the cosine similarity between the semantic embedding vector in step S1 and the vector in the language knowledge base; if the maximum similarity is greater than the threshold, update the corresponding vector and determine the skill code; if it is less than the threshold, expand the language knowledge base and initialize the new skill code.

[0098] Based on skill encoding, latent features are extracted from the logical relationship between the adaptive language knowledge base and dynamic feature mapping in the skill-specific evolutionary planner (SEP). Skill-specific parameters are adapted to the voxel representation aligned in step S3 in the low-rank space through the LoRA layer, decoupling skill sharing and skill-specific knowledge, and outputting skill-specific latent features and adapted voxel features.

[0099] The model is trained based on the weight update formula of the low-rank adaptation layer (LoRA) and the skill-specific evolutionary planner loss to adapt to new skill requirements.

[0100] The weight update formula for the low-rank adaptation layer is:

[0101] ;

[0102] in, It's the learning rate. It's about the loss of skill-specific evolution planners. gradient, These are the weights of the low-rank adaptation layer.

[0103] The skill-specific evolution planner receives language instructions and multimodal inputs, continuously embedding new skill-specific knowledge to achieve the learning of new skills.

[0104] S4. The voxel representations aligned in step S2, the skill-specific latent features output in step S3, and the adapted voxel features are concatenated. The concatenated features are then fused with cross-attention blocks and self-attention blocks to obtain comprehensive features.

[0105] S5. Input the comprehensive features into the Q function header, combine them with the semantic embedding vector from step S1, predict the robot's discrete translation, rotation, gripper opening and closing, and collision avoidance action parameters, and output the action prediction results.

[0106] S6. Based on the motion prediction results of step S5, generate robot operation control instructions to control the robot to perform movement, grasping, and placement actions to complete the specified task, and record the task execution data for subsequent model iteration and optimization.

[0107] Figure 3This visualization showcases the prediction results of the proposed method and different models across various operational skills. It displays the prediction results for different models, including the proposed model Ours, the proposed model Ours-w / oSRD (without the skill-shared representation distillation module), the proposed model Ours-w / oSEP&SRD (without both the skill-specific evolutionary planner and the skill-shared representation distillation module), initial state, experience replay (ER), and fine-tuning. The visualization comparison clearly demonstrates the differences in performance among the models in robot operation tasks, particularly the comparison between the proposed model (Ours) and other baseline methods. Experimental results show that the complete model (Ours) proposed in this invention performs excellently in tasks such as watering plants, stacking wine bottles, hanging picture frames, and picking up cups, accurately predicting and executing the tasks. In contrast, the model without the skill-shared representation distillation module (Ours-w / oSRD) shows a decline in performance on older skills, indicating that the skill-shared representation distillation module plays a crucial role in reducing skill forgetting. The model without the skill-specific evolutionary planner and skill-shared representation distillation module (Ours-w / oSEP&SRD) performed poorly in new skill learning, further validating the importance of the skill-specific evolutionary planner and skill-shared representation distillation module in continuous learning and skill-specific knowledge embedding. By comparing the prediction results of different models on various operational skills, it can be seen that the proposed model Ours has significant advantages in reducing skill forgetting and improving the ability to learn new skills. Especially in complex operational tasks, the proposed model Ours can better understand and execute the task, validating the effectiveness of the skill-shared semantic rendering module, the skill-shared representation distillation module, and the skill-specific evolutionary planner.

[0108] The following experiments are conducted based on the method of this invention. The experimental method in this embodiment is as described above, and the specific implementation steps will not be elaborated in detail. The results will only be presented to demonstrate the effectiveness of the experiments. An agent method and system model for the continuous evolution of robot operational skills are evaluated through several empirical comparisons of two benchmark datasets.

[0109] To verify the effectiveness of the method, this embodiment designed a robot operation benchmark including two scenarios: a kitchen and a living room. Each scenario contains multiple operation skills, with each skill having a training set of 20 episodes and a test set of 25 episodes. This benchmark allows for a comprehensive evaluation of the robot's operational capabilities in different scenarios. Experimental results show that using the method of this invention, the average success rate of robot operations in the kitchen and living room scenarios is improved by 2.5%–41.4% and 0.4%–23.8%, respectively, verifying the effectiveness of the method in robot operation tasks. Furthermore, ablation experiments further verified the impact of skill-sharing attributes and skill-specific attributes on model performance. Specifically, the skill-sharing representation distillation module and the skill-specific evolutionary planner improve the model's learning ability for old and new skills, respectively.

[0110] By adjusting the size of the memory buffer, it was verified that increasing the memory buffer size can significantly reduce catastrophic forgetting, but it also increases the memory load. Specific experimental results show that when the memory buffer size is 6, the forgetting rate of the model is significantly reduced, but the memory load also increases accordingly.

[0111] Tables 1, 2, and 3 respectively show the success rates of each model on the kitchen and living room datasets, the comparative results of the invention method on the kitchen dataset under skill range 5-1 settings, and the comparative results of the invention method on the living room dataset in terms of robot operation success rate (%) when setting different sizes of memory buffer M.

[0112] Table 1. Success rates of each model on the kitchen and living room datasets.

[0113]

[0114] In the table, 5-1, 6-3, etc., represent the skill range learned at different stages. For example, 5-1 (6 steps) indicates that the agent is first trained on a basic skill learning task, which includes five operational skills from all data skills (a total of 10 skills). 1 indicates that each subsequent incremental learning session involves learning one skill, requiring a total of 6 steps including basic learning. ALL represents the average success rate of all skill learning tasks; Avg. For. represents the average forgetting rate. 1-5 6-10 ALL Avg. For. indicates a total of 10 skills, 1-5 indicates the first five are basic tasks, and 6-10 are incremental learning tasks, used to measure the success rates of basic and incremental tasks.

[0115] In Table 2, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 represent different skill learning tasks (the specific task needs to be combined with the research background, such as different tasks in a kitchen setting, such as chopping vegetables, plating, and heating); ALL represents the average success rate of all skill learning tasks; Imp. represents performance improvement. 1.5 indicates that compared to a certain benchmark (usually the default or base model), the corresponding model's performance has improved by 1.5 units (the units here, combined with the average success rate, can be understood as an improvement in the success rate by 1.5 percentage points).

[0116] Table 2 Comparison results of the kitchen dataset under the 5-1 setting of the method of the present invention

[0117]

[0118] Table 3 compares the success rate (%) of the living room dataset using the method of the present invention with different memory buffer sizes M.

[0119]

[0120] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.

Claims

1. An agent method for the continuous evolution of robot operational skills, characterized in that, The method comprises the following steps: S1, transferring 2D semantic information in the environment to a 3D space to obtain 3D scene semantic features; The 2D semantic information comprises an RGB-D image and a language instruction describing a task; S2, inputting the 3D scene semantic features into a skill sharing representation distillation module to output aligned voxel representations and key frame prediction intermediate results; comprising the following steps: inputting a current multi-modal key frame into a student model, processing the 3D scene semantic features through an encoder of the student model to obtain a 3D voxel representation of the current task, and calling a Q function head to generate a key frame prediction intermediate result, wherein the key frame prediction intermediate result comprises probability distributions of discrete translation, rotation, gripper opening and collision avoidance; wherein the perceiver model of the current task is used as the student model; inputting the same current multi-modal key frame into a teacher model to extract soft labels output by the teacher model, wherein the perceiver model trained for a historical task is used as the teacher model; introducing a temperature parameter to perform probability calibration on the soft labels and the key frame prediction intermediate result of the student model through a softmax function, and calculating the KL divergence of the soft labels and the key frame prediction intermediate result of the student model as a distillation loss, wherein the softmax function ensures that only the key frame of the historical task is optimized, and forces the voxel representation of the student model to align with the teacher model in the semantic space; combining the distillation loss with a cross-entropy loss and a loss of a neural radiance field model to form a total loss to update parameters of the student model, and outputting aligned voxel representations and optimized key frame prediction intermediate results after convergence; S3, based on the aligned voxel representations and the key frame prediction intermediate results output in step 2, performing skill-specific knowledge learning through a skill-specific evolutionary planner to output skill-specific latent features and adapted voxel features; specifically comprising: firstly determining a current skill code through an adaptive language knowledge base, extracting latent features in a dynamic skill-specific latent space based on the skill code, and performing skill-specific parameter adaptation on the voxel representations of step S2 through a LoRA layer in a low-rank space to decouple skill sharing and skill-specific knowledge, and outputting skill-specific latent features and adapted voxel features; S4, splicing the aligned voxel representations of step S2, the skill-specific latent features and the adapted voxel features output in step S3, fusing the spliced features to obtain comprehensive features; S5, inputting the comprehensive features into a Q function head to predict discrete translation, rotation, gripper opening and collision avoidance action parameters of the robot in combination with semantic embedding vectors, and outputting action prediction results; S6, controlling the robot to perform moving, grasping and placing actions according to the action prediction results of step S5 to complete a specified task.

2. The agent method for continuous evolution of robotic task skills according to claim 1, wherein, Step S1 comprises the following steps: S11, a voxel encoder converts an RGB-D image into a 3D voxel representation, and inputs the 3D voxel representation into a neural radiance field model to generate RGB colors and semantic features of a new view based on 3D voxel features and camera parameters; S12, a language instruction describing a task is processed by an encoding operation of a language encoder to obtain a semantic embedding vector; S13, input the semantic embedding vector and the noisy image into the visual base model to extract visual language semantic features; S14, calculate the difference between the semantic features generated by the neural radiance field model and the visual language semantic features through the loss function to supervise the accuracy of the 3D scene semantic features.

3. The agent method for continuous evolution of robotic task skills according to claim 1, wherein, The current skill code is determined through the adaptive language knowledge base, and the specific process is as follows: the cosine similarity between the semantic embedding vector and the vector in the knowledge base is calculated; if the maximum similarity is greater than a threshold, the corresponding vector is updated and the skill code is determined; If it is less than the threshold, the knowledge base is expanded and a new skill code is initialized.

4. An agent system for continuous evolution of robotic job skills for implementing the agent method of claim 1, characterized by The NBAgent model comprises: The skill sharing semantic rendering module comprises a voxel encoder, a neural radiance field model, a visual base model and a language encoder, and receives an RGB-D scene and a language instruction as input, wherein the voxel encoder converts the RGB-D image into a 3D voxel representation, the neural radiance field model is used to generate the RGB color and semantic features of a new view, the diffusion module of the visual base model is used to extract and diffuse semantic information from the 3D voxel representation, and the language encoder converts the language instruction describing the task into a semantic embedding vector; finally, the semantic embedding vector is combined with the 3D voxel representation to output the semantic representation of the 3D scene; The skill sharing representation distillation module comprises a channel attention block, a spatial attention block and a perception model, the channel attention block comprises a channel attention mechanism, and the spatial attention block comprises a spatial attention mechanism; the channel attention mechanism is used to extract and align the channel information of the key frame, and the spatial attention mechanism is used to extract and align the spatial information of the key frame; the spatial channel information of the key frame is processed through the perception model to output the aligned voxel representation; The skill-specific evolutionary planner comprises an adaptive language knowledge base and a low-rank adaptive layer; the skill-specific evolutionary planner module receives a language instruction and a voxel representation, and outputs skill-specific latent features and adapted voxel features through the adaptive language knowledge base and the low-rank adaptive layer; The cross-attention block and the self-attention block are used to fuse the features after the aligned voxel representation and the skill-specific latent features and the adapted voxel features output in step S3 are spliced to obtain comprehensive features; based on the comprehensive features, the discrete translation, rotation, gripper opening and closing and collision avoidance action parameters of the robot are predicted to output the action prediction result; The memory buffer is used to dynamically store the key data samples of historical tasks to provide data support.

5. The agent system for continuous evolution of robotic task skills according to claim 4, wherein, The continuous robot operation benchmark module also comprises a task sequence and an evaluation module, which receives the specific description and scene information of the robot operation task as input, processes the specific description and scene information of the robot operation task through the task sequence and the evaluation module, and outputs the success rate and performance index of the robot in different operation tasks.

6. The agent system for continuous evolution of robotic task skills according to claim 5, wherein, The evaluation process of the evaluation module comprises setting three types of indexes of semantic alignment, action prediction and overall task, calculating the indexes in task execution, and outputting the calculation results in the form of quantitative reports and visual charts to feedback and guide the iterative optimization of the NBAgent model and the task sequence.

7. A computer device comprising a memory and a processor, the memory and the processor being electrically connected, the memory storing a computer program, characterized in that: The computer program, when executed by the processor, causes the processor to implement the agent method of any one of claims 1-4.

8. A computer readable storage medium storing a computer program, characterized in that: The computer program, when executed by the processor, causes the processor to implement the agent method of any one of claims 1-4.

Images