Intelligent agent training method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide an agent training method and device, electronic equipment and storage medium, relating to the technical field of artificial intelligence, the agent comprising a shared feature extraction layer, a policy head and a value head, the policy head and the value head being connected with the shared feature extraction layer, the method comprising: inputting first perception information in sample data into the shared feature extraction layer for feature extraction to obtain first information features; inputting the first information features into the policy head to predict a first action trajectory, inputting the first information features and the first action trajectory into the value head to evaluate a first excellence degree of the first action trajectory, inputting the first information features and a sample action trajectory in the sample data into the value head to evaluate a second excellence degree of the sample action trajectory; determining a loss based on the first excellence degree, the second excellence degree and a third excellence degree; and updating parameters of the shared feature extraction layer, the policy head and the value head according to the loss. The required storage resources and computing resources for agent training can be reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent agent training method, apparatus, electronic device, and storage medium. Background Technology

[0002] Reinforcement learning (RL) is a machine learning paradigm that aims to enable an agent (algorithm model) to autonomously learn the optimal decision-making strategy through trial and error. Currently, reinforcement learning can be used to train an agent to obtain a robot action prediction model, which then outputs actions for the robot to execute, thereby completing various tasks.

[0003] In related technologies, the Actor-Critic algorithm (a commonly used training method in reinforcement learning) is typically used to train the agent. The agent consists of a policy network and a value network. During training, the network parameters of the two networks are collaboratively optimized to improve the agent's learning ability and training efficiency.

[0004] However, policy networks and value networks contain a large number of network parameters, resulting in a large amount of storage and computing resources required during agent training. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, electronic device, and storage medium for training intelligent agents, thereby reducing the storage and computing resources required during the training process. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide an agent training method, wherein the agent includes a shared feature extraction layer, a policy head, and a value head, and both the policy head and the value head are connected to the shared feature extraction layer. The method includes:

[0007] The first perceptual information in the sample data is input into the shared feature extraction layer for feature extraction to obtain the first information feature;

[0008] The first information feature is input into the strategy head to predict the first action trajectory to be executed when the first robot perceives the first perception information. The first information feature and the first action trajectory are input into the value head to evaluate the first excellence level representation information of the first action trajectory. The first information feature and the sample action trajectory in the sample data are input into the value head to evaluate the second excellence level representation information of the sample action trajectory.

[0009] Based on the first level of excellence representation information, the strategy loss is determined, and based on the difference between the second and third level of excellence representation information, the value loss is determined, wherein the third level of excellence representation information is determined based on the sample excellence representation information of the sample action trajectory in the sample data and the second perception information.

[0010] Based on the policy loss, update the network parameters of the shared feature extraction layer and the policy head; based on the value loss, update the network parameters of the shared feature extraction layer and the value head to obtain the robot action prediction model.

[0011] In a second aspect, embodiments of the present invention provide an agent training apparatus, wherein the agent includes a shared feature extraction layer, a policy head, and a value head, and both the policy head and the value head are connected to the shared feature extraction layer. The apparatus includes:

[0012] The feature extraction module is used to input the first perceptual information in the sample data into the shared feature extraction layer for feature extraction to obtain the first information features;

[0013] The information prediction module is used to input the first information feature into the strategy head to predict the first action trajectory to be executed by the first robot when it perceives the first perception information, input the first information feature and the first action trajectory into the value head to evaluate the first degree of excellence representation information of the first action trajectory, and input the first information feature and the sample action trajectory in the sample data into the value head to evaluate the second degree of excellence representation information of the sample action trajectory.

[0014] The loss determination module is used to determine the strategy loss based on the first level of excellence representation information, and to determine the value loss based on the difference between the second level of excellence representation information and the third level of excellence representation information, wherein the third level of excellence representation information is determined based on the sample excellence representation information of the sample action trajectory in the sample data and the second perception information.

[0015] The parameter update module is used to update the network parameters of the shared feature extraction layer and the policy head according to the policy loss, and to update the network parameters of the shared feature extraction layer and the value head according to the value loss, so as to obtain the robot action prediction model.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, a communication bus, and an image acquisition device, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0017] Memory, used to store computer programs;

[0018] The processor, when executing a program stored in memory, implements the method steps of the first aspect.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method steps of the first aspect.

[0020] Fifthly, embodiments of the present invention provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method steps of the first aspect.

[0021] As can be seen from the above, in the agent training scheme provided by this embodiment of the invention, the shared feature extraction layer in the agent is connected to both the policy head and the value head. After the robot's perception information is processed by the shared feature extraction layer, information features are obtained. These information features are then input into the policy head and the value head, respectively, and the policy head and the value head output action trajectory and excellence level representation information, respectively. It is evident that the policy head and the value head share the same feature extraction layer, achieving feature extraction layer reuse. Thus, compared to the traditional training process, by designing a shared feature extraction layer, it is no longer necessary to design independent feature extraction layers for the policy module and the value module in the agent, simplifying the agent architecture during training, reducing the number of agent parameters, and consequently reducing the storage and computing resources required during agent training.

[0022] In addition, both policy loss and value loss are used to adjust the network parameters of the shared feature extraction layer, so that the shared feature extraction layer can learn general feature representations that are beneficial to trajectory prediction and value assessment at the same time. This improves the rationality of the agent training process and helps to improve the performance of the robot action prediction model obtained in the final training.

[0023] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the above advantages at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0025] Figure 1 This is a flowchart illustrating an agent training method provided in an embodiment of the present invention.

[0026] Figure 2 A flowchart illustrating a method for obtaining sample data according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an interactive data acquisition and agent training process provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an intelligent agent training device provided in an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0031] First, the application scenarios of the solutions provided in the embodiments of the present invention will be introduced.

[0032] The application scenario of the solution provided in this embodiment of the invention is: a scenario in which a robot action prediction model is obtained by training an intelligent agent using reinforcement learning.

[0033] Reinforcement learning is a machine learning technique that uses reward signals to guide policy optimization through the interaction of an agent with its environment, and its application in the field of robotics is becoming increasingly widespread. During training, an agent can autonomously learn the action decision-making logic for tasks such as robotic arm grasping, mobile navigation, and industrial assembly, without the need for manually writing a large amount of fixed control code. This effectively improves the robot's adaptability to complex and dynamic scenarios and reduces task adaptation costs.

[0034] In related technologies, the Actor-Critic algorithm is commonly used to train agents. The agent consists of a policy network and a value network. During training, the network parameters of the two networks are collaboratively optimized to improve the agent's learning ability and training efficiency. However, the large number of network parameters in the policy and value networks leads to significant storage and computational resources required during agent training.

[0035] In view of the above, embodiments of the present invention provide an agent training scheme to reduce the storage and computing resources required during agent training.

[0036] In the agent training scheme provided by this invention, the agent includes a shared feature extraction layer, a policy head, and a value head. Both the policy head and the value head are connected to the shared feature extraction layer. The overall training process is as follows:

[0037] First, the first perception information from the sample data is input into the shared feature extraction layer for feature extraction, resulting in the first information feature. Then, the first information feature is input into the strategy head to predict the first action trajectory to be executed by the robot when it perceives the first perception information. The first information feature and the first action trajectory are input into the value head to evaluate the first excellence representation information of the first action trajectory. The first information feature and the sample action trajectory from the sample data are input into the value head to evaluate the second excellence representation information of the sample action trajectory. Second, based on the first excellence representation information, the strategy loss is determined, and based on the difference between the second and third excellence representation information, the value loss is determined. The third excellence representation information is determined based on the sample excellence representation information of the sample action trajectory from the sample data and the second perception information. Finally, based on the strategy loss, the network parameters of the shared feature extraction layer and the strategy head are updated; based on the value loss, the network parameters of the shared feature extraction layer and the value head are also updated, resulting in the robot action prediction model.

[0038] The sample data used to train the intelligent agent is based on the interaction data generated by the second robot interacting with its working environment, as detailed in the following explanation.

[0039] It should be noted that the motion parameters of the operating components and the sensor layout of the second robot are compared with those of the first robot. The operating components include a robotic arm and an end effector, and their motion parameters include the type, size, installation position, and range of motion of the robotic arm and end effector. The sensor layout refers to the installation position of the sensors used to collect interactive data on the robot.

[0040] In short, the second robot has the same environmental perception range and object manipulation capabilities as the first robot. Therefore, the model trained using the interaction data from the second robot can also be applied to predicting the actions of the first robot.

[0041] Of course, the second robot can be the same model as the first robot; preferably, the second robot and the first robot can be the same robot.

[0042] The intelligent agent training scheme provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] See Figure 1 The above is a flowchart illustrating an agent training method provided in an embodiment of the present invention. The method includes the following steps S101 to S104.

[0044] Step S101: Input the first perceptual information in the sample data into the shared feature extraction layer in the agent for feature extraction to obtain the first information feature.

[0045] The aforementioned sample data is obtained from interaction data pre-collected in the sampling experience replay buffer; the specific acquisition method is described later. The sample data includes at least: first perceptual information, sample motion trajectory, sample motion trajectory quality representation information, and second perceptual information. Specifically, the first perceptual information is the perceptual information collected before the second robot moves based on the sample motion trajectory; the second perceptual information is the perceptual information collected after the second robot moves based on the sample motion trajectory; the sample quality representation information can also be called the reward information of the sample motion trajectory, obtained based on the set task objective. This embodiment of the invention does not limit the method of obtaining the reward information.

[0046] The aforementioned perception information, also known as state information, may include the second robot's body perception information and environmental perception information (external perception information). The body perception information may include, but is not limited to, the second robot's joint angles q, joint velocities q', and the pose of the second robot's end-effector. The environmental perception information may include, but is not limited to, data such as images and point clouds of the second robot's working environment. Preferably, the environmental perception information may include an RGB depth (RGB-D) image of the second robot's working environment.

[0047] Since agent training is an iterative process, for the t-th sample data used in the training process, the first perceptual information (current state information), sample action trajectory, sample excellence representation information, and second perceptual information (next state information) included in the sample data can be divided into: s t v t r t、 s t+1 .

[0048] The shared feature extraction layer, also known as the shared backbone layer, is used to extract features from the first perceptual information of the second robot in the sample data. This process maps the multimodal perceptual information to first information features, which can be referred to as the high-dimensional latent feature vector h. t .

[0049] Step S102: Input the first information feature into the strategy head connected to the shared feature extraction layer to predict the first action trajectory to be executed when the first robot perceives the first perception information; input the first information feature and the first action trajectory into the value head connected to the shared feature extraction layer to evaluate the first excellence representation information of the first action trajectory; input the first information feature and the sample action trajectory in the sample data into the value head to evaluate the second excellence representation information of the sample action trajectory.

[0050] In the solution provided by this embodiment of the invention, the strategy head and the value head are connected to the same shared feature extraction layer. Specifically, the strategy head and the value head are lightweight neural network layers connected to the end of the shared backbone network layer, used to convert the features extracted by the shared backbone network into specific set outputs.

[0051] The strategy head is used for trajectory prediction, obtaining motion trajectories that include the target number of actions. These predicted motion trajectories can also be called action chunks. The target number can be set according to the actual scenario and / or the type of task to be performed by the first robot, such as 10, 15, 20, etc.

[0052] As can be seen, the strategy head is designed to output a future k-step action sequence, rather than a single action. Thus, in subsequent application stages, the trained robot action prediction model can output an action block after completing one inference, thereby controlling the first robot to execute the action sequence within that action block sequentially. In high-frequency robot control scenarios, this reduces the model's inference frequency requirement. For example, assuming the inference frequency required for the model to output a single action is f, the solution provided in this application can reduce the inference frequency by f / k. Furthermore, it effectively utilizes the temporal correlation within the action sequence, improving the smoothness of the robot's motion trajectory.

[0053] In one possible implementation, the idea of ​​a diffusion model and velocity field can be used to generate the predicted action trajectory: the strategy head first generates a noisy initial random trajectory, and calculates the corresponding velocity field for each action in the initial trajectory; based on the velocity field, the first action of the initial trajectory is corrected to obtain a new action; the velocity field is updated according to the new action, and the correction operation is repeated for the next action of the trajectory; the correction steps are iteratively executed until the trajectory meets the requirements, and finally the predicted action trajectory is obtained.

[0054] The value head is used to evaluate the value of a motion trajectory, obtaining information representing the quality of the motion trajectory. This quality representation information can also be referred to as the value of the motion trajectory.

[0055] Step S103: Determine the policy loss based on the first level of excellence representation information, and determine the value loss based on the difference between the second and third level of excellence representation information.

[0056] The methods for determining the aforementioned strategy losses and value losses will be introduced below.

[0057] Regarding strategy loss:

[0058] The training objective of the policy head can be to output the action that maximizes value. Therefore, the policy loss can be obtained based on the value information of the predicted trajectory output by the policy head. Specifically:

[0059] h of the shared feature extraction layer output t The data is input into the strategy head, which predicts the first robot's current state s. t The first action trajectory v to be executed pred Current state s t and the first motion trajectory v pred The information is input into the value head, which evaluates the first degree of excellence of the current state and the predicted trajectory. At this point, the first degree of excellence reflects the value of the action trajectory currently predicted by the strategy head. Therefore, the strategy loss can be determined based on the value.

[0060] In this way, based on the policy loss, the network parameters of the shared feature extraction layer and the policy head can be updated with the goal of maximizing the value of the action trajectory, so that the policy head can output action trajectories with higher value in the future.

[0061] Regarding value loss:

[0062] The training objective of the value head can be to make its self-assessed predicted value closer to the true value. Predictive value (i.e., second-best representation information) can be obtained by processing the sample action trajectory v... t and the first information feature h t The input value head yields the true value (i.e., the third-degree excellence representation information), which includes the reward r in the sample data. t It consists of two parts: future value and future value. The future value can be obtained as follows: by extracting the next state information s from the sample data. t+1 The shared feature extraction layer is input, and the policy head predicts the first robot's trajectory in the next state based on the output of the shared feature extraction layer. The output of the shared feature extraction layer and the predicted next trajectory are then input into the value head to obtain the future value output by the value head. At this point, the value loss can be determined based on the difference between the predicted value and the actual value.

[0063] In this way, based on the value loss, the network parameters of the shared feature extraction layer and the value head can be updated with the goal of minimizing the difference between the predicted value and the true value, so that the value head can more accurately evaluate the value of the action trajectory in the future.

[0064] Step S104: Update the network parameters of the shared feature extraction layer and the policy head according to the policy loss, and update the network parameters of the shared feature extraction layer and the value head according to the value loss to obtain the robot action prediction model.

[0065] Specifically, after obtaining the policy loss and value loss, gradient descent can be used to adjust the corresponding network parameters.

[0066] Of course, it is understandable that agent training is often an iterative process. Therefore, the process of adjusting network parameters can be repeated based on multiple sample data until the set training completion conditions are met.

[0067] To improve the continuity and smoothness of the action trajectories predicted by the policy head, one possible implementation could be to introduce an additional action smoothing loss term to adjust the network parameters of the shared feature extraction layer and the policy head.

[0068] Specifically, a first difference can be determined between every two adjacent actions included in the first action trajectory, and a second difference can be determined between the first action included in the first action trajectory and the last action included in the previous action trajectory output by the policy head; then, based on the first difference and the second difference, the action smoothness loss is calculated; and the network parameters of the shared feature extraction layer and the policy head are updated according to the policy loss and the action smoothness loss.

[0069] It is evident that the first difference reflects the difference in the connection between adjacent action trajectories output by the strategy head, while the second difference reflects the difference between adjacent actions within the action trajectory.

[0070] In this way, by adjusting the network parameters of the feature extraction layer and the policy head according to the gradient descent method, the actions between adjacent action trajectories output by the policy head and between adjacent actions in each action trajectory can be more continuous and smooth, reducing action abrupt changes and jitter, and improving the quality of the predicted action trajectory.

[0071] As can be seen from the above, in the agent training scheme provided by this embodiment of the invention, the shared feature extraction layer in the agent is connected to both the policy head and the value head. After the robot's perception information is processed by the shared feature extraction layer, information features are obtained. These information features are then input into the policy head and the value head, respectively, and the policy head and the value head output action trajectory and excellence level representation information, respectively. It is evident that the policy head and the value head share the same feature extraction layer, achieving feature extraction layer reuse. Thus, compared to the traditional training process, by designing a shared feature extraction layer, it is no longer necessary to design independent feature extraction layers for the policy module and the value module in the agent, simplifying the agent architecture during training, reducing the number of agent parameters, and consequently reducing the storage and computing resources required during agent training.

[0072] In addition, both policy loss and value loss are used to adjust the network parameters of the shared feature extraction layer, so that the shared feature extraction layer can learn general feature representations that are beneficial to trajectory prediction and value assessment at the same time. This improves the rationality of the agent training process and helps to improve the performance of the robot action prediction model obtained in the final training.

[0073] The following section introduces one of the methods for obtaining the sample data mentioned earlier.

[0074] See Figure 2 The above is a flowchart illustrating a sample data acquisition method provided by an embodiment of the present invention. The method includes the following steps S201 to S204.

[0075] Step S201: For the interaction data generated by the interaction between the second robot and the working environment, which is pre-recorded in the experience playback pool, determine the target number of consecutive sampling data from the interaction data.

[0076] The interaction data is recorded in units of individual actions. Each interaction data includes: initial perception information before the second robot performs the action, the action of the second robot, the action quality representation information, and updated perception information after the second robot completes the action. The target number is the number of actions included in the action trajectory predicted by the strategy head.

[0077] In this step, you can select any number of consecutively collected sample data from the interactive data.

[0078] Step S202: Obtain the motion trajectory including the action in each sampled data, as the sample motion trajectory in the sample data.

[0079] That is, the individual actions in the sampled data are combined into sample action trajectories in the sample data. The order of the actions included in the sample action trajectory is the same as the order in which the actions in the sampled data were collected.

[0080] Step S203: Based on the action excellence characterization information in each sampled data, obtain the sample excellence characterization information of the sample action trajectory.

[0081] Step S204: Take the initial sensing information in the earliest sampled data as the first sensing information in the sample data, and take the updated sensing information in the latest sampled data as the second sensing information in the sample data.

[0082] That is, the initial perception information in the first sampled data is used as the current state information in the sample data, and the updated perception information in the last sampled data is used as the next state information in the sample data.

[0083] By repeating the above steps multiple times, sufficient sample data can be obtained. Of course, different sampling data can be determined each time the above steps are performed to improve the diversity of the obtained sample data.

[0084] It is evident that when interaction data is recorded at the action level, continuous interaction data can be sampled. By analyzing and combining continuous interaction data, a large amount of diverse sample data recorded at the action trajectory level can be obtained efficiently and quickly.

[0085] In one possible implementation, a sliding window approach can be used to sample from the interactive data, thereby obtaining the richest possible sample data.

[0086] The length of the sliding window can be set to the number of targets of the predicted motion trajectory, the sliding step size of the sliding window is set according to the number of samples to be acquired, and the starting position of the sliding window can be located at the first interactive data.

[0087] Taking a target quantity of 10 and a sliding step size of 1 as an example, the initial interactive data within the sliding window consists of the first 10 interactive data. After the sliding window slides according to the sliding step size, the interactive data within the sliding window consists of the 2nd to 11th interactive data. After the sliding window slides again according to the sliding step size, the interactive data within the sliding window consists of the 3rd to 12th interactive data, and so on. The interactive data within each sliding window can be determined as the sampled data. The sample data is determined based on the sampled data in the aforementioned manner until the sliding window covers the last interactive data.

[0088] The following describes one method for collecting interactive data from the experience replay pool mentioned earlier:

[0089] The current perception information of the second robot is input into the shared feature extraction layer for feature extraction to obtain the second information feature. This second information feature is then input into the strategy head to predict the second action trajectory to be executed by the second robot. The second robot is controlled to sequentially execute each action included in the second action trajectory. Perception information is obtained after each action is completed. For each action executed by the second robot, based on the set task objective, the action's performance level is determined. The perception information before the action is executed is used as the initial perception information, and the perception information after the action is executed is used as the updated perception information. The initial perception information, the action, the action's performance level, and the updated perception information are recorded to obtain interactive data. Here, the initial perception information is the current state information before the action is executed, the updated perception information is the next state information after the action is executed, and the action's performance level is called the action's reward information.

[0090] It can be seen that after the strategy head predicts the motion trajectory, it controls the second robot to execute each action included in the motion trajectory, and records the interaction data for each action with the action as the granularity.

[0091] In this way, after the intelligent agent outputs the predicted action trajectory based on the environmental state information, it controls the robot to execute each action in the predicted action trajectory. For each action, it records the initial perception information, the action itself, the action's quality representation information, and the updated perception information, which can quickly obtain interactive data.

[0092] It should be noted that after the second robot has completed all the actions in the predicted action trajectory, it can return to the step of inputting the perception information of the second robot at the current moment into the shared feature extraction layer for feature extraction, thereby cyclically collecting a large amount of interaction data and storing it in the experience replay pool until the amount of interaction data in the experience replay pool meets the requirements.

[0093] In one possible implementation, during the process of acquiring interactive data, in response to receiving a motion intervention command, the electronic device controls the second robot to pause executing the actions included in the second motion trajectory, and then controls the second robot to execute the action indicated by the motion intervention command. The aforementioned motion intervention command can be a command input by a human expert, or a command generated by the electronic device when it detects that pre-set intervention conditions are met.

[0094] During the data collection process, if a human expert perceives an anomaly (the robot's actions do not meet expectations and / or may pose a danger), they can input an intervention command. At this point, the electronic device completely blocks the agent's output, no longer controlling the robot to perform actions based on the agent's output, but directly controlling the robot to perform actions according to the intervention command (the human expert's action). In this way, the robot's actions completely replicate the human expert's operational instructions, rather than randomly selecting human expert actions or the output actions of the policy head through weighted averaging or random gate functions. In this case, the collected interaction data, including human expert actions, can be called expert demo data.

[0095] It is evident that when human experts attempt to correct a robot's erroneous actions, the electronic device can immediately sever the strategy head's control over the robot, completely handing over control of the robot to the human expert. This reduces the probability of excessive human error correction due to inadequate execution of actions instructed by the human expert, thereby reducing the probability of system oscillation, improving the rationality of agent training, and enhancing the safety of real-machine training.

[0096] In one possible implementation, in response to receiving an intervention termination command, the electronic device returns to the step of inputting the current perception information of the second robot into the shared feature extraction layer in the agent for feature extraction. That is, after receiving a clear "cancel takeover" command, the electronic device can restore the strategy head's control over the second robot's actions. In this way, during the interactive data acquisition process, human experts can intervene fully in the robot's control at any time, and can also restore the agent's control over the robot at any time after intervention, improving the flexibility of collecting interactive data.

[0097] In one possible implementation, during the process of collecting interactive data, for a certain action performed by the second robot, in addition to recording the initial perception information, the action, the action's quality representation information, and the updated perception information, an intervention flag can also be recorded. The intervention flag indicates whether the action was instructed by a human intervention command.

[0098] In other words, the interaction data carries an intervention flag, which indicates whether the action is an action instructed by an action intervention command. The information included in the interaction data can then be represented as: (Current State, Action, Reward, Next_State, Intervention Flag Is_Expert_Flag).

[0099] In this scenario, to obtain sample data including expert demonstration data, a target number of sample data points can be determined from the interaction data generated by the interaction between the second robot and the working environment. These sample data points must be collected in a continuous order and include demonstration data. The demonstration data refers to the interaction data where the intervention markers represent actions indicated by human intervention commands, i.e., expert demonstration data. In this case, the target number of sample data points acquired will include the demonstration data.

[0100] In this way, the obtained sample data can also include expert demonstration data with human intervention. Therefore, in the process of training the agent based on the obtained sample data, the agent can learn the robot control mode in abnormal situations, thereby improving the accuracy and rationality of the agent's predicted motion trajectory.

[0101] Considering that human intervention is rare, the amount of expert demonstration data in the interactive data may be small. Therefore, in one possible implementation, sample data can be sampled from the interactive data stored in the experience replay pool according to a set proportion of expert demonstration data, so that the amount of expert demonstration data included in the obtained sample data meets the requirements.

[0102] In one possible implementation, the agent can be pre-trained using behavior cloning (BC) based on sample data, including expert demonstration data, so that the agent roughly learns the robot's expected motion trajectory. Then, the agent can be further trained using the method described above, thereby improving the training convergence speed and obtaining the robot prediction model.

[0103] Based on the foregoing description, and with reference to the accompanying drawings, a specific data acquisition and agent training process provided by an embodiment of the present invention will be described below.

[0104] See Figure 3 , Figure 3 Solid arrows represent control flow, such as command issuance and signal triggering, emphasizing real-time performance and action execution; dashed arrows represent data flow, indicating the recording, storage, or retrieval of information, which does not directly drive physical actions. The overall process will be described below from two aspects: interactive data collection and agent training.

[0105] Interactive data collection:

[0106] The perception input layer is responsible for multimodal data preprocessing of robot body perception information and environmental perception information to obtain the current state. On the one hand, the current state obtained each time is input into the shared backbone network layer in the agent. The feature h output by the shared backbone network layer is processed by the policy head to obtain the robot action block predicted by the policy head. On the other hand, each state information is stored in the experience playback pool.

[0107] Human-computer interaction devices, such as spatial mice and robotic telescopic devices, are used to input intervention signals when human experts detect anomalies. The intervention detection module at the control arbitration and execution layer determines whether human intervention has occurred based on the intervention signal. If not, it receives the action block output from the strategy head and sets a flag value to represent non-human intervention, thus obtaining flag information. If yes, it receives human action instructions, sets a flag value to represent human intervention, and obtains flag information. This flag information is stored in the experience replay pool. Thus, interaction data generated during human intervention is automatically flagged as expert demonstration data when stored in the experience replay pool.

[0108] It should be noted that only one data flow is active at any given time among the two dashed control flows pointing from the set flag value to the experience replay pool. Specifically, when a human expert takes over, the actions output by the strategy module are ignored, and the flag information of the human intervention actions is stored in the experience replay pool; when the human expert is not in the takeover state, the flag information of the actions output by the strategy module is stored in the experience replay pool.

[0109] Both action blocks output by the strategy head and manual action instructions are cached in a timing execution buffer. The timing execution buffer sequentially sends each action to the robot controller according to the set robot control frequency. The robot controller then executes each sent action sequentially. For each action executed by the robot, an action reward is calculated, and the next state after the action is determined through the aforementioned perception input layer. For each action executed by the robot, the state before the action, the action itself, the reward for the action, and the next state after the robot completes the action are stored in the experience replay pool.

[0110] In this way, for each action performed by the robot, complete interaction data (current state information, action, reward information, next state information, intervention flag Is_Expert_Flag) is recorded in the experience replay pool.

[0111] Agent training:

[0112] After collecting the interaction data, the data storage and training layer can sample the interaction data stored in the experience replay pool. Based on the sampling results, sample data is obtained and used to train the agent offline. During the gradient update phase of offline training, the feature h output from the agent's shared backbone network layer is input to the policy head for action block prediction and then processed by the value head to obtain the trajectory value. The gradients of the policy head and value head are determined based on the obtained values ​​and are used to update the agent's own parameters and the network parameters of the shared backbone network layer. The specific processes for obtaining sample data and offline training have been explained above and will not be repeated here.

[0113] In the sample data sampling stage, expert demonstration data can be assigned a higher sampling weight to ensure that the amount of expert demonstration data included in the obtained sample data meets the requirements. Furthermore, as mentioned earlier, the agent can be pre-trained using expert demonstration data to roughly learn the robot's expected movement trajectory, and then further trained using the sampled data, thereby improving the training convergence speed.

[0114] In one possible implementation, after training a robot motion prediction model, the trained model can be used to control the robot to perform tasks.

[0115] Specifically, the third sensory information of the third robot is first obtained; then, the third sensory information is input into the robot motion prediction model to obtain the predicted motion sequence output by the robot motion prediction model; finally, the third robot is controlled to execute each motion in the predicted motion sequence in sequence so that the third robot performs the operation on the object to be operated.

[0116] The third perception information includes: body perception information and environment perception information. The body perception information includes: the joint pose and end effector pose of the third robot. The environment perception information includes at least: the visual information and / or spatial information of the object to be operated in the working environment of the third robot. The motion parameters of the operating parts and the sensor layout of the third robot are the same as those of the first robot.

[0117] The aforementioned ontological perception information and environmental perception information have been described in detail above, and will not be repeated here.

[0118] This is equivalent to using a trained robot motion prediction model to control the robot to perform tasks, which improves the robot's task execution efficiency compared to staff manually writing motion control instructions.

[0119] The advantages of the solutions provided by the embodiments of the present invention compared with existing solutions will be described in detail below.

[0120] Firstly, the construction of a shared backbone network layer:

[0121] In existing Actor-Critic architectures, the Actor and Critic modules typically each have independent feature extraction layers. When processing high-dimensional inputs (such as images and multimodal sensor data), these feature extraction layers consume significant computational resources, leading to a substantial increase in GPU memory usage and making it difficult to meet the stringent real-time constraints of training on real machines. Furthermore, during training, the two independent feature extraction layers update their parameters independently, which may result in ineffective alignment between their learning of environmental representations.

[0122] In the solution provided by this invention, the Actor head and Critic are connected to the same shared backbone network layer, which is equivalent to the Actor module and the Critic module sharing the same shared backbone network layer. By designing a shared feature extraction layer, it is no longer necessary to design separate feature extraction layers for the Actor module and the Critic module in the agent, simplifying the agent architecture during training, reducing the number of parameters in the agent, and thus reducing the storage and computing resources required during agent training. Experiments have shown that the solution provided by this invention can reduce the number of parameters and GPU memory usage during agent training by nearly 50%, making online training of agents on a single GPU more efficient.

[0123] Furthermore, during training, policy loss and value loss are superimposed on the shared backbone network layer, jointly updating the model parameters of the shared backbone network layer. This allows the shared backbone network layer to simultaneously learn general feature representations beneficial for action generation and value assessment. It is evident that the design of the shared backbone network layer enables the Actor and Critic modules to mutually assist in learning features. The Critic module's gradient for judging the value of sparse rewards helps the Actor module better understand state features, thus improving model performance.

[0124] Secondly, the action block prediction strategy:

[0125] Existing Actor-Critic algorithms typically employ the standard paradigm of Markov Decision Processes (MDPs) for single-step action prediction, i.e., inputting state st at time t and outputting action at a single time. For robots requiring high-frequency control, the inference latency of the model often leads to robot control lag. Furthermore, the lack of temporal consistency in single-step prediction can easily cause high-frequency jitter in the robot's end effector, resulting in hardware wear and tear and a decrease in task success rate.

[0126] The solution provided in this invention abandons the traditional single-step action output mode and adopts action block prediction technology. After the agent completes one inference, it can output an action block, thereby controlling the robot to execute the action sequence within that block sequentially. This reduces the agent's inference frequency while maintaining a high control frequency, making it possible to deploy large models on edge devices with limited computing power. Furthermore, the temporal continuity within the action block naturally eliminates the discrete jitter of control commands, improves the smoothness of the robot's motion trajectory, reduces hardware wear and tear, and increases the task success rate.

[0127] Thirdly, a complete takeover mechanism is introduced:

[0128] Most existing solutions employ a human-in-the-loop reinforcement learning strategy to train the agent. During the data collection process, if a human expert intervenes, their actions are used as corrections to the policy module's output, or the human expert's actions or the policy module's output are randomly selected through weighted averaging or random gate functions. This approach keeps the agent in a semi-autonomous state. Once a human expert intervenes, the input distribution of the policy module changes drastically (covariance shift), making it difficult for the agent to distinguish between its own errors and human intentions. Furthermore, the human expert's actions may not be executed properly, leading to over-correction and system oscillations.

[0129] In the solution provided by this invention, when entering expert takeover mode, the agent's output is completely masked. The intervention action commands input by the human expert are directly sent to the robot, effectively "completely replacing" the agent's output with human expert actions, rather than "overlaying" or "mixing" them, thus achieving full control by the human expert. This reduces the probability of over-correction due to inadequate execution of human expert instructions, improving the rationality of agent training. Furthermore, by designing a clear "complete takeover" mechanism, unsafe policy outputs can be completely masked in abnormal situations, ensuring the safety of real-machine training.

[0130] Given the above advantages, the solutions provided in this embodiment of the invention are applicable to a variety of robot application scenarios, which will be described below with examples.

[0131] Example 1: Industrial Assembly Scenario:

[0132] Examples of scenarios include the insertion and removal of automotive parts and the assembly of electronic components. In such scenarios, robot tasks require extremely high positional accuracy and involve complex contact forces. The motion block prediction strategy provided in this embodiment of the invention can achieve smooth insertion force control, improving motion stability and accuracy. Furthermore, in the event of robot jamming, a worker can immediately intervene to take over and adjust the situation, ensuring safety before handing over control of the robot back to the model.

[0133] Example 2: Object grasping and sorting scenarios in unstructured environments:

[0134] Examples include home service robots and logistics sorting. In such scenarios, the shapes and positions of objects in the environment are highly variable. In the solution provided by this invention, the shared backbone network layer can effectively extract visual features, providing a good foundation for subsequent action prediction; in addition, the full takeover mechanism allows users to provide high-quality teaching data when the robot encounters objects it has never seen before, enabling continuous learning.

[0135] Example 3: Remote operation assistance scenarios in hazardous environments:

[0136] Examples include nuclear waste disposal and remote medical surgery. In such scenarios, automation is auxiliary, while human safety is paramount. The solution provided by this invention ensures that, in any abnormal situation, the operator can instantly gain complete control at the underlying level to handle emergencies.

[0137] Corresponding to the above-described agent training method, this embodiment of the invention also provides an agent training device.

[0138] See Figure 4 This is a schematic diagram of a smart agent training device provided in an embodiment of the present invention. The smart agent includes a shared feature extraction layer, a policy head, and a value head. The policy head and the value head are both connected to the shared feature extraction layer. The device includes the following modules:

[0139] Feature extraction module 401 is used to input the first perceptual information in the sample data into the shared feature extraction layer for feature extraction to obtain the first information feature;

[0140] Information prediction module 402 is used to input the first information feature into the strategy head to predict the first action trajectory to be executed by the first robot when it perceives the first perception information, input the first information feature and the first action trajectory into the value head to evaluate the first excellence level characterization information of the first action trajectory, and input the first information feature and the sample action trajectory in the sample data into the value head to evaluate the second excellence level characterization information of the sample action trajectory.

[0141] The loss determination module 403 is used to determine the strategy loss based on the first excellence level representation information, and to determine the value loss based on the difference between the second excellence level representation information and the third excellence level representation information, wherein the third excellence level representation information is determined based on the sample excellence level representation information of the sample action trajectory in the sample data and the second perception information.

[0142] The parameter update module 404 is used to update the network parameters of the shared feature extraction layer and the policy head according to the policy loss, and to update the network parameters of the shared feature extraction layer and the value head according to the value loss, so as to obtain the robot action prediction model.

[0143] As can be seen from the above, in the agent training scheme provided by this embodiment of the invention, the shared feature extraction layer in the agent is connected to both the policy head and the value head. After the robot's perception information is processed by the shared feature extraction layer, information features are obtained. These information features are then input into the policy head and the value head, respectively, and the policy head and the value head output action trajectory and excellence level representation information, respectively. It is evident that the policy head and the value head share the same feature extraction layer, achieving feature extraction layer reuse. Thus, compared to the traditional training process, by designing a shared feature extraction layer, it is no longer necessary to design independent feature extraction layers for the policy module and the value module in the agent, simplifying the agent architecture during training, reducing the number of agent parameters, and consequently reducing the storage and computing resources required during agent training.

[0144] In addition, both policy loss and value loss are used to adjust the network parameters of the shared feature extraction layer, so that the shared feature extraction layer can learn general feature representations that are beneficial to trajectory prediction and value assessment at the same time. This improves the rationality of the agent training process and helps to improve the performance of the robot action prediction model obtained in the final training.

[0145] In one possible implementation, sample data is obtained according to the following modules:

[0146] The sampling module is used to determine a target number of consecutive sampling data from the interaction data generated by the interaction between the second robot and the working environment in which the second robot is located. Each of the interaction data includes: initial perception information before the second robot performs an action, the action of the second robot, the action quality representation information, and updated perception information after the second robot completes the action. The target number is the number of actions included in the action trajectory predicted by the strategy head. The motion parameters of the operating parts and the sensor layout of the second robot are the same as those of the first robot.

[0147] The sample data acquisition module is used to acquire the motion trajectory of the action in each of the sampled data as the sample motion trajectory in the sample data, wherein the order of the actions included in the sample motion trajectory is the same as the acquisition order of the actions in each of the sampled data; to acquire the sample excellence characterization information of the sample motion trajectory based on the action excellence characterization information in each of the sampled data; to take the initial perception information in the sampled data with the earliest acquisition order as the first perception information in the sample data, and to take the updated perception information in the sampled data with the latest acquisition order as the second perception information in the sample data.

[0148] It is evident that when interaction data is recorded at the action level, continuous interaction data can be sampled. By analyzing and combining continuous interaction data, a large amount of diverse sample data recorded at the action trajectory level can be obtained efficiently and quickly.

[0149] In one possible implementation, interaction data is collected according to the following modules:

[0150] The interactive data acquisition module is used to input the current perception information of the second robot into the shared feature extraction layer for feature extraction to obtain second information features; input the second information features into the strategy head to predict the second action trajectory to be executed by the second robot; control the second robot to execute each action included in the second action trajectory in sequence; obtain the perception information after the second robot has completed each action; for each action executed by the second robot, based on the set task objective, determine the action excellence representation information of the action, use the perception information before the action is executed as the initial perception information, use the perception information after the action is executed as the updated perception information, and record the initial perception information, the action, the action excellence representation information, and the updated perception information to obtain interactive data.

[0151] In this way, after the intelligent agent outputs the predicted action trajectory based on the environmental state information, it controls the robot to execute each action in the predicted action trajectory. For each action, it records the initial perception information, the action itself, the action's quality representation information, and the updated perception information, which can quickly obtain interactive data.

[0152] In one possible implementation, the device further includes:

[0153] The motion intervention module is used to respond to receiving a motion intervention command, control the second robot to pause the execution of the motion included in the second motion trajectory, and control the second robot to execute the motion indicated by the motion intervention command.

[0154] It is evident that when human experts attempt to correct a robot's erroneous actions, the electronic device can immediately sever the strategy head's control over the robot, completely handing over control of the robot to the human expert. This reduces the probability of excessive human error correction due to inadequate execution of actions instructed by the human expert, thereby reducing the probability of system oscillation, improving the rationality of agent training, and enhancing the safety of real-machine training.

[0155] In one possible implementation, the device further includes:

[0156] The termination intervention module is used to respond to receiving an intervention termination command and return to the step of inputting the current perception information of the second robot into the shared feature extraction layer in the agent for feature extraction.

[0157] In this way, during the interactive data collection process, human experts can intervene in the control of the robot at any time, and can also restore the control of the robot by the intelligent agent at any time after intervention, which improves the flexibility of collecting interactive data.

[0158] In one possible implementation, the interaction data carries an intervention identifier, wherein the intervention identifier indicates whether the action is an action indicated by an action intervention command; the target number of sampled data includes demonstration data, wherein the demonstration data is: interaction data in which the included intervention identifier indicates that the action is an action indicated by an action intervention command.

[0159] In this way, the obtained sample data can also include expert demonstration data with human intervention. Therefore, in the process of training the agent based on the obtained sample data, the agent can learn the robot control mode in abnormal situations, thereby improving the accuracy and rationality of the agent's predicted motion trajectory.

[0160] In one possible implementation, the parameter update module is specifically used to determine a first difference between every two adjacent actions included in the first action trajectory; determine a second difference between the first action included in the first action trajectory and the last action included in the previous action trajectory output by the policy head; calculate the action smoothness loss based on the first difference and the second difference; and update the network parameters of the shared feature extraction layer and the policy head according to the policy loss and the action smoothness loss.

[0161] In this way, by adjusting the network parameters of the feature extraction layer and the policy head according to the gradient descent method, the actions between adjacent action trajectories output by the policy head and between adjacent actions in each action trajectory can be more continuous and smooth, reducing action abrupt changes and jitter, and improving the quality of the predicted action trajectory.

[0162] In one possible implementation, the device further includes:

[0163] A robot control module is used to obtain third perception information of a third robot, wherein the third perception information includes: body perception information and environment perception information. The body perception information includes: the joint pose and end effector pose of the third robot. The environment perception information includes at least: the visual information and / or spatial information of the object to be operated in the working environment of the third robot. The motion parameters of the operating parts and the sensor layout of the third robot are the same as those of the first robot. The third perception information is input into the robot motion prediction model to obtain a predicted motion sequence output by the robot motion prediction model after processing. The module controls the third robot to execute each motion in the predicted motion sequence in sequence so that the third robot performs an operation on the object to be operated.

[0164] This is equivalent to using a trained robot motion prediction model to control the robot to perform tasks, which improves the robot's task execution efficiency compared to staff manually writing motion control instructions.

[0165] Corresponding to the above-described intelligent agent training method, embodiments of the present invention also provide an electronic device, a storage medium, and a program product.

[0166] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0167] Memory 503 is used to store computer programs;

[0168] The processor 501 is used to implement the aforementioned agent training method when executing the program stored in the memory 503.

[0169] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0170] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0171] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0172] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0173] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described intelligent agent training methods.

[0174] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the intelligent agent training methods described above.

[0175] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0176] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0177] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0178] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for training an intelligent agent, characterized in that, The intelligent agent includes a shared feature extraction layer, a policy head, and a value head. Both the policy head and the value head are connected to the shared feature extraction layer, which is a Shared Backbone network layer used to map multimodal perceptual information into high-dimensional latent feature vectors. The method includes: The first perceptual information in the sample data is input into the shared feature extraction layer for feature extraction to obtain the first information feature; The first information feature is input into the strategy head to predict the first action trajectory to be executed when the first robot perceives the first perception information. The first information feature and the first action trajectory are input into the value head to evaluate the first excellence level representation information of the first action trajectory. The first information feature and the sample action trajectory in the sample data are input into the value head to evaluate the second excellence level representation information of the sample action trajectory. Based on the first excellence level representation information, a strategy loss is determined, and based on the difference between the second and third excellence level representation information, a value loss is determined. The third excellence level representation information is determined based on the sample excellence level representation information of the sample action trajectory in the sample data and second perception information. The third excellence level representation information includes: rewards and future value in the sample data. The future value is obtained as follows: the next state information in the sample data is input into the shared feature extraction layer; the strategy head predicts the action trajectory of the first robot under the next state information based on the output of the shared feature extraction layer; the output of the shared feature extraction layer and the obtained next action trajectory are input into the value head to obtain the future value output by the value head. Based on the policy loss, update the network parameters of the shared feature extraction layer and the policy head; based on the value loss, update the network parameters of the shared feature extraction layer and the value head to obtain the robot action prediction model. The step of updating the network parameters of the shared feature extraction layer and the policy head according to the policy loss includes: Determine a first difference between every two adjacent actions included in the first action trajectory; determine a second difference between the first action included in the first action trajectory and the last action included in the previous action trajectory output by the policy head; obtain an action smoothness loss based on the first difference and the second difference; update the network parameters of the shared feature extraction layer and the policy head according to the policy loss and the action smoothness loss.

2. The method according to claim 1, characterized in that, The sample data was obtained in the following manner: From the interaction data generated by the interaction between the second robot and its working environment, a target number of sampling data points with a continuous collection order are determined. Each of the interaction data points includes: initial perception information before the second robot performs an action, the action of the second robot, the action quality representation information, and updated perception information after the second robot completes the action. The target number is the number of actions included in the action trajectory predicted by the strategy head. The motion parameters of the operating parts and the sensor layout of the second robot are the same as those of the first robot. The motion trajectory of each action in each of the sampled data is obtained as the sample motion trajectory in the sample data, wherein the order of the actions included in the sample motion trajectory is the same as the collection order of the actions in each of the sampled data; the sample excellence characterization information of the sample motion trajectory is obtained based on the action excellence characterization information in each of the sampled data. The initial sensing information in the earliest collected sample data is used as the first sensing information in the sample data, and the updated sensing information in the latest collected sample data is used as the second sensing information in the sample data.

3. The method according to claim 2, characterized in that, The interaction data is collected in the following manner: The perception information of the second robot at the current moment is input into the shared feature extraction layer for feature extraction to obtain the second information feature; The second information feature is input into the strategy head to predict the second action trajectory to be executed by the second robot; Control the second robot to sequentially execute each action included in the second motion trajectory; Obtain the perception information after the second robot has completed each action; For each action performed by the second robot, based on the set task objective, the action excellence representation information is determined. The perception information before the action is performed is used as the initial perception information, and the perception information after the action is performed is used as the updated perception information. The initial perception information, the action, the action excellence representation information, and the updated perception information are recorded to obtain interactive data.

4. The method according to claim 3, characterized in that, The method further includes at least one of the following: In response to receiving a motion intervention command, control the second robot to pause executing the motion included in the second motion trajectory, and control the second robot to execute the motion indicated by the motion intervention command; In response to receiving an intervention termination command, the process returns to the step of inputting the current perception information of the second robot into the shared feature extraction layer in the agent for feature extraction.

5. The method according to claim 4, characterized in that, The interactive data carries an intervention identifier, wherein the intervention identifier indicates whether the action is an action indicated by an action intervention command; The target number of sampled data includes demonstration data, wherein the demonstration data is: interactive data in which the intervention identifier represents the action indicated by the action intervention command.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain third perception information of a third robot, wherein the third perception information includes: body perception information and environment perception information, the body perception information includes: joint pose and end effector pose of the third robot, and the environment perception information includes at least: visual information and / or spatial information of the object to be operated in the working environment of the third robot, and the motion parameters of the operating parts and the sensor layout of the third robot are the same as those of the first robot. The third perception information is input into the robot motion prediction model to obtain the predicted motion sequence output by the robot motion prediction model after processing. The third robot is controlled to execute each action in the predicted action sequence in sequence, so that the third robot performs an operation on the object to be operated.

7. An intelligent agent training device, characterized in that, The intelligent agent includes a shared feature extraction layer, a policy head, and a value head. Both the policy head and the value head are connected to the shared feature extraction layer, which is a Shared Backbone network layer used to map multimodal perceptual information into high-dimensional latent feature vectors. The device includes: The feature extraction module is used to input the first perceptual information in the sample data into the shared feature extraction layer for feature extraction to obtain the first information feature; The information prediction module is used to input the first information feature into the strategy head to predict the first action trajectory to be executed by the first robot when it perceives the first perception information, input the first information feature and the first action trajectory into the value head to evaluate the first excellence level representation information of the first action trajectory, and input the first information feature and the sample action trajectory in the sample data into the value head to evaluate the second excellence level representation information of the sample action trajectory. A loss determination module is used to determine a strategy loss based on the first excellence level representation information, and to determine a value loss based on the difference between the second excellence level representation information and the third excellence level representation information. The third excellence level representation information is determined based on the sample excellence level representation information of the sample action trajectory in the sample data and second perception information. The third excellence level representation information includes: a reward and a future value in the sample data. The future value is obtained as follows: the next state information in the sample data is input into the shared feature extraction layer; the strategy head predicts the action trajectory of the first robot under the next state information based on the output of the shared feature extraction layer; the output of the shared feature extraction layer and the obtained next action trajectory are input into the value head to obtain the future value output by the value head. The parameter update module is used to update the network parameters of the shared feature extraction layer and the policy head according to the policy loss, and to update the network parameters of the shared feature extraction layer and the value head according to the value loss, so as to obtain the robot action prediction model. The parameter update module is specifically used to determine the first difference between every two adjacent actions included in the first action trajectory; determine the second difference between the first action included in the first action trajectory and the last action included in the previous action trajectory output by the policy head; calculate the action smoothness loss based on the first difference and the second difference; and update the network parameters of the shared feature extraction layer and the policy head according to the policy loss and the action smoothness loss.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, a communication bus, and an image acquisition device. The processor, communication interface, and memory communicate with each other through the communication bus. Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

Images