A reinforcement learning-based adaptive training method, device, and medium for robots.

Abstract

This invention relates to the technical field of robot training. Specifically, it relates to a robot adaptive training method, apparatus, and medium based on reinforcement learning. The method includes generating task sub-objective information using a high-level policy network, inputting the task sub-objective information into a low-level execution network, generating action control commands based on the task sub-objective information, collecting interaction feedback information during execution, calculating a reward value based on the interaction feedback information, associating the reward value with scene complexity parameters, performing dynamic reward shaping operations to generate an adjusted reward signal, generating a policy model optimized by meta-learning based on the adjusted reward signal, loading the policy model into a simulation environment to generate a target policy model optimized by simulation training, loading the target policy model into the robot, and controlling the robot to perform task operations in an actual interactive scenario. This invention achieves the effect of adaptive task learning for robots in multi-interaction scenarios.

Description

Technical Field

[0001] This invention relates to the technical field of robot training, and in particular to a robot adaptive training method, apparatus and medium based on reinforcement learning. Background Technology

[0002] With the rapid development of intelligent robot technology, the complexity of tasks performed by robots in manufacturing, service operations, and complex environments is constantly increasing, placing higher demands on the adaptive and cross-scenario generalization capabilities of policy models. Existing training methods are mostly based on supervised learning or traditional reinforcement learning frameworks, but both suffer from structural technical bottlenecks when dealing with multi-scenario task switching: supervised learning relies on a large number of manually labeled samples, making it difficult to cover changes in multiple scene environments, resulting in insufficient generalization ability of the model in new environments; while traditional reinforcement learning can optimize policies through environmental interactions, it often suffers from sparse reward signals, unclear task objectives, and difficulty in stable convergence of parameter updates. Especially when complex scenes change frequently, the policy often cannot adapt quickly, leading to unstable task performance.

[0003] Furthermore, existing reinforcement learning training processes typically rely on single-scene or weakly perturbated simulation environments, making it difficult to simulate the multi-distribution changes in real operations; policy updates usually adopt a fixed gradient iteration method, lacking a hierarchical understanding of task structure and scene changes, and also lacking meta-learning capabilities that can be transferred across tasks. Therefore, it is difficult to obtain consistent and transferable training results in multi-task and multi-scene environments. Summary of the Invention

[0004] To enable robots to learn adaptive tasks in multi-interaction scenarios, this invention provides a robot adaptive training method, device, and medium based on reinforcement learning.

[0005] The above-mentioned objective of this invention is achieved through the following technical solution:

[0006] An adaptive training method for robots based on reinforcement learning, comprising:

[0007] The robot acquires environmental perception information in multiple interactive scenarios, inputs the environmental perception information into the scene encoder, and obtains scene feature information.

[0008] Based on scene feature information, a high-level policy network is used to generate task sub-objective information;

[0009] The task sub-objective information is input into the low-level execution network, action control instructions are generated based on the task sub-objective information, and interactive feedback information is collected during the execution process;

[0010] The reward value is calculated based on the interactive feedback information, and the reward value is correlated with the scene complexity parameter. A dynamic reward shaping operation is performed to generate an adjusted reward signal.

[0011] Based on the adjusted reward signal, meta-learning optimization is performed on the parameters of the high-level policy network and the low-level execution network. The parameters are updated using an optimization mechanism that combines inner and outer loops to generate a policy model optimized by meta-learning.

[0012] The strategy model is loaded into the simulation environment, and randomized training is performed based on the multi-distributed perturbation execution domain to generate a target strategy model optimized by simulation training. The target strategy model is then loaded into the robot to control the robot to perform task operations in the actual interaction scenario.

[0013] By adopting the above technical solution, adaptive training and policy optimization of robots in multi-scenario interactive environments can be achieved, solving the problems of low training efficiency, sparse reward signals, and insufficient generalization ability of policy models in existing technologies. This method extracts environmental feature information through a scene encoder and uses high-level and low-level policy networks to generate task sub-objectives and action control instructions hierarchically, giving the training process a clear hierarchical structure. By introducing a dynamic reward shaping mechanism and a meta-learning optimization process, the model can converge quickly and maintain stable performance in diverse scenarios. Combined with domain randomization training, the adaptability of the policy model to perturbed environments is further enhanced, thereby significantly improving the robot's autonomous execution and collaborative operation capabilities in complex and variable scenarios.

[0014] Preferably, the step of generating task sub-objective information based on scene feature information using a high-level policy network includes:

[0015] The scene feature information is input into the feature encoding layer of the high-level policy network to extract high-level semantic features;

[0016] High-level semantic features are input into the task decomposition layer of the high-level policy network, and the interaction task is decomposed according to the semantic relevance of the scene to obtain several task candidate targets.

[0017] The candidate targets for tasks are input into the strategy decision layer, and the probability distribution of the task targets is calculated based on the joint state representation of the current scene feature information and the historical task execution information.

[0018] From the probability distribution of task objectives, select candidate task objectives with probability values ​​greater than a preset task selection threshold, determine the corresponding task objectives, and obtain task sub-objective information.

[0019] By adopting the above technical solution, the high-level policy network can possess a clear hierarchical structure and task decision-making ability when processing complex scene information, thereby effectively solving the problems of unstable target selection and low task planning efficiency of existing robots in multi-scene tasks. This solution extracts high-level semantic features through a feature encoding layer, ensuring that scene information is effectively abstracted and compressed; the task decomposition layer breaks down complex tasks into several executable sub-objectives based on semantic relevance, improving task understanding accuracy; the policy decision-making layer calculates the probability distribution of task objectives by jointly considering current and historical states, making the decision-making process both real-time and empirical; finally, key task objectives are filtered through a task selection threshold to generate task sub-objective information, enabling the robot to achieve task recognition and adaptive planning capabilities in dynamic environments.

[0020] Preferably, the step of inputting task sub-objective information into the low-level execution network, generating action control instructions based on the task sub-objective information, and collecting interactive feedback information during execution includes:

[0021] The task sub-objective information is input into the instruction parsing layer of the lower-level execution network to parse and obtain the corresponding control vector information;

[0022] The robot's current state information is obtained, and the control vector information and the current state information are input into the action generation layer of the lower-level execution network. The action output parameters are calculated based on the correspondence between the control vector information and the current state information, and action control commands are generated.

[0023] Motion control commands are sent to the robot. During the execution of the motion, the robot's operating status information and environmental change information are collected. The collected operating status information and environmental change information are aggregated and processed to obtain interactive feedback information.

[0024] By adopting the above technical solution, fine-grained analysis and dynamic response control of task sub-objectives can be achieved at the robot execution layer, solving the problem that action generation in existing technologies relies on fixed strategies and is difficult to adaptively adjust to environmental changes. This solution transforms task sub-objective information into computable control vector information through the instruction parsing layer, enabling accurate mapping of task intent at the lower execution layer. Then, combined with current state information, the action generation layer calculates action output parameters based on the correspondence between the two, achieving temporal consistency and control continuity in action generation. During execution, real-time collection and aggregation of operational state information and environmental change information form interactive feedback information, providing precise basis for subsequent reward calculation and strategy optimization, thereby improving the robot's action coordination and execution stability in complex dynamic scenarios.

[0025] Preferably, the step of calculating the reward value based on the interactive feedback information, associating the reward value with the scene complexity parameter, performing a dynamic reward shaping operation, and generating an adjusted reward signal includes:

[0026] The interactive feedback information is parsed to extract feedback parameter information that characterizes the task execution status.

[0027] The reward value is calculated based on the values ​​of the feedback parameters and the preset evaluation rules.

[0028] The reward value and the scene complexity parameter are normalized to obtain the normalized result;

[0029] Based on the normalization results, a correspondence is established between the reward value and the scene complexity parameter. Dynamic reward shaping is performed according to the correspondence to adaptively adjust the reward value and generate the adjusted reward signal.

[0030] By adopting the above technical solution, dynamic adjustment and precise characterization of reward signals can be achieved during robot training, solving the problem of slow model convergence caused by singular reward design and sparse signals in existing reinforcement learning methods. This solution extracts feedback parameters representing the task execution state by parsing interactive feedback information, ensuring that reward calculation is based on the quantitative foundation of real-world behavior. Reward values ​​are calculated according to preset evaluation rules and normalized using scene complexity parameters to ensure comparability of feedback under different difficulty scenarios. Furthermore, a non-linear mapping relationship between reward and complexity is established through dynamic reward shaping operations, adaptively adjusting the reward value to maintain balanced updates under different environmental conditions, thereby improving the stability and training efficiency of policy learning.

[0031] Preferably, the step of performing meta-learning optimization on the parameters of the high-level policy network and the low-level execution network based on the adjusted reward signal, and updating the parameters using an optimization mechanism combining inner and outer loops to generate a meta-learning optimized policy model includes:

[0032] The target parameters are updated based on the adjusted reward signal;

[0033] Based on the parameter update objective, an inner loop update is performed on the parameters of the high-level policy network and the low-level execution network to obtain the inner loop update parameters;

[0034] The inner loop update parameters are used as input to the outer loop update. The outer loop update is performed based on the changes in the adjusted reward signal to obtain the outer loop update parameters.

[0035] The inner loop update parameters and the outer loop update parameters are jointly adjusted to determine the optimized parameters;

[0036] Based on the optimized parameters, the policy model optimized by eigenvalue learning is obtained.

[0037] By adopting the above technical solution, adaptive joint optimization of high-level and low-level policy networks can be achieved during robot training, solving the problems of single model updates, slow parameter convergence, and insufficient cross-scene generalization ability in existing reinforcement learning methods. This solution determines the parameter update target based on the adjusted reward signal, ensuring the optimization direction aligns with training feedback; it achieves rapid local adaptation through inner-loop updates, enabling the model to respond promptly to short-term task changes; and it integrates multi-round training feedback through outer-loop updates to achieve long-term cross-task optimization. In the joint adjustment phase, the results of inner and outer loop updates are merged to obtain the globally optimal parameter configuration. Finally, the optimized parameters are used to generate a policy model optimized by meta-learning, enabling the training process to possess continuous self-learning and multi-scene transfer capabilities, thereby enhancing the stability and generalization of the policy model.

[0038] Preferably, the step of loading the policy model in the simulation environment, performing randomized training based on multi-distribution perturbation execution domain, and generating a target policy model optimized through simulation training includes:

[0039] Load the strategy model into the simulation environment to create a virtual training scenario that corresponds to the actual interaction scenario;

[0040] In a virtual training scenario, multiple distribution perturbation parameters are set, and the simulation environment is perturbed multiple times based on the distribution perturbation parameters. The training iteration of the policy model is then performed to obtain the corresponding training results.

[0041] The performance stability of the policy model under different perturbation distributions is evaluated based on the training results, and the training parameters are adjusted to obtain the adjusted training parameters.

[0042] Based on the adjusted training parameters, the policy model is updated to obtain the target policy model optimized through simulation training.

[0043] By adopting the above technical solution, multi-dimensional perturbation training and adaptive parameter optimization of the policy model can be achieved in a simulation environment, solving the problems of weak model generalization ability and unstable performance in complex environments in traditional robot training methods. This solution establishes a virtual training scenario corresponding to the actual interaction scenario in the simulation environment, enabling the model to possess perceptual consistency in real interaction contexts; introduces environmental diversity through multi-distributed perturbation training, allowing the policy model to learn stably under different physical conditions; dynamically optimizes training parameters during evaluation and convergence adjustment, gradually bringing the model parameters closer to the global optimum; and finally, updates the model based on the adjusted training parameters to obtain the target policy model optimized through simulation training, thereby improving the robot's policy adaptability and execution stability in real-world scenarios.

[0044] Preferably, the step of setting multiple distribution perturbation parameters in the virtual training scenario, performing multiple rounds of perturbation on the simulation environment based on the distribution perturbation parameters, executing training iterations of the policy model, and obtaining the corresponding training results includes:

[0045] In a virtual training scenario, determine the range of disturbance distribution and generate distribution disturbance parameter information based on the range of disturbance distribution;

[0046] The distributed perturbation parameter information is applied to the simulation environment, and the physical attribute information and perception condition information of the simulation environment are perturbed to obtain the perturbed simulation environment.

[0047] The policy model is loaded into the perturbed simulation environment, and multiple rounds of training iterations are performed based on the distributed perturbation parameter information to obtain the model training output information under each perturbation condition.

[0048] The training output information of the model is aggregated to obtain the training results.

[0049] By adopting the above technical solution, distributed parameter generation and comprehensive model performance evaluation under multiple perturbation conditions can be achieved during the simulation training phase, solving the problems of single environment and insufficient model adaptability in traditional simulation training. This solution first determines the perturbation distribution range in the virtual training scene and generates distributed perturbation parameter information, enabling controllable variation in the physical properties and perception conditions of the simulation environment. By applying the distributed perturbation parameter information to the simulation environment, multiple perturbed environment instances are formed, ensuring that the model can be trained under different physical and perception conditions. Multiple rounds of training iterations are performed in the perturbed simulation environment, collecting model training output information under each perturbation condition, and then aggregating the output data to generate training results, achieving the evaluation of model stability and convergence under multiple distribution scenarios.

[0050] The second objective of this invention is achieved through the following technical solution:

[0051] A reinforcement learning-based adaptive training device for robots, comprising:

[0052] The scene feature extraction module is used to acquire environmental perception information of the robot in multiple interactive scenarios, and input the environmental perception information into the scene encoder to obtain scene feature information;

[0053] The task sub-objective generation module is used to generate task sub-objective information based on scene feature information and using a high-level policy network.

[0054] The motion control and feedback module is used to input task sub-objective information into the low-level execution network, generate motion control instructions based on the task sub-objective information, and collect interactive feedback information during the execution process.

[0055] The reward shaping module is used to calculate the reward value based on the interactive feedback information, associate the reward value with the scene complexity parameter, perform dynamic reward shaping operation, and generate the adjusted reward signal.

[0056] The meta-learning optimization module is used to perform meta-learning optimization on the parameters of the high-level policy network and the low-level execution network based on the adjusted reward signal. It updates the parameters using an optimization mechanism that combines inner and outer loops to generate a policy model optimized by meta-learning.

[0057] The simulation training and execution module is used to load the policy model in the simulation environment, generate the target policy model optimized by simulation training based on randomized training of the multi-distributed perturbation execution domain, load the target policy model into the robot, and control the robot to perform task operations in the actual interaction scenario.

[0058] By adopting the above technical solution, adaptive training and policy optimization of robots in multi-scenario interactive environments can be achieved, solving the problems of low training efficiency, sparse reward signals, and insufficient generalization ability of policy models in existing technologies. This method extracts environmental feature information through a scene encoder and uses high-level and low-level policy networks to generate task sub-objectives and action control instructions hierarchically, giving the training process a clear hierarchical structure. By introducing a dynamic reward shaping mechanism and a meta-learning optimization process, the model can converge quickly and maintain stable performance in diverse scenarios. Combined with domain randomization training, the adaptability of the policy model to perturbed environments is further enhanced, thereby significantly improving the robot's autonomous execution and collaborative operation capabilities in complex and variable scenarios.

[0059] The above-mentioned objective three of the present invention is achieved through the following technical solution:

[0060] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aforementioned reinforcement learning-based adaptive training method for robots.

[0061] The above-mentioned objective four of the present invention is achieved through the following technical solution:

[0062] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based adaptive training method for robots.

[0063] In summary, the present invention has at least one of the following beneficial technical effects:

[0064] 1. This method enables adaptive training and policy optimization for robots in multi-scene interactive environments, solving the problems of low training efficiency, sparse reward signals, and insufficient generalization ability of policy models in existing technologies. The method extracts environmental feature information through a scene encoder and uses high-level and low-level policy networks to generate task sub-objectives and action control instructions hierarchically, giving the training process a clear hierarchical structure. By introducing a dynamic reward shaping mechanism and a meta-learning optimization process, the model can converge quickly and maintain stable performance in diverse scenarios. Combined with domain randomization training, the adaptability of the policy model to perturbed environments is further enhanced, thereby significantly improving the robot's autonomous execution and collaborative operation capabilities in complex and variable scenarios. Attached Figure Description

[0065] Figure 1 This is a flowchart of a robot adaptive training method based on reinforcement learning in one embodiment of the present invention.

[0066] Figure 2 This is a flowchart illustrating the implementation of step S20 in a reinforcement learning-based adaptive training method for robots according to an embodiment of the present invention.

[0067] Figure 3 This is a flowchart illustrating the implementation of step S30 in a reinforcement learning-based adaptive training method for robots according to an embodiment of the present invention.

[0068] Figure 4 This is a flowchart illustrating the implementation of step S40 in a reinforcement learning-based adaptive training method for robots according to an embodiment of the present invention.

[0069] Figure 5 This is a flowchart illustrating the implementation of step S50 in a reinforcement learning-based adaptive training method for robots according to an embodiment of the present invention.

[0070] Figure 6 This is a flowchart illustrating the implementation of step S60 in a reinforcement learning-based adaptive training method for robots according to an embodiment of the present invention.

[0071] Figure 7 This is a flowchart illustrating the implementation of step S602 in a reinforcement learning-based adaptive training method for robots according to an embodiment of the present invention.

[0072] Figure 8 This is a block diagram illustrating the principle of a robot adaptive training device based on reinforcement learning, according to one embodiment of the present invention. Detailed Implementation

[0073] The present invention will be further described in detail below with reference to the accompanying drawings.

[0074] In one embodiment, such as Figure 1As shown, this invention discloses a robot adaptive training method based on reinforcement learning, which specifically includes the following steps:

[0075] S10: Acquire environmental perception information of the robot in multiple interactive scenarios, input the environmental perception information into the scene encoder, and obtain scene feature information.

[0076] In this embodiment, environmental perception information refers to the set of data that can represent the state of the scene collected by the robot through the perception unit in different interaction scenarios. The scene encoder refers to the computational structure that performs feature mapping and encoding processing on the environmental perception information. Scene feature information refers to the multi-dimensional vector data that represents the spatial structure and dynamic state of the scene after being transformed by the scene encoder.

[0077] Specifically, when acquiring environmental perception information of the robot in multiple interactive scenarios, multiple types of sensor data are simultaneously collected by a sensing unit installed on the robot's shell. The sensing unit includes a camera, a depth sensor, a microphone array, and an inertial measurement unit. The camera is used to collect image frame sequences of the interactive scene. The depth sensor measures the distance matrix through infrared or laser scanning. The microphone array collects ambient sound signals and calculates the direction of the sound source. The inertial measurement unit detects the robot's posture angle and acceleration information. The sensor data is marked in the form of timestamps and time alignment is performed by the sampling frequency calibration module to keep the multi-source data consistent in the time dimension. When the time-aligned environmental perception information is input into the scene encoder, it is input in the form of a multi-channel tensor. The input layer of the scene encoder receives image signals, depth matrices, and sound feature vectors according to the data type. Normalization operations are performed on each type of input data to unify the dimensions and numerical range. The scene encoder uses convolution operations to extract spatial texture features and sequence mapping operations to extract temporal change features. In the feature fusion stage, spatial and temporal features are combined into a joint feature representation through weighted summation operations. The output joint feature vector is the scene feature information.

[0078] S20: Based on scene feature information, generate task sub-objective information using a high-level policy network.

[0079] In this embodiment, the high-level policy network refers to the policy structure used for task planning and target reasoning based on scene feature information, and the task sub-target information refers to the phased task target data obtained by the high-level policy network based on the scene feature information.

[0080] Specifically, when generating task sub-target information using a high-level policy network based on scene feature information, the scene feature information output from the scene encoder is input into the input layer of the high-level policy network. The high-level policy network includes a feature encoding layer, a task inference layer, and a target output layer. The feature encoding layer performs dimension mapping and normalization operations on the input scene feature information, transforming feature data from different sources into a unified vector representation. The task inference layer constructs a task semantic association graph based on this unified vector representation, calculates the degree of association between scene features and potential task targets through a weight matrix, and calculates the degree of association through matrix multiplication and a nonlinear activation function to obtain the response value of each potential task target. The obtained response value is input into the target output layer, which performs normalization operations to form a probability distribution of task targets and sorts the candidate targets according to their probability values. The candidate targets with probability values ​​above a preset task threshold are selected, and their identification information and association parameters are used to form task sub-target information.

[0081] S30: Input the task sub-objective information into the low-level execution network, generate action control instructions based on the task sub-objective information, and collect interactive feedback information during the execution process.

[0082] In this embodiment, the low-level execution network refers to the execution layer structure used to generate control action sequences based on task sub-objective information. The action control command refers to the set of control parameters output by the low-level execution network that can drive the robot to perform specific operations. The interactive feedback information refers to the data collected by sensors during the action execution process that can reflect the execution state and environmental changes.

[0083] Specifically, when the task sub-objective information is input into the low-level execution network, the task identifier parameter and target position parameter in the task sub-objective information are used as input vectors to the input layer of the low-level execution network. The input layer converts the input vector into a feature representation with a uniform amplitude range through a normalization function. The low-level execution network includes an instruction parsing layer and an action generation layer. The instruction parsing layer parses the input feature vector through matrix mapping operations, extracts the corresponding action control variables, and passes the parsed control variables to the action generation layer. The action generation layer calculates the action output parameters based on the control variables. During the calculation process, the input variables are linearly transformed through the control matrix, and the range of output parameters is limited by the activation function. The output parameters include control quantities such as velocity, pose angle, and execution torque, which are combined into complete action control instructions. During the execution of the action control instructions, the state detection unit and the environment perception unit configured on the robot body synchronously collect the running state information and environmental change information. The state detection unit acquires pose, angular velocity, and motor speed data, and the environment perception unit acquires external obstacle distance and touch pressure data. The collected running state information and environmental change information are aggregated and processed in the form of a time series to form interactive feedback information.

[0084] S40: Calculate the reward value based on the interactive feedback information, associate the reward value with the scene complexity parameter, perform dynamic reward shaping operation, and generate the adjusted reward signal.

[0085] In this embodiment, the reward value refers to a quantitative scoring parameter calculated based on the task execution status in the interactive feedback information. It is used to characterize the degree of completion of the robot's actions and the rationality of its behavior during the task execution process. The scene complexity parameter includes at least one of the following: target object quantity parameter, obstacle density parameter, environmental dynamics parameter, sensor noise intensity parameter, and external interference parameter. It is used to reflect the comprehensive difficulty of the scene in terms of spatial congestion, structural uncertainty, sensor noise level, and external interference intensity. The dynamic reward shaping operation refers to the process of establishing a correspondence between the reward value and the scene complexity parameter and making adaptive adjustments. The adjusted reward signal refers to the reward input data that can be used for subsequent learning and optimization after dynamic shaping.

[0086] Specifically, when calculating the reward value based on the interactive feedback information, the running status information and environmental change information in the interactive feedback information are analyzed to extract feedback parameters representing action stability, energy consumption, pose deviation, and task completion degree. The extraction of feedback parameters is achieved by setting an analytical function to read and filter the raw values ​​of the interactive feedback information. According to the preset evaluation rules, the feedback parameters are mapped to scores within a numerical range. The scores are calculated by weighted summation, and the weights corresponding to each feedback parameter are obtained from historical training samples. The calculated scores are the reward value. When associating the reward value with the scene complexity parameter, a normalization operation is first performed based on the value range of the scene complexity parameter to make the reward value and the scene complexity parameter comparable in the same numerical space. Then, an interpolation mapping method is used to establish the correspondence between the two to obtain the reward adjustment function. When performing dynamic reward shaping, the reward value is input into the reward adjustment function, and the reward value is adaptively adjusted by calculating the rate of change of the function output. The adjustment result is output as the adjusted reward signal.

[0087] S50: Based on the adjusted reward signal, meta-learning optimization is performed on the parameters of the high-level policy network and the low-level execution network. The parameters are updated using an optimization mechanism that combines inner and outer loops to generate a policy model optimized by meta-learning.

[0088] In this embodiment, meta-learning optimization refers to the optimization method of rapidly adapting and generalizing network parameters across tasks based on the adjusted reward signal during training. The inner loop refers to the process of updating parameters multiple times on fixed task samples, while the outer loop refers to the optimization process of adjusting the initial parameter distribution globally based on the update results of the inner loop. The policy model optimized by meta-learning refers to the policy model with adaptive learning capabilities obtained after joint updates by the inner and outer loops.

[0089] Specifically, when performing meta-learning optimization on the parameters of the high-level policy network and the low-level execution network based on the adjusted reward signal, the adjusted reward signal is used as the optimization driving quantity input to the parameter update module. The parameter update module reads the current parameter values ​​of the high-level policy network and the low-level execution network and calculates the gradient information of the reward signal relative to the parameters. The gradient calculation is implemented through the backpropagation algorithm. When executing the inner loop, a fixed task sample set is used as the training benchmark, and the learning rate coefficient is used to control the parameter update step size. The parameter weights are updated sequentially according to the gradient descent formula. After multiple rounds of iterative updates, the inner loop updated parameters are obtained. When executing the outer loop, the changes in the inner loop updated parameters and the adjusted reward signal are input into the outer loop optimization structure to calculate the outer loop gradient direction. Based on this direction, the initial parameter distribution is adjusted as a whole to obtain global parameters that are suitable for different tasks. When the inner loop updated parameters and the outer loop updated parameters are jointly adjusted, the update results of the two loops are fused by weighted averaging to achieve a balance between local convergence and global generalization of the parameters. Based on the jointly adjusted parameters, the policy model structure is configured, and after the parameter assignment operation is completed, the policy model optimized by meta-learning is obtained.

[0090] S60: Load the policy model in the simulation environment, perform randomized training based on the multi-distributed perturbation execution domain, generate the target policy model optimized by simulation training, load the target policy model into the robot, and control the robot to perform task operations in the actual interaction scenario.

[0091] In this embodiment, domain randomization training refers to a training method that improves the adaptability of the policy model to various scenarios by changing the distribution of environmental parameters in a simulation environment. The target policy model refers to a policy model that can stably perform tasks in actual interactive scenarios after simulation training and optimization.

[0092] Specifically, when loading the policy model in the simulation environment, the policy model optimized by meta-learning is imported into the virtual simulation platform. After loading the model structure and parameter files, the simulation platform establishes a virtual training scene consistent with the actual interaction scenario. The virtual training scene includes adjustable parameters such as object layout, lighting conditions, friction coefficient, and noise distribution. During randomized training based on multi-distribution perturbations, multiple sets of distributed perturbation parameters are randomly generated before training begins. Each set of parameters corresponds to a set of environmental variable configurations. These perturbation parameters are applied one by one to the simulation environment to adjust the physical properties and perceptual inputs of the simulation environment. The policy model is trained iteratively under each set of perturbation conditions. During the training iteration, the model output actions and corresponding reward signals are recorded. The model parameters are adjusted based on the reward signals using the gradient descent algorithm. The system is optimized to ensure the model can complete tasks under various perturbation conditions. After training with multiple sets of perturbations, the training results of each set are converged. The final parameter values ​​are calculated based on the stable interval of parameter changes. The converged model parameters and structure are saved together to obtain the target policy model optimized by simulation training. When the target policy model is loaded into the robot, the model file is transferred to the storage space in the robot control unit through the communication interface. The control unit loads the model structure and initializes the running state. The input environmental perception information is input into the target policy model in real time, and the model outputs a sequence of motion control commands. The robot executes the task operation according to the motion control commands. During the execution process, sensor data is continuously received and the motion sequence is updated according to the model output, realizing the task control execution of the robot in the actual interactive scenario.

[0093] In one embodiment, such as Figure 2 As shown, in step S20, based on scene feature information, task sub-objective information is generated using a high-level policy network, including:

[0094] S201: Input scene feature information into the feature encoding layer of the high-level policy network to extract high-level semantic features.

[0095] In this embodiment, high-level semantic features refer to abstract representation data that reflects the semantic relationships of a scene, formed by the feature encoding layer of a high-level policy network when encoding scene feature information.

[0096] Specifically, when scene feature information is input into the feature encoding layer of the high-level policy network, the spatial and temporal feature vectors contained in the scene feature information are used as input signals to the input nodes of the feature encoding layer. The input nodes are first processed according to a normalization function. The input data undergoes amplitude normalization to unify the dimensionality of feature values ​​from different sources. The resulting feature vector is then fed into the linear mapping unit of the feature encoding layer, where it is processed according to matrix operation formulas. A weighted summation operation is performed on the input vector to obtain the primary feature mapping result, where W1 is the mapping weight matrix and b1 is the bias vector. The mapping weights are initialized according to the previously trained parameters. After obtaining the primary mapping result, the output is fed into the nonlinear transformation unit, which uses an activation function. A nonlinear transformation is performed on the linear output to enhance the discriminative power and expressive hierarchy of the features. Feature aggregation is then performed on the activated feature vectors using a weighted average method, employing the formula... The multi-layer output features are weighted and fused, where the weights are... This represents the importance coefficient of each layer's features. This coefficient is obtained by normalizing the magnitude of the feature layer's output. The final set of vectors generated is the high-level semantic feature.

[0097] S202: Input the high-level semantic features into the task decomposition layer of the high-level policy network, decompose the interaction task according to the semantic relevance of the scene, and obtain several task candidate targets.

[0098] In this embodiment, the task decomposition layer refers to the network layer used to analyze the semantic relationships between high-level semantic features and decompose complex tasks into several sub-target structures. The task candidate target refers to the set of multiple task targets with potential execution significance output by the task decomposition layer after calculation.

[0099] Specifically, when inputting high-level semantic features into the task decomposition layer of the high-level policy network, the high-level semantic features are first used as the input matrix. The input matrix is ​​fed into the task decomposition layer. After reading the input matrix, the task decomposition layer calculates the correlation between each feature vector using the semantic similarity calculation unit. The semantic similarity calculation uses the cosine similarity formula. This is used to measure the similarity between different feature vectors in the semantic space. A scene semantic relationship graph is constructed based on the semantic similarity matrix. In this graph, nodes represent high-level semantic features, and edge weights represent semantic relevance values, based on a relevance threshold. Filter the edges; when the edge weight is greater than... When two features are related, a node clustering algorithm in the relationship graph is used to perform feature aggregation during task decomposition. Density clustering is employed for aggregation, and semantic subsets are divided based on the cluster centers of the similarity matrix. Each semantic subset corresponds to a candidate task space. An average fusion calculation is performed on the feature vectors within each semantic subset to obtain the subset center vector. The center vector is used to represent the semantic representation features of the task subspace, and the set of all center vectors is defined as the task candidate target.

[0100] S203: Input the candidate task targets into the strategy decision layer, and calculate the probability distribution of the task targets based on the joint state representation of the current scene feature information and the historical task execution information.

[0101] In this embodiment, the strategy decision layer refers to the network structure used to establish a task target selection probability model based on input features. The joint state representation refers to the combination of state vectors composed of current scene feature information and historical task execution information, which is used to describe the task context state at the current interaction moment. The task target probability distribution refers to the set of probabilities of each candidate task being selected for execution output by the strategy decision layer.

[0102] Specifically, when inputting candidate task targets into the policy decision layer, the sequence of candidate task target vectors is first input into the input node of the policy decision layer, and the current scene feature information and historical task execution information are input simultaneously. The three types of inputs are connected along the feature dimension through data concatenation operations to form a joint input matrix. Where T represents the candidate target vector for the task, F represents the scene feature information, and H represents the historical task execution information, the joint input matrix is ​​input into the feature fusion unit, which performs a linear transformation operation on the input matrix using the weight matrix Wf. This transformation achieves input dimension compression and feature fusion. The fused feature representation is then input into the hidden layer, where it is activated by a non-linear activation function. A nonlinear mapping is performed on the feature relationships to obtain an intermediate feature vector S representing the joint state. This intermediate feature vector is then input into the task selection unit, which calculates the task target probability distribution based on the Softmax function. , where Pi represents the selection probability of the i-th candidate target and n is the total number of candidate tasks. This probability distribution reflects the relative importance of each task in the current joint state, and the obtained task target probability distribution serves as the input for the subsequent task sub-target determination stage.

[0103] S204: Select candidate targets from the task target probability distribution whose probability values ​​are greater than the preset task selection threshold, determine the corresponding task targets, and obtain task sub-target information.

[0104] In this embodiment, the task selection threshold refers to the lower probability limit parameter used to filter candidate task targets, which limits the selection range of task targets. The task sub-target information refers to the set of target parameters output by the policy network after the task target is determined, which describes the specific target characteristics and operational constraints of the robot performing the task.

[0105] Specifically, when selecting candidate targets whose probability values ​​are greater than a preset task selection threshold from the task target probability distribution, the probability value Pi corresponding to each candidate target is first read from the task target probability distribution vector and compared one by one with the preset task selection threshold Pth. The comparison operation is performed according to the judgment condition Pi>Pth. When this condition is met, the candidate target Ti is marked as a valid task target. The value of the preset task selection threshold is determined based on the statistical distribution of historical task execution results. The average probability of successfully executing tasks in multiple past decisions is used as the threshold benchmark, so that the threshold can adapt to the importance of tasks in different scenarios. After completing all probability comparisons, all the candidate targets marked as valid task targets are combined into a candidate set. Then, tasks in the candidate set are prioritized based on semantic similarity and spatial relevance, with the ranking determined by a weighted scoring function. Where Si represents the semantic correlation coefficient between tasks. and As a weighting factor, one or more tasks with the highest ranking are selected as the final task objectives based on the scoring results. After determining the corresponding task objectives, the semantic description information, spatial location parameters and constraints corresponding to the task objectives are extracted and combined to form task sub-objective information.

[0106] In one embodiment, such as Figure 3 As shown, in step S30, the task sub-objective information is input into the low-level execution network, action control instructions are generated based on the task sub-objective information, and interactive feedback information is collected during execution, including:

[0107] S301: Input the task sub-objective information into the instruction parsing layer of the lower-level execution network to parse and obtain the corresponding control vector information.

[0108] In this embodiment, the instruction parsing layer refers to the functional layer in the lower-level execution network used for semantic and parameter mapping of task sub-target information, and the control vector information refers to the multi-dimensional vector set used to describe the direction, magnitude and constraints of the execution action, calculated by the instruction parsing layer based on the target feature parameters in the task sub-target information.

[0109] Specifically, when inputting task sub-objective information into the instruction parsing layer of the lower-level execution network, the semantic labels, spatial coordinates, and operational constraint parameters in the task sub-objective information are first used as input variables and input to the input nodes of the instruction parsing layer. The input nodes encode the task sub-objective information into a structured input matrix according to a preset input format. Each column element corresponds to a feature dimension. After receiving the input matrix, the instruction parsing layer first encodes the semantic label information through the embedding mapping unit, and then embeds the discrete semantic labels through the embedding matrix. Convert to continuous vector representation This process is used to obtain dense feature representations in the semantic space. These dense features are then concatenated with spatial coordinate parameters from the task sub-target information, and matrix multiplication is performed through a linear mapping unit. Where pi is the target spatial location parameter, and W1 and b1 are the weight matrix and bias term, respectively, used to establish the association mapping between semantic features and spatial features. After obtaining the preliminary parsing results, the instruction parsing layer uses a normalization function. The output amplitude is standardized to keep the values ​​of different dimensions at a uniform magnitude. The standardized feature vector is input to the feature decoding unit. The decoding unit maps the information of each dimension to specific control variables, such as joint angle change, motion direction vector and operation force parameter, according to the feature mapping table. The final control vector information is generated through feature dimension arrangement and numerical concatenation.

[0110] S302: Obtain the robot's current state information, input the control vector information and the current state information into the action generation layer of the lower-level execution network, calculate the action output parameters based on the correspondence between the control vector information and the current state information, and generate action control commands.

[0111] In this embodiment, the action generation layer refers to the structural layer in the lower-level execution network that is used to fuse control vector information with current state information and output executable action parameters. The action output parameters refer to the numerical vectors calculated by the action generation layer that describe the robot's action amplitude, direction, and speed. The action control command refers to the robot control signal generated based on the action output parameters, which is used to drive the robot to perform specific actions.

[0112] Specifically, when acquiring the robot's current state information, the robot's built-in sensing units read the data outputs from the joint angle sensors, inertial measurement units, and position detection units to form a set of state vectors containing attitude angles, angular velocities, and end-effector positions and velocities. The state information is input to the data receiver of the lower-level execution network through a unified encoding interface. When the control vector information and the current state information are input to the action generation layer, the two types of inputs are concatenated into a composite input matrix in the vector dimension through a feature concatenation operation. C represents the control vector information. After concatenation, the data is input to the input node of the action generation layer. The action generation layer contains several linear computation units and nonlinear activation units. The linear computation units perform matrix multiplication operations. The nonlinear activation unit uses the hyperbolic tangent function. A nonlinear mapping is performed on the linear output to enhance the nonlinear expressiveness between the input and output. After obtaining the mapping result, weight normalization is performed to ensure the stability of the output value. Then, the action output parameters are calculated at the output node. The action output parameters are scaled by the output vector using a scaling factor matrix K, as shown in the formula. Where A represents the motion output parameter vector, the motion control command is finally generated based on the values ​​of each dimension in the motion output parameter. The command generation process is completed by the command encoder, which converts the motion amplitude, direction and speed parameters into the robot control signal format. The output motion control command is used to guide the robot to perform the corresponding motion operation.

[0113] S303: Sends motion control commands to the robot, collects robot operating status information and environmental change information during motion execution, aggregates and processes the collected operating status information and environmental change information to obtain interactive feedback information.

[0114] In this embodiment, the running status information refers to the dynamic data related to posture, speed, torque and position generated by the robot during the execution of actions; the environmental change information refers to the change data of conditions such as light, obstacle distance and ground friction in the external environment when the robot performs operations; the convergence processing refers to the processing method of uniformly integrating sensor data from different sources in the time and feature dimensions; and the interactive feedback information refers to the data set that can reflect the interactive state between the robot's actions and the environmental response after convergence processing.

[0115] Specifically, when motion control commands are sent to the robot, they are input as digital signals to the execution port of the robot control unit via a communication interface. After receiving the commands, the control unit parses each control field and maps the motion amplitude, direction, and speed parameters to the motor drive unit. The corresponding joint movements are achieved by adjusting the drive current. During the execution of the motion, when collecting robot running status information, the attitude sensor, angular velocity sensor, and position detection device output attitude angle data, angular velocity data, and end-effector position coordinate data, respectively. The data sampling module synchronously samples the sensor output execution time at a fixed sampling period t_s to generate a running status data sequence. At the same time, when collecting environmental change information, the distance sensor, light sensor, and temperature sensor output corresponding physical quantities. The signal preprocessing module performs noise reduction and amplitude correction operations on the raw signals to generate an environmental change data sequence. After the data acquisition is completed, the running status information and environmental change information are converged. The convergence processing achieves timestamp synchronization through a time alignment algorithm, and then combines the state vector and environment vector at the corresponding time in the data dimension using a feature concatenation method to form a joint feature matrix. Where R_t represents the running status information and Et represents the environmental change information, the joint matrix is ​​finally weighted and fused using a feature integration function, as shown in the formula: Weighting coefficient Based on the confidence level of each feature source signal, the fusion result is the interactive feedback information, which is used in the subsequent reward value calculation steps.

[0116] In one embodiment, such as Figure 4 As shown, in step S40, the reward value is calculated based on the interactive feedback information, the reward value is correlated with the scene complexity parameter, a dynamic reward shaping operation is performed, and an adjusted reward signal is generated, including:

[0117] S401: Analyze the interactive feedback information and extract feedback parameter information to characterize the task execution status from the interactive feedback information.

[0118] In this embodiment, feedback parameter information refers to a set of quantifiable parameters extracted by structured parsing of interactive feedback information, which is used to describe the task execution status characteristics of the robot, such as motion accuracy, trajectory deviation, energy consumption, and degree of completion during task execution.

[0119] Specifically, when parsing interactive feedback information, the interactive feedback information is first input into the parsing module. The parsing module performs a sequence reconstruction operation on the information stream according to the time sequence, and recombines the state and environmental data within the continuous time period into a time window sequence. To ensure data consistency during the parsing process, the state components within each time window are dimensionally divided, with attitude, velocity, displacement, torque, and external environment parameters stored separately. A structured template is used to define the field types and position indices. Feature extraction is then performed within each field, and the attitude field is calculated using a three-dimensional angle change calculation function. The attitude change is obtained. The velocity field is used to calculate the acceleration component using a difference algorithm. The displacement field is used to obtain the path offset using coordinate difference. The torque field is used to calculate the average power consumption from the instantaneous values ​​output by the sensors. The environmental field is used to calculate the rate of change of illumination and the rate of change of distance using mean smoothing filtering. After extracting features from each field, the results are combined to form a feature vector set. This feature vector set is the feedback parameter information used to describe the task execution status. The feedback parameter information is arranged according to the time index to form a sequence data structure, which is used in the subsequent reward value calculation stage.

[0120] S402: Calculate the reward value based on the values ​​of the feedback parameters and the preset evaluation rules.

[0121] In this embodiment, the preset evaluation rule refers to the numerical scoring criteria set for different types of feedback parameter information, which is used to map the feedback parameters to a unified evaluation range. The reward value refers to the numerical quantification result obtained after calculating the feedback parameter information according to the preset evaluation rule, which is used to reflect the robot's comprehensive performance in the task execution process.

[0122] Specifically, when calculating the reward value based on the values ​​of the feedback parameters and the preset evaluation rules, the feedback parameters are first categorized by type, and a parameter index table is established. These correspond to indicators such as motion accuracy, trajectory deviation, energy consumption, task completion, external disturbance response, and time efficiency. Then, based on the characteristics of each indicator, a corresponding evaluation function is selected. Motion accuracy is evaluated using a linear scoring function. The trajectory deviation is expressed using an inverse function. Energy consumption adopts an exponential decay function. The task completion rate is calculated using a proportional function. The external disturbance response uses a difference function. Time efficiency is achieved using a ratio function. , among which , and As preset parameters, Tstd represents the standard completion time, and Tact represents the actual time taken. The output of each evaluation function is normalized to ensure all scores fall within the interval [0,1]. Then, vectors are assigned according to their weights. The six rating results are weighted and summed using the following formula: R is the comprehensive reward value. Finally, a smoothing filter algorithm is used to perform a moving average on the reward value sequence to eliminate the interference of instantaneous fluctuations. The smoothed result is the final reward value, which is used for subsequent dynamic reward shaping operations.

[0123] S403: Normalize the reward value and the scene complexity parameter to obtain the normalized result.

[0124] In this embodiment, the scene complexity parameter refers to a numerical indicator used to quantify the dynamics and interference level of the environment in the interactive scene, and the normalization result refers to the standardized output data obtained by numerically mapping and proportionally adjusting the reward value and the scene complexity parameter.

[0125] Specifically, when normalizing the reward value and the scene complexity parameter, the reward value sequence is first read from the data storage unit. With the corresponding scenario complexity parameter sequence The two sets of data are mapped one-to-one using time indices. Then, the statistical boundary values ​​for the reward value and the scene complexity parameter are calculated, including the maximum, minimum, and mean values. The calculation formulas are as follows: , , ,as well as , , After obtaining the statistical boundary, a linear normalization operation is performed on the reward value, calculated using the following formula: This is used to map the reward value to the interval [0,1]. The scene complexity parameter is normalized in the same way to obtain... After obtaining two sets of normalized data, a standardized matching structure is established between the two using a numerical fusion function, and a proportional balance coefficient is adopted. The two are weighted and fused together, and the calculation formula is as follows: Where Ni represents the normalized result at time i, and the scaling factor is... The value is determined based on the sensitivity of reward changes to complexity changes in historical training samples.

[0126] S404: Based on the normalization result, establish the correspondence between the reward value and the scene complexity parameter, perform dynamic reward shaping operation according to the correspondence, adaptively adjust the reward value, and generate the adjusted reward signal.

[0127] In this embodiment, dynamic reward shaping refers to the process of real-time correction and proportional adjustment of the reward value based on the numerical correspondence between the reward value and the scene complexity parameter. The adjusted reward signal refers to the corrected reward data calculated and output by the dynamic reward shaping operation and used as input for the subsequent optimization stage.

[0128] Specifically, when establishing the correspondence between reward values ​​and scene complexity parameters based on the normalization results, the normalized values ​​of reward values ​​and scene complexity parameters are read from the normalized data, and a set of sample points is formed according to the corresponding time index. And establish the functional relationship between the two through a fitting algorithm. The coefficients a, b, and c are calculated using least squares to characterize the influence trend of scene complexity changes on the reward value. When performing dynamic reward shaping based on the established functional relationship, the current reward value is mapped to the function space, and the difference is calculated. The correction strength is controlled by an adaptive adjustment factor α, and the correction calculation is performed. ,in The value range is limited to [0,1] to ensure adjustment stability. After the correction is completed, the correction result is subjected to inverse normalization. The output Rfinal is the adjusted reward signal.

[0129] In one embodiment, such as Figure 5As shown, in step S50, based on the adjusted reward signal, meta-learning optimization is performed on the parameters of the high-level policy network and the low-level execution network. An optimization mechanism combining inner and outer loops is used to update the parameters, generating a meta-learning optimized policy model, including:

[0130] S501: Determine the parameter update target based on the adjusted reward signal.

[0131] In this embodiment, the parameter update target refers to deterministic target information calculated based on the adjusted reward signal, which guides the direction and magnitude of parameter adjustments for the high-level policy network and the low-level execution network.

[0132] Specifically, when determining the parameter update target based on the adjusted reward signal, the numerical information of the adjusted reward signal is first read. and compared with the preset reward benchmark value Compare and calculate reward bias This is used to characterize the degree of difference between the current policy performance and the baseline level. Subsequently, based on the current parameter states of the high-level policy network and the low-level execution network, a relationship function between reward changes and parameter changes is established. The sensitivity index of each parameter to reward changes is obtained by calculating the partial derivative of the reward function with respect to the parameters. After obtaining the sensitivity index, a normalization operation is performed to obtain the parameter weight allocation vector. The calculation formula is: This is used to represent the relative influence of each parameter in the current training phase. Based on the parameter weights and reward bias, the adjustment direction and magnitude of each parameter are calculated. The calculation formula is as follows: ,in The learning rate coefficient is used to control the overall update rate, and ultimately the adjustment direction and magnitude of all parameters are uniformly encoded into the parameter update target.

[0133] S502: Based on the parameter update target, perform an inner loop update on the parameters of the high-level policy network and the low-level execution network to obtain the inner loop update parameters.

[0134] In this embodiment, the inner loop update parameter refers to the intermediate parameter result obtained after multiple iterations of calculation based on the parameter update target within a fixed task sample range. It is used to characterize the short-term adaptability of the model parameters in a local task scenario.

[0135] Specifically, when performing an inner loop update on the parameters of the high-level policy network and the low-level execution network based on the parameter update objective, the network parameter set for the current training epoch is first loaded from the storage unit. And read the corresponding parameters to update the target information. The two sets of data are matched one-to-one through an index, and then a fixed task sample dataset is selected. As training samples for the inner loop, each parameter is updated according to the gradient descent rule, and the update formula is: ,in This represents the learning rate of the inner loop. This represents the gradient of the loss function with respect to the current parameters. This represents the influence coefficient of the parameter update target on the inner loop update, and the parameter change magnitude is calculated after each iteration. When the change amplitude of two consecutive iterations is lower than a preset threshold Stop updating the parameter when it is timed to repeat the above update process until all parameters converge. Convergence is determined based on the gradient magnitude. Less than the threshold Once the conditions are determined and all parameters have been updated, the updated parameter set is defined as the inner loop update parameter.

[0136] S503: Use the inner loop update parameters as input for the outer loop update, perform the outer loop update based on the changes in the adjusted reward signal, and obtain the outer loop update parameters.

[0137] In this embodiment, the outer loop update parameters refer to the set of parameters obtained after performing global optimization calculations with the inner loop update parameters as the initial state and combined with the trend of the adjusted reward signal. These parameters are used to characterize the overall generalization characteristics of the model at the cross-task level.

[0138] Specifically, when using the inner loop update parameters as input for the outer loop update, the inner loop update parameter set is first... The parameters are loaded into the optimization calculation module as initial parameters for the outer loop. When performing the outer loop update based on the changes in the adjusted reward signal, the reward signal values ​​under multiple consecutive training tasks are first recorded. By calculating the rate of change of rewards between adjacent tasks To obtain the reward change trend, the average gradient direction of each parameter across task scenarios is calculated using the following formula:

[0139] ,in This represents the loss function value on the k-th task dataset, where m is the number of outer loop tasks. After obtaining the average gradient direction, the comprehensive updated weights are calculated by combining the reward change rate. ,in and The balancing coefficient controls the ratio of the reward signal to the gradient contribution. The outer loop parameters are updated based on the comprehensive update weights, using the following formula:

[0140] ,in The outer loop learning rate is used. After each round of outer loop calculation, the variation amplitude of all parameters is normalized to ensure the numerical stability of gradients across tasks. After all updates are completed, the calculated parameter set is defined as the outer loop update parameters.

[0141] S504: Jointly adjust the update parameters of the inner loop and the outer loop to determine the optimized parameters.

[0142] In this embodiment, the optimized parameters refer to the final set of parameters obtained by joint weighted adjustment after comprehensively considering the convergence characteristics and global adaptability of the inner loop update parameters and the outer loop update parameters.

[0143] Specifically, when jointly adjusting the inner loop update parameters and the outer loop update parameters, the inner loop update parameter set is first read separately. Update parameter set with outer loop A one-to-one correspondence is established based on the parameter index, and then a difference vector is calculated for each set of corresponding parameters. And judge the convergence consistency between the inner and outer loops based on the difference magnitude, when If a parameter is marked as stable, it is marked as adjustable. A weighted fusion calculation is performed on all adjustable parameters using a joint weighting formula: , where λ is the joint adjustment weight coefficient, with a value range of [0,1], used to balance the contribution ratio of the inner and outer loop update results. During the calculation process, the weight coefficient is adaptively adjusted according to the parameter change rate, which is calculated by the formula λ = λfrac{|Delta λtheta_i|}{|λtheta_i^{out}|+|λtheta_i^{in}|}. When the change rate is large, the value of λ increases to increase the weight of the outer loop parameter in the adjustment. After the weighted fusion of all parameters is completed, a regularization operation is performed on the fusion result to prevent numerical bias. The regularization formula is:

[0144] To maintain the overall numerical stability of the parameters, the final set of all parameters that have been jointly adjusted and regularized is defined as the optimized parameters.

[0145] S505: Based on the optimized parameters, the policy model optimized by econometric learning is obtained.

[0146] In this embodiment, the policy model optimized by meta-learning refers to a policy network model that has task adaptability and cross-scenario generalization ability, formed by loading optimized parameters into the structure of the high-level policy network and the low-level execution network.

[0147] Specifically, when obtaining the policy model optimized by emaphore learning based on the optimized parameters, the structure definition files of the high-level policy network and the low-level execution network are first read from the storage module to establish a model computation graph and a parameter mapping table. The parameter mapping table records the storage location of each network node and its corresponding parameter. Then, the optimized parameters are written into the network according to the order of the mapping table, replacing the original parameter values ​​of the high-level policy network and the low-level execution network. The replacement process is achieved through matrix assignment operations, with matrix elements updating the weight matrix and bias vector in the network structure one by one. After the parameters are written, the integrity of the entire model is checked, including the matching of the weight matrix dimensions and the consistency of the number of bias terms. After passing the check, parameter binding is performed in the computation graph to fix the new parameter state as the computation benchmark of the current network. After the parameter binding is completed, a forward computation test is performed on the model, inputting validation sample data to check the model output dimension, numerical range, and activation layer response. After confirming that the computation process is correct, the model is saved as the policy model optimized by emaphore learning.

[0148] In one embodiment, such as Figure 6 As shown, in step S60, i.e., loading the policy model in the simulation environment, performing randomized training based on multi-distributed perturbations in the execution domain, and generating a target policy model optimized through simulation training, the following steps are taken:

[0149] S601: Load the strategy model into the simulation environment to create a virtual training scenario that corresponds to the actual interaction scenario.

[0150] In this embodiment, a virtual training scenario refers to a virtualized task space configured in a simulation environment based on the physical characteristics, spatial layout, and perception conditions of the actual interactive scenario, used for executing strategy training and testing.

[0151] Specifically, when loading the strategy model into the simulation environment, the simulation process is first initialized in the computing resource node, and a data package containing the strategy model structure definition and parameter file is loaded. The simulation environment reads the model file content through the file parsing interface, parses the structural relationships and parameter mapping positions of each layer according to the model description language definition, and instantiates the model computation graph into a callable object of the simulation environment. Subsequently, a virtual training scene is built in the simulation environment. During the building process, the geometric model file and scene configuration file are imported according to the structural information of the actual interaction scene. The geometric model file contains the three-dimensional coordinate data of obstacles, operation targets, and ground boundaries in the scene, and the scene configuration file contains physical characteristic parameters such as lighting parameters, friction coefficient, environmental noise, and gravity direction. The simulation environment executes the scene construction function based on these inputs, divides the interactive area and obstacle area through a spatial meshing algorithm, and places virtual robot objects in the scene to simulate the actual interactive subject. In the virtual training scene, to ensure correspondence with the actual interaction scene, a parameter mapping operation is performed, and the physical parameters measured in the real scene are proportionally converted to the parameters set in the simulation environment. The conversion function is defined as follows: Where kp is a scaling factor used to keep the physical response in the simulation environment consistent with the actual scene. After all configurations are completed, the virtual training scene is saved as a running instance in the simulation environment.

[0152] S602: Set multiple distribution perturbation parameters in the virtual training scenario, perform multiple rounds of perturbation on the simulation environment based on the distribution perturbation parameters, execute training iterations of the policy model, and obtain the corresponding training results.

[0153] In this embodiment, the distributed perturbation parameter refers to a set of parameters used to describe the range of changes in physical properties, perception conditions, and dynamic disturbances in the simulation environment. This is used to introduce different environmental distribution differences during the training phase to enhance the adaptability of the policy model.

[0154] Specifically, when setting multiple distributed perturbation parameters in a virtual training scenario, a parameter sample set is first generated based on a preset perturbation distribution range. Each distributed perturbation parameter consists of three parts: parameter name, value range, and change step size. The parameter name includes friction coefficient, gravity direction angle, sensor noise amplitude, ambient light intensity, and target object mass, etc. The value range is determined by the actual measured scene statistical data. The change step size is set using a piecewise linear function to ensure the continuity of perturbation changes. After generating the parameter sample set, the simulation environment is initialized. Before each training round, a set of distributed perturbation parameters is randomly selected from the parameter set. The simulation environment performs perturbation configuration operations based on the selected parameters, affecting the friction matrix, gravity vector, and lighting rendering modules of the physics engine. Parameter replacement is performed to create environment instances with different physical properties. The policy model is then loaded into the perturbed simulation environment for training iterations. The training iterations are performed using a cyclic scheduling mechanism. In each iteration, the simulation environment adjusts the physical feedback signal in real time according to the distributed perturbation parameters. For example, when the gravity direction angle changes, the robot's motion trajectory changes accordingly. The model updates the parameter weights through the gradient backpropagation algorithm. During the multi-round perturbation training, the loss function value and policy output distribution of each round of training are recorded. The training results of each round are classified and stored according to the perturbation condition label. After training, all output information is aggregated and calculated. The training results under different perturbation distributions are fused and calculated using a weighted average method to obtain the corresponding training results.

[0155] S603: Evaluate the performance stability of the policy model under different perturbation distributions based on the training results, and adjust the training parameters to obtain the adjusted training parameters.

[0156] In this embodiment, performance stability refers to the degree of fluctuation in the output of the policy model under different distribution perturbation conditions. It is used to measure the model's ability to maintain consistent performance in diverse environments. Convergence adjustment refers to the process of gradually stabilizing the model's output under perturbation conditions by optimizing training parameters.

[0157] Specifically, when evaluating the performance stability of the policy model under different perturbation distributions based on the training results, the policy output information and loss function values ​​corresponding to each perturbation condition are first extracted from the training results. A correspondence table between perturbation labels and output data is established. Training records with the same type of perturbation are aggregated into data groups, each containing the mean and variance of the loss from several training iterations. Subsequently, a performance stability index is calculated for each data group. The performance stability index is expressed by the formula...

[0158] The calculation yielded that, among which This represents the output variance under the i-th perturbation distribution. This indicates that the output is the mean. To prevent the use of tiny constants with a denominator of zero, the closer Si is to 1, the more stable the performance under the perturbation distribution. After obtaining the performance stability indices corresponding to all perturbation distributions, global statistics are performed to calculate the overall stability mean and standard deviation. Based on the statistical results, a set of perturbation distribution labels with lower stability is identified. For the training parameters corresponding to these perturbation distribution labels, a convergence adjustment operation is performed. The convergence adjustment is carried out using the gradient decay method for each parameter. According to the formula: Update, among which For learning rate, This parameter represents the gradient value. If the gradient direction is consistent across multiple consecutive iterations after the update, the learning rate is dynamically reduced to accelerate convergence. After all parameters are updated, the mean squared error of the training parameter set is verified. When the verification result is less than the preset convergence threshold, the current parameter set is determined to be the adjusted training parameter set.

[0159] S604: Based on the adjusted training parameters, update the policy model to obtain the target policy model optimized through simulation training.

[0160] In this embodiment, the target policy model optimized by simulation training refers to the model structure formed after replacing the adjusted training parameters in the policy model and verifying it through simulation training, which is used to execute control tasks in subsequent actual interaction scenarios.

[0161] Specifically, when updating the policy model based on the adjusted training parameters, the original policy model structure file is first loaded into the computing environment. The parameter mapping tables for each layer in the model are then read. These mapping tables record the parameter indices and storage locations of each network node. Subsequently, the adjusted training parameters are written into the policy model according to the index order of the mapping tables, replacing the corresponding original parameter values. This replacement operation is performed using matrix assignment to ensure that the dimensions of the weight matrix and bias vector are consistent. After the parameter replacement is complete, a structural consistency check is performed on the policy model. This check includes the matching of inter-layer connections, input / output tensor dimensions, and the number of parameters. If the check results meet the preset requirements... After the consistency condition is met, the parameter solidification operation is performed to fix the newly written parameter state as the effective calculation parameters of the current model. Then, the updated strategy model is subjected to simulation verification test. The simulation verification is completed by running a round of perturbationless training in a virtual training scenario and recording the difference between the output value and the expected value. The convergence degree of the model after updating the parameters is calculated using mean square error. When the mean square error is less than the preset verification threshold, it is determined that the model has completed the effective update. Finally, the verified model is saved as the target strategy model optimized by simulation training. The target strategy model optimized by simulation training serves as the control basis for the robot to perform task operations in actual interactive scenarios.

[0162] In one embodiment, such as Figure 7As shown, in step S602, multiple distribution perturbation parameters are set in the virtual training scenario. Based on these parameters, the simulation environment is perturbed multiple times, and the training iteration of the policy model is performed to obtain the corresponding training results, including:

[0163] S6021: Determine the perturbation distribution range in the virtual training scenario, and generate distribution perturbation parameter information based on the perturbation distribution range.

[0164] In this embodiment, the disturbance distribution range refers to the set of numerical boundaries used to define the variable range of each physical property parameter in the simulation environment, and the distributed disturbance parameter information refers to the parameter dataset used to perform environmental disturbance operations, calculated based on the disturbance distribution range.

[0165] Specifically, when determining the perturbation distribution range in a virtual training scenario, a set of perturbation parameters is first established based on the types of physical attribute information contained in the simulation environment. This set includes attributes such as friction coefficient, gravity direction angle, sensor noise intensity, light intensity, and ground roughness. For each type of attribute parameter, statistical samples are extracted from measurement data in the actual interactive scenario, and their maximum, minimum, and standard deviations are calculated. The upper and lower boundaries of the perturbation are determined using the three-standard-deviation interval method. ,in This represents the mean of the attribute in the actual scenario. This represents the standard deviation, which will ultimately be... Defined as the perturbation distribution range of the corresponding attribute, after obtaining the entire perturbation distribution range, the parameter discretization operation is performed, dividing each continuous interval into several discrete points according to a set step size. The step size is determined based on the simulation calculation accuracy to ensure the controllability and continuity of perturbation changes. Based on the discretized perturbation distribution range, the distributed perturbation parameter information is generated.

[0166] S6022: Apply the distributed disturbance parameter information to the simulation environment, perform disturbance operations on the physical attribute information and perception condition information of the simulation environment, and obtain the disturbed simulation environment.

[0167] In this embodiment, the perturbed simulation environment refers to an environment instance formed in the simulation environment after adjusting the physical attribute information and perception condition information based on the distributed perturbation parameter information, which is used to simulate interactive scenarios under different perturbation conditions.

[0168] Specifically, when applying distributed disturbance parameter information to the simulation environment, the distributed disturbance parameter information file is first read in the parameter configuration module of the simulation environment. Based on the parameter identifier, the corresponding physical attribute information item and sensing condition information item are retrieved, and a parameter mapping table is established, creating a one-to-one correspondence between parameter names and attribute variables. Then, a disturbance replacement operation is performed to adjust the friction coefficient, gravity direction, surface roughness, and air resistance parameters in the physical attribute information. The disturbance adjustment uses a combination of additive and multiplicative disturbances. The additive disturbance is performed according to the formula: The calculation is performed, where P is the original parameter value. The offset value is randomly sampled from the perturbation distribution range, and the multiplicative perturbation is calculated according to the formula: Perform calculations, where As a perturbation scaling factor, the same perturbation calculation is performed on the light intensity, sensor noise amplitude, and ambient temperature parameters in the perception condition information to ensure that the perturbed perception input maintains physical consistency. After the parameter replacement is completed, the simulation environment makes the new perturbation parameters effective by overloading the initialization function of the physics engine, and recalculates the gravity field distribution matrix and friction matrix to update the scene dynamics model. After all perturbation configurations are applied, a physical stability verification is performed on the simulation environment. The verification steps include detecting whether objects in the scene are in a balanced state, whether the collision response is within the preset range, and whether the sensor signal remains within the readable range. When the verification result meets the stability conditions, the current environmental state is saved as the perturbed simulation environment, which serves as the input environment for subsequent strategy model training iterations.

[0169] S6023: Load the policy model in a perturbed simulation environment, perform multiple rounds of training iterations based on the distributed perturbation parameter information, and obtain the model training output information under each perturbation condition.

[0170] In this embodiment, the model training output information refers to the set of output data containing loss values, gradient vectors, and action decision sequences generated by the policy model after completing training iterations in a perturbed simulation environment. It is used to describe the learning response of the model under different perturbation conditions.

[0171] Specifically, when loading the policy model into the perturbed simulation environment, the policy model runtime instance is first initialized in the simulation computing node. The model parameters and computation graph are imported into the simulation environment's memory space. Parameter binding is completed through interface functions to ensure that the model can directly access environmental state information during simulation execution. Subsequently, multiple training iterations are initiated based on the distributed perturbation parameter information. Before each iteration, a set of parameter configurations is selected from the perturbation parameter set and loaded into the simulation environment, maintaining the physical attribute information and perception condition information in this perturbation state. Under this condition, the forward inference and backward update processes of the policy model are executed. In the forward inference stage, the state variable input returned by the simulation environment is received and the policy network computation action output is processed. In the backward update stage, the gradient is calculated based on the loss function, and the model parameters are updated. The loss function is defined as:

[0172] ,in For trajectory reward value, Let be the regularization coefficient and 'a' be the model action output. After each iteration, the loss value, action output, and gradient change information corresponding to the perturbation parameter are recorded and encapsulated as single-round training record data. After multiple iterations, the training record data under all perturbation conditions are classified and summarized. An index table is built according to the perturbation parameter label, and training records under the same perturbation condition are merged into a data group. Each data group represents the model training output information under that perturbation condition.

[0173] S6024: Aggregate the model training output information to obtain the training results.

[0174] In this embodiment, convergence processing refers to the process of performing aggregation calculations and parameter fusion on the model training output information under various perturbation conditions according to preset statistical rules, in order to generate unified data results that can characterize the overall training performance.

[0175] Specifically, when processing the model training output information, the model training output information corresponding to each perturbation condition is first extracted based on the perturbation label index. Each set of model training output information includes a loss value sequence, a gradient vector sequence, and an action output sequence. Then, a time step alignment operation is performed on each set of model training output information, using the training iteration epoch as the time index to ensure that the loss value and gradient information under the same epoch are comparable in calculation. After alignment, the loss value under each perturbation condition is averaged to obtain the average loss value corresponding to the perturbation distribution. Then, a global average calculation is performed on the average loss value under all perturbation conditions to reflect the overall training error level. At the gradient information level, a weighted fusion operation is performed on the gradient vectors under each perturbation condition. The weights are determined according to the stability index of the perturbation condition, and the weighted result is used as the global gradient representation. At the action output level, statistical analysis is performed on the action distribution under different perturbation conditions to extract the mean and variance of the action output to reflect the central tendency of the policy output. Finally, the average loss, fused gradient, and action statistics are combined to form a unified data structure. After the aggregation calculation is completed, a training result file is generated to obtain the training result.

[0176] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0177] In one embodiment, a reinforcement learning-based robot adaptive training device is provided, which corresponds one-to-one with the reinforcement learning-based robot adaptive training method described in the above embodiments. For example... Figure 8 As shown, this robot adaptive training device based on reinforcement learning includes a scene feature extraction module, a task sub-objective generation module, an action control and feedback module, a reward shaping module, a meta-learning optimization module, and a simulation training and execution module.

[0178] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A robot adaptive training method based on reinforcement learning, characterized in that, The aforementioned reinforcement learning-based adaptive training method for robots includes: The robot acquires environmental perception information in multiple interactive scenarios, inputs the environmental perception information into the scene encoder, and obtains scene feature information. Based on scene feature information, a high-level policy network is used to generate task sub-objective information; The task sub-objective information is input into the low-level execution network, action control instructions are generated based on the task sub-objective information, and interactive feedback information is collected during the execution process; The reward value is calculated based on the interactive feedback information, and the reward value is correlated with the scene complexity parameter. A dynamic reward shaping operation is performed to generate an adjusted reward signal. Based on the adjusted reward signal, meta-learning optimization is performed on the parameters of the high-level policy network and the low-level execution network. The parameters are updated using an optimization mechanism that combines inner and outer loops to generate a policy model optimized by meta-learning. The strategy model is loaded into the simulation environment, and randomized training is performed based on the multi-distributed perturbation execution domain to generate a target strategy model optimized by simulation training. The target strategy model is then loaded into the robot to control the robot to perform task operations in the actual interaction scenario.

2. The robot adaptive training method based on reinforcement learning according to claim 1, characterized in that, The generation of task sub-objective information based on scene feature information and using a high-level policy network includes: The scene feature information is input into the feature encoding layer of the high-level policy network to extract high-level semantic features; High-level semantic features are input into the task decomposition layer of the high-level policy network, and the interaction task is decomposed according to the semantic relevance of the scene to obtain several task candidate targets. The candidate targets for tasks are input into the strategy decision layer, and the probability distribution of the task targets is calculated based on the joint state representation of the current scene feature information and the historical task execution information. From the probability distribution of task objectives, select candidate task objectives with probability values ​​greater than a preset task selection threshold, determine the corresponding task objectives, and obtain task sub-objective information.

3. The robot adaptive training method based on reinforcement learning according to claim 1, characterized in that, The process of inputting task sub-objective information into the low-level execution network, generating action control commands based on the task sub-objective information, and collecting interactive feedback information during execution includes: The task sub-objective information is input into the instruction parsing layer of the lower-level execution network to parse and obtain the corresponding control vector information; The robot's current state information is obtained, and the control vector information and the current state information are input into the action generation layer of the lower-level execution network. The action output parameters are calculated based on the correspondence between the control vector information and the current state information, and action control commands are generated. Motion control commands are sent to the robot. During the execution of the motion, the robot's operating status information and environmental change information are collected. The collected operating status information and environmental change information are aggregated and processed to obtain interactive feedback information.

4. The robot adaptive training method based on reinforcement learning according to claim 1, characterized in that, The step of calculating the reward value based on interactive feedback information, associating the reward value with the scene complexity parameter, performing dynamic reward shaping, and generating an adjusted reward signal includes: The interactive feedback information is parsed to extract feedback parameter information that characterizes the task execution status. The reward value is calculated based on the values ​​of the feedback parameters and the preset evaluation rules; The reward value and the scene complexity parameter are normalized to obtain the normalized result; Based on the normalization results, a correspondence is established between the reward value and the scene complexity parameter. Dynamic reward shaping is performed according to the correspondence to adaptively adjust the reward value and generate the adjusted reward signal.

5. The robot adaptive training method based on reinforcement learning according to claim 1, characterized in that, The process involves performing meta-learning optimization on the parameters of the high-level policy network and the low-level execution network based on the adjusted reward signal. This optimization uses a combination of inner and outer loops to update the parameters, generating a meta-learning optimized policy model. This includes: The target parameters are updated based on the adjusted reward signal; Based on the parameter update objective, an inner loop update is performed on the parameters of the high-level policy network and the low-level execution network to obtain the inner loop update parameters; The inner loop update parameters are used as input to the outer loop update. The outer loop update is performed based on the changes in the adjusted reward signal to obtain the outer loop update parameters. The inner loop update parameters and outer loop update parameters are jointly adjusted to determine the optimized parameters; Based on the optimized parameters, the policy model optimized by eigenvalue learning is obtained.

6. The robot adaptive training method based on reinforcement learning according to claim 1, characterized in that, The process of loading a policy model in a simulation environment, performing randomized training based on multi-distributed perturbations in the execution domain, and generating a target policy model optimized through simulation training includes: Load the strategy model into the simulation environment to create a virtual training scenario that corresponds to the actual interaction scenario; In a virtual training scenario, multiple distribution perturbation parameters are set, and the simulation environment is perturbed multiple times based on the distribution perturbation parameters. The training iteration of the policy model is then performed to obtain the corresponding training results. The performance stability of the policy model under different perturbation distributions is evaluated based on the training results, and the training parameters are adjusted to obtain the adjusted training parameters. Based on the adjusted training parameters, the policy model is updated to obtain the target policy model optimized through simulation training.

7. The robot adaptive training method based on reinforcement learning according to claim 6, characterized in that, The process involves setting multiple distribution perturbation parameters in the virtual training scenario, applying multiple rounds of perturbation to the simulation environment based on these parameters, performing training iterations of the policy model, and obtaining the corresponding training results, including: In a virtual training scenario, determine the range of disturbance distribution and generate distribution disturbance parameter information based on the range of disturbance distribution; The distributed perturbation parameter information is applied to the simulation environment, and the physical attribute information and perception condition information of the simulation environment are perturbed to obtain the perturbed simulation environment. The policy model is loaded into the perturbed simulation environment, and multiple rounds of training iterations are performed based on the distributed perturbation parameter information to obtain the model training output information under each perturbation condition. The training output information of the model is aggregated to obtain the training results.

8. A robot adaptive training device based on reinforcement learning, characterized in that, The aforementioned reinforcement learning-based adaptive training device for robots includes: The scene feature extraction module is used to acquire environmental perception information of the robot in multiple interactive scenarios, and input the environmental perception information into the scene encoder to obtain scene feature information; The task sub-objective generation module is used to generate task sub-objective information based on scene feature information and using a high-level policy network. The motion control and feedback module is used to input task sub-objective information into the low-level execution network, generate motion control instructions based on the task sub-objective information, and collect interactive feedback information during the execution process. The reward shaping module is used to calculate the reward value based on the interactive feedback information, associate the reward value with the scene complexity parameter, perform dynamic reward shaping operation, and generate the adjusted reward signal. The meta-learning optimization module is used to perform meta-learning optimization on the parameters of the high-level policy network and the low-level execution network based on the adjusted reward signal. It updates the parameters using an optimization mechanism that combines inner and outer loops to generate a policy model optimized by meta-learning. The simulation training and execution module is used to load the policy model in the simulation environment, generate the target policy model optimized by simulation training based on randomized training of the multi-distributed perturbation execution domain, load the target policy model into the robot, and control the robot to perform task operations in the actual interaction scenario.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the reinforcement learning-based adaptive training method for robots as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the reinforcement learning-based adaptive training method for robots as described in any one of claims 1 to 7.

Images