A robot operation task meta-reinforcement learning decision method, system, device and storage medium

Abstract

The application relates to a robot operation task meta-reinforcement learning decision method, system, device and storage medium, the method comprising the following steps: encoding a natural language description into an initial neural skill; generating an individual hierarchical neural skill based on the initial neural skill; decoding the first hierarchical neural skill into a weight matrix and a bias vector of a first full connection layer of a policy network; configuring the first full connection layer in the policy network layer by layer by using the weight matrix and the bias vector to obtain the policy network; inputting a current environment state into the policy network, and updating the policy network based on an execution result by using an SAC algorithm. By encoding task semantics from a natural language description into a hierarchical neural skill sequence, task knowledge is used in a layered form to participate in policy network construction, so that task description information can pass through each layer of the policy network, and step-by-step mapping and structured expression from a high-level task intention to a low-level control parameter are realized.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method, system, device, and storage medium for reinforcement learning decision-making in robot operation tasks. Background Technology

[0002] In existing robotic manipulation tasks, different tasks may appear different, but they often share basic operational structures. For example, tasks such as closing a drawer, closing a door, and opening a window may all involve underlying operational mechanisms such as pushing, rotating, or contact holding. Meta-reinforcement learning aims to leverage these shared structures to enable robots to quickly adapt when faced with new objects, new targets, or new task variations.

[0003] However, existing technologies have at least the following drawbacks.

[0004] Existing skill-based hierarchical reinforcement learning methods typically define skills as low-level action primitives or sub-policies at the action level and rely on pre-defined intrinsic reward functions for skill discovery. Skills obtained in this way are often task-independent, making it difficult to efficiently absorb task semantic knowledge. Furthermore, the manual design of intrinsic reward functions is costly and has weak transferability, making it difficult to fine-tune the policy network to meet specific task requirements.

[0005] Existing context-based meta-reinforcement learning methods typically infer task representations from historical transition experiences through a context encoder, and then use these representations as policy condition inputs. This approach is susceptible to Markov decision process ambiguity, meaning that spurious correlations may form between historical experience and the real task, leading to biases in policy learning and making it difficult to stably distinguish heterogeneous tasks with similar state transitions but different task objectives.

[0006] Existing language-conditional reinforcement learning methods often use a simple concatenation of language embedding vectors and environmental states before inputting them into the policy network. Language semantics mainly participate in decision-making as additional state features, failing to form a direct mapping relationship between "task semantics and network layer parameters". This results in limited ability of task knowledge to shape the internal parameter structure of the policy network.

[0007] Traditional strategy network parameters are usually updated explicitly through gradient optimization. There is no clear and interpretable correspondence between network parameters and task semantics, making it difficult to form reusable and composable hierarchical parameter templates. This limits the robot's generalization efficiency in multi-task, sparse reward, and task description changing scenarios.

[0008] During the deployment phase, if each new task requires relearning the complete strategy or re-inferring context variables, it will increase the cost of sample collection and online adaptation, which is not conducive to the rapid switching and stable execution of robots in real-world scenarios such as industrial assembly, service robots, warehousing and logistics, and precision grasping.

[0009] To address the aforementioned technical problems, this application provides a method, system, device, and storage medium for reinforcement learning decision-making in robot operation tasks. Summary of the Invention

[0010] The purpose of this invention is to provide a method, system, device, and storage medium for reinforcement learning decision-making in robot operation tasks, in order to solve the technical problems mentioned in the background art. This purpose is achieved through the following technical solutions: A reinforcement learning decision-making method for robot operation tasks includes the following steps: Step S1. Receive the natural language description of the robot's operation task. Natural language description Encoding as initial neural skills ; Step S2. Based on initial neural skills Generate according to the hierarchical order of the policy network. Level of neural skills , ; Step S3. [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Level of neural skills Decoding into the policy network A fully connected layer weight matrix and bias vector ; Step S4. Using the weight matrix and bias vector The first layer in the configuration policy network A fully connected layer , The policy network is obtained; Step S5. Set the current environment state. Input policy network, output action control signal ; Step S6. Execute the motion control signal Get the next environment state and external rewards and will Store in the experience replay pool ; Step S7. Based on the experience replay pool The policy network is updated using the SAC algorithm.

[0011] Furthermore, hierarchical neural skills in step S2 The generation method is as follows:

[0012] In the formula, For semantic encoding model, These are the network parameters for the semantic coding model; For neural skill generation models, These are the network parameters for the neural skill generation model.

[0013] Furthermore, the semantic coding model is one of the following: MiniLM model, BERT model, RoBERTa model, Sentence-BERT model, DistilBERT model, TinyBERT model, CLIP text encoder, T5 encoder, GPT-type text embedding model, or self-trained task text coding network.

[0014] Furthermore, the neural skill generation model is one of the following: Transformer Encoder, Transformer Decoder, Bidirectional Transformer, Self-Attention Network, GRU, LSTM, Temporal Convolutional Network, or MLP-Mixer.

[0015] Furthermore, the weight matrix in step S3 The method for obtaining it is as follows: right Perform a 3×3 convolution operation and reshape the convolution output to obtain the result similar to the first convolution. A fully connected layer Weight matrix matching connection dimensions ; bias vector in step S3 The method for obtaining it is as follows: Through the fully connected layer right Perform a linear mapping to obtain the first... Bias vectors of fully connected layers .

[0016] Furthermore, the current environmental state in step S5 This includes robot joint angles, joint velocities, end effector pose, gripper state, object pose, target position, force / tactile information, camera visual features, and language target embedding or multimodal fusion state; the motion control signals. This includes continuous motion, Gaussian policy distribution parameters, discrete motion probability, end pose, joint angle, joint velocity, joint torque, or gripper opening / closing amount.

[0017] Furthermore, the updated parameters of the policy network in step S7 include state value estimation. Target Q value Critics' losses Frame parameters Temperature parameters and target commentator parameters, where: State value estimation ; Target Q value ; critics' loss By minimizing renew ; By maximizing Update framework parameters ; By maximizing Update temperature parameters ,in The target entropy; Target commentator parameters Perform a soft update.

[0018] Furthermore, the SAC algorithm described in step S7 can be replaced by the DDPG algorithm, TD3 algorithm, PPO algorithm, A2C / A3C algorithm, DQN variant, offline reinforcement learning algorithm, or imitation learning combined with reinforcement learning algorithm.

[0019] Furthermore, it also includes: Step S8. Before testing, build the parameter recovery set. The parameter recovery set Used to store the weight matrix and bias vector For the same task description, the restored parameter recovery set is reused during task execution. .

[0020] A system for implementing any of the above-described robot operation task reinforcement learning decision-making methods includes: The task description input module is used to receive natural language descriptions of robot operation tasks. ; The semantic encoding module is used to encode natural language descriptions. Encoding as initial-level neural skills ; Neural skill generation module, used to generate neural skills based on initial neural skills Generate according to the hierarchical order of the policy network. Level of neural skills , ; The neural skill decoding module includes hierarchical neural skills. The corresponding neural skill decoders , Each neural skill decoder Used to the first Level of neural skills Decoding into the policy network Weight matrices of each fully connected layer and bias vector ; The policy network building module is used to utilize the decoded weight matrix. and bias vector Fully Connected Layer in Layer-by-Layer Configuration Policy Network , ; The interactive execution module is used to receive the environment status. It is then input into the constructed policy network, which outputs action control signals. and the motion control signal Send the command to a robotic arm or simulated robot for execution. The experience replay module is used to store the experience replay pool formed during robot interactions. ; The reinforcement learning optimization module is used for experience replay pool-based learning. The policy network is updated using the SAC algorithm.

[0021] A robot operation task reinforcement learning decision-making device, comprising: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement a robot operation task meta-reinforcement learning decision method as described above.

[0022] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a robot operation task meta-reinforcement learning decision-making method as described above.

[0023] The technical solutions provided in this application have at least the following technical effects or advantages: 1. This invention encodes task semantics from natural language descriptions into hierarchical neural skill sequences, enabling task knowledge to participate in the construction of policy networks in a hierarchical form. This solves the problem that traditional methods only concatenate task semantics with the state at the input layer and cannot deeply influence the representation capabilities of each layer of the network. This allows task description information to penetrate every layer of the policy network, achieving a hierarchical mapping and structured expression from high-level task intent to low-level control parameters.

[0024] 2. This invention utilizes the weight matrix obtained through decoding. and bias vector By constructing a policy network layer by layer, the network parameters are dynamically determined by the task description, rather than simply concatenating language vectors from state inputs or inferring task variables through a context encoder. For different task descriptions, the system can construct policy networks with different combinations of hierarchical parameters. These parameters are not a pre-fixed set of ordinary trainable parameters, but are dynamically configured by hierarchical neural skills triggered by the task description. This fundamentally changes the traditional paradigm of fixed policy network parameters, enabling the same network architecture to flexibly adjust its parameters according to differences in task descriptions, significantly enhancing adaptability to task variations and generalization performance.

[0025] 3. This invention uses only external rewards during training, without relying on manually designed intrinsic reward functions. In existing technologies, to guide agents in exploring sparse reward environments, carefully designed intrinsic reward functions are typically required. This not only increases the complexity of algorithm design but also means that the weights and forms of intrinsic rewards are highly task-dependent, requiring readjustment when transferring to new tasks. This invention, through a task-description-driven hierarchical parameter configuration mechanism, enables the policy network to naturally possess structured priors, allowing for effective learning in sparse reward scenarios without additional intrinsic rewards. This reduces reward design costs and improves task transfer applicability.

[0026] 4. This invention does not use a context encoder to extract task representations from historical transition trajectories, thus reducing policy bias caused by ambiguity in Markov decision processes and spurious correlations from historical experience. Existing meta-reinforcement learning methods generally employ context encoders to encode historical trajectories to infer task variables. However, due to partial observability, insufficient state representation, or accidental correlations in historical trajectories, the encoder may extract false information irrelevant to the real task, leading to systematic biases in the policy on new tasks. This invention directly generates policy network parameters from the task description ψ, and the decision-making process does not rely on historical trajectory encoding, fundamentally avoiding the above problems and improving the robustness of the policy when deployed on new tasks.

[0027] 5. In the meta-testing or deployment phase, this invention generates and caches the strategy layer recovery parameter set first for the test task description. For the same task description, there is no need to repeatedly execute the semantic encoding, neural skill generation, and decoding processes during task execution; the stored set of recovery parameters can be directly reused. This significantly improves the response speed when switching tasks. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of the decision-making method in an embodiment of this application. Detailed Implementation

[0030] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0031] Example 1

[0032] A robot operation task meta-reinforcement learning decision-making system, comprising: The task description input module is used to receive natural language descriptions of robot operation tasks. This description can be generated by user input, task library calls, or upper-level planning modules, such as "slide the plate into the cabinet," "press the button," or "turn the faucet clockwise."

[0033] In other implementations, the natural language task description can be in English, Chinese, or multiple languages, or it can be a task label, a structured task instruction, action target text, a text command generated by a task planner, or task semantic text output by a visual language model.

[0034] The semantic encoding module is used to encode natural language descriptions. Encoding as initial-level neural skills In this embodiment, the semantic encoding module internally loads a pre-trained MiniLM model.

[0035] In other implementations, the semantic encoding module may also be a BERT model, RoBERTa model, Sentence-BERT model, DistilBERT model, TinyBERT model, CLIP text encoder, T5 encoder, GPT-like text embedding model, or a self-trained task text encoding network; the semantic encoding module may have fixed parameters or may be jointly trained with the subsequent neural skill generation module.

[0036] Neural skill generation module, used to generate neural skills based on initial neural skills Generate I-level neural skills according to the hierarchical order of the policy network. (1≤i≤I). The neural skill generation module internally loads a neural skill generation model, which is preferably composed of a single Transformer Encoder layer.

[0037] In other implementations, the neural skill generation model may employ a Transformer Decoder model, a bidirectional Transformer, a self-attention network, a GRU model, an LSTM model, a temporal convolutional network, an MLP-Mixer, or other sequence modeling networks.

[0038] The neural skill decoding module includes I-level neural skills. The corresponding neural skill decoders (1≤i≤I). Each neural skill decoder Used to transfer neural skills at the i-th level Decoded as the weight matrix of the i-th fully connected layer of the policy network and bias vector .

[0039] The policy network building module is used to utilize the decoded weight matrix. and bias vector Fully Connected Layer in Layer-by-Layer Configuration Policy Network (1≤i≤I), so that the structural parameters of the policy network are dynamically determined by the task semantics through hierarchical neural skills.

[0040] The interactive execution module is used to receive the current environment state. It is then input into the constructed policy network, which outputs action control signals. and the motion control signal Send the command to the robotic arm or simulated robot for execution. The interactive execution module includes a driver interface for communicating with the robot, supporting various robot models and simulation environments, such as the Franka Panda robotic arm, UR series robotic arms, Gazebo simulation environment, MuJoCo simulation environment, etc.

[0041] The experience replay module is used to store the experience replay pool formed during robot interactions. Each experience is ( , , , , The quintuple form of ) where This refers to external rewards.

[0042] The reinforcement learning optimization module is used for experience replay pool-based learning. The module updates the policy network number using the SAC algorithm. This module implements the core optimization logic of the SAC algorithm, including state value estimation, target Q-value calculation, critic loss calculation, frame parameter update, temperature parameter update, and target critic soft update.

[0043] Example 2

[0044] like Figure 1 The robot operation task reinforcement learning decision-making method shown includes the following steps: Step S1. Receive the natural language description of the robot's operation task. Natural language description Encoding as initial neural skills ; Step S2. Based on initial neural skills Generate according to the hierarchical order of the policy network. Level of neural skills , 1≤i≤I; In this embodiment, hierarchical neural skills are neither action primitives nor predefined low-level control sub-policies, but rather multidimensional latent vectors used to characterize the parameter structure of a certain layer in the policy network. Each hierarchical neural skill... Encode high-level task abstractions related to robot tasks and establish a correspondence with the i-th network layer in the policy network.

[0045] Specifically, let the task be described in natural language as follows: The number of hierarchical neural skills is First, Input the semantic encoding module to obtain the initial neural skills. Then, the neural skill generation module generates neural skills according to the policy network hierarchy, layer by layer. As input, generate neural skills at the current level. This process can be represented as:

[0046] in, This represents a pre-trained MiniLM semantic encoding model. These are the network parameters for the semantic coding model; χ represents the neural skill generation model, where χ is the network parameter of the neural skill generation model.

[0047] The aforementioned sequential generation mechanism enables the prior semantic information in the task description to be passed layer by layer along the data propagation direction of the policy network, thereby forming a more stable one-to-one correspondence between hierarchical neural skills and policy network layers.

[0048] The neural skill generation unit employs a Transformer Encoder layer, whose self-attention mechanism can reorganize and propagate task-related information from the previous level neural skill, generating a current level neural skill that is more suitable for subsequent policy layer parameter decoding.

[0049] Step S3. [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full Level of neural skills Decoding into the policy network A fully connected layer weight matrix and bias vector ; For each level of neural skill In this embodiment, a corresponding neural skill decoder is set up. Neural Skill Decoder take over As input, the i-th fully connected layer of the output policy network. Required weight matrix and bias vector ,Right now:

[0050]

[0051] in, Represents the i-th neural skill decoder The parameters.

[0052] Neural Skill Decoder This includes weight generation branches and bias generation branches. The weight generation branch pairs... Perform a 3×3 convolution operation and reshape the convolution output to obtain the result connected to the i-th fully connected layer. Weight matrix matching connection dimensions The bias generation branch is passed through the fully connected layer. Perform mapping to obtain the i-th fully connected layer. bias vector .

[0053] Through the above decoding method, the high-level semantic information in the task description is no longer just part of the policy network input, but is transformed into low-level specific parameters that can be directly configured in the policy network layer, realizing a direct mapping from "task semantics" to "policy layer parameters".

[0054] Step S4. Using the weight matrix and bias vector The first layer in the configuration policy network A fully connected layer , The policy network is obtained; The policy network consists of I sequentially connected network layers. to Composition. In the i-th network layer, its connection weights are set to the values ​​obtained from neural skill decoding. Its neuron bias is set to the value obtained from neural skill decoding. .

[0055] Step S5. Set the current environment status. Input policy network, output action control signal ; The robot receives the current environmental state at time t. Afterwards, As input to the policy network, the action control signal is output after sequential forward propagation through each layer. The process is represented as:

[0056] Current environmental status This can include robot joint angles, joint velocities, end effector pose, gripper state, object pose, target position, force / haptic information, camera visual features, language target embedding, or multimodal fusion state. Motion control signals. It can be a continuous control quantity, or it can be converted into discrete motion commands, target pose of the robotic arm end effector, joint angular velocity, joint torque, gripper opening and closing commands, or other robot-executable control quantities, depending on the task scenario.

[0057] The core of this embodiment lies in the fact that the policy network parameters are not a pre-fixed set of ordinary trainable parameters, but are dynamically configured by hierarchical neural skills triggered by the task description. Therefore, for different task descriptions, the system can construct policy networks with different combinations of hierarchical parameters, thereby enhancing its adaptability to task changes.

[0058] Step S6. Execute the motion control signal Get the next environment state and external rewards and will Store in the experience replay pool ; Step S7. Based on the experience replay pool The policy network is updated using the SAC algorithm.

[0059] By combining hierarchical neural skill generation, neural skill decoding, and policy network construction, the overall decision function of this invention can be expressed as:

[0060] in, This represents the semantic encoding module, the neural skill generation module, and the decoders for each neural skill. The set of parameters, Represented in natural language description A frame-wide function with parameter 𝜃 as a condition.

[0061] The reinforcement learning optimization module uses the SAC algorithm and sets up a bi-objective critic network. and the dual critics network This reduces overestimation of the soft Q value and stabilizes the training process.

[0062] Among them, state value estimation Target Q value Critics' losses Frame parameters Temperature parameters The optimization methods for the target reviewer parameters are as follows: State value estimation , This represents the overall function of the frame that takes 𝜃 as a parameter. Parameters for a bi-objective critic network; Target Q value , For the reward function, This is the discount factor; critics' loss By minimizing renew , For the parameters of the dual-critic network; By maximizing Update framework parameters ; By maximizing Update temperature parameters ,in The target entropy; Target commentator parameters Perform a soft update.

[0063] The above optimization relies solely on external rewards and does not require the design of an additional internal reward function. Furthermore, it avoids inferring task variables from historical trajectories using a context encoder, thus reducing the risk of task ambiguity caused by spurious correlations in historical transitions. For the external rewards used in this reinforcement learning optimization, in a sparse reward environment, the reward upon task completion can generally be set to +1, while the reward at other times can be 0. Alternatively, task-related stage rewards can be set according to the task's needs.

[0064] In other embodiments of this application, the number of hierarchical neural skills I can be equal to, greater than, or less than, the number of policy network layers. When the two are not equal, parameters can be generated for the policy layer through shared decoders, parameter aggregation, interpolation, pooling, or multi-skill fusion.

[0065] As another implementation of this application, the neural skill dimension can be set to 32, 64, 128, 256, 512 or other dimensions according to the task complexity; different levels of neural skills can use the same dimension or different dimensions.

[0066] In other embodiments of this application, the 3×3 convolution in the weight generation branch can be replaced with 1×1 convolution, 5×5 convolution, multi-scale convolution, deconvolution, fully connected network, low-rank matrix factorization, tensor factorization, super network, attention mapping or a combination thereof; the bias generation branch can adopt fully connected layer, convolutional layer, attention pooling or normalized mapping.

[0067] As another embodiment of this application, the policy network can be a fully connected layer, a convolutional layer, a graph neural network layer, a Transformer layer, a gating layer, a normalization layer, or a combination thereof.

[0068] As other embodiments of this application, the SAC algorithm can be replaced by the DDPG algorithm, TD3 algorithm, PPO algorithm, A2C / A3C algorithm, DQN variant, offline reinforcement learning algorithm, imitation learning combined with reinforcement learning algorithm, or other algorithms that can optimize strategies based on environmental rewards. The dual-commentator network can be replaced by a single-commentator, multi-commentator, or distributed commentator structure.

[0069] As another implementation of this application, this application can be deployed in a simulation robot, industrial robotic arm, service robot, mobile operating robot, or cloud training / edge inference system.

[0070] Example 3

[0071] This embodiment discloses a meta-training method, including the following steps: Initialize the experience replay pool (at this time For the newly constructed empty set), training task distribution Semantic coding model Neural skill generation model Neural skill decoder set Policy network layer set Dual-critic networks and dual-objective critic networks. Task distribution. It is the data source and fundamental premise of the entire training process. It does not refer to a specific task, but rather to the distribution of all possible training tasks in the probability space.

[0072] From the distribution of training tasks Sample training tasks and obtain the corresponding natural language task descriptions. and reward function .

[0073] At each time step Read the current environment status The current environmental state As a potential state .according to Generate initial neural skills ;for to Through neural skill generation models Recursive generation of hierarchical neural skills .

[0074] For each hierarchical neural skills Input the corresponding neural skill decoder The weight matrix is ​​obtained. and bias vector And using the weight matrix and bias vector Configuration number Each policy network layer .

[0075] The latent state is input sequentially into each policy network layer, and the output of each layer is then... Finally, the motion control signal is obtained. .

[0076] Execute action control signal Get the next state and external rewards and will Store in the experience replay pool .

[0077] If the training update condition is met (i.e., the neural network performance has not yet reached an ideal state), then the empirical replay pool is used. Update critic network and frame parameters Temperature parameters And the network parameters of the target commentator.

[0078] Example 4

[0079] This embodiment discloses a meta-testing / deployment method, including the following steps: Obtain natural language descriptions of test tasks or tasks to be deployed. .

[0080] Initialize parameter recovery set (at this time The newly constructed empty set is used to store the network parameters obtained from subsequent decoding. .

[0081] according to Generate initial neural skills And generate recursively through a neural skill generation model to .

[0082] Each Input the corresponding neural skill decoder , obtain parameters And store all parameters into the parameter recovery set. .

[0083] At each test time step Read environment status Input it into the parameter recovery set Configured policy network.

[0084] Policy network output actions The robot performs the action and enters the next state.

[0085] For the same task description, the stored set of recovery parameters can be reused during task execution. This reduces the overhead of repeated generation and decoding. The recovery parameter set F during the testing phase can be generated in real time or pre-generated offline and stored.

[0086] Example 5

[0087] This embodiment provides a robot operation task reinforcement learning decision-making device, including: One or more processors; A storage device for storing one or more programs and an experience playback pool D; When one or more programs are executed by one or more processors, the one or more processors implement the robot operation task element reinforcement learning decision method as described in Example 2.

[0088] The processor can be one or more of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), or an application-specific integrated circuit (ASIC). In this embodiment, to ensure the computational efficiency of neural network inference and reinforcement learning training, a heterogeneous computing architecture of CPU and GPU is preferably adopted, wherein the CPU is responsible for control logic and data processing, and the GPU is responsible for forward propagation and gradient calculation of the policy network.

[0089] The storage device includes non-volatile memory and volatile memory. Non-volatile memory is used for long-term storage of program code, model parameters, and parameter recovery set F; volatile memory is used for runtime caching of policy network parameters, experience replay pool data, and intermediate calculation results.

[0090] The device can be specifically implemented as an industrial robot controller, a mobile robot onboard computing platform, a cloud inference server, or an edge computing device. When implemented as an industrial robot controller, the device is integrated with the robot body and communicates with the servo drive via an industrial fieldbus. When implemented as a cloud inference server, the device receives task descriptions and environmental states via a network, calculates the actions, and then transmits them back to the robot's actuators via the network.

[0091] The device also includes a communication interface for receiving natural language descriptions ψ from a host computer or remote server, and / or sending motion control signals to the robot actuators. The communication interface can be an Ethernet interface, WiFi module, Bluetooth module, CAN bus interface, or RS-485 serial interface, depending on the deployment scenario.

[0092] The storage device also stores semantic coding model parameters. Parameters of the neural skill generation model and parameters of each neural skill decoder (1≤i≤I). The processor automatically loads the above parameters to construct the complete policy network when executing the program.

[0093] Example 6

[0094] This embodiment provides a computer-readable storage medium storing a computer program or computer-executable instructions thereon. When the computer program or computer-executable instructions are executed by one or more processors, they implement the robot operation task element reinforcement learning decision-making method as described in Embodiment 2.

[0095] Computer-readable storage media can be any of the following: read-only memory (ROM), random access memory (RAM), disk, optical disk, USB flash drive, portable hard drive, solid-state drive (SSD), embedded multi-media card (eMMC), or server cloud storage space.

[0096] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0097] Scenario 1: In a robot manipulation task under the Meta-World ML1 protocol, the robot environment uses the Sawyer robotic arm simulation platform. The tasks include Plate-Slide, Handle-Press, Button-Press, Faucet-Close, Door-Close, and Door-Unlock. Task descriptions are input in natural language, such as "Slide a plate into a cabinet," "Press a handle down," "Press a button," "Rotate the faucet clockwise," "Close a door with a revolving joint," and "Unlock the door by rotating the lock counter-clockwise." This scenario is used to verify the generalization ability of the policy to new objects or target variations within the same task.

[0098] Scenario 2: In the Panda Gym environment, the robot uses the Franka Emika Panda robotic arm simulation platform. Tasks include PandaReach, PandaPush, and PandaPickAndPlace. PandaReach involves the end effector reaching the target position, PandaPush involves pushing a cube to the target position on the table, and PandaPickAndPlace involves moving the cube to the target position on the tabletop. This scenario is used to verify the applicability of the present invention in multi-objective sparse reward tasks.

[0099] Scenario 3: The preferred implementation parameters are shown in the table below. Those skilled in the art can make equivalent adjustments to the following parameters based on the robot's state dimension, action dimension, task complexity, and computing resources.

[0100]

[0101] Scenario 4: For training hardware, a single NVIDIA GeForce RTX 3090 GPU and an Intel Xeon Platinum 8358P CPU can be used for training and testing. The evaluation metric can be the average success rate, which measures the stability of the robot in completing the task under multiple random initializations or multiple test rounds.

[0102] Under the Meta-World ML1 protocol, this invention achieved an average success rate of 100.00% in both the meta-training and meta-testing phases for the Handle-Press, Button-Press, Facet-Close, and Door-Close tasks; a meta-training success rate of 96.67% and a meta-testing success rate of 90.00% for the Plate-Slide task; and a meta-training success rate of 93.33% and a meta-testing success rate of 100.00% for the Door-Unlock task. These results demonstrate that this invention, through its hierarchical neural skill parameter decoding mechanism, can achieve strong generalization ability across different operational tasks.

[0103]

[0104] In the Panda Gym multi-objective sparse reward task, this invention achieves an average success rate of 100.00% in the Reach task, 95.33% in the Push task, and 91.33% in the PickAndPlace task. These results demonstrate that this invention is not only applicable to meta-learning tasks but can also be extended to sparse reward multi-objective robot operation tasks.

[0105]

[0106] In the ablation embodiment of the neural skill generation unit, the success rate of the Plate-Slide task meta-test was 50.00% without a neural skill generation model, 26.67% with GRU, 80.00% with LSTM, and 90.00% with TransformerEncoder. This result indicates that the preferred TransformerEncoder-type neural skill generation model is beneficial for the propagation of task-related information between hierarchical neural skills.

[0107]

[0108] In the ablation study of policy network depth, when using a 2-layer policy network on the Door-Close task, the average success rate of meta-training and meta-testing was 100.00%; while the success rates of 3-layer, 4-layer, and 5-layer policy networks decreased. This result indicates that a 2-layer policy network is the preferred structure in this embodiment, but the present invention does not limit the number of policy network layers and can adjust them according to the task and action space complexity.

[0109]

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A reinforcement learning decision-making method for robot operation tasks, characterized in that, Includes the following steps: Step S1. Receive the natural language description of the robot's operation task. The natural language description Encoding as initial neural skills ; Step S2. Based on initial neural skills Generate according to the hierarchical order of the policy network. Level of neural skills , ; Step S3. [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Level of neural skills Decoding into the policy network A fully connected layer weight matrix and bias vector ; Step S4. Using the weight matrix and bias vector The first layer in the configuration policy network A fully connected layer , The policy network is obtained; Step S5. Set the current environment state. Input the policy network and output the action control signal. ; Step S6. Execute the motion control signal Get the next environment state and external rewards and will Store in the experience replay pool ; Step S7. Based on the experience playback pool The policy network is updated using the SAC algorithm.

2. The robot operation task element reinforcement learning decision-making method according to claim 1, characterized in that, Hierarchical neural skills in step S2 The generation method is as follows: In the formula, For semantic encoding model, These are the network parameters for the semantic coding model; For neural skill generation models, These are the network parameters for the neural skill generation model.

3. The robot operation task element reinforcement learning decision-making method according to claim 2, characterized in that, The semantic coding model is one of the following: MiniLM model, BERT model, RoBERTa model, Sentence-BERT model, DistilBERT model, TinyBERT model, CLIP text encoder, T5 encoder, GPT-type text embedding model, or self-trained task text coding network.

4. The robot operation task element reinforcement learning decision-making method according to claim 2, characterized in that, The neural skill generation model is one of the following: Transformer Encoder, Transformer Decoder, Bidirectional Transformer, Self-Attention Network, GRU, LSTM, Temporal Convolutional Network, or MLP-Mixer.

5. The robot operation task element reinforcement learning decision-making method according to claim 1, characterized in that, The weight matrix in step S3 The method for obtaining it is as follows: right Perform a 3×3 convolution operation and reshape the convolution output to obtain the result similar to the first convolution. A fully connected layer Weight matrix matching connection dimensions ; bias vector in step S3 The method for obtaining it is as follows: Through the fully connected layer right Perform a linear mapping to obtain the first... Bias vectors of fully connected layers .

6. The robot operation task element reinforcement learning decision-making method according to claim 1, characterized in that, The current environmental state described in step S5 This includes robot joint angles, joint velocities, end effector pose, gripper state, object pose, target position, force / tactile information, camera visual features, and language target embedding or multimodal fusion state; the motion control signals. This includes continuous motion, Gaussian policy distribution parameters, discrete motion probability, end pose, joint angle, joint velocity, joint torque, or gripper opening / closing amount.

7. The robot operation task element reinforcement learning decision-making method according to claim 1, characterized in that, The updated parameters of the policy network in step S7 include state value estimation. Target Q value Critics' losses Frame parameters Temperature parameters and target commentator parameters, where: State value estimation ; Target Q value ; critics' loss By minimizing renew ; By maximizing Update framework parameters ; By maximizing Update temperature parameters ,in The target entropy; Target commentator parameters Perform a soft update.

8. The robot operation task element reinforcement learning decision-making method according to claim 1, characterized in that, The SAC algorithm mentioned in step S7 can be replaced by the DDPG algorithm, TD3 algorithm, PPO algorithm, A2C / A3C algorithm, DQN variant, offline reinforcement learning algorithm, or imitation learning combined with reinforcement learning algorithm.

9. The robot operation task element reinforcement learning decision-making method according to claim 1, characterized in that, Also includes: Step S8. Before testing, build the parameter recovery set. The parameter recovery set Used to store the weight matrix and bias vector For the same task description, the restored parameter recovery set is reused during task execution. .

10. A system for implementing the reinforcement learning decision-making method for robot operation tasks according to any one of claims 1-9, characterized in that, include: The task description input module is used to receive natural language descriptions of robot operation tasks. ; The semantic encoding module is used to encode natural language descriptions. Encoding as initial-level neural skills ; Neural skill generation module, used to generate neural skills based on initial neural skills Generate according to the hierarchical order of the policy network. Level of neural skills , ; The neural skill decoding module includes hierarchical neural skills. The corresponding neural skill decoders , Each neural skill decoder Used to the first Level of neural skills Decoding into the policy network Weight matrices of each fully connected layer and bias vector ; The policy network building module is used to utilize the decoded weight matrix. and bias vector Fully Connected Layer in Layer-by-Layer Configuration Policy Network , ; The interactive execution module is used to receive the current environment state. It is then input into the constructed policy network, which outputs action control signals. and the motion control signal Send the command to a robotic arm or simulated robot for execution. The experience replay module is used to store the experience replay pool formed during robot interactions. ; The reinforcement learning optimization module is used for experience replay pool-based learning. The policy network is updated using the SAC algorithm.

11. A robot operation task element reinforcement learning decision-making device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the robot operation task meta-reinforcement learning decision method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the robot operation task element reinforcement learning decision method as described in any one of claims 1-9.

Images