Robot smart operation method based on mean flow strategy and reinforcement learning and electronic equipment
By combining mean flow strategy and reinforcement learning, and employing a two-stage training method, the problems of long inference time and low success rate in robot dexterity operation are solved, achieving efficient and reliable real-time operation results.
Patent Information
- Application Number
- CN202511965443.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-24
- Publication Date
- 2026-02-10
AI Technical Summary
Existing robot dexterity methods suffer from long reasoning times and low success rates, making it difficult to meet real-time requirements. Furthermore, imitation learning lacks environmental interaction feedback and task success evaluation mechanisms, resulting in a high failure rate.
A two-stage training method based on mean-flow strategy and reinforcement learning is adopted. First, the core neural network of mean-flow strategy is trained by teleoperation data acquisition. Then, the operation strategy is optimized by combining exploration generative network and reinforcement learning algorithm. Environmental feedback is used to improve the robustness and success rate of the strategy.
It improves the efficiency and success rate of robot operation, reduces the action reasoning time by an order of magnitude, meets the requirements of real-time operation, and enhances the robustness and generalization ability in complex scenarios.
Smart Images
Figure CN121492041A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot intelligent control technology, and in particular to a robot dexterity operation method and electronic device based on mean flow strategy and reinforcement learning. Background Technology
[0002] Dexterous manipulation of robots is a core technology for achieving precise interaction between robots and their environment. Its goal is to enable robots to perform complex tasks such as grasping, placing, and assembling with the same dexterity as humans. With the widespread application of robotics in industry, healthcare, and service sectors, increasingly higher demands are being placed on the real-time performance and success rate of dexterous manipulation.
[0003] Currently, generative methods have attracted widespread attention in the field of dexterous robot manipulation due to their strong task adaptability. Typical examples include diffusion policy and flow matching. However, these generative methods have inherent drawbacks in the inference process: to generate the operation that meets the task requirements, multiple iterative calculations are required (usually around 10 iterations). The accumulation of iterations leads to a long inference time for the generation of action instructions, making it difficult to meet the real-time requirements of dexterous manipulation.
[0004] Furthermore, imitation learning methods, which focus on mimicking demonstration actions acquired through teleoperation and generate operational commands by learning the action mapping relationships in the demonstration data, lack a feedback mechanism for interaction with the environment. They cannot dynamically adjust actions based on environmental changes (such as object position shifts or surface condition changes), nor do they have an evaluation and optimization mechanism for "task success." This results in a low success rate, especially in scenarios not covered by demonstration data, where the failure rate increases significantly and generalization ability is weak. Therefore, there is an urgent need for a dexterous robot manipulation method that can balance inference speed and operational success rate to overcome the bottlenecks of existing technologies. Summary of the Invention
[0005] This invention addresses the aforementioned technical problems by overcoming multiple technical shortcomings in the field of robot dexterity manipulation, such as the long inference time and low success rate of generative methods (e.g., diffusion policy, flow matching) due to multiple iterations, the lack of task success rate guarantee and environmental adaptation mechanism in simple imitation learning, and the low sample efficiency and unstable training of traditional reinforcement learning. It provides a robot dexterity manipulation method and electronic device based on mean flow policy and reinforcement learning. Through two-stage training, it ensures the high efficiency of action inference and improves the robustness and success rate of the manipulation strategy through environmental feedback, ultimately solving the core technical defects of existing generative methods.
[0006] In a first aspect, the present invention provides a method for dexterous robot manipulation based on mean flow strategy and reinforcement learning, specifically including the following steps:
[0007] High-quality robot operation data is collected through teleoperation to train the core neural network in the mean flow strategy. The core neural network includes an image encoding network, a state encoding network, and an average velocity field prediction network.
[0008] An exploration-generative network is introduced on the basis of the trained average velocity field prediction network to construct a learnable Gaussian action distribution. Data is collected through the interaction between the robot and the environment. The collected data and reinforcement learning algorithms are used to jointly optimize the parameters of the average velocity field prediction network and the exploration-generative network to obtain the optimized robot operation strategy.
[0009] Based on the optimized average velocity field prediction network, the robot's motion commands are generated through single-step forward calculation to enable dexterous robot operation.
[0010] Furthermore, the image encoding network is used to extract feature information from visual observations of the environment, the state encoding network is used to fuse feature information of the robot's own state and the physical state of the environment, and the average velocity field prediction network is used to predict the average velocity of the generated robot action based on the feature information output by the image encoding network and the state encoding network.
[0011] Furthermore, when collecting high-quality robot operation data, the collected dataset is denoted as... ;in For robot visual observation, In robot state, For the corresponding action instructions, The total number of samples.
[0012] Furthermore, the specific steps for training the core neural network include:
[0013] With visual images As input, visual features are first extracted using a ResNet-18 convolutional neural network, then further encoded using a Transformer encoder, and finally, an average pooling operator is used. Obtain the feature vector of the image:
[0014] ;
[0015] robot status Extracting state features using a fully connected network ;
[0016] The time is positionally encoded using Sinusoidal Position Encoding, and then a fully connected network is used to extract the temporal features:
[0017] ;
[0018] Fusion of visual features With state characteristics Joint observation features were obtained Based on the average velocity definition of the mean flow strategy, the network is trained. Predict the average speed, where and For time variables, yes The state at any given moment, For network parameters; , For robot actions, .
[0019] Furthermore, the convolutional neural network is ResNet-18.
[0020] Furthermore, the loss function for training the average velocity field prediction network is:
[0021] ,
[0022] in ,in The instantaneous velocity of the action. To stop the gradient operation, and These are the partial derivatives of the average velocity field with respect to the state and time variables, respectively.
[0023] Furthermore, in the reinforcement learning training phase, constructing a learnable Gaussian action distribution specifically involves:
[0024] In the trained average velocity field prediction network Based on this, a new exploratory generative network was added. Output noise scale Define action distribution
[0025] ,
[0026] in, The actual actions can be obtained by sampling from the action distribution: .
[0027] Furthermore, the data collection through the interaction between the robot and the environment specifically includes:
[0028] Based on current observations get ,Will Send the command to the robot for execution and receive environmental feedback rewards. And the observation of the next moment. ;
[0029] Store and transfer tuples To the experience playback buffer .
[0030] Furthermore, methods for dexterous robot operation also include periodically extracting... Data from the middle sampling batch;
[0031] The SAC (Soft Actor-Critic) algorithm is used to jointly update the expected cumulative reward. and Parameters in:
[0032] ,
[0033] in, , For critic networks, For temperature coefficient,
[0034] Once reinforcement learning training is complete, deployment only requires forward computation. One-time generation action:
[0035] .
[0036] In a second aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any one of the first aspects.
[0037] The beneficial effects of this invention are:
[0038] (1) The robot dexterity operation method based on mean flow strategy and reinforcement learning provided by the present invention integrates the mean flow strategy and reinforcement learning hybrid strategy framework. For the first time, the generative modeling method of mean flow strategy is combined with reinforcement learning to construct a new hybrid strategy framework. The mean flow strategy is used to quickly generate high-quality initial motion trajectories as policy priors, and then the policy is fine-tuned and optimized online through reinforcement learning, thereby taking into account both generation efficiency and task success rate.
[0039] (2) Based on the mean flow policy optimization mechanism of reinforcement learning, this invention designs an exploration generation network module and uses the SAC algorithm for training, so that reinforcement learning can optimize the actions generated by the mean flow policy, significantly improving the robustness and generalization ability of the policy in complex contact and high-dimensional state space.
[0040] (3) Traditional generative methods (such as diffusion models) require multiple iterations during the inference stage, resulting in high latency and poor real-time performance. This invention generates an approximately optimal trajectory in one step through the mean flow strategy and combines it with a lightweight RL fine tuner. While ensuring a high success rate, it reduces the inference time by one order of magnitude, meeting the real-time dexterity operation requirements of robots. These innovations together solve the key bottlenecks in current generative dexterity operation methods, such as high inference latency, low task success rate, and sensitivity to environmental disturbances. This provides an efficient and reliable new paradigm for high-precision operation of robots in complex and unstructured scenarios. Attached Figure Description
[0041] Figure 1 This is a flowchart illustrating the robot dexterity operation method in a specific embodiment of the present invention.
[0042] Figure 2 This is a schematic diagram of the UNet1D network structure in the robot dexterity operation method of a specific embodiment of the present invention.
[0043] Figure 3 This is a schematic diagram of the task execution process in the embodiment (consisting of three parts). Detailed Implementation
[0044] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific implementation methods in the embodiments are detailed descriptions of the technical solution of this application, rather than limitations on the technical solution of this application. In the absence of conflict, the technical features of the embodiments of this application and the implementation methods can be combined with each other.
[0045] The robot dexterity manipulation method based on mean flow strategy and reinforcement learning provided by this invention is suitable for robot systems with high degrees of freedom dexterity hands to perform fine manipulation tasks (such as workpiece grasping, workpiece assembly, plugging and unplugging, etc.) in unstructured environments, and is especially suitable for real-time operation scenarios that need to balance inference efficiency and operation success rate.
[0046] In one specific embodiment, this invention proposes a two-stage training framework that organically integrates the efficient action generation capability of the mean-flow strategy with the feedback optimization mechanism of reinforcement learning: The first stage is the imitation learning training stage, in which high-quality robot operation data is collected through teleoperation to train the core neural networks (image encoding network, state encoding network, and average velocity field prediction network) in the mean-flow strategy, establishing an efficient mapping from environmental observation to operation actions, providing a good policy foundation for subsequent reinforcement learning; The second stage is the reinforcement learning training stage, in which an exploration generation network is introduced on the basis of the mean-flow strategy, a Gaussian distribution action sampling mechanism is constructed, reward signals are obtained through robot-environment interaction, interaction data is stored using an experience replay buffer, and the parameters of the average velocity field prediction network and the exploration generation network are updated periodically to achieve dynamic optimization of the strategy.
[0047] Through two-stage training, the efficiency of action reasoning is ensured, and the robustness and success rate of the operational strategy are improved through environmental feedback, ultimately solving the core technical defects of existing generative methods.
[0048] The technical solution of this invention includes an imitation learning training stage and a reinforcement learning training stage, and the overall framework is as follows: Figure 1 As shown, the core components and their interactions at each stage are as follows:
[0049] The core components include: image coding network, state coding network, average velocity field prediction network, and exploration generation network.
[0050] Among them, the image coding network is used to extract feature information from visual observations of the environment, the state coding network is used to fuse the robot's own state and the physical state features of the environment, the average velocity field prediction network generates the average velocity of the action based on the mean flow strategy, and the exploration generation network generates the noise term of the action.
[0051] The core of the imitation learning training phase is to train the core neural network of the mean-stream strategy using teleoperated data. The specific steps are as follows:
[0052] 1. Teleoperation Data Acquisition: Operators control the robot to perform target dexterous manipulation tasks (such as object grasping and precision assembly) via teleoperation devices, simultaneously acquiring multimodal data, including: environmental visual images (acquired through the robot's onboard camera), robot status data (joint angles, joint velocities), and manipulation commands (joint position movement commands). The acquired dataset is denoted as... in For robot visual observation, In robot state, For the corresponding action instructions, The total number of samples.
[0053] 2. Neural Network Training:
[0054] 2.1 Image Coding Networks: Based on visual images As input, visual features are first extracted using a ResNet-18 convolutional neural network, then further encoded using a Transformer encoder, and finally, an average pooling operator is used. Obtain the feature vector of the image: .
[0055] 2.2 State Coding: Transcoding the robot's state Extracting state features using a fully connected network .
[0056] 2.3 Time Encoding: First, time is encoded using Sinusoidal Position Encoding, then a fully connected network is used to extract time features.
[0057] .
[0058] 2.4 Mean Velocity Field Prediction Network: Integrating Visual Features With state characteristics Joint observation features were obtained Based on the average velocity definition of the mean flow strategy, the network is trained. Predict the average speed, where and For time variables, yes The state at any given moment, For network parameters; , For robot actions, The average velocity field prediction network adopts the UNet1D model, and the network structure is as follows: Figure 2 As shown.
[0059] 2.5 Training Loss Function:
[0060] ,
[0061] in ,in The instantaneous velocity of the action. To stop the gradient operation, and These are the partial derivatives of the average velocity field with respect to the state and time variables, respectively.
[0062] It should be noted that the reinforcement learning training phase, based on imitation learning training, optimizes the strategy through environmental feedback. The specific steps are as follows:
[0063] 1. Construct a learnable Gaussian action distribution
[0064] In the trained mean velocity field prediction network Based on this, a new exploratory generative network was added. Output noise scale Define action distribution
[0065] ,
[0066] in, The actual actions can be obtained by sampling from the action distribution: .
[0067] 2. Interactive data collection and playback
[0068] Based on current observations get ,Will Send the command to the robot for execution and receive environmental feedback rewards. And the observation of the next moment. ,
[0069] Store and transfer tuples To the experience playback buffer .
[0070] 3. Joint Strategy Optimization
[0071] Regularly from Data from the middle sampling batch,
[0072] The SAC (Soft Actor-Critic) algorithm is used to jointly update the expected cumulative reward. and Parameters in:
[0073] ,
[0074] in, , For critic networks, This is the temperature coefficient.
[0075] Once reinforcement learning training is complete, deployment only requires forward computation. One-time generation action:
[0076] .
[0077] In the two-stage training framework proposed in the aforementioned implementation, the imitation learning stage provides a good policy foundation for reinforcement learning, reduces the exploration space of reinforcement learning, and accelerates the policy convergence speed. At the same time, the experience replay mechanism makes full use of interactive data, improves data utilization efficiency, and accelerates training convergence speed.
[0078] Furthermore, the scheme employs a mean flow strategy for average velocity field prediction, combined with single-step Gaussian distribution sampling to generate action commands. This eliminates the need for multi-step iterative reasoning, significantly shortening the action generation time for robot dexterity operations, meeting real-time operation requirements, and demonstrating high reasoning efficiency.
[0079] This invention introduces an environmental feedback mechanism through reinforcement learning and uses reward signals to guide strategy optimization, thereby improving the robustness of action generation to environmental noise and uncertainty, and significantly increasing the success rate of operations in complex scenarios.
[0080] The framework proposed in this invention is applicable to tasks in any high-dimensional continuous motion space and has been verified on platforms such as 7-DoF robotic arms and dexterous hands, demonstrating strong versatility.
[0081] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned robot dexterity operation method based on mean flow strategy and reinforcement learning.
[0082] When implementing the foregoing embodiments, the following hardware environment configuration can be provided:
[0083] 1. Humanoid robot platform: It adopts a dual-arm 7-DOF (7DoF) humanoid robot, with each arm having 3 degrees of freedom in the shoulder, 1 degree of freedom in the elbow, and 3 degrees of freedom in the wrist, to meet the multi-posture adjustment requirements in assembly tasks; both hands are equipped with 6-DOF dexterous hands (the thumb has 2 independent degrees of freedom, and the other four fingers each have 1 degree of freedom); the head and each of the left and right wrists are equipped with a high-definition RGB camera.
[0084] 2. Teleoperation Equipment: A collaborative control kit consisting of Xsens motion capture equipment and Manus data gloves is used. After wearing the motion capture equipment and data gloves, the operator controls the robot's arm movements synchronously by using upper limb movements near the robot, controls the opening and closing of the dexterous hand by bending the fingers, and adjusts the operation by visually observing the robot's actual movements, realizing a "operation-observation-adjustment" closed loop to ensure accurate reproduction of teleoperation actions.
[0085] 3. Operating Environment and Workpieces: The operating table is a horizontal workbench (150cm × 80cm). A material frame is set on the table: workpiece A is on the left and workpiece B is on the right. All workpieces to be assembled are made of metal and are cylindrical in shape. The assembly task requires picking up workpiece A from the left material frame and workpiece B from the right material frame, then coaxially embedding workpiece A into the groove of workpiece B, and finally storing the assembled product in the middle material frame.
[0086] 4. Computing Equipment: ① Training-Interactive Computer: A workstation with two NVIDIA A6000 GPUs (48GB VRAM per card, PCIe 4.0 x16 interconnect between GPUs, Intel Xeon Gold 6426R CPU, 128GB DDR5 RAM, and 2TB NVMe SSD storage), serving three main functions: First, it handles remote operation dataset processing and offline neural network training (parallel computing with two cards improves training efficiency); second, it acts as a reward receiver for real-machine reinforcement learning, acquiring 1 / 0 reward feedback from operator input via keyboard; third, it receives real-time visual and state data transmitted from the robot, executes model inference, generates action commands, and sends them to the robot via Ethernet; ② Inference Platform: A Jetson AGX Orin 64G edge computing module (GPU computing power 275 TOPS, 8-core ARM Cortex-A78 CPU, 64GB RAM). The LPDDR5 is integrated inside the robot's torso and is responsible for receiving real-time image data from three cameras and joint status data, performing motion inference, and outputting control commands, ensuring an inference latency of ≤30ms to meet the real-time operation requirements on site.
[0087] During data acquisition, operators wearing Xsens motion capture equipment and Manus data gloves remotely manipulate the robot by directly observing its movements from nearby, completing the "dual-arm collaborative metal part assembly-storage task." The operation process is consistent with the previous steps (grabbing → lifting → aligning → assembling → storing). During data acquisition, images from the three cameras and joint status data (7 joint angles of both arms + 6 joint angles of the dexterous hand), along with remote manipulation commands (joint position / torque commands), are simultaneously transmitted to a 2-card A6000 workstation. The server stores and constructs a dataset with 200 valid samples.
[0088] In the image encoding network of the neural network structure, a "ResNet18 + Transformer encoder" architecture is adopted. A 640×480×3 RGB image is input, and the low-level feature map is extracted by ResNet18 (with the last fully connected layer removed). After flattening, the feature map is mapped to 256 dimensions through a linear projection layer and input to a 6-layer Transformer encoder (each layer contains one 8-head self-attention module and one feedforward network, with a 2048-dimensional hidden layer in the feedforward network), outputting a 512-dimensional feature vector. State encoding and temporal encoding are each implemented using 2 fully connected layers, with feature output dimensions of 64 and 256, respectively. The average velocity field prediction network adopts the UNet1D network structure. The exploration and generation network is implemented by 2 fully connected layers, with the feature output dimension equal to the dimension of the robot's actions.
[0089] Imitation learning training phase: Teleoperation is used to collect robot operation data and train the neural networks (image encoding network, state encoding network, and average velocity field prediction network) in the mean-flow strategy. The training process is based on a 2-card A6000 server and adopts a hybrid training mode of model parallelism and data parallelism, using the ADAM optimizer uniformly, with a total training step of 200,000 steps.
[0090] Basic training parameters: The batch size is fixed at 128, and the data is evenly distributed between the two GPUs (64 samples / batch per GPU) to ensure balanced load on both GPUs. The optimizer parameters are uniformly configured as follows: optimizer_type=ADAM, optimizer_lr=1e-4, optimizer_weight_decay=1e-6, optimizer_betas=(0.9, 0.999), optimizer_eps=1e-10.
[0091] Reinforcement learning and training phase:
[0092] Step 1: Based on the mean flow strategy, an exploration generation network is superimposed; the average velocity generated by the mean velocity field prediction network and the noise generated by the exploration generation network are used to form a Gaussian distribution, and action commands are sampled from the Gaussian distribution.
[0093] Step 2: Send the instructions to the robot for execution. After execution, give a reward (if the task is successful, the operator presses the "s" key on the keyboard and the reward is 1; if the task fails, the operator presses the "f" key on the keyboard and the reward is 0), and store the relevant data in the experience playback buffer.
[0094] Step 3: Every so often, sample a batch of interactive data from the experience replay buffer and use this data to adjust the parameters in the mean velocity field prediction network and the exploration generation network.
[0095] Two test scenarios were designed to evaluate the performance of the method. In all scenarios, a human reward was obtained by inputting "s / f" via the server keyboard. Task execution consists of three parts: workpiece grabbing, workpiece assembly, and finished product storage (e.g., ...). Figure 3 As shown in the image, from left to right, the steps are: workpiece grabbing, workpiece assembly, and finished product storage. Successful finished product storage indicates that the task has been successfully executed.
[0096] 1. Standard environmental test: Workpieces A and B are randomly placed in the material frame (without stacking) and there is no change in ambient light to evaluate basic operational performance.
[0097] 2. Illumination disturbance test: The brightness of the light above the workbench is randomly varied between 400 and 2000 lux, and there is local reflection on the surface of the workpiece and the material frame to evaluate the environmental robustness.
[0098] The test indicators include workpiece grabbing success rate, workpiece assembly success rate, and finished product storage success rate.
[0099] The test results (100 times each for standard / material frame position offset / lighting disturbance scenarios) are shown in Table 1 below.
[0100] Table 1
[0101] Test Scenario Workpiece gripping success rate Workpiece assembly success rate Finished product storage success rate Standard Environment 98% 95% 95% Light disturbance 96% 92% 92%
[0102] The implementation results show that, under the two test scenarios mentioned above, the success rates for workpiece grabbing, workpiece assembly, and finished product storage all remained above 92%. Furthermore, the model's time consumption during the inference phase... 28ms, meeting the requirements for real-time operation.
[0103] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.
Claims
1. A method for dexterous robot manipulation based on mean flow strategy and reinforcement learning, characterized in that, Includes the following steps: Robot operation data is collected through teleoperation to train the core neural network in the mean flow strategy. The core neural network includes an image encoding network, a state encoding network, and an average velocity field prediction network. An exploration-generative network is introduced on the basis of the trained average velocity field prediction network to construct a learnable Gaussian action distribution. Data is collected through the interaction between the robot and the environment. The collected data and reinforcement learning algorithms are used to jointly optimize the parameters of the average velocity field prediction network and the exploration-generative network to obtain the optimized robot operation strategy. Based on the optimized average velocity field prediction network, the robot's motion commands are generated through single-step forward calculation to enable dexterous robot operation.
2. The robot dexterity operation method according to claim 1, characterized in that, The image encoding network is used to extract feature information from visual observations of the environment, the state encoding network is used to fuse feature information of the robot's own state and the physical state of the environment, and the average velocity field prediction network is used to predict the average velocity of the generated robot action based on the feature information output by the image encoding network and the state encoding network.
3. The robot dexterity operation method according to claim 1, characterized in that, When collecting robot operation data, the collected dataset is denoted as... ;in For robot visual observation, In robot state, For the corresponding action instructions, The total number of samples.
4. The robot dexterity operation method according to claim 1, characterized in that, The specific steps for training the core neural network include: With visual images As input, visual features are first extracted using a ResNet-18 convolutional neural network, then further encoded using a Transformer encoder, and finally average pooling is applied. Obtain the feature vector of the image: ； robot status Extracting state features using a fully connected network ; The time is positionally encoded using Sinusoidal Position Encoding, and then a fully connected network is used to extract the temporal features: ； Fusion of visual features With state characteristics Joint observation features were obtained Based on the average velocity definition of the mean flow strategy, the network is trained. Predict the average speed, where and For time variables, yes The state at any given moment, For network parameters; , For robot actions, .
5. The robot dexterity operation method according to claim 4, characterized in that, The convolutional neural network is ResNet-18.
6. The robot dexterity operation method according to claim 4, characterized in that, The loss function for training the average velocity field prediction network is: ， in ,in The instantaneous velocity of the action, To stop the gradient operation, and These are the partial derivatives of the average velocity field with respect to the state and time variables, respectively.
7. The robot dexterity operation method according to claim 4, characterized in that, In the reinforcement learning training phase, the construction of a learnable Gaussian action distribution specifically involves: In the trained mean velocity field prediction network Based on this, a new exploratory generative network was added. Output noise scale Define action distribution ， in, The actual actions can be obtained by sampling from the action distribution: .
8. The robot dexterity operation method according to claim 7, characterized in that, The data collection through the interaction between the robot and the environment specifically includes: Based on current observations get ,Will Send the command to the robot for execution and receive environmental feedback rewards. And the observation of the next moment. ; Storage transfer tuples To the experience playback buffer .
9. The robot dexterity operation method according to claim 8, characterized in that, Also includes: Regularly from Data from the middle sampling batch; The SAC algorithm is used to jointly update the expected cumulative reward. and Parameters in: , in, , For critic networks, For temperature coefficient, Once reinforcement learning training is complete, deployment only requires forward computation. One-time generation action: 。 10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 9.