Multi-degree-of-freedom robot continuous action control method and system based on flow strategy

Abstract

This invention discloses a continuous motion control method and system for a multi-degree-of-freedom robot based on a streaming policy. The continuous motion control method includes a motion generation step to generate a final motion command, and drives the multi-degree-of-freedom robot to perform the corresponding action based on the final motion command. The motion generation step includes: A1: acquiring the current state observation information, current intermediate motion variables, and current flow time step information of the multi-degree-of-freedom robot; A2: using a velocity network with a sequence model structure to calculate the output velocity field based on the information acquired in A1; A3: acquiring initial noise samples, and iteratively executing steps A1 to A2 based on the initial noise samples to perform a motion update step, generating the final motion command. In each iteration, the velocity field is used to update the current intermediate motion variables. This invention avoids the problems of training instability and policy convergence difficulties caused by ill-conditioned gradients in multi-step flow expansion when training streaming policies using off-policy reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method and system for controlling the continuous motion of a multi-degree-of-freedom robot based on a streaming strategy. Background Technology

[0002] In the continuous motion control of multi-degree-of-freedom robots (such as humanoid bipedal robots, multi-legged crawling robots, and industrial robotic arms), reinforcement learning is an effective method for autonomously learning optimal control strategies through environmental interaction. Flow-based policies, as a generative model, can represent rich multimodal motion distributions and show potential in robot control.

[0003] However, combining streaming policies with sample-efficient off-policy reinforcement learning algorithms (such as SAC and TD3) presents significant challenges. Streaming policy action generation requires multiple "stream unrolling" steps, a process computationally equivalent to forward propagation of a deep residual recurrent neural network (RNN). When backpropagating this deep computational graph using the off-policy loss function to optimize the policy, the classic gradient explosion or vanishing problem in RNNs is easily triggered, leading to unstable training, ineffective policy convergence, and ultimately preventing the robot from learning stable, high-performance motion or operational control policies.

[0004] To circumvent this problem, existing technologies typically employ two compromise solutions: one is to use a surrogate objective to avoid differentiation through complete flow unrolling, but this leads to bias in the optimization objective; the other is to distill the streaming policy into a simple single-step policy, but this sacrifices the inherent multimodal action representation capability of the streaming policy. Therefore, there is an urgent need in this field for a technical solution that can achieve stable streaming policy training and end-to-end off-policy training.

[0005] The above background information is provided only to aid in understanding the concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above information was disclosed on the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a continuous motion control method and system for multi-degree-of-freedom robots based on streaming policies. This avoids the problems of training instability and policy convergence difficulties caused by gradient ill-conditioning (gradient explosion or vanishing) during multi-step flow expansion when training streaming policies using off-policy reinforcement learning.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, this invention discloses a continuous motion control method for a multi-degree-of-freedom robot based on a streaming strategy, comprising a motion generation step, generating a final motion instruction through the motion generation step, and driving the multi-degree-of-freedom robot to perform a corresponding motion according to the final motion instruction, wherein the motion generation step includes: A1: Obtain the current state observation information, current intermediate action variables, and current flow time step information of the multi-degree-of-freedom robot; A2: Input the current state observation information, the current intermediate action variables, and the current flow time step information into a velocity network with a sequence model structure to calculate the output velocity field; A3: Obtain initial noise samples, and based on the initial noise samples, iteratively execute steps A1 to A2 to perform action update steps and generate final action instructions, wherein the velocity field is used to update the current intermediate action variables during each iteration.

[0008] Preferably, the speed network adopts a sequence model structure based on a gating mechanism or a decoder mechanism.

[0009] Preferably, the velocity network adopts a sequence model structure based on a gate mechanism. The sequence model structure based on the gate mechanism includes parallel gate networks and candidate networks, wherein the velocity field is calculated by the sequence model structure based on the gate mechanism in the following manner: The gate network calculates the gating coefficients based on the current state observation information, the current intermediate action variables, and the current flow time step information. The candidate network calculates candidate velocities based on the current state observation information, the current intermediate action variables, and the current flow time step information. The velocity field is calculated based on the gating coefficient, the candidate velocity, and the current intermediate action variable.

[0010] Preferably, the velocity network adopts a sequence model structure based on a decoder mechanism, wherein the velocity field is calculated by the sequence model structure based on the decoder mechanism in the following manner: The current state observation information is encoded into a state embedding vector; The current intermediate action variable and the current flow time step information are combined into an action time representation; By using a cross-attention mechanism, the action time representation is used as a query vector, and the state embedding vector is used as a key vector and a value vector for information fusion. The fused information is processed by layer normalization and residual connection, and then projected through the output layer to obtain the velocity field.

[0011] Preferably, the continuous motion control method for the multi-degree-of-freedom robot further includes a strategy optimization step, wherein the strategy optimization step includes: B1: In the action update step, Gaussian noise is injected into the intermediate action variable, and a drift correction term is introduced to calculate the joint probability density of the flow unfolding trajectory from the initial noise sample to the final action command. B2: Sample data from the experience replay pool, and based on the sampled data and the joint probability density, construct and optimize the policy objective function and loss function to update the velocity network.

[0012] Preferably, in step B1, the drift correction term is calculated based on the velocity field and the flow time step, and is used to offset the influence of the injected Gaussian noise on the final action command distribution.

[0013] Preferably, in step B2, the policy objective function is a policy loss function based on the soft actor-critic algorithm, and its entropy term is calculated from the joint probability density.

[0014] Preferably, in step B2, when optimizing the policy objective function, a proximity regularization term is added. The proximity regularization term is calculated based on the difference in action output between the current policy and the historical policy or expert policy under the same state, and is used to constrain the policy update magnitude in sparse reward tasks.

[0015] Preferably, before step B2, a pre-training step is included, which includes: pre-training the velocity network by minimizing a flow matching objective function using an offline dataset containing expert demonstration data.

[0016] Preferably, driving the multi-degree-of-freedom robot to perform corresponding actions according to the final action command includes: processing the final action command through a squeeze function to map it to the actual control range of the multi-degree-of-freedom robot joints.

[0017] In a second aspect, the present invention discloses a control system for a multi-degree-of-freedom robot based on a streaming strategy, comprising a memory and a processor, wherein the processor stores a computer program, and when the computer program is executed by the processor, it implements the steps of the continuous motion control method for a multi-degree-of-freedom robot based on a streaming strategy as described in the first aspect.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention uses a velocity network with a sequence model structure to calculate the velocity field for updating actions, and formalizes the action update steps of multi-step iterative execution into a deep sequence generation process; wherein the robot control system that uses a velocity network with a sequence model structure to stabilize the gradient propagation of multi-step flow expansion fundamentally solves the gradient ill-conditioning problem when training streaming policies from a policy perspective, enabling the robot to stably learn high-performance control policies through reinforcement learning, thereby avoiding the problems of training instability and policy convergence difficulty caused by gradient ill-conditioning (gradient explosion or vanishing) of multi-step flow expansion when training streaming policies using reinforcement learning from a policy perspective.

[0019] In a further embodiment, the present invention also has the following beneficial effects: (1) By reconstructing the velocity network of the streaming policy into a sequence model structure based on gating mechanisms (such as GRU variants) or decoder mechanisms (such as Transformer decoders), gating or residual mechanisms are introduced into the physical structure, fundamentally ensuring the numerical stability of multi-step backpropagation. On the other hand, the gradient flow characteristics during multi-step backpropagation are fundamentally improved at the network architecture level. The gating mechanism adaptively adjusts the information flow, while the decoder mechanism effectively suppresses gradient explosion or vanishing through layer normalization and residual connections, thereby achieving stable, end-to-end training of the streaming policy under the policy reinforcement learning framework. This further enables the robot control policy to learn efficiently from environmental interactions without sacrificing multimodal expressive capabilities, significantly improving sample efficiency and final performance in complex continuous control tasks.

[0020] (2) By combining noise-enhanced unfolding, complete and unbiased end-to-end policy optimization is achieved. Furthermore, by introducing noise-enhanced flow unfolding and drift correction techniques, the deterministic flow process is transformed into a stochastic process, enabling the joint probability density of the policy to be accurately calculated. This allows for seamless integration into maximum entropy reinforcement learning frameworks (such as SAC), providing an accurate entropy regularization term for policy optimization.

[0021] (3) By designing an offline-to-online training process that includes pre-training and proximity regularization, the method can effectively utilize expert demonstration data and stably optimize the strategy in a sparse reward environment, achieving a smooth transition from imitation learning to reinforcement learning, which is particularly suitable for complex robotic arm operation tasks.

[0022] Thanks to the aforementioned structural and algorithmic innovations, this invention significantly improves sample efficiency without sacrificing the multimodal expressive power of the flow model. Experiments show that on complex control tasks, the performance of this invention is significantly better than existing off-policy flow training methods.

[0023] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the motion generation steps in the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy according to a preferred embodiment of the present invention. Figure 2 This is a schematic diagram of the strategy optimization steps in the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy according to a preferred embodiment of the present invention. Figures 3a to 3c A schematic diagram of the parameterization of a streaming policy velocity network from the perspective of a sequence model; Figures 4a to 4f This is a schematic diagram of the de novo training performance curve for the motion control task of a bionic robot in the MuJoCo simulation platform.

[0025] Figures 5a to 5c This is a schematic diagram of the offline to online training performance curves of the UR-5 industrial robotic arm for multi-object grasping and placement tasks in the OGBench simulation platform. Detailed Implementation

[0026] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0027] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for both fixing and circuit / signal connectivity.

[0028] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0029] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0030] The instability of directly optimizing the policy loss is not accidental, but stems from a fundamental theoretical connection. Research reveals that the Euler integral used to generate robot control actions is algebraically identical to the iterative computation of an RNN using residual connections. This involves intermediate actions... Treating it as a hidden state, the stream update becomes a residual RNN step: The unit function is Therefore, training streaming policies using off-policy loss requires... Backpropagation through a deep recurrent stack of layers is prone to the classic gradient explosion and vanishing problems in RNNs. This insight reveals that the instabilities observed in previous work are structural problems of the sequential computation graph, implying a need to redesign the velocity parameterization, rather than simply modifying the objective function.

[0031] Based on the above research, a preferred embodiment of the present invention discloses a continuous motion control method for a multi-degree-of-freedom robot based on a streaming strategy, including a motion generation step, which generates a final motion command and drives the multi-degree-of-freedom robot to perform the corresponding motion according to the final motion command. The motion generation step includes: A1: Obtain the current state observation information, current intermediate action variables, and current flow time step information of the multi-degree-of-freedom robot; A2: Input the current state observation information, the current intermediate action variables, and the current flow time step information into the velocity network with a sequence model structure to calculate the output velocity field; The speed network employs a sequence model structure based on either a gating mechanism or a decoder mechanism.

[0032] In some embodiments, the velocity network employs a gate-based sequence model structure. This gate-based sequence model structure includes parallel gate networks and candidate networks. The velocity field is calculated as follows: the gate network calculates gating coefficients based on current state observation information, current intermediate action variables, and current flow time step information; the candidate network calculates candidate velocities based on current state observation information, current intermediate action variables, and current flow time step information; and the velocity field is calculated based on the gating coefficients, candidate velocities, and current intermediate action variables. Specifically, the velocity field is calculated as follows: the product of the gating coefficients and candidate velocities element-wise, plus the product of (1 minus the gating coefficients) and the intermediate action variables element-wise.

[0033] In other embodiments, the velocity network employs a sequence model structure based on a decoder mechanism, wherein the sequence model structure based on the decoder mechanism calculates the velocity field as follows: the current state observation information is encoded into a state embedding vector; the current intermediate action variable and the current flow time step information are combined into an action time representation; through a cross-attention mechanism, the action time representation is used as a query vector, and the state embedding vector is used as a key vector and a value vector, respectively, for information fusion; the fused information is processed through layer normalization and residual connections, and then projected through the output layer to obtain the velocity field.

[0034] A3: Obtain initial noise samples, and based on the initial noise samples, iteratively execute steps A1 to A2 to perform action update steps and generate the final action command. In each iteration, the velocity field is used to update the current intermediate action variables.

[0035] Here, the initial noise sample z0 is a multidimensional random vector, the dimension of which is the same as the motion space dimension d of the multi-degree-of-freedom robot. a The same applies. In some embodiments, the initial noise samples can be sampled from a standard multivariate Gaussian distribution, i.e., z0 ~ N(0, I), where I is the d a 3D identity matrix.

[0036] The process of driving a multi-degree-of-freedom robot to perform corresponding actions according to the final motion command includes: processing the final motion command through a squeeze function to map it to the actual control range of the multi-degree-of-freedom robot joints.

[0037] In some embodiments, the continuous motion control method for the multi-degree-of-freedom robot further includes a strategy optimization step, wherein the strategy optimization step includes: B1: In the action update step, Gaussian noise is injected into the intermediate action variables, and a drift correction term is introduced to calculate the joint probability density of the flow unfolding trajectory from the initial noise sample to the final action command. The drift correction term, calculated based on the velocity field and flow time step, is used to counteract the impact of injected Gaussian noise on the final action command distribution. The flow time step refers to the time interval between two iterations, and its value is coordinated with the output scale of the velocity network. It is usually determined by conducting small-scale experiments in the target simulation environment, and is generally taken as a small positive value, such as 0.1s or 0.01s.

[0038] B2: Sample data from the experience replay pool, and based on the sampled data and joint probability density, construct and optimize the policy objective function and loss function to update the velocity network.

[0039] Among them, when optimizing the objective function of the policy, a proximity regularization term is added. The proximity regularization term is calculated based on the difference in action output between the current policy and the historical policy or expert policy in the same state, and is used to constrain the policy update magnitude in sparse reward tasks.

[0040] In some embodiments, prior to step B2, a pre-training step is further included, which includes: pre-training the velocity network by minimizing a flow matching objective function using an offline dataset containing expert demonstration data.

[0041] The following detailed description of the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy proposed in the preferred embodiment of the present invention, with reference to specific embodiments, provides further details.

[0042] This continuous motion control method for multi-DOF robots based on streaming policies is a control approach based on velocity reparameterization sequence modeling and streaming policy reinforcement learning. It addresses the continuous motion space control task of multi-DOF robots by formalizing the motion generation process of the streaming policy into a sequence model and incorporating a noise-enhanced unfolding mechanism. This solves the gradient ill-conditioning problem encountered when directly training the streaming model in off-policy reinforcement learning, enabling the robot to efficiently learn stable and high-performance control policies. The robot control simulation system involved includes: a simulation environment module, built on physical simulation engines such as MuJoCo and OGBench, used to simulate the dynamic behavior of the robot body and provide current state observation information, including but not limited to joint angles, joint angular velocities, trunk posture (quaternions or Euler angles), center of mass velocity, and end effector pose; a control policy module, the core of this invention, which receives the state observation information provided by the simulation environment based on the streaming generation model and outputs continuous control commands (i.e., final motion commands), including but not limited to torque commands or position increment commands for each joint; and a simulation execution module, used to input control commands into the simulation engine, drive the simulated robot to perform corresponding actions, and return the state observation and reward signals for the next moment, forming a closed-loop control circuit.

[0043] The preferred embodiment of this invention proposes a multi-degree-of-freedom robot continuous motion control method based on a streaming strategy, which solves the following technical problems: (1) In continuous control tasks of multi-degree-of-freedom robots, flowing strategies have significant advantages due to their powerful multimodal action representation capabilities. However, when using off-policy reinforcement learning to train them, existing technologies often lead to divergence or collapse of control policy training due to gradient ill-conditioning (gradient explosion or vanishing) in multi-step flow expansion, making it impossible for the robot to learn stable and effective motion control or operation control policies. A preferred embodiment of this invention proposes a velocity network reparameterization method based on a sequence model perspective. By formalizing K-step flow expansion into residual RNN (Recurrent Neural Network) computation, the fundamental cause of gradient instability is revealed. Two stable velocity network designs are proposed using modern sequence architectures: Flow-G (based on GRU gating) and Flow-T (based on Transformer decoding). This method effectively solves the gradient ill-conditioning problem, enabling direct end-to-end optimization of the robot control policy and completely eliminating the need for proxy targets or policy distillation.

[0044] (2) In practical applications of robot control, different task scenarios have different requirements for training methods: for motion control tasks with clear and continuous feedback signals (such as gait optimization of bionic robots, where the reward signal is directly related to the forward speed), a training method that can learn efficiently from scratch is needed; while for complex operation tasks with extremely sparse feedback signals (such as multi-object grasping and placement by a robotic arm, where a success signal is only obtained when all objects are placed correctly), a training method that can utilize existing human demonstration data for pre-training and gradually transition to autonomous learning is needed. This invention develops the SAC Flow algorithm, which solves the key technical obstacle of intractable likelihood calculation in SAC (Soft Actor-Critic) objectives by introducing a noise-enhanced unfolding mechanism, and derives two robust training programs for different robot control scenarios: a de novo training process for dense reward motion control tasks, and an offline-to-online training process for sparse reward operation control tasks.

[0045] (3) This invention has been extensively experimentally verified on various robot control simulation platforms. In the de novo training of MuJoCo bionic robot motion control tasks (including the Hopper monopod, Walker2d bipod, HalfCheetah quadruped, Ant multipod crawling robot, and Humanoid robot), this invention achieved a motion reward improvement of up to 130% compared to the existing best methods. In the offline-to-online training of OGBench industrial robotic arm operation control tasks (including the UR-5 robotic arm's grasping and precise placement of multiple objects), this invention achieved a task success rate improvement of up to 60%. These results verify the superior sample efficiency and generalization ability of this invention in different types of robot control tasks.

[0046] The following sections provide further detailed explanations of the sequence modeling-based velocity network module used in the action generation step and the noise-enhanced SAC training and control algorithm used in the policy optimization step.

[0047] The velocity network module based on sequence modeling used in the action generation step is a core component of the robot control strategy subsystem. Its function is to measure the robot's current state observation information (denoted as the environmental state). Specifically, this includes sensor readings of the robot's joint angles, angular velocities, trunk posture, and center of mass velocity, etc., which are generated through a multi-step flow unfolding process to produce continuous control commands (referred to as actions). Specifically, this refers to the torque command or position increment command for each joint.

[0048] Existing techniques typically treat the velocity network in flow unrolling as a simple residual connection, leading to gradient explosion during deep backpropagation. This makes it impossible for robot control policies to be effectively trained end-to-end through policy reinforcement learning.

[0049] The embodiments of the present invention reconstruct the speed network by introducing the gating or decoding mechanism of modern sequence models, specifically including the following two specific embodiments.

[0050] Figures 3a to 3c This is a schematic diagram illustrating the parameterization of a streaming policy velocity network from the perspective of a sequence model. Figure 3a The RNN unit in the code represents a standard streaming policy, where the speed... It is the direct output of the neural network, and this simple formula is prone to gradient instability. Figure 3b The GRU (Gated Recurrent Unit) in the code uses a GRU-style gating mechanism for computational speed, and the gate network... Adaptive control from candidate networks The update strength is adjusted to stabilize the gradient flow. Here, NN represents a Neural Network, used to implement the nonlinear mapping between the gate network and the candidate network. It calculates the original output and candidate velocity of the gating coefficients based on the current intermediate action variables, the environmental state, and the flow time step information; Sig represents the Sigmoid activation function (i.e., the logistic function). This is used to map the original values ​​of the NN output in the gate network element by element to the interval between 0 and 1, thereby obtaining the gate coefficients with a limited range of values. This ensures that it can effectively and adaptively adjust the update intensity of candidate velocities. Figure 3c The decoder in the code uses a Transformer decoder to model speed, where intermediate action states... pass Layer state conditions are refined through cross-attention to improve decoding speed.

[0051] Example 1: Speed ​​Network Based on GRU Gating Mechanism (Flow-G) like Figure 3b As shown, this embodiment constructs the velocity field calculation process as a variant of GRU to stabilize the gradient flow during multi-step unfolding.

[0052] In this embodiment, for each time step of the flow unrolling The system receives three inputs: (1) The intermediate action state at the current moment , represents an intermediate quantity in the process of evolving from the initial Gaussian noise to the final control command, and its dimension is consistent with the robot's action space dimension (e.g., for a bipedal walking robot (Walker2d) with 6 controlled joints). For the UR-5 robotic arm with 5 degrees of freedom, ), Represents the set of real numbers; (2) Temporal embedding encoding, representing the current flow unpacking step; (3) Environmental conditions This refers to the state observation information collected by the robot's sensors.

[0053] The system internally constructs two parallel neural network branches: a gate network and a candidate network. The gate network is configured to output a gating coefficient with a value between 0 and 1, which is used to adaptively adjust the degree of retention from the action at the previous time step to the current update. The candidate network is configured to output the suggested speed update amount, i.e., the candidate speed, in the current state.

[0054] The system calculates the final velocity field using the following formula. Output: (1) in, This represents the gating coefficients output by the gate network. Its dimension is consistent with the motion space, and each component controls the update amplitude of the corresponding joint control command. This represents the candidate velocity output by the candidate network. This indicates element-wise multiplication. express Intermediate action variables at time points, s It represents the environmental state, that is, the current state observation information.

[0055] The significant technical advantage of this structure lies in the fact that when the gating coefficient approaches 0, the Jacobian matrix of the flow unrolling step approaches the identity matrix. This mechanism physically constructs a high-speed gradient channel, similar to the constant error propagation mechanism in LSTM (Long Short-Term Memory) networks, thereby effectively suppressing... Figure 3a The gradient explosion phenomenon commonly seen in traditional velocity networks (RNN units) enables robot control strategies to maintain training stability under deep decomposition.

[0056] Example 2: Speed ​​Network Based on Transformer Decoding Mechanism (Flow-T) like Figure 3c As shown, this embodiment utilizes the structural advantages of the Transformer decoder to parameterize the velocity field and leverages its layer normalization and residual connection characteristics to stabilize gradient propagation in deep networks.

[0057] In this embodiment, the system first projects the environmental state through a linear projection layer. (i.e., sensor readings such as joint angles, angular velocities, and torso posture of the robot) are mapped to a global state embedding vector. At the same time, the current intermediate action state With time step Mapped to action tag vectors Unlike traditional autoregressive sequence models, this invention employs a specific cross-attention mechanism. In each decoding block, the action tag vector serves as the query vector, while the global state embedding vector serves as the key and value vectors. Simultaneously, a diagonal mask or independent processing mechanism is used in the self-attention layer to ensure no causal confusion occurs between time steps, preserving the Markov property of the flow model.

[0058] The system calculates the velocity field using the following formula. Output: (2) The system stacks L layers of pre-normalized (Pre-Norm) residual blocks, and the action update is finally calculated using the following formula: (3) in, This represents the action characteristics output by the Lth layer decoded block. LN Representation layer normalization, This represents the output projection matrix. This structure ensures the benignity of gradients in the deep computational graph through pre-normalization and residual connections, enabling the model to withstand [various challenges / challenges]. Long-distance back propagation caused by the unfolding of the flow.

[0059] In this invention, a velocity network with a sequence model structure is adopted, and a mechanism that can stabilize gradient propagation (such as gating, attention, residual connections, etc.) is introduced. The "gating mechanism" and "decoder mechanism" mentioned above are two preferred specific implementation methods. In other embodiments, LSTM (Long Short-Term Memory) units or deep MLP networks with skip connections can also be used, as long as their structural design can alleviate the gradient vanishing / exploding problem in deep networks.

[0060] This invention also introduces a policy optimization step to optimize the above-mentioned speed network by combining the SAC (Soft Actor-Critic) algorithm.

[0061] To address the problem of incalculable probability density caused by deterministic flow expansion, this invention proposes a specific noise-enhanced training process. To clearly illustrate the training process of this embodiment, the key symbols involved are first uniformly defined:

[0062] The weight parameters represent the actor network (i.e., the velocity network), which is responsible for generating control commands based on the robot's state. The weight parameters represent the Critic (Q-network) network, which are responsible for evaluating the long-term cumulative reward of executing a certain control instruction in the current state; This represents the weight parameters of the Target Critic network.

[0063] This refers to the experience playback pool, used to store historical data generated by the interaction between the robot and the simulation environment; Indicates the state of the environment; Indicates an action; This represents a reward, which is a scalar feedback signal returned by the simulation environment based on the robot's current performance (e.g., related to forward speed and standing stability in motion control tasks, and related to the correct number of objects placed in operation control tasks).

[0064] This indicates the index of the batch data sampled from the playback pool; Represents the virtual time step when the streaming model generates actions. ; This indicates the time step of the flow unrolling.

[0065] Different training processes are used for training from scratch under dense rewards and for offline-to-online training under sparse rewards.

[0066] Table 1. De novo training process under dense rewards Table 1 shows the de novo training process under dense rewards. This process is used to train a control strategy from scratch in a robot motion control simulation task with dense reward signals. The specific steps are as follows:

[0067] S101: Noise-enhanced flow unrolling: During the training phase of the Actor network, the system in Each step of the flow unwinding process actively injects isotropic Gaussian noise, transforming the deterministic ODE (Ordinary Differential Equation) solution process into a stochastic process. This allows the initial noise to gradually evolve into physically meaningful joint control commands. To counteract the diffusion effect of the noise, the system introduces a drift correction term, defined as follows:

[0068] (4) in, Indicates flow expansion in Intermediate action variables at time points, The parameter representing the standard deviation of noise, This is the velocity field output by the velocity network in its current state. This drift correction term... This ensures that the marginal probabilities of the final generated action distribution remain consistent with the deterministic unfolding.

[0069] S102: Processable Likelihood Calculation: Based on the noise enhancement in step S101, the joint probability density function of the complete flow unfolding path from the initial noise to the final joint control command can be decomposed into the product of independent Gaussian transition probabilities at each step. The system calculates the log-likelihood using the following formula for entropy regularization of the SAC:

[0070] (5) in, Represents the complete flow generation trajectory , This represents the probability density function of the basic distribution (usually a standard Gaussian distribution). This represents the Gaussian transition probability at each step. This indicates the range from step 0 to step 1. The joint probability density of the entire flow unfolded trajectory is decomposed into the product of the individual Gaussian transition probabilities by multiplying all the single-step Gaussian transition probabilities together. Indicates that given the current state Current intermediate action variable and flow expansion step size Under the condition that, the intermediate action variable at the next moment The Gaussian transition probability density, whose mean is composed of the current intermediate action variable superimposed with the velocity increment with drift correction, and whose variance is determined by the standard deviation of the injected Gaussian noise and the step size. This represents the final action command obtained after processing by the squeeze function, i.e. ; This represents the Jacobian matrix of the squeezing function with respect to its input; It represents the inverse of the Jacobian determinant of the action squashing (e.g., Tanh function) operation, used to map the unbounded flow expansion output to the effective range of robot joint control commands (e.g., upper and lower limits of joint torque), and correct the mapped probability density.

[0071] S103: The Actor and Critic Network parameter update system samples historical interaction data between the robot and the simulation environment from the experience replay pool. ,in For the robot in the Step state observation, For the robot in the Step-by-step execution of joint control commands, For the robot in the The instant reward returned by the simulation environment. For the robot in the The next state observation is performed after the control command is executed. Using the path likelihood calculated above as the entropy term, the actor network (Actor) loss function is constructed:

[0072] (6) in, This is the entropy regularization coefficient (Temperature), used to balance exploration and exploitation; The action is generated through reparameterization of the flow model. Represent the critic network. Simultaneously, construct the critic network (Critic) loss function:

[0073] (7) in, This serves as a discount factor to control the robot's emphasis on long-term future returns. A target commentator network is used to calculate the target Q-value; This is the new action sampled for the next state. It should be noted that the log-likelihood term in formula (6)... With superscript The log-likelihood term in formula (7) Without superscript The two are fundamentally different in meaning and gradient propagation method, and cannot be substituted for each other:

[0074] (i) In formula (6), Indicates the current actor network parameters Below, by analyzing the current state Perform noise enhancement The complete flow unfolding trajectory generated by step flow unfolding (i.e., reparameterized sampling) explicitly depends on the current parameters of the actor network. Regarding the loss of actors' online presence. When calculating the gradient and backpropagating, the gradient will follow... Passing through the entire Step flow unfolds and is transmitted back to the actor's network parameters This directly drives the parameter updates of the velocity network. The entropy term here is used to reward policy randomness in maximum entropy reinforcement learning, thus promoting exploration.

[0075] (ii) In formula (7), Indicates the next state The following are flow unrolling trajectories generated through the same flow unrolling process, but with different commentator network losses. It appears only as a component of the Temporal Difference (TD) objective. When calculating the gradient of the commentator network loss and backpropagating, the gradient only travels along... Return to critic network parameters Without going through the log-likelihood term and returning it to the actor's network parameters --Right now This loss is considered to be related to Irrelevant constants (typically implemented in engineering by stopping gradient operations using detach). (No superscript here.) That is, explicitly indicating that the item does not participate in the evaluation. The gradient is calculated. The entropy term here is used to correct the TD objective value so that it is consistent with the maximum entropy objective.

[0076] The difference in superscripts mentioned above reflects the division of labor between Actor loss and Critic loss in terms of optimization objectives: the optimization parameters of Actor loss are the actor network parameters. Therefore, with The relevant trajectory variables need to be explicitly labeled; the optimization parameters for the Critic loss are the critic network parameters. All of them All relevant variables are treated as constants in this loss.

[0077] Target commentator network parameters By analyzing the parameters of the critic network An EMA (Exponential Moving Average) is used for updates to maintain training stability. By minimizing the above loss function, the system achieves end-to-end updates of policy parameters.

[0078] Table 2. Offline to Online Training Process under Sparse Rewards Table 2 shows the offline-to-online training process under sparse rewards, which is used for complex tasks with sparse rewards and expert demonstration data. The main difference from the process shown in Table 1 is the introduction of proximity regularization and a pre-training stage.

[0079] Typical application scenarios include the UR-5 industrial robotic arm multi-object grasping and placement task (Cube-Double / Triple / Quadruple, with state dimensions including robotic arm joint states and object poses, and motion dimension 5) in the OGBench (Open Graph Benchmark) simulation platform. The robotic arm needs to accurately place multiple cubes into target positions, and only receives reward feedback (reward value is...) when the objects are placed correctly. , (Number of incorrectly placed objects). In these tasks, the simulation environment provides a pre-defined expert demonstration dataset. This dataset was generated by running control algorithms with a basic success rate in a simulation environment (such as pre-trained strategies based on behavior cloning or human teleoperators controlling robots), and includes state-action trajectories and corresponding reward information.

[0080] S201: Stream Matching Pre-training: Before entering online interaction, the system first utilizes an expert dataset. The velocity network is pre-trained using supervised learning by minimizing the flow matching objective function to initialize the policy distribution. This pre-training phase enables the velocity network to initially learn to mimic robot operation patterns (such as the grasping trajectory and placement posture of a robotic arm) in expert demonstrations, providing a policy initialization with basic operational capabilities for subsequent online optimization through reinforcement learning.

[0081] S202: Online interaction and data integration: The system uses the current policy. It interacts with the environment and stores newly collected data in a playback pool containing expert data. At this time, the playback pool It includes initial expert data. Newly collected data can be interacted with online by robots.

[0082] S203: Regularized Actor Update: When updating the actor parameters, the system adds a proximity regularization term to the original SAC loss to constrain the update magnitude of the policy and prevent the policy from deviating drastically from the learned effective operating pattern under sparse reward signals. The modified actor loss function is as follows:

[0083] (8) in, The regularization coefficient is . This is expert or historical data sampled from the replay pool. This regularization term constrains the policy update magnitude, preventing catastrophic forgetting under sparse rewards and ensuring a smooth transition from offline to online policies.

[0084] The continuous motion control method for multi-degree-of-freedom robots based on streaming strategies in specific embodiments of the present invention provides the following important architectures or methods.

[0085] First, the multi-DOF robot continuous motion control method based on streaming policy in specific embodiments of the present invention provides a reparameterized architecture for streaming policy velocity networks for continuous control tasks of multi-DOF robots. Existing technologies, when using off-policy reinforcement learning to train streaming policies to control robots, suffer from severe gradient ill-conditioning in the backpropagation of multi-step flow unrolling due to the use of a common MLP (Multi-Layer Perceptron) structure in the velocity network. This leads to divergent training of the robot control policy, preventing the learning of stable motion or operational skills. The present invention equates flow unrolling to residual cyclic computation and innovatively proposes two stable velocity network architectures: Flow-G employs a GRU-style gating mechanism, where the gating coefficients adaptively adjust the update amplitude of control commands joint by joint, constructing a high-speed channel for gradient propagation in the physical structure; Flow-T employs a Transformer decoding mechanism, fusing high-dimensional state observation information of the robot (such as joint angles, angular velocities, trunk posture, end effector pose, etc.) through a cross-attention module, and using pre-normalized residual connections to ensure the benign gradient of deep unrolling. Both architectures serve as direct replacements for velocity networks, enabling stable end-to-end policy training for streaming policies in tasks such as biomimetic robot motion control and industrial robotic arm operation control, completely eliminating the need for agent targets or policy distillation.

[0086] Secondly, the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy in a specific embodiment of the present invention provides a robot control strategy likelihood calculation method based on noise enhancement unrolling and drift correction. Existing maximum entropy reinforcement learning algorithms such as SAC face a technical bottleneck when applied to streaming strategy-controlled robots: the probability density of control commands cannot be calculated due to deterministic flow unrolling. The present invention actively injects isotropic Gaussian noise at each step of the flow unrolling and, in conjunction with a precise drift correction term, decomposes the joint probability density of the complete flow trajectory from the initial noise to the final joint control command into a product of independent Gaussian transition probabilities at each step. While ensuring that the final control command distribution remains unchanged, it achieves accurate likelihood evaluation and entropy regularization with extremely low algebraic computation cost. This method enables seamless integration of robot control strategies into the highly efficient SAC framework, allowing effective control skills to be learned in a simulation environment with fewer interaction steps.

[0087] Third, the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy in a specific embodiment of the present invention provides a de novo training method for dense reward robot motion control tasks. This invention combines the aforementioned stable velocity network architecture with a noise-enhanced SAC training process to construct a complete de novo training framework. In this framework, the robot starts from a random initial state in a simulation environment such as MuJoCo (Multi-Joint dynamics with Contact). The control strategy generates joint torque commands based on state observations (joint angles, angular velocities, torso posture, etc.) collected by sensors, interacts with the simulation environment, and obtains dense reward feedback directly related to motion indicators such as forward speed and standing stability. The control strategy is continuously optimized through off-policy learning. Experimental verification shows that this method achieves superior control performance compared to existing methods in various biomimetic robot motion control tasks, including bipedal walking robots (Walker2d), quadrupedal running robots (HalfCheetah), and humanoid standing robots (HumanoidStandup). In the humanoid standing robot task, the motion reward improvement is as high as 130%.

[0088] Fourth, the multi-degree-of-freedom robot continuous motion control method based on streaming strategies in specific embodiments of the present invention provides an offline-to-online training method for sparse reward robotic arm operation control tasks. Addressing the characteristic that feedback signals are only obtained upon task completion in sparse reward tasks such as multi-object grasping and placement in industrial robotic arms, the present invention proposes a unified offline-to-online training framework. This framework first utilizes expert demonstration datasets pre-collected by the basic control algorithm in a simulation environment, and uses streaming matching pre-training to enable the velocity network to initially learn the basic operation motion patterns of the robotic arm. Then, it transitions to the online stage, where the robotic arm autonomously executes grasping and placement tasks in the simulation environment. The system introduces a proximity regularization term into the loss function of the SAC-updated Actor network, constraining the control strategy to explore better operation strategies without deviating from existing successful operation experiences, achieving a smooth transition from imitation learning to reinforcement learning. Experimental results show that this method achieves a significant improvement in task success rate compared to the existing state-of-the-art methods FQL (Flow Q-Learning) and QC-FQL (Q-chunking with Flow Q-Learning) in the UR-5 robotic arm Cube-Double / Triple / Quadruple multi-object placement task on the OGBench simulation platform. In particular, the success rate in the three-object placement task (Cube-Triple) is increased from less than 15% to 60%-70% compared to existing methods.

[0089] The following description, in conjunction with specific application examples, further illustrates the preferred embodiment of the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy proposed in this invention.

[0090] Application Example 1: Motion Control of a Bipedal Walking Robot in the MuJoCo Simulation Platform This embodiment applies the SAC Flow-T method of the present invention to the de novo training motion control task of a bipedal walking robot (Walker2d) in the MuJoCo simulation platform, and demonstrates the complete application process of the de novo training process (Table 1).

[0091] (1) Task Description: Walker2d is a simulation model of a planar bipedal walking robot with 6 controlled joints. The control objective is to enable the robot to walk forward as quickly as possible without falling. The simulation environment returns the state observation vector at each control step. This includes: trunk top height, trunk angle, angles of six joints (right hip, right knee, right ankle, left hip, left knee, left ankle), trunk horizontal velocity, trunk vertical velocity, trunk angular velocity, and angular velocities of the six joints. The control strategy requires the output motion vector. The torque commands for the above 6 joints are normalized to a range after being compressed using the Tanh function (hyperbolic tangent function). Then, the simulation engine scales the output to the actual torque range of each joint. (Reward signal) It consists of three parts: forward speed reward, standing reward, and control cost, and can be obtained in every simulation step, belonging to dense rewards.

[0092] (2) Velocity network configuration: The Flow-T architecture is adopted, which is a velocity network based on the Transformer decoding mechanism. State observation vector The observation encoder (2-layer MLP, 64 hidden dimensions, SiLU activation function) is mapped to a global state embedding vector. Intermediate action state With time step After embedding and concatenation, the vectors are mapped to action tag vectors through linear projection. The Transformer decoder is set to... Layers, each containing 4 cross-attention modules (model dimension) The motion marker vectors are fused with the robot's current motion state information by querying the state embedding. The final output layer projects the decoded features into a 6-dimensional velocity vector. The flow unpacking step count is set to... .

[0093] (3) Training process: Follow the training from scratch process shown in Table 1. Initialize the experience replay pool. (capacity After the robot randomly initializes its posture in the simulation environment, it generates joint torque commands based on the current control strategy and interacts with the environment, transmitting the interaction data at each step. Store in the replay pool. Training uses batch size... Policy learning rate Critics' learning rate Discount factor Noise standard deviation It uses an independent network for adaptive learning. In the first 50,000 steps, only data is collected without updating the network. After that, data is collected and the network parameters are updated once in each subsequent step.

[0094] (4) Control effect: such as Figure 4b As shown, after approximately After interacting with the simulation environment, the bipedal walking robot (Walker2d) trained by the SAC Flow-T method can achieve a motion reward of about 5000-6000 (close to the convergence level), which means that the robot has learned a stable bipedal alternating walking gait and can move forward continuously at a high speed without falling. In comparison, existing FlowRL (Flow-based Reinforcement Learning) methods yield a reward of approximately 4000 with the same number of interaction steps, corresponding to poor gait stability. The standard SAC (Soft Actor-Critic) method, based on Gaussian policies, converges even slower. The Proximal Policy Optimization (PPO) method requires several times more interaction steps to achieve a similar performance. QSM (Q-Score Matching, an early representative method for diffusion policy gradient optimization by matching the gradients of the policy score function and the Q-score function with respect to actions) uses a diffusion model structure in its policy network, requiring numerous denoising steps for single-step inference. Furthermore, its indirect optimization method based on score matching has limited convergence performance in this task. Therefore, the bipedal walking robot trained with the same number of simulated interaction steps yields significantly lower motion rewards than the method described in this invention, making it difficult for the robot to learn a stable gait.

[0095] Figures 4a to 4f The horizontal axis of the graph represents the number of interaction steps in the simulation environment, and the vertical axis represents the motion return. Figure 4a This represents the de novo training performance curve of the Hopper monopod hopping robot on the MuJoCo simulation platform for motion control tasks. Figure 4b This represents the de novo training performance curve of the bipedal walking robot (Walker2) d in the MuJoCo simulation platform for the motion control task. Figure 4cThis represents the de novo training performance curve of the quadrupedal running robot (HalfCheetah) in the motion control task on the MuJoCo simulation platform. Figure 4d This represents the de novo training performance curve of the multi-legged crawling robot (Ant) in the motion control task on the MuJoCo simulation platform. Figure 4e This represents the de novo training performance curve of a humanoid robot for motion control tasks on the MuJoCo simulation platform. Figure 4f This represents the de novo training performance curve of a humanoid standing robot in the MuJoCo simulation platform for motion control tasks. From... Figures 4a to 4f It can be concluded that the SAC Flow-G (a combination of a velocity network based on GRU gating mechanism and a soft actor-critic network) and SAC Flow-T (a combination of a velocity network based on Transformer decoding mechanism and a soft actor-critic network) methods of the present invention achieve control performance that is superior to or equal to that of existing methods in most motion control tasks. The improvement in reward is most significant in the standing task of the humanoid standing robot. All results are the average of 5 random seeds, and the shaded area represents the 95% confidence interval.

[0096] refer to Figures 4a to 4f In de novo training of six biomimetic robot motion control tasks on the MuJoCo simulation platform, the SAC Flow-G and SAC Flow-T methods of this invention achieved control performance superior to or on par with existing best methods on most tasks. Specifically: Figure 4f In the humanoid standing up task, the SAC Flow-T method of this invention achieved a final motion reward of approximately 350,000, while the existing baseline FlowRL was approximately 150,000, representing an improvement of approximately 130% in motion reward. This demonstrates that the robot can stand up from a supine position more quickly and stably. Figure 4a In the Hopper task, the invention achieved a final reward of approximately 3000, exceeding FlowRL's approximately 2500; Figure 4b In the bipedal walking task (Walker2d), the invention achieved a final reward of approximately 5000-6000, exceeding FlowRL's approximately 4000. In both tasks, the invention demonstrated faster convergence speed, meaning the robot can learn stable jumping and walking skills with fewer simulation interactions. Figure 4c In the Half-Cheetah quadrupedal running task, the invention achieved a final return of approximately 13,000-14,000, comparable to the best baseline DIME and exceeding FlowRL; Figure 4dIn the multi-legged crawling task (Ant), this invention achieves a reward level of approximately 5000-6000, which is on par with the performance of baseline methods. Compared to the policy-based method PPO, this invention, as an off-policy method, has a significant advantage in sample efficiency, greatly reducing the number of simulation environment interaction steps required to achieve the same motion control performance, and significantly improving the task success rate in industrial robotic arm operation control tasks.

[0097] Application Example 2: Multi-object grasping and placement control of the UR-5 robotic arm in the OGBench simulation platform This example applies the SAC Flow-T method of the present invention to the Cube-Triple three-object grasping and placement task of the UR-5 robotic arm in the OGBench simulation platform, demonstrating the complete application process from offline to online training (Table 2).

[0098] (1) Task Description: UR-5 is a 6-axis industrial robotic arm simulation model. In this task, 5 degrees of freedom are controlled. The task objective is to control the robotic arm to sequentially grasp 3 cubes (Cube-Triple) and accurately place them into their respective target positions. The simulation environment provides state observation vectors. This includes: the angles and angular velocities of the robotic arm's five joints, the pose (position and orientation) of the end effector, the gripper's opening and closing states, and the current and target positions of the three cubes. The motion vectors output by the control strategy are also included. This provides position increment commands for the five joints of the robotic arm. The reward signal is a sparse reward, defined as... ,in This represents the number of cubes that were not correctly placed in the target position, only if all 3 cubes are placed correctly. (Maximum reward), otherwise It can be a negative value. The maximum number of moves per round is 1000.

[0099] (2) Expert demonstration dataset: expert dataset Provided by the OGBench simulation platform, this dataset contains 3 million state-action transition data points. It was generated by running a pre-trained behavior clone strategy with a baseline success rate within the simulation environment. The dataset covers various trajectory patterns of a robotic arm performing grasping and placing operations from different initial configurations.

[0100] (3) Speed ​​network configuration: The Flow-T architecture is adopted, and the Transformer decoder is set to Layers, 4-head cross-attention, model dimensions Number of flow expansion steps The noise standard deviation is fixed at 1. By employing action chunking technology, the robotic arm outputs a continuous sequence of control commands with each decision, thereby improving the execution consistency of long field-of-view operations.

[0101] (4) Training process: Follow the offline to online training process shown in Table 2. First, initialize the experience replay pool as an expert dataset. In the offline phase ( (Step update), the system uses expert data from the replay pool, and simultaneously performs stream matching pre-training (step S201) and SAC update with proximity regularization (step S203), with regularization coefficients... To ensure the strategy doesn't deviate too far from the expert demonstration. During the online phase ( (Step S202) The system uses the current strategy to control the robotic arm to perform grasping and placement tasks in the simulation environment, appends the newly collected interaction data to the playback pool, and continues to perform SAC updates with regularization. Other training parameters include: batch size. Discount factor Target network update rate .

[0102] (5) Control effect: such as Figure 5b As shown, at the end of the offline phase ( (Step 1), the robotic arm's manipulation strategy trained by SAC Flow-T has already demonstrated a certain ability to place three objects. After entering the online phase, through continuous interaction with the simulation environment and reinforcement learning optimization, the success rate has rapidly increased, reaching approximately [missing information]. After a total update, the success rate reaches approximately 60%-70% and tends to converge. This means that the robotic arm has learned a reliable sequence of operations to sequentially grasp three cubes and accurately place them in their respective target positions. In contrast, existing FQL (Flow Q-Learning) methods have a success rate close to 0% with the same number of training steps, and QC-FQL (Q-chunking Flow Q-Learning, an improved method that introduces an action chunking mechanism on top of FQL to update the policy in a temporally extended action space for handling long-viewpoint operations) methods have a success rate of approximately 10%-15%, while this invention achieves a significant improvement in success rate. This result verifies that this invention can effectively optimize the basic operational experience from expert demonstrations through off-policy reinforcement learning, enabling the robotic arm to achieve significantly higher control accuracy in complex multi-object manipulation tasks.

[0103] Figures 5a to 5c The horizontal axis represents the total number of training steps (including...). Step-by-step offline updates and (Step-by-step online interaction), with the vertical axis representing the average success rate of tasks. Figure 5a This diagram illustrates the offline to online training performance curves (average success rate of 5 sub-tasks) for the Cube-Double object placement task on the OGBench simulation platform. Figure 5b This diagram illustrates the offline to online training performance curves of the Cube-Triple object placement task on the OGBench simulation platform (average success rate of the five sub-tasks). Figure 5c This diagram illustrates the offline to online training performance curves (average success rate of the 5 sub-tasks) for the Cube-Quadruple task on the OGBench simulation platform. Figures 5a to 5c As can be seen, the SAC Flow-T method of this invention significantly outperforms the existing methods FQL and QC-FQL in all three difficulty levels of robotic arm operation tasks, especially in high-difficulty tasks such as the three-object placement task (Cube-Triple) and the four-object placement task (Cube-Quadruple).

[0104] like Figures 5a to 5c As shown, in the offline-to-online training of the UR-5 industrial robotic arm multi-object grasping and placement task on the OGBench simulation platform, this invention achieved significant performance advantages. Specifically: Figure 5a In the cube-double object placement task, the SAC Flow-T method of this invention achieves an average success rate of approximately 85%-90%, while existing methods FQL achieve only about 5%-10% and QC-FQL about 35%-40%, demonstrating a significant lead. In more challenging tasks... Figure 5b In the cube-triple object placement task, the SAC Flow-T method of this invention achieved a final success rate of approximately 60%-70%, while FQL approached 0% and QC-FQL was approximately 10%-15%. Figure 5c In the cube-quadruple placement task, SAC Flow-T achieved a final success rate of approximately 55%-65%, while FQL approached 0% and QC-FQL approximately 5%-10%. This invention achieves a significant improvement in success rate across all difficulty levels of robotic arm operation tasks. This demonstrates that the method of this invention can effectively utilize pre-collected expert demonstration data and continuously improve the robotic arm's operation strategy through online reinforcement learning, enabling the robotic arm to exhibit higher control precision and success rate in complex placement tasks requiring multi-step accurate operations.

[0105] Based on the above application examples, the multi-degree-of-freedom robot continuous motion control method based on streaming strategy in the preferred embodiment of the present invention can achieve the following effects: (1) The multi-degree-of-freedom robot continuous motion control method based on streaming policy proposed in this invention is based on velocity reparameterization sequence modeling streaming policy reinforcement learning (SAC streaming algorithm), and has been extensively experimentally verified on the MuJoCo bionic robot motion control simulation platform and the OGBench industrial robotic arm operation control simulation platform. Compared with the prior art, this invention solves the gradient instability problem of robot control policy in off-policy training, and realizes stable and reliable control policy learning.

[0106] (2) In the prior art, when applying streaming policies to off-policy reinforcement learning training for robot control, the control policy often diverges or collapses due to the ill-conditioned gradients of multi-step flow expansion, resulting in the robot failing to learn effective motion or operational skills. This invention, through the gating mechanism of Flow-G and the pre-normalized residual structure of Flow-T, ensures the numerical stability of gradients during multi-step backpropagation from the physical structure of the velocity network. For example... Figures 4a to 4f As shown, SAC Flow-G and SAC Flow-T of this invention achieved stable convergence in all six biomimetic robot motion control tasks on the MuJoCo simulation platform, without any training divergence or crashes, verifying the reliability of the architecture design of this invention. This significantly improves the control performance and sample efficiency in biomimetic robot motion control tasks.

[0107] (3) This invention abandons policy distillation and proxy objectives, while retaining the advantages of streaming policies in multimodal action representation in robot control: Existing technologies, such as policy distillation (e.g., FQL) used to avoid gradient problems, compress streaming policies into single-step policies, losing the ability to express multimodal action distributions. This means that in robot tasks with multiple feasible operation paths, the distilled policy may only learn a single operation mode and cannot flexibly select the optimal path based on different initial conditions. This invention achieves end-to-end direct optimization through a stable sequence architecture, fully preserving the multimodal representation capabilities of streaming policies, enabling robot control policies to adaptively handle multi-solution task scenarios.

[0108] Another preferred embodiment of the present invention discloses a multi-degree-of-freedom robot control system based on a streaming strategy, including a memory and a processor. The processor stores a computer program, and when the computer program is executed by the processor, it implements the steps of the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy as described in the preferred embodiment above.

[0109] Optionally, the aforementioned storage media may include, but are not limited to, USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks, and other media capable of storing computer programs.

[0110] The background section of this invention may include background information about the problems or circumstances surrounding the invention, rather than a description of prior art by others. Therefore, the content included in the background section is not an admission of prior art by the applicant.

[0111] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate different embodiments or examples and features of different embodiments or examples described in this specification without contradiction. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope defined by the appended claims.

Claims

1. A method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy, characterized in that, The process includes a motion generation step, through which a final motion command is generated, and the multi-degree-of-freedom robot is driven to perform a corresponding motion based on the final motion command. The motion generation step includes: A1: Obtain the current state observation information, current intermediate action variables, and current flow time step information of the multi-degree-of-freedom robot; A2: Input the current state observation information, the current intermediate action variables, and the current flow time step information into a velocity network with a sequence model structure to calculate the output velocity field; A3: Obtain initial noise samples, and based on the initial noise samples, iteratively execute steps A1 to A2 to perform action update steps and generate final action instructions, wherein the velocity field is used to update the current intermediate action variables during each iteration.

2. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 1, characterized in that, The speed network adopts a sequence model structure based on a gating mechanism or a decoder mechanism.

3. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 1, characterized in that, The velocity network adopts a gated sequence model structure, which includes parallel gate networks and candidate networks. The velocity field is calculated by the gated sequence model structure as follows: The gate network calculates the gating coefficients based on the current state observation information, the current intermediate action variables, and the current flow time step information. The candidate network calculates candidate velocities based on the current state observation information, the current intermediate action variables, and the current flow time step information. The velocity field is calculated based on the gating coefficient, the candidate velocity, and the current intermediate action variable.

4. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 1, characterized in that, The velocity network employs a decoder-based sequence model structure, wherein the velocity field is calculated in the following manner: The current state observation information is encoded into a state embedding vector; The current intermediate action variable and the current flow time step information are combined into an action time representation; By using a cross-attention mechanism, the action time representation is used as a query vector, and the state embedding vector is used as a key vector and a value vector, to perform information fusion. The fused information is processed by layer normalization and residual connection, and then projected through the output layer to obtain the velocity field.

5. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 1, characterized in that, It also includes a strategy optimization step, wherein the strategy optimization step includes: B1: In the action update step, Gaussian noise is injected into the intermediate action variable, and a drift correction term is introduced to calculate the joint probability density of the flow unfolding trajectory from the initial noise sample to the final action command. B2: Sample data from the experience replay pool, and based on the sampled data and the joint probability density, construct and optimize the policy objective function and loss function to update the velocity network.

6. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 5, characterized in that, In step B1, the drift correction term is calculated based on the velocity field and the flow time step, and is used to counteract the influence of the injected Gaussian noise on the final action command distribution.

7. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 5, characterized in that, In step B2, the policy objective function is a policy loss function based on the soft actor-critic algorithm, and its entropy term is calculated from the joint probability density.

8. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 5, characterized in that, In step B2, when optimizing the policy objective function, a proximity regularization term is added. The proximity regularization term is calculated based on the difference in action output between the current policy and the historical policy or expert policy under the same state, and is used to constrain the policy update magnitude in sparse reward tasks.

9. The method for continuous motion control of a multi-degree-of-freedom robot based on a streaming strategy according to claim 5, characterized in that, Before step B2, a pre-training step is included, which includes: pre-training the velocity network by minimizing a flow matching objective function using an offline dataset containing expert demonstration data.

10. A control system for a multi-degree-of-freedom robot based on a streaming strategy, characterized in that, It includes a memory and a processor, the processor storing a computer program that, when executed by the processor, implements the steps of the multi-degree-of-freedom robot continuous motion control method based on a streaming strategy as described in any one of claims 1 to 9.

Images