A flow matching reinforcement learning method for jump control of humanoid robots

Abstract

The application provides a flow matching reinforcement learning method for humanoid robot jumping control, and the method comprises the following steps: obtaining training samples comprising task instructions, color images, depth images, body states and future joint action blocks; respectively encoding and mapping each mode to a unified hidden space to splice a joint condition tensor; using an action expert network to predict the velocity field of the noise action block to the real action block, and performing flow matching pre-training based on joint-level physical weight, phase sampling, action smoothing and modal balance constraint; further training the joint action block as a macro action in a physical simulation closed loop, obtaining joint log-likelihood through random reparameterization, and performing proximal policy optimization after training. The application can improve the visual contact ability, closed loop stability and action smoothness of the humanoid robot jumping action generation, and reduce the distribution deviation between offline pre-training and actual execution.

Description

Technical Field

[0001] This invention belongs to the field of robot motion control and artificial intelligence technology, and particularly relates to a flow matching reinforcement learning method for jump control of a humanoid robot. More specifically, this invention relates to a humanoid robot control method that integrates natural language task instructions, color images, depth images, and robot body states into a unified conditional representation, and generates future joint motion blocks through physical prior flow matching pre-training and macro-motion reinforcement learning post-training. Background Technology

[0002] Humanoid robots need to perform movement, obstacle crossing, and manipulation tasks in environments designed for humans. For scenarios involving stepping stones, steps, tray blocks, and spaced obstacles, control strategies relying solely on the robot's own state struggle to obtain external information such as the target landing point, terrain boundaries, and the relative pose of obstacles. Decision-making methods relying solely on visual recognition also struggle to directly generate high-frequency joint sequences that conform to full-body dynamics constraints. Therefore, high-dynamic jump control requires unifying visual grounding, task semantic understanding, and full-body motion generation into a single closed-loop control chain.

[0003] Existing vision-language-action models can map images and natural language to robot actions, but related solutions are mostly geared towards desktop robotic arm operations or low-speed movement tasks. For humanoid robots standing and jumping, the action process includes phases such as charging, takeoff, airborne, landing cushioning, and recovery, presenting challenges such as rapid switching of contact states, underactuation during the airborne phase, impact absorption upon landing, and precise landing point control. Directly transferring action regression, diffusion sampling, or frame-by-frame reinforcement learning training methods from low-dynamic tasks to this scenario can easily lead to problems such as motion averaging, amplified landing point deviations, high-frequency motion jitter, and unstable closed-loop execution.

[0004] Furthermore, existing generative action strategies typically treat each frame of action or each denoising step as an independent optimization object during online post-training, leading to difficulties in credit allocation within long action blocks and easily disrupting the overall structure of continuous actions. Modality utilization degradation may also occur during the multimodal fusion stage, where the strategy over-relies on text or single-path visual features while failing to fully utilize depth geometry and target boundary information. Therefore, a high-dynamic action generation method for humanoid robots that combines multimodal condition construction, physical prior flow matching pre-training, and macro-action reinforcement learning post-training is needed. Summary of the Invention

[0005] The purpose of this invention is to provide a flow matching reinforcement learning method for humanoid robot jumping control, which can improve the visual grounding ability, closed-loop stability and motion smoothness of humanoid robot jumping action generation, and reduce the distribution offset between offline pre-training and actual execution.

[0006] The core technical problem to be solved by this invention is: how to enable a humanoid robot to generate continuous, smooth, and loop-executable jumping action blocks based on natural language, color images, depth images, and body state without modifying the underlying hardware of the humanoid robot, by utilizing upper-level multimodal visual language action strategies.

[0007] Specifically, the following issues are included: (1) Multimodal condition construction problem: How to convert language instructions, RGB color images, depth images and robot body states into conditional representations that can be uniformly utilized by action generation networks.

[0008] (2) High dynamic motion generation problem: How to generate joint targets for multiple future frames at once in a jumping task so that the charging, take-off, airborne, landing buffer and recovery phases remain as a whole.

[0009] (3) Physical prior injection problem: How to reflect the different importance of different joints to the jumping task in the flow matching pre-training, and avoid the use of the same supervision intensity for all joints, which leads to insufficient learning of key lower limb movements.

[0010] (4) Modal collapse suppression problem: How to avoid the model relying solely on textual instructions or fixed scene statistics while ignoring visual / depth landing information.

[0011] (5) Closed-loop distribution offset problem: How to put the action prior obtained by offline pre-training into the physical simulation closed loop for further reinforcement learning training, so that the model can better adapt to the state offset, landing point error and landing impact in real execution.

[0012] (6) Deployment cost issues: How to leverage the grounding capabilities of multi-perspective teacher strategies while reducing input perspectives and computational load during actual deployment.

[0013] A flow matching reinforcement learning method for jump control of a humanoid robot includes the following steps: Step S1: First, obtain multimodal training samples, which include natural language task instructions, at least one color image, at least one depth image, robot body state, and future joint motion blocks.

[0014] Step S2: Perform heterogeneous feature encoding on the natural language task instructions, color images, and depth images respectively to obtain feature tokens for each input modality, and map the features of each modality to a unified latent space through a projector that corresponds to each modality and does not share parameters.

[0015] Specifically, a task-level semantic encoder, a visual encoder, and a deep encoder are used to obtain language semantic features, color visual features, and deep geometric features, respectively. Then, a non-shared weight projector corresponding to each modality is used to map each feature to a unified latent space.

[0016] Step S3: Concatenate the mapped feature tokens along the sequence dimension to obtain a joint conditional tensor, and construct the input tensor of the motion expert network based on the robot body state and the noisy action blocks.

[0017] Step S4: In the pre-training phase, the motion expert network receives noisy motion blocks, continuous time steps, robot body states, and joint conditional tensors, and predicts the velocity field from the noisy motion blocks to the real motion blocks. The training objective adopts a physical prior flow matching loss, which includes joint-level weighting, kinematic perception variance weighting, phase sampling strategy, third-order difference smoothing constraint for motion blocks, and modal equalization constraint based on attention proportion.

[0018] Step S5: In the post-training phase, the pre-trained motion expert network is placed in a physical simulation closed-loop environment, and the future joint actions output by a single control decision are divided into blocks as indivisible macro-actions. By introducing a noise injection network during the flow-matched Euler integral process, the deterministic ordinary differential equation sampling is transformed into stochastic differential equation sampling with computable joint log-likelihood. Based on the joint log-likelihood, a probability ratio for proximal policy optimization is constructed, and parameters are efficiently updated by combining the value network, behavior clone anchors, and action smoothing loss.

[0019] Step S6: Utilize the post-trained motion expert network to output future joint motion blocks based on current multimodal observations, in order to control the humanoid robot to perform jumping or obstacle-crossing actions.

[0020] In the optional deployment phase, a multi-view post-training strategy is used as the teacher strategy to construct a student strategy that retains only single-view color images and single-view depth images as inputs. Distillation is performed through velocity field alignment, action output alignment, and intermediate feature alignment to reduce the number of sensors and computational overhead in the deployment phase.

[0021] Preferably, the future joint motion block in step S1 includes the joint target sequence of the future H frames, where H is 8 to 30.

[0022] Preferably, the heterogeneous feature encoding in step S2 includes: Language semantic features are obtained through a task-level semantic encoder; Color visual features are obtained through at least one visual encoder; Depth geometric features are obtained through at least one depth encoder; The task-level semantic encoder, visual encoder, and deep encoder are at least partially frozen during the pre-training and post-training phases.

[0023] Preferably, the projector in step S2 is a multilayer perceptron with non-shared weights configured according to modality; The projector includes a linear layer, a nonlinear activation layer, and a normalization layer connected in sequence, used to map language features, color visual features, and deep geometric features of different dimensions to the same latent space dimension D.

[0024] Preferably, step S3 further includes maintaining a modal slice dictionary for the joint condition tensor; The modality slice dictionary records the start and end positions of each modality feature in the joint conditional tensor, and is used to perform modality independent discarding, cross-attention proportion equalization, and modality contribution diagnosis.

[0025] Preferably, the motion expert network is a Transformer-based motion generation network, including an motion sequence encoding layer, a self-attention layer, a cross-attention layer, an adaptive normalization layer, a feedforward network layer, and a velocity field output layer; The cross-attention layer uses the action token as the query and the joint condition tensor as the key and value.

[0026] The action expert network is a Transformer-based action generation network, comprising a sequentially connected action sequence encoding layer, multiple action generation blocks, and a velocity field output layer. The action sequence encoding layer receives noisy action blocks and encodes them as action tokens. The multiple action generation blocks are stacked sequentially along the action sequence propagation direction. Each action generation block includes a sequentially connected self-attention layer, a cross-attention layer, an adaptive normalization layer, and a feedforward network layer. The self-attention layer models the temporal dependencies between action tokens. The cross-attention layer uses the action tokens normalized by the aforementioned adaptive layer as queries, and the joint conditional tensor as keys and values, to inject linguistic, visual, and depth conditions into the action generation process. The velocity field output layer is connected to the output of the last action generation block and outputs a predicted velocity field of the same dimension as the noisy action block.

[0027] Preferably, the flow matching loss in step S4 includes a velocity field regression loss based on a linear interpolation path; The linear interpolation path is divided into blocks based on noise action. Segmentation with real motion The supervised velocity field learned by the action expert network is constructed as follows: .

[0028] Preferably, the joint-level physical weights in step S4 are determined based on the dynamic effects of different joints in the jumping task and / or the kinematic variance in the training samples, so that the lower limb take-off related joints, the waist posture adjustment joints and the upper limb arm swing joints have different supervision intensities. The motion smoothing constraint in step S4 includes a third-order difference smoothing constraint calculated along the time dimension of the motion block, which is used to suppress high-frequency jitter in the future joint motion block. The modal equalization constraints in step S4 include: performing independent random discarding on the text modality, color visual modality, and depth modality; calculating the attention percentage of each visual modality in the cross-attention matrix based on the modality slice dictionary; and introducing a penalty term when the attention percentage of any visual modality is lower than a preset lower limit.

[0029] Preferably, the randomization reparameterization in step S5 includes introducing a noise injection network during the flow-matched Euler integral process, converting deterministic ordinary differential equation sampling into random differential equation sampling, and obtaining the joint log-likelihood of the macro-action based on the sum of the transition probabilities of each integration step.

[0030] Preferably, the post-training includes constructing the probability ratio of the new and old policies based on the joint log-likelihood, and efficiently bypassing the parameters of the action expert network by updating the pruning target, value network loss, behavior clone anchor loss and action smoothing loss through proximal policy optimization. The parameter efficient bypass is a low-rank adaptive bypass, which is injected into at least one linear layer of the action expert network. The dense parameters obtained during pre-training are kept frozen or partially frozen during the post-training stage. The value network is a generative value network, which reuses flow matching conditions and / or hidden representations of action experts to estimate the value of the closed-loop simulation state.

[0031] This invention also proposes a flow matching reinforcement learning device for humanoid robot jump control, comprising: a sample acquisition module, a heterogeneous feature encoding module, a joint condition construction module, an action generation module, a flow matching pre-training module, a reinforcement learning post-training module, and an action output control module; wherein each module is configured to execute a flow matching reinforcement learning method for humanoid robot jump control.

[0032] This invention also proposes a humanoid robot control system, including a perception interface, a body state interface, a processor, a memory, and a control interface for connecting to an existing robot underlying controller. The memory stores a computer program, which, when executed by the processor, implements a flow matching reinforcement learning method for humanoid robot jump control. The computer program outputs future joint motion blocks or joint targets obtained by parsing the future joint motion blocks to the existing robot underlying controller through the control interface. The control interface does not limit the hardware structure of the existing robot underlying controller.

[0033] Compared with the prior art, the advantages of the present invention are: (1) By using heterogeneous modal independent encoding and unified projection, language, RGB, depth and ontology state are converted into a unified conditional representation, enabling motion experts to utilize task semantics, visual appearance, depth geometry and robot state at the same time.

[0034] (2) By generating future motion blocks through flow matching velocity field, the multi-step noise reduction overhead of traditional diffusion motion head is reduced, which is conducive to meeting the high-frequency control requirements of humanoid robots.

[0035] (3) By adjusting the joint-level physical weights and kinematic variance, key joints such as the lower limbs can be more fully supervised in jumping tasks, thereby improving the quality of take-off and landing.

[0036] (4) By using phase sampling, key high-dynamic phases such as take-off, flight and landing cushioning can be more covered in training, reducing the risk of only learning steady-state movements.

[0037] (5) By using third-order difference smoothing constraints, the high-frequency jitter of the joint target sequence is reduced, and the continuity of the action and landing stability are improved.

[0038] (6) By modal independent discarding, attention balance and spatial attention diagnosis, the excessive dependence of multimodal models on a single modality is reduced and the visual grounding robustness is improved.

[0039] (7) By dividing the action blocks into macro actions and training them after Flow-PPO, the discontinuity problem caused by frame-by-frame credit allocation is reduced, and the pre-trained action prior can be corrected in the closed-loop physical environment.

[0040] (8) By reparameterizing ODE to SDE, the joint log-likelihood can be calculated for flow matching action blocks, so that generative action strategies can be combined with PPO updates.

[0041] (9) Through teacher-student distillation, multi-view information can be utilized during the training phase, and the number of sensors and model computation can be reduced during the deployment phase.

[0042] (10) Since the present invention outputs action blocks through the existing robot underlying control interface, it does not depend on the hardware structure of a specific manufacturer, and can be transferred to humanoid robots with different degrees of freedom, different sizes or different brands. Attached Figure Description

[0043] Figure 1 The flowchart shows a flow matching reinforcement learning method for jump control of humanoid robots.

[0044] Figure 2 This is a diagram of the control system architecture for a humanoid robot.

[0045] Figure 3 This is a five-way heterogeneous feature alignment diagram.

[0046] Figure 4 This is a diagram of the motion expert network structure.

[0047] Figure 5 Diagram to prevent modal collapse mechanism.

[0048] Figure 6 This is a flowchart of the training process after Flow-PPO. Detailed Implementation

[0049] The following will describe in more detail a flow matching reinforcement learning method for jump control of a humanoid robot according to the present invention, with reference to the schematic diagrams, which illustrate preferred embodiments of the invention. It should be understood that those skilled in the art can modify the invention described herein while still achieving its advantageous effects. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the invention.

[0050] Explanation of terms and symbols: Vision-Language-Action (VLA) refers to a control strategy that simultaneously utilizes visual observations, natural language task instructions, and the robot's own state to generate robot actions. In this paper, "Vision-Language-Action" refers only to the upper-level action generation algorithm and does not represent the robot's underlying servo hardware.

[0051] Flow Matching (FM): A generative modeling method. Instead of directly regressing a single action point, it learns the velocity field that gradually flows from random noise actions to real actions, used to generate continuous action blocks.

[0052] Future action block: refers to the sequence of joint targets in the next H frames output by a single policy inference. For example, in a preferred embodiment, the output of H=15 future frames, with 29-dimensional joint targets per frame, is collectively referred to as an action block.

[0053] Macro-actions: This refers to breaking down a complete action into blocks as an action unit in reinforcement learning, rather than breaking down each frame's joint target into independent small actions. This design is used to maintain the continuity of jumping actions.

[0054] Ordinary Differential Equations (ODE) and Stochastic Differential Equations (SDE): ODE is used to describe deterministic flow matching sampling paths; SDE is a stochastic sampling process obtained by introducing controllable random noise into the path, used to calculate action probabilities and perform reinforcement learning updates.

[0055] Proximal Policy Optimization (PPO) is a reinforcement learning policy update method that improves training stability by limiting the magnitude of changes between the old and new policies. This paper uses a macro-action PPO post-training method modified for stream matching action blocks.

[0056] Joint conditional tensor: refers to the conditional input formed by mapping language features, color image features, depth image features and robot body state features to the same latent space and then splicing them together, for use by motion expert networks.

[0057] Robot body state: refers to the state information of the humanoid robot itself, including but not limited to root posture, root velocity, joint angle, joint velocity, foot contact state, etc.; the body state comes from the sensor / state interface of the robot or simulator.

[0058] In the following text, "robot body state" will be referred to as "body state".

[0059] Motion expert network: refers to a motion generation network based on Transformer. Its inputs are noisy motion blocks, time steps, body states, and joint condition tensors, and its output is a velocity field with the same shape as the motion blocks.

[0060] Physics priors: These refer to training constraints set based on the physical characteristics of humanoid robot jumping tasks, such as lower limb joint weights, motion phase sampling, third-order differential smoothing constraints, and landing stability rewards.

[0061] Modal collapse refers to a multimodal model becoming overly reliant on a single type of input after training, such as relying solely on textual instructions while ignoring visual context information, resulting in insufficient visual grounding ability in real-world scenarios.

[0062] Teacher-student distillation refers to first training a multi-view teacher policy with more complete input information, and then transferring its action outputs and intermediate features to a single-view student policy with less input and a smaller model for deployment.

[0063] Robot underlying controller: refers to the existing motor servo, joint drive, and communication control system of robot manufacturers. This invention does not modify this underlying hardware, but only outputs upper-level motion blocks or joint targets to it.

[0064] "Stream matching reinforcement learning method" refers to the training and reasoning method of upper-level visual language action strategies; "Humanoid robot jump control" is used to define the application scenario. This name does not imply any improvement to the underlying hardware structure of Unitree Robotics or other robot manufacturers.

[0065] Table 1. Explanation of Symbols

[0066] This invention can be applied to the motion generation and control of humanoid robots, bipedal robots, or other multi-jointed legged robots in highly dynamic obstacle-crossing scenarios such as stepping stones, steps, tray blocks, gaps, and other obstacles requiring visual grounding. The method can be implemented on simulation training platforms, upper-level robot controllers, edge computing devices, or cloud-based training devices, and can output joint targets through existing low-level control interfaces, demonstrating clear engineering application value.

[0067] 1. Overall methodology and process: like Figure 1 As shown in the figure, from top to bottom, the process is as follows: multimodal sample acquisition, heterogeneous feature encoding, joint condition construction, flow matching pre-training, Flow-PPO post-training, teacher-student distillation, and action output control.

[0068] The controller acquires multimodal observations of the humanoid robot in the target jumping scenario.

[0069] Multimodal observations include natural language task commands, color images, depth images, and robot body state. Training samples also include future joint motion blocks as supervisory signals. The controller inputs the multimodal observations into a trained motion generation strategy, which outputs joint target motion blocks for future H frames. Based on these joint target motion blocks, the controller drives the humanoid robot to perform standing jumps, stepping stone crossings, or stair obstacle crossings.

[0070] The humanoid robot control system adopts a dual-system hierarchical structure, such as Figure 2 As shown in the figure, the low-frequency task-level semantic encoder is responsible for extracting language and visual semantic conditions of the first frame, while the high-frequency motion expert is responsible for generating motion blocks based on the current RGB-D, ontology state, and semantic conditions.

[0071] System 2 completes semantic-level encoding at the start of the task, encoding the task instructions and initial visual observations into task-level semantic conditions. System 1 continuously receives the current RGB-D observations, the ontology state, and the semantic conditions output by System 2 in the control loop, generating action blocks at high frequency. If the visual sensor frequency is lower than the control frequency, System 1 can reuse the previous visual condition tensor through zero-order hold, thereby reducing visual encoding overhead without reducing the control output frequency.

[0072] 2. Construction of multimodal conditions, including steps S1 to S3.

[0073] like Figure 3 As shown, the multimodal conditional construction includes five types of feature sources: text or first frame semantic features, first-view color visual features, second-view color visual features, first-view depth geometric features, and second-view depth geometric features.

[0074] Figure 3 The text, front view RGB, top view RGB, front view depth, and top view depth are mapped to a unified latent space through independent encoders and projectors, respectively, and then concatenated into a joint conditional tensor, and a modal slice dictionary is recorded.

[0075] Training samples can consist of simulation data, redirected motion data, or data acquired from a real device. Each sample must include at least: task instructions, visual observations, depth observations, ontological state, and future action blocks.

[0076] Task instructions are used to describe the jumping objective, such as "jump to the center of the tray in front," "jump over the rocks in front," or "jump to the stepping stone to the right front." Task instructions do not need to be limited to a fixed sentence structure; they only need to express the target object, direction, distance, or landing point requirement.

[0077] Visual observations provide visual information about the appearance of stepping stones, trays, obstacles, or landing areas. Depth observations provide metric distances, edge heights, top surface geometry, and relative positions. Combining the two can reduce the scale uncertainty caused by relying solely on RGB images.

[0078] The body state represents the robot's current physical state and may include root position / pose, root velocity, joint angles, joint velocities, foot contact state, and historical actions. In practice, some or all of these parameters can be selected based on the robot's state interface.

[0079] The future action block is a supervisory signal that represents the joint target to be output in the next H frames starting from the current moment. This action block can come from human action redirection, simulation expert strategies, manually designed trajectories, or training strategies from the previous stage.

[0080] It should be noted that the five-channel feature source is the preferred implementation; in other embodiments, one channel of color image, one channel of depth image, or more than two channels of visual / depth input can also be used.

[0081] To avoid forcibly mixing different modalities, this invention uses corresponding encoders for language, RGB images, and depth images. The language or task-level semantic encoder obtains task semantic features, the color image encoder obtains texture and appearance features, and the depth encoder obtains distance and geometric features.

[0082] Since different encoders output different dimensions, lengths, and statistical distributions, this invention sets up an independent projector for each modality, mapping it to a unified latent space. Each projector does not share weights, in order to adapt to different feature distributions such as language, RGB, and depth.

[0083] For any mode Its projector can be represented as:

[0084] In formula (1): Indicates the first The projector function corresponding to each mode. The mode It can be a text modal, a first-view color image modal, a second-view color image modal, a first-view depth image modal, or a second-view depth image modal.

[0085] Indicates the input number The original modal feature sequence of a projector can come from a task-level semantic encoder, visual encoder, or depth encoder.

[0086] Indicates the first The weight matrix of the first linear layer in each projector is used to represent the original input dimension of the modality. Mapped to the intermediate latent space dimension.

[0087] Indicates the first The bias vector of the first linear layer in each projector.

[0088] Indicates the first The weight matrix of the second linear layer in each projector is used to map intermediate latent space features to a unified latent space dimension. .

[0089] Indicates the first The bias vector of the second linear layer in each projector.

[0090] This represents the Gaussian error linear unit activation function, used to enhance nonlinear expressive power.

[0091] The representation layer normalization operation is used to adjust the features projected from different modalities to a similar statistical scale.

[0092] The mapped modal features are concatenated along the sequence dimension to form a joint conditional tensor:

[0093] In formula (2): This represents the joint conditional tensor, which serves as the source of keys and values ​​for the cross-attention layers of the action expert network.

[0094] This indicates an operation that splices along the sequence dimension.

[0095] Represents a sequence of semantic features at the text or task level.

[0096] This represents the feature sequence of a first-person color image after processing by a visual encoder and projector.

[0097] This represents the feature sequence of a second-view color image after processing by a visual encoder and projector.

[0098] This represents the feature sequence of a first-view depth image after processing by a depth encoder and projector.

[0099] This represents the feature sequence of a second-view depth image after processing by a depth encoder and projector.

[0100] This indicates that the splicing occurs at the token sequence level, rather than the feature channel level.

[0101] If the actual deployment only has one image or one depth signal, the above stitching can be performed by deleting missing modalities or replacing them with zero features / masks.

[0102] Simultaneously maintain the modal slice dictionary Used to record each mode in The start and end positions in the slice dictionary. This slice dictionary is invoked in modality-independent discarding, attention-based balancing, and ablation diagnostics.

[0103] If the first The length of the projected sequence for each modality is Then it can be placed in The slice range in the code is represented as follows:

[0104] In the formula: Indicates the first Each mode in the joint conditional tensor The start and end index range in the index.

[0105] Indicates the first The number of tokens after modal projection.

[0106] Indicates the first The starting position of each modal slice.

[0107] Indicates the first The end position of a modal slice.

[0108] Half-open interval This indicates that the starting index is included, but the ending index is not.

[0109] For example, text features occupy tokens 1 to 64, and front-look RGB features occupy tokens 65 to 1264. This slice dictionary is used for subsequent modality-independent discarding, attention percentage statistics, and modality contribution diagnosis.

[0110] The body state may include at least one of the following: root node pose, joint angle, joint velocity, IMU observation, and foot contact state.

[0111] In one embodiment, the body state encoder concatenates the root node pose and joint angles into a state vector:

[0112] The state vector is mapped to the unified latent space via a multilayer perceptron to obtain the state token. .

[0113] In equation (4): Indicates time The robot's body state vector.

[0114] Indicates the robot root node at time [time]. The attitude quaternions, in order are , , , .

[0115] Indicates the robot at a certain moment The controlled joint angle vector.

[0116] This indicates a vector concatenation operation.

[0117] The state vector is mapped to a state token by the state encoder:

[0118] In equation (5): This represents the body state token, used to provide the motion expert network with the current robot pose and joint state.

[0119] and These represent the weight matrix and bias vector of the first linear layer of the state encoder, respectively.

[0120] and These represent the weight matrix and bias vector of the second linear layer of the state encoder, respectively.

[0121] and The meaning is the same as in the aforementioned projector formula.

[0122] The future noise-adding action is represented in blocks as follows:

[0123] In formula (6): Indicates from time Start, length is The future noise-adding action block.

[0124] Indicates the first Noisy action vectors at future time positions, where .

[0125] , and The meaning of is given in the aforementioned symbol table.

[0126] In one embodiment, for the first The sample, the first The future time location, the first Joint-level affine calibration is performed on the noise-added motion values ​​of each joint:

[0127] In equation (7): This indicates the calibrated action value.

[0128] Indicates the first noisy action block The sample, the first The future time location, the first The original noise-added motion values ​​of each joint.

[0129] Indicates the first The learnable scale coefficient of each joint.

[0130] Indicates the first The learnable bias coefficients of each joint.

[0131] For batch sample index, Indexing for future action frames, For joint indexing.

[0132] The action sequence is then encoded into an action token and concatenated with the state token:

[0133] In equation (8): This represents the input tensor of the action expert network.

[0134] This represents the aforementioned ontology state token.

[0135] This represents the action token obtained by the action encoder from the noisy action block.

[0136] 3. Physical prior flow matching pre-training, i.e. step S4.

[0137] Motion expert networks can be understood as "core networks responsible for translating multimodal conditions into future joint movements".

[0138] like Figure 4 As shown, the motion expert network includes an action sequence encoding layer, multiple action generation blocks, and a velocity field output layer. This figure illustrates that the motion expert network comprises action encoding, self-attention, cross-attention, adaptive layer normalization, a feedforward network, and a velocity field output.

[0139] Action sequence coding layer receives noisy action blocks This is mapped to an action token, and time-position encoding and joint topology embedding are added.

[0140] The action generation blocks are connected in the order of "self-attention layer, cross-attention layer, feedforward network layer".

[0141] The noisy action blocks are intermediate action samples constructed during stream matching training, which contain both noise information and some real action trends.

[0142] In the In each action generation block, the self-attention layer first models the temporal dependencies between future action frames:

[0143] Subsequently, the cross-attention layer uses the action token as the query and the joint conditional tensor as the key and value to inject language, vision, depth, and ontology conditions into the action generation process:

[0144] Finally, the feedforward network layer updates the hidden state:

[0145] In the formula: This indicates the layer index of the action generation block.

[0146] Indicates the first The hidden state of the layer input.

[0147] This represents the hidden state after updating the self-attention residual.

[0148] This represents the hidden state after updating the residuals through cross-attention.

[0149] Indicates the first The hidden state of the layer output.

[0150] , and These represent the gating coefficients of the self-attention branch, the cross-attention branch, and the feedforward branch, respectively.

[0151] This indicates element-wise multiplication.

[0152] This represents multi-head self-attention, used to model the temporal dependencies between different future frames within an action block.

[0153] This represents cross-attention, used to query with an action token and a joint condition tensor. Inject language, vision, and depth conditions into keys and values.

[0154] This indicates the adaptive layer normalization operation, whose modulation amount is generated by time step embedding and conditional embedding.

[0155] This represents a modulation embedding generated from continuous time steps and conditional information.

[0156] The mask representing the joint conditional tensor is used to mask invalid tokens for padding positions or discarded modalities.

[0157] This represents a feedforward network.

[0158] The adaptive normalization layer adjusts the network's hidden state based on time steps and conditional embeddings to keep the velocity field prediction stable across different sampling times. The action expert network then passes through an output projection layer (velocity field output layer) to obtain a velocity field with the same shape as the action blocks.

[0159] In equation (12): This indicates that the motion expert network is in continuous time. Intermediate action samples and conditions The predicted velocity field is obtained.

[0160] This represents the parameters of the motion expert network.

[0161] This indicates the total number of layers in the action generation block.

[0162] This indicates the output projection layer, whose output shape is consistent with the shape of the action block.

[0163] This velocity field indicates which direction a noisy action should move in order to gradually become a real, executable action under the current time and multimodal conditions.

[0164] The basic idea of ​​flow matching pre-training is to start from random noise actions and gradually generate realistic actions by learning the velocity field. Compared with direct action regression, flow matching is more suitable for representing a variety of actionable patterns; compared with traditional diffusion models, flow matching can complete sampling in fewer steps.

[0165] Physics-prior flow matching pre-training based on noisy action blocks Segmentation with real motion Constructing a linear path:

[0166] In equation (13): Representing continuous time Intermediate action samples at the location.

[0167] This represents a noise action block obtained from sampling a standard Gaussian distribution.

[0168] This represents the corresponding real action block in the training sample.

[0169] This indicates that the stream matches continuous-time variables, with a value range of [value range missing]. .

[0170] and These represent the weights of the noisy action block and the real action block in the interpolation path, respectively. The supervised velocity field corresponding to this path is:

[0171] In equation (14): This represents the supervised velocity field of a linear path.

[0172] and The meaning of is as described above.

[0173] because right The derivative is Therefore, the aforementioned supervised velocity field can be directly used as the regression target for motion expert networks to predict the velocity field.

[0174] Action expert network output and with error Constructing a joint-level weighted flow matching loss:

[0175] In equation (15): This represents the joint-level weighted flow matching loss.

[0176] Indicates the number of samples in the batch. Indicates the length of the action block. Indicates the number of controlled joints.

[0177] Indicates the batch sample index. Indicates the index of the future action frame. This represents a joint index.

[0178] Indicates the first The physical weights of each joint are used to reflect the dynamic contribution of that joint in the jumping task.

[0179] This represents the normalization factor, which can be set to... Or its equivalent normalization constant.

[0180] Indicates the predicted velocity field in the th... The sample, the first Frame, First The components at each joint.

[0181] Indicates the supervised velocity field at the th The sample, the first Frame, First The components at each joint.

[0182] This represents the squared norm 2; since it acts on the scalar component, it can also be understood as the squared error.

[0183] For humanoid robot jumping tasks, the ankle, knee, and hip joints related to jumping can be given higher weights, the waist posture adjustment joints and upper limb arm swing joints can be given medium weights, and auxiliary joints that contribute less to the landing point can be given lower weights. These weights can be determined either by prior rules or adaptively by the kinematic variance in the training samples.

[0184] In jumping tasks, the lower limb joints such as the ankle, knee, and hip directly determine the takeoff momentum and landing cushioning, and should therefore have high [performance / capacity]. The waist joint affects takeoff posture and landing stability, so it can be set to a medium weight; the upper limb swing joint affects angular momentum adjustment, so it can also be set to a task-related weight; auxiliary joints that contribute less to the landing point can be set to a lower weight.

[0185] To enhance the supervision density of high dynamic phases, kinematic perception weights can be constructed based on acceleration or phase annotations within action blocks:

[0186] In equation (16): Indicates the first The sample, the first Kinematic perception weights of frames.

[0187] This represents the kinematic weighting adjustment coefficient.

[0188] Indicates the first The sample, the first The second-order difference of the actual motion in the time dimension can approximately reflect the degree of drastic change in motion acceleration or phase.

[0189] This represents the smallest positive number that prevents the denominator from being zero.

[0190] It represents the average scale of acceleration amplitude within the same action block.

[0191] This indicates the index used when summing the time positions within an action block.

[0192] Combining joint-level weights and kinematic perception weights, we obtain the physical-weighted flow matching loss:

[0193] In the formula: This represents the physical prior weighted flow matching loss.

[0194] Used to emphasize the time and position of dramatic changes in motion, such as takeoff, flight, and landing cushioning.

[0195] Used to emphasize the different contributions of different joints to jump dynamics.

[0196] All other symbols have been defined above.

[0197] To suppress high-frequency jitter in the joint target sequence, this invention estimates the motion... Apply smoothing constraints to the third-order difference:

[0198] In formula (18): This represents the third-order difference smoothing loss.

[0199] The action block representing the policy prediction The sample, the first Frame, First The target value for each joint.

[0200] This indicates that the third-order difference is in length . The number of calculable positions on the action block.

[0201] The combination term in parentheses represents the third-order difference along the time dimension, which physically corresponds to the rate of change of acceleration, i.e., jerk.

[0202] A jumping motion is not a smooth, uniform movement, but rather consists of phases such as steady state, power accumulation, takeoff, flight, landing, and recovery. The takeoff and landing phases have a greater impact on the success rate, and if training segments are sampled uniformly, the model may not learn these key phases sufficiently.

[0203] Therefore, this invention can adjust the sampling probability according to the motion phase, thereby achieving higher supervision density during the takeoff, flight, and landing cushioning phases. The phase label can be determined by foot contact, center of mass height, root velocity, knee flexion, or manual annotation.

[0204] To suppress high-frequency motion jitter, this invention applies a smoothing constraint to the third-order difference of the motion block along the time dimension. The third-order difference can be understood as the rate of change of the motion change rate, used to penalize sudden jitter.

[0205] A common problem with multimodal models is modal collapse. For example, in training data, if text instructions are highly correlated with a fixed scene, the model may predict actions based solely on the text without actually reading visual / depth information. In real-world deployments, the model is prone to failure if the position or shape of the stepping stone changes.

[0206] To this end, the present invention employs three mechanisms: First, independent modality discarding, that is, randomly occluding a certain modality during training, so that the model cannot rely on a single input for a long time; Second, attention ratio balancing, that is, statistically analyzing the distribution of cross-attention on different modalities based on the modality slice dictionary, constraining the model to maintain reasonable multimodal utilization; Third, spatial attention diagnosis, that is, checking whether visual attention is concentrated near stepping stones, landing points, or obstacle edges.

[0207] like Figure 5 As shown, the anti-modal collapse mechanism consists of three parts: independent modal discarding, attention ratio balancing, and spatial attention diagnosis.

[0208] Modal independent discard can be represented as:

[0209] In equation (19): This indicates the number of items after random discarding. Modal features.

[0210] Indicates the number before being discarded. Modal features.

[0211] Indicates the first A Bernoulli random variable with multiple modes; This indicates that the mode is retained. This indicates that the modality is discarded.

[0212] Indicates the first The probability of dropping each modality.

[0213] This represents the set of modalities participating in the training.

[0214] Represented by probability Values The Bernoulli distribution.

[0215] Based on modal slice dictionary The first one can be calculated The proportion of each modality in cross-attention:

[0216] In equation (20): Indicates the first The proportion of cross-attention across modalities.

[0217] This represents the cross-attention matrix.

[0218] Indicates the first The action query token is for the first Attention weights for each condition token.

[0219] This indicates the index of the action query token.

[0220] Indicates the index of the condition token.

[0221] Indicates the first The set of condition token indexes corresponding to each modality.

[0222] The molecule represents all actions. The query token points to the first... The sum of attention weights for each modal conditional token.

[0223] The denominator represents the sum of attention weights for all action query tokens pointing to all condition tokens.

[0224] When the attention share of any visual or depth modality falls below a preset lower limit, a balance penalty is introduced:

[0225] In equation (21): This indicates the loss of modal attention balance.

[0226] This represents the set of visual and depth modalities that require attentional balance constraints.

[0227] Indicates the first The minimum attention percentage threshold for each modality.

[0228] The meaning of is as described above.

[0229] Indicates when Below the threshold A second penalty is generated when This item is zero when it is not lower than the threshold.

[0230] The total pre-training loss is:

[0231] In equation (22): This represents the total loss during the pre-training phase.

[0232] This represents the physical prior weighted flow matching loss.

[0233] This represents the third-order difference smoothing loss.

[0234] This indicates the loss of modal attention balance.

[0235] The weighting coefficients represent the smoothing loss.

[0236] The weighting coefficients represent the modal equalization loss.

[0237] 4. Flow-PPO reinforcement learning followed by training, i.e., step S5.

[0238] The pre-training phase primarily involves the model mimicking demonstrated actions, but a robot's actual jump is a closed-loop physical process: the takeoff angle, aerial posture, and landing impact all affect subsequent states. To reduce the distributional offset between offline pre-training and closed-loop execution, this invention places the pre-trained motion expert into a physical simulation environment for further post-training.

[0239] The key aspect of this invention is that the entire H-frame action is divided into blocks as a single macro-action. Upon receiving a macro-action, the environment executes these joint objectives sequentially according to the control cycle. After execution, it returns the reward, next state, and termination information. This avoids disrupting the continuity within the action block by updating PPO frame by frame.

[0240] Figure 6 It represents the data flow between the simulation environment, motion experts, macro-motion blocks, ODE to SDE reparameterization, joint log-likelihood, PPO update, and value network.

[0241] like Figure 6 As shown, post-Flow-PPO training divides the future actions of a single control decision output into blocks as macro-actions. Let the macro-action span... For each environmental step, the returns and advantages can be calculated based on the macro-action scale:

[0242] In equation (23): Indicating from macro decision-making moments Start, length is Macro-action discount return.

[0243] This indicates the number of low-level environment steps covered by the macro action.

[0244] This represents the discount factor.

[0245] Indicates low-level environment step Instant rewards.

[0246] This indicates the low-level environment step index within the macro action.

[0247] The timing difference error of the macro-motion scale is:

[0248] In equation (24): This represents the timing difference error of the macroscopic motion scale.

[0249] This indicates the value network at macro decision-making moments. For conditional states Value estimate.

[0250] Indicates the conditional state after the macro action has finished executing. Value estimate.

[0251] This represents the parameters of the value network.

[0252] To enable the flow matching strategy to be used for near-end policy optimization, this invention incorporates a noise injection network into the Euler integral to form a stochastic differential equation sampling:

[0253] In equation (25): Representing continuous time The minute changes in the action sample.

[0254] This represents the velocity field predicted by the motion expert network.

[0255] This represents the continuous-time differential step size.

[0256] This represents the diffusion coefficient of the noise injection network output.

[0257] This represents the parameters of the noise injection network.

[0258] This represents the Wiener process increment, used to introduce controlled randomness.

[0259] Its discrete form is:

[0260] In equation (26): Indicates the first A sample of motion steps in an Euler integral.

[0261] Indicates the first A sample of motion steps in an Euler integral.

[0262] Indicates the first The continuous time corresponding to each integration step.

[0263] This represents the Euler integral step size.

[0264] Indicates the first Standard Gaussian noise sampled in one integral step.

[0265] This represents a standard Gaussian distribution with zero mean and covariance equal to the identity matrix.

[0266] The meanings of the other symbols have been defined above.

[0267] Based on the transition density of each integration step, the joint log-likelihood of the action block can be obtained by summing them up:

[0268] In equation (27): Indicates the state The following strategy generates macro actions The joint log-likelihood.

[0269] This represents the macro action block obtained from the final stream matching sampling.

[0270] It represents the state or observation in a closed-loop environment of a physical simulation.

[0271] This represents the number of Euler integral steps for flow matching.

[0272] Indicates the first Each integration step from Transferred to The conditional probability density.

[0273] This represents a stochastic policy jointly determined by the action expert network and the noise injection network.

[0274] Construct the probability ratio using the difference in the joint log-likelihood of the old and new strategies:

[0275] In equation (28): Indicates time The probability ratio between the old and new strategies.

[0276] This indicates the strategy that needs to be optimized.

[0277] The old strategy used when representing sampled data.

[0278] Indicates time Macro action blocks that are generated and executed.

[0279] Indicates time Environmental conditions or joint observation.

[0280] The PPO trimming target is:

[0281] In equation (29): This indicates the proxy loss of the PPO pruning strategy.

[0282] This indicates that the empirical average is calculated over the sampled macro-decision moments.

[0283] Indicates time The advantage estimate.

[0284] This means restricting the probability ratio to an interval. Inside.

[0285] This represents the PPO pruning threshold.

[0286] The overall goal of post-training is:

[0287] In equation (30): This represents the overall training objective after Flow-PPO.

[0288] This indicates the proxy loss of the PPO pruning strategy.

[0289] This indicates a loss in the value network.

[0290] This represents the behavior cloning anchor loss, used to limit the deviation of the post-trained policy from the prior pre-trained actions.

[0291] This indicates a loss of smoothness in motion.

[0292] This represents the auxiliary loss of the generative value network.

[0293] , , and These represent the weighting coefficients of the corresponding loss terms.

[0294] In one optional embodiment, to improve visual robustness, visual distractors similar to the task target but which should not be used as landing sites can be randomly introduced into the physical simulation environment. These visual distractors maintain a preset safe distance from the robot's root node, take-off block, and target block to avoid interfering with the actual landing area. The probability of distractor injection can increase with the training iteration progress, allowing the strategy to first learn basic jumping capabilities and then gradually adapt to complex visual scenes. Furthermore, the lighting, material, friction coefficient, mass, initial position, and initial yaw angle can be randomized within the domain.

[0295] During the deployment phase, the multi-view strategy trained after Flow-PPO can be used as the teacher strategy to construct the single-view student strategy.

[0296] After distillation, the student strategy can continue to be trained following the Flow-PPO process to enhance the closed-loop control capability under single-view conditions.

[0297] The parameters of the preferred embodiment can be as follows: action block length H=15; joint dimension J=29; control frequency is 50Hz to 60Hz; flow matching sampling Euler steps K=4 to 10; visual input can be one or more RGB-D channels; projection dimension D can be set to 256, 512, 768 or 1024 depending on the model size; PPO clipping factor can be 0.1 to 0.3; behavior clone anchor weight can be gradually decayed in the post-training stage.

[0298] The above values ​​are preferred embodiments and are not intended to limit the scope of protection. Other degrees of freedom for humanoid robots, other action block lengths, other visual encoders, other depth encoders, other integrators, or other proximal reinforcement learning variants, as long as they adopt the multimodal flow matching action block and macro-action post-training concept of this invention, can all be used as alternative implementation methods.

[0299] Optional implementation methods: (1) Robot platform replacement: The preferred embodiment is a 29-degree-of-freedom humanoid robot, but it can also be replaced by other degrees of freedom humanoid robots, bipedal robots or legged robots with jumping / obstacle crossing capabilities.

[0300] (2) Sensor replacement: The preferred embodiment uses RGB-D input, but RGB image plus monocular depth estimation, binocular vision, lidar height map, event camera or multi-sensor fusion input can also be used.

[0301] (3) Encoder replacement: Visual encoders, depth encoders and language encoders can use different pre-trained models or lightweight encoders trained from scratch. It is not recommended to limit the specific model name in the claims.

[0302] (4) Action representation alternatives: Action blocks can be represented as joint angles, joint velocities, joint position increments, joint target offsets, or other action interfaces acceptable to the underlying controller.

[0303] (5) Flow matching path substitution: The preferred embodiment uses a linear interpolation path, but a nonlinear path, optimal transmission path or noise scheduling path can also be used.

[0304] (6) Alternative integration methods: Euler integrals can be replaced by the midpoint method, Heun method, Runge-Kutta method or a few-step sampling solver.

[0305] (7) Reinforcement learning algorithm replacement: PPO can be replaced by policy optimization algorithms with proximal constraints or probability ratio constraints, such as TRPO variants, KL-constrained policy optimization, or advantage-weighted policy update.

[0306] (8) Value network replacement: FlowCritic can be replaced by a common multilayer perceptron value network, Transformer value network, state value network or action value network.

[0307] (9) Efficient parameter training alternatives: LoRA can be replaced by adapters, prefix tuning, partial layer unfreezing, or full parameter fine-tuning.

[0308] (10) Distillation substitution: Teacher-student distillation can distill only the action output, or it can distill intermediate features, attention maps or value estimates at the same time.

[0309] (11) Application scenario replacement: In addition to standing jump, it can also be extended to crossing steps, stepping stones to overcome obstacles, narrow platform jumps, turning jumps with angled targets, or other high dynamic full-body movements.

[0310] The robot's deployment control loop includes an RGB-D camera, body sensors, an upper-level policy controller, a single-view motion policy, and a control interface for connecting to the existing robot's lower-level controller. The upper-level policy controller performs time synchronization and normalization processing on the acquired RGB-D images and body state, inputs the motion policy to obtain future joint motion blocks, and writes these blocks into the motion buffer. The existing robot's lower-level controller receives the joint targets parsed from the motion blocks according to the control frequency and drives the robot to execute them.

[0311] It should be noted that the innovation of this invention lies in the multimodal condition construction of the upper-level visual language action strategy, the physical prior flow matching pre-training, the macro-action reinforcement learning post-training, and the distillation deployment process; the robot's underlying servo controller, motor driver, communication protocol, and actuator can be implemented using existing hardware and are not considered as necessary improvements to this invention.

Claims

1. A flow matching reinforcement learning method for jump control of a humanoid robot, characterized in that, Including the following steps: Step S1: Obtain multimodal training samples of the humanoid robot in jumping or obstacle crossing tasks. The multimodal training samples include natural language task instructions, at least one color image, at least one depth image, robot body state, and future joint motion blocks corresponding to the robot body state. Step S2: Perform heterogeneous feature encoding on the natural language task instructions, color images and depth images respectively to obtain feature tokens for each input modality, and map the features of each modality to a unified latent space through a projector that corresponds to each modality and does not share parameters. Step S3: Concatenate the feature tokens of each mapped input modality along the sequence dimension to obtain a joint conditional tensor, and construct the input tensor of the motion expert network based on the robot body state and the noisy action block. Step S4: The motion expert network receives the input tensor, continuous time steps and joint condition tensor. The motion expert network predicts the velocity field from the noisy motion block to the real motion block and pre-trains the motion expert network based on the flow matching loss including joint-level physical weights, kinematic perception weights, motion smoothing constraints and modal equalization constraints. Step S5: Place the pre-trained motion expert network in a physical simulation closed-loop environment, divide the future joint motion into blocks as macro motions, obtain the joint log-likelihood of the macro motions through randomization reparameterization of the convection matching sampling process, and post-train the motion expert network based on the proximal policy optimization objective. Step S6: Utilize the post-trained motion expert network to output future joint motion blocks based on current multimodal observations, in order to control the humanoid robot to perform jumping or obstacle-crossing actions.

2. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, The future joint motion block in step S1 includes future... The joint target sequence of the frame, The value ranges from 8 to 30.

3. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, The heterogeneous feature encoding in step S2 includes: Language semantic features are obtained through a task-level semantic encoder; Color visual features are obtained through at least one visual encoder; Depth geometric features are obtained through at least one depth encoder; The task-level semantic encoder, visual encoder, and deep encoder are at least partially frozen during the pre-training and post-training phases.

4. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, The projector in step S2 is a multilayer perceptron with non-shared weights configured according to modality; The projector includes a linear layer, a nonlinear activation layer, and a normalization layer connected in sequence, used to map language features, color visual features, and deep geometric features of different dimensions to the same latent space dimension D.

5. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, Step S3 further includes maintaining a modal slice dictionary for the joint condition tensor; The modality slice dictionary records the start and end positions of each modality feature in the joint conditional tensor, and is used to perform modality independent discarding, cross-attention proportion balancing, and modality contribution diagnosis.

6. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, The motion expert network is a Transformer-based motion generation network, including an action sequence encoding layer, a self-attention layer, a cross-attention layer, an adaptive normalization layer, a feedforward network layer, and a velocity field output layer. The cross-attention layer uses the action token as the query and the joint condition tensor as the key and value.

7. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, The flow matching loss in step S4 includes velocity field regression loss based on the linear interpolation path; The linear interpolation path is divided into blocks based on noise action. Segmentation with real motion The supervised velocity field learned by the action expert network is constructed as follows: .

8. The flow matching reinforcement learning method for humanoid robot jump control according to claim 5, characterized in that, The joint-level physical weights in step S4 are determined based on the dynamic role of different joints in the jumping task and / or the kinematic variance in the training samples, so that the lower limb take-off related joints, waist posture adjustment joints and upper limb arm swing joints have different supervision intensities. The motion smoothing constraint in step S4 includes a third-order difference smoothing constraint calculated along the time dimension of the motion block, which is used to suppress high-frequency jitter in the future joint motion block. The modal equalization constraints in step S4 include: performing independent random discarding on the text modality, color visual modality, and depth modality; calculating the attention percentage of each visual modality in the cross-attention matrix based on the modality slice dictionary; and introducing a penalty term when the attention percentage of any visual modality is lower than a preset lower limit.

9. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, The randomization reparameterization in step S5 includes introducing a noise injection network during the flow-matched Euler integral process, converting deterministic ordinary differential equation sampling into random differential equation sampling, and obtaining the joint log-likelihood of the macro-action based on the sum of the transition probabilities of each integration step.

10. The flow matching reinforcement learning method for humanoid robot jump control according to claim 1, characterized in that, The post-training includes constructing the probability ratio of the new and old policies based on the joint log-likelihood, and efficiently bypassing the parameters of the action expert network by updating the pruning target, value network loss, behavior clone anchor loss and action smoothing loss through near-end policy optimization. The parameter efficient bypass is a low-rank adaptive bypass, which is injected into at least one linear layer of the action expert network. The dense parameters obtained during pre-training are kept frozen or partially frozen during the post-training stage. The value network is a generative value network, which reuses flow matching conditions and / or hidden representations of action experts to estimate the value of the closed-loop simulation state.

Images