A flow potential field-based strategy generation and value evaluation integrated robot self-guiding operation method

Abstract

The present application belongs to the technical field of robot learning and intelligent control, and particularly relates to a strategy generation and value evaluation integrated robot self-guiding operation method based on a flow field. The method comprises the following steps: collecting robot operation mixed quality offline data sets, and performing stage binary success labeling on each trajectory in the data set; constructing a unified generative backbone network based on conditional flow matching, sampling noise samples from a Gaussian prior distribution, and interpolating the data endpoints along the optimal transport straight line path to obtain interpolation states; setting the backbone network to take the interpolation states, interpolation time variables and conditions as inputs, and outputting predicted velocities; obtaining multi-modal observations at the current time as conditional inputs, generating candidates along the optimal flow path starting from Gaussian noise, and selecting the optimal trajectory according to the predicted potential ranking.

Description

Technical Field

[0001] This invention belongs to the field of robot learning and intelligent control technology, and specifically relates to a robot self-guided operation method that integrates strategy generation and value assessment based on flow potential field. Background Technology

[0002] In the field of autonomous robot operation, intelligent agents need to autonomously generate continuous action sequences based on multimodal perception information (such as visual images, language commands, and proprioception) to complete complex multi-stage operation tasks, such as object grasping, handling, sorting, and assembly.

[0003] In recent years, Vision-Language-Action (VLA) models, as a class of large-scale pre-trained generative policy models, have shown great potential in the field of autonomous robot operation. VLA models can parse natural language commands, fuse multi-view visual perception and ontological perception information, and generate complete action sequences suitable for various operational tasks. Regarding the backbone architecture of VLA models, continuous generative methods based on flow matching and diffusion models have become the mainstream policy generation paradigm due to their stable likelihood training objectives, flexible sampling strategies, and efficient ODE solving characteristics. Behavior cloning (BC) is the most commonly used training method for VLA policies, learning operational skills by minimizing the deviation between the policy output and the expert demonstration.

[0004] However, in actual deployments, robots inevitably accumulate interaction experiences of varying quality, including high-quality expert demonstrations, imperfect autonomous rollbacks, and partially failed execution trajectories. How to effectively extract optimal behavior and improve policies from this mixed-quality offline data is a core challenge facing robot policy learning. Existing methods have the following limitations in addressing this problem: ① Standard behavior cloning methods indiscriminately mimic all behavioral trajectories in the dataset, including suboptimal and failed trajectories, resulting in policy performance being limited by the overall data quality. Simply filtering and retaining only successful data for training, while eliminating interference from failed trajectories, discards useful sub-behavioral information contained within them (e.g., a failed trajectory may still contain high-quality approach and grasp actions), potentially reducing policy performance in long-term tasks. ② Offline reinforcement learning (OffRL) methods can extract useful information from mixed-quality data through a value function, but existing methods typically require training an independent critic network outside the policy network to estimate the state-action value function and updating it using an iterative temporal-difference (TD) method. This separate Actor-Critic architecture introduces significant additional network structure and training overhead to large-scale generative models. Furthermore, it faces instability issues during TD training under sparse rewards and long-term task settings, increasing the system's architectural complexity and optimization difficulty. ③ When policy improvement methods such as Advantage-Weighted Regression (AWR) are directly applied to an architecture that unifies policy and value in the same network, a value illusion problem arises due to weight coupling. Specifically, when the action and value dimensions share the same exponential advantage weights, the weights of failed samples (with negative advantages) approach zero, causing the gradient of the value estimation head on failed data to be obscured, preventing it from learning and correcting errors. This leads the value head to continuously output excessively high success predictions, creating an "illusion" that misleads the direction of policy improvement in subsequent training, severely affecting the effectiveness and stability of policy optimization. Summary of the Invention

[0005] To address the shortcomings and deficiencies of the existing technologies, this invention aims to provide a robot self-guided operation method that integrates strategy generation and value assessment based on flow potential field. This method aims to solve at least the problems in existing robot policy learning methods, such as the inability of behavior cloning to effectively utilize mixed quality data, the additional training overhead and instability caused by the separate Actor-Critic architecture, and the value illusion caused by weight coupling under a unified architecture.

[0006] The technical solution for implementing the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for training a robot operation strategy model based on a flow potential field, comprising: Data Acquisition and Labeling: Collect a mixed-quality offline dataset of robot operations and perform phased binary success labeling on each trajectory in the dataset; concatenate the action sequences in the trajectories with the binary success labels to construct training data endpoints. Using the multimodal observation sequence in the trajectory as a condition ; Policy learning: Construct a unified generative backbone network based on conditional flow matching to sample noise samples from a Gaussian prior distribution. For data endpoints Interpolation is performed along the optimal transmission straight path to obtain the interpolation state. Configure the backbone network in interpolation mode. Interpolation time variables and conditions Input: Prediction speed At the noise endpoint At this point, a single forward propagation is performed using a stopping gradient approach to obtain the predicted endpoint of the augmented flow state. From the predicted endpoint Extract the success potential component and aggregate it into a scalar baseline value along the time step. ; the scalar baseline value As an approximate estimate of the value function of the behavioral strategy, the advantage estimate is calculated, and the backbone network parameters are updated by using a decoupled advantage-weighted regression objective to optimize the strategy of the backbone network. Robot self-guided reasoning: Obtain multimodal observations at the current moment as conditional input, generate candidates along the optimal flow path starting from Gaussian noise, and select the optimal trajectory based on prediction potential ranking.

[0008] Optionally, the present invention describes the data endpoint Interpolation is performed along the optimal transmission straight path to obtain the interpolation state. for:

[0009] in, This is the interpolation time variable.

[0010] Optionally, the present invention constructs a single-step value estimator at the noisy endpoint. Perform a single forward propagation using the stopping gradient method to obtain the predicted endpoints of the augmented flow state:

[0011] in, This indicates that the gradient operation is stopped. A snapshot of network parameters that is updated periodically; From the prediction endpoint Extracting potential components for success The scalar baseline values ​​aggregated along the time step are:

[0012] in, Indicates the predicted endpoint Extract the potential for success from it; Indicates the temporal length of the action block; The calculated advantage estimate is:

[0013] in, This indicates a binary success label.

[0014] Optionally, the present invention employs a decoupling advantage weighted regression objective to optimize and update the backbone network parameters. The specific process is as follows: Calculate the indexed importance weights:

[0015] in, For temperature parameters; This is the cutoff constant; Predicting the velocity field of the backbone network Decomposed into motion components And the potential for success and target speed Decomposed into motion components And the potential for success The total loss function is defined as: .

[0016] Optionally, the specific process of the robot's self-guided reasoning in this invention is as follows: Obtain multimodal observations at the current time And use this as a condition for generating candidate actions, setting the number of candidate samples to be... Independent sampling from a standard Gaussian distribution Initial noise state Each initial noise state corresponds to the starting point for generating a candidate action; During the candidate action generation process, the backbone network is based on the initial noise state. Continuous time variables and conditions It predicts and generates a continuous generation path from initial noise to candidate actions; The final result Each candidate endpoint is represented as , Candidate action sequences, This is the step-by-step success potential score vector corresponding to the action sequence; Based on the stepwise success potential scoring vector, the quality of different candidate action sequences can be compared, and the better action to be executed under the current observation conditions can be selected.

[0017] Optionally, the multimodal observation described in this invention includes multi-view RGB images, language commands, and ontology-aware information.

[0018] Optionally, the multi-view RGB images of the present invention extract image feature sequences through a visual encoder, process the RGB images from the front, left, and right perspectives respectively, and encode each image into a visual token sequence; the language commands are embedded by extracting language features through a language encoder; the ontology perception information includes the robot's joint angles and end effector poses, which are normalized and mapped into compact state vectors through a linear projection layer; the visual token sequence, language feature embedding, and state vectors are concatenated into a heterogeneous token sequence, input into a pre-trained multimodal Transformer backbone network, and cross-modal interactions are managed through attention masks, outputting a fused representation as the conditional input for conditional flow matching. .

[0019] Optionally, the backbone network of the present invention includes two independent linear heads; wherein, the action head outputs the action component corresponding to the augmented velocity field; the success potential outputs the success potential component corresponding to the augmented velocity field; the outputs of the two independent linear heads are concatenated along the feature dimension to form a complete velocity field prediction of the augmented flow state. . Attached Figure Description

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 The diagram shows the overall structure of the flow potential field framework according to an embodiment of the present invention, which includes three parts: data acquisition and reward labeling, policy learning, and inference. Figure 2 A flowchart of the decoupled advantage weighted regression training process according to an embodiment of the present invention includes a single-step value estimation stage and a flow matching update stage; Figure 3 This is a schematic diagram of a task scenario in a simulated and real environment according to an embodiment of the present invention; Figure 4 This is a comparison chart of average completion scores at each stage in a real robot task, based on an embodiment of the present invention. Figure 5This is a comparative analysis diagram of value calibration under the coupling and decoupling weighting methods according to an embodiment of the present invention; Figure 6 The figure shows the ablation experiment results of the efficiency and accuracy of the single-step value estimator according to an embodiment of the present invention. Detailed Implementation

[0021] To make the technical solutions contained in this invention application clearer and easier to understand, the invention application will be described and introduced in detail below with reference to specific embodiments. When the description involves drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Based on the embodiments in this invention application, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of this invention.

[0022] Figure 1 The diagram shown illustrates the overall structure of the flow potential field framework provided in this embodiment of the invention. This framework consists of three parts: data acquisition and annotation, policy learning, and robot self-guided inference.

[0023] During the data collection and annotation phase, a mixed-quality dataset was obtained through expert demonstration and autonomous strategy rollback, and each trajectory was successfully annotated with stage-level binary data.

[0024] During the strategy learning phase, the successful trajectory enhances the high-quality flow through a decoupling weighting mechanism, while the action flow of the failed trajectory is suppressed, and the potential success head continues to learn real value.

[0025] During the robot's self-guided reasoning phase, candidates are generated along the optimal flow path starting from Gaussian noise, based on multimodal observations (language commands, images, and ontology perception), and the optimal trajectory is selected according to the predicted potential ranking.

[0026] The process of the above three stages is explained in detail below. The data collection and labeling stage corresponds to steps S11-S12, the policy learning stage corresponds to steps S13-S15, and the inference stage corresponds to steps S21-S23. S11: Collect a mixed-quality offline dataset including expert demonstrations and autonomous policy rollbacks. Each trajectory It consists of a multimodal observation sequence and a corresponding action sequence, wherein the multimodal observation... This includes multi-view RGB images, language commands, and ontology-aware information; each trajectory is successfully annotated at a stage level, providing a basis for the next stage. Trajectory Each time step Successful allocation of binary labels: (1) in, This indicates that the current phase of the task has been completed. This indicates that the task at this stage is not yet complete.

[0027] S12: Construct the augmented flow state. (This involves adjusting the action sequence dimension.) With the dimension of success potential Concatenate them into a unified implicit variable, represented in the following form: (2) in, For single-step action space dimension, The action sequence is the time-domain length of the action block. To include the future The flattening vector of the step movement, the potential for success Each element Indicates the first The conditional success probability estimate for the step, with a range of values ​​of [value range missing]. The binary success label broadcast generates a success potential target vector. Construct training data endpoints: (3) in, For trajectory At time step The beginning Step movement unfolds into a flat vector.

[0028] S13: Construct a unified generative backbone network based on conditional flow matching, using the aforementioned multimodal observations. As a conditional input, By multimodal observation Composition, from the Gaussian prior distribution Medium-sampled noise samples Interpolation is performed along the optimal straight transmission path: (4) in, The interpolation time variable is used. The linear interpolation path has a time-independent and constant generated velocity field property, and the target velocity is: (5) The final backbone network is in an interpolated state. Interpolation time variables and conditions Input: Prediction speed The basic conditional flow matching loss is: (6) S14: Construct a single-step value estimator. Utilize optimal transport condition flow matching in... The property of the optimal velocity field pointing to the conditional mean of the data: (7) Equation (7) shows that, under the optimal transmission condition flow matching linear interpolation path, when the interpolation time... At that time, the flow state degenerates into a noise endpoint. At this point, the optimal velocity field points to the data endpoint. Enter conditions The conditional mean is given. Therefore, the noise endpoints are... Adding this to the predicted velocity at that point yields a single-step approximate estimate of the conditional mean of the data endpoints. Based on this property, equation (8) utilizes periodically updated network parameter snapshots. exist Perform a stopping gradient forward propagation once to obtain the predicted endpoint of the augmented flow state. Since the augmented flow state contains both action and success potential components, it can be derived from... Extracting potential components for success And along the time domain length of the action block using equation (9) Aggregate to scalar baseline values This baseline value is used for subsequent calculations of the advantage estimate. This allows for the construction of advantage-weighted regression weights, thereby enhancing high-quality action segments and suppressing low-quality action segments.

[0029] At the noise endpoint Perform a single forward propagation using the stopping gradient method to obtain the predicted endpoints of the augmented flow state: (8) in, This indicates that the gradient operation is stopped. A snapshot of network parameters that is updated periodically. From the prediction endpoint. Extracting potential components for success Aggregates along the time step into a scalar baseline value: (9) The scalar baseline value serves as the value function of the behavioral strategy. An approximate estimate. Calculate the advantage estimate: (10) This single-step estimation requires only one network forward propagation (NFE=1), avoiding the computational overhead of iterative integration of ordinary differential equations, making the computational efficiency of reinforcement learning fine-tuning comparable to that of standard supervised learning.

[0030] S15: Optimize the strategy using a decoupling advantage weighted regression objective.

[0031] Calculate the indexed importance weights: (11) in, For temperature parameters, the regularization strength of the control strategy relative to the behavior distribution; This is a truncation constant used to maintain numerical stability. The velocity field of the backbone network is predicted. Decomposed into motion components And the potential for success and target speed Decomposed into motion components And the potential for success The total loss function is defined as: (12) The first item is based on importance weight. Weighted action-dimensional flow matching loss: Minimizing the weighted action loss is equivalent to projecting the policy onto the KL regularized optimal policy distribution. (13) in, For the distribution of behavioral strategies, The first term is the action-value function. The second term is the success potential dimension flow matching loss calculated with uniform weights (weights are always 1). The decoupling design avoids the gradient masking effect caused by the exponential weights on failed samples approaching zero: in the coupled architecture, failed samples ( If the model over-predicts success ( ), advantages Weight Without this approach, the gradient of the successful potential head is masked, making it impossible to correct the illusion. However, in the decoupled architecture, the successful potential head always receives non-zero gradients with uniform weights, enabling continuous calibration of value estimation and forming a stable policy iterative improvement loop. The backbone network parameters are updated using the gradient of the total loss. .

[0032] Secondly, embodiments of the present invention provide a robot self-guided operation reasoning method based on a flow potential field, including: S21: Obtain the multimodal observations at the current moment. This includes multi-view RGB images, language commands, and ontology-aware information, which are used as conditions for generating candidate actions.

[0033] S22: Set the number of candidate samples Independent sampling from a standard Gaussian distribution Initial noise state Each initial noise state corresponds to the starting point for generating a candidate action; During the candidate action generation process, the backbone network is based on the initial noise state. Continuous time variables and conditions Predict how this state should change. This change is represented by the following ordinary differential equation: (14) In other words, equation (14) describes the continuous generation path from initial noise to candidate actions. This equation is then applied to... Points to This means that the initial noise state is gradually transformed into the final candidate result according to the direction of change predicted by the backbone network. The final result is the... Each candidate endpoint is represented as .in, Candidate action sequences, This is the step-by-step success potential score vector corresponding to the action sequence.

[0034] Therefore, the candidate endpoint not only provides the candidate action sequence itself, but also simultaneously provides an evaluation result of the success potential of the candidate action sequence. Based on this evaluation result, the quality of different candidate action sequences can be compared, and the better action to be executed under the current observation conditions can be selected.

[0035] S23: Average the stepwise success potential scores of each candidate group over time steps and aggregate them to form the overall quality score of that candidate: (15) Select the candidate action sequence with the highest score as the optimal action to execute at the current moment: (16) The optimal action sequence is executed using a sliding window approach. This ranking and selection process improves upon the previous greedy strategy by eliminating low-quality behaviors at the tail of the generated distribution, achieving self-guided decision-making without the need for an external evaluation network.

[0036] Furthermore, in this embodiment of the invention, the specific construction method and working mechanism of the multimodal observation processing method are as follows: The multi-view RGB images are processed by a visual encoder to extract image feature sequences. The visual encoder adopts a visual Transformer (ViT) architecture to process RGB images from three perspectives: front, left, and right, and encodes each image into a visual token sequence. The language instructions are extracted and embedded by a language encoder, and the natural language instructions are mapped to fixed-dimensional language feature embeddings to provide task goal guidance for the strategy. The body perception information includes the angles of each joint of the robot and the pose of the end effector, which are normalized and then mapped into a compact state vector through a linear projection layer. The visual token sequence, language feature embeddings, and state vectors are concatenated into a heterogeneous token sequence, which is then input into a pre-trained multimodal Transformer backbone network. Cross-modal interactions are managed through attention masks, and the output fused representation serves as the conditional input for conditional flow matching. .

[0037] Furthermore, in this embodiment of the invention, the specific construction method and working mechanism of the output layer structure of the backbone network are as follows: The original unified output projection layer is replaced with two independent linear heads: the action head transforms the hidden layer representation of the backbone network from the model dimension. Mapping to the action space dimension ,Right now The output corresponds to the motion components of the augmented velocity field; the successful potential head will transform the hidden layer representation of the backbone network from the model dimension. Mapping to a one-dimensional scalar, i.e. The outputs correspond to the success potential component of the augmented velocity field. The outputs of the two independent linear heads are concatenated along the feature dimension to form a complete velocity field prediction of the augmented flow state. The two heads share all the parameters of the backbone network, separating only at the final output layer, thus achieving independent output of actions and values ​​while maintaining the compactness of the architecture.

[0038] Furthermore, in this embodiment of the invention, the generation mechanism of value illusion and the role of the decoupling strategy in the decoupling advantage weighted regression are as follows: set up For the set of hallucination samples, i.e., the failure trajectory ( The model predicted successfully in ( ) For samples of this type, the odds are negative. At smaller temperature parameters Weighting In a coupled architecture, the total loss is... Shared weight The gradient of the potential for success is: (17) The near-zero weights prevent the value head from learning and correcting from failed data, an illusion. This persists during subsequent training and, in turn, misleads the weighting of the policy head. In a decoupled architecture, the weights... Only the action component loss is applied; the success potential component loss is always weighted uniformly. (18) Among them, the first item The strategy ignores failed samples; the second item This is a non-zero correction gradient.

[0039] Uniform weighting ensures the success of the potential head by continuously receiving corrective gradients, driving... Once the value estimation is calibrated, the advantage estimation tends to be more accurate in subsequent training, and the policy head can correctly suppress failed actions through weighted summarization to achieve a stable policy iterative improvement cycle.

[0040] Furthermore, in this embodiment of the invention, the theoretical basis of the single-step value estimator is as follows: In optimal transmission condition stream matching, when interpolation time At that time, the flow state degenerates into pure noise. The optimal speed at this point is: (19) Because in hour And noise With data Independent sampling, conditional expectation simplifies to: (20) Therefore, predict the endpoints. It converges to the conditional mean of the data distribution. For the success potential component, the objective is a binary result. The conditional expectation is: (twenty one) Equivalent to behavioral strategies Under given conditions The state value function under the given conditions. Therefore, the single-step estimate... for The unbiased estimator requires only a single network forward propagation (NFE=1), avoiding the computational overhead of iterative ODE integration.

[0041] The robot self-guided operation method based on flow potential field-integrated strategy generation and value assessment provided in this application improves the shortcomings of current related methods in the robot strategy learning process.

[0042] First, to address the issue that existing offline reinforcement learning methods require training independent Critic networks, a unified flow potential architecture is proposed. By embedding the success potential dimension into the flow state of conditional flow matching, policy generation and value evaluation are integrated into a single generative backbone network, avoiding the additional architectural overhead and training instability caused by independent evaluation networks.

[0043] Secondly, to address the value illusion problem caused by weight coupling under a unified architecture, a decoupling advantage weighted regression strategy is proposed. Different weighting methods are used for the action dimension and the success potential dimension, which effectively suppresses the gradient masking effect on failed samples, enabling continuous calibration of value estimation and achieving stable strategy iterative improvement.

[0044] Furthermore, by utilizing the straight-line trajectory property of optimal transmission condition flow matching, a single-step value estimator is proposed, which simplifies the calculation of the baseline value to a single forward propagation, making the computational cost of reinforcement learning fine-tuning comparable to that of standard supervised learning, and significantly improving training efficiency.

[0045] Example: Figure 1 The diagram shows the overall training and inference process provided in this embodiment of the invention, which specifically includes: S11: Acquire mixed-quality offline datasets Each trajectory contains Multimodal observations at each time step ,action and binary success labels .

[0046] Specifically, the hybrid quality dataset contains two types of data sources: (1) high-quality expert demonstration data, collected through human demonstrations in VR teleoperation or simulation environments; and (2) autonomous strategy rollback data, generated by pre-trained strategy models autonomously executing in the environment, covering a variety of results from complete success to complete failure. A binary success label is assigned to each trajectory based on the completion status of stage-level semantic sub-objectives. For example, in the "turn on the radio" task, the approach stage, grasping stage, and pressing stage correspond to three sub-objectives, and the completion of each sub-objective corresponds to a time step marked as... .

[0047] S12: Construct augmented flow states. Concatenate the action sequence dimension and the success potential dimension into a unified latent variable.

[0048] Specifically, the action sequence To include the future The flattening vector of the step motion, where This refers to a single-step motion dimension (including end effector displacement, rotation, and gripper opening and closing, etc.). The length of the action block; the success potential Each element Indicates the first Estimation of the conditional success probability of the step; augmented flow state Concatenate the two along the feature dimension. Success potential target vector. Generated by stage-level binary label broadcasting, with training data endpoints as follows: .

[0049] S13: Construct a unified generative backbone network based on conditional flow matching.

[0050] Specifically, optimal transport conditional flow matching (OT-CFM) is employed, starting from the Gaussian prior. Sampling noise Define the linear interpolation path ,in The interpolation time variable is used. The velocity field generated by this straight path is a time-independent constant. The backbone network is in an interpolated state. ,time and conditions Input: Prediction speed By minimizing the mean square error Conduct training.

[0051] S14: Construct a single-step value estimator.

[0052] Specifically, using OT-CFM in Optimal speed The property pointing to the conditional mean, in a snapshot of network parameters. Perform a single-stop gradient forward propagation at the noise endpoint: Extracting potential components for success. Aggregate baseline value .

[0053] Figure 2 The diagram shown is a flowchart of the decoupled advantage weighted regression training process provided in an embodiment of the present invention. This training process includes two stages: the first stage (first forward propagation, no gradient), in... The first stage involves predicting the potential value at the boundary conditions by stopping gradient forward propagation, and deriving the dominance estimate; the second stage (second forward propagation, with gradient) involves randomly sampling... The flow matching is updated at the point of application, with the action dimension weighted by advantage and the success potential dimension weighted evenly.

[0054] S15: Optimize the strategy using a decoupling advantage weighted regression objective.

[0055] Specifically, the indexed importance weights are calculated. The velocity field prediction is decomposed into motion components. And the potential for success The total loss is:

[0056] The weighted action loss minimizes the KL divergence, projecting the policy onto the optimal energy distribution. The successful potential loss of uniform weighting ensures that the value head is continuously calibrated across the entire data spectrum.

[0057] Figure 2 The specific steps of the training process are as follows: For each gradient step, sample the trajectory index. and time step , Construction conditions and data endpoints Sampling noise and time interpolation Perform single-step estimation: Aggregate baseline Computational advantages and weight Predicted speed Calculate the decoupling loss Update parameters Periodically update parameter snapshots .

[0058] Figure 3 The figures illustrate the task scenarios in simulated and real-world environments of this invention. The simulated tasks were conducted in the OmniGibson environment of the BEHAVIOR-1K benchmark, using a Galaxea R1 Pro humanoid robot. These included five representative tasks: turning on a radio (a), picking up trash (b), spraying fruit trees (c), cooking hot dogs (d), and washing a baseball cap (e), covering hand-eye coordination and long-term motion manipulation. The real-world robot tasks were conducted on the DexTeleop TeleAvatar Lite dual-arm platform, including five tabletop tasks: rock-paper-scissors (f), sorting trash (g), stacking blocks (h), transferring food (i), and wiping a whiteboard (j), covering precision assembly, contact manipulation, and multi-stage hand-eye coordination.

[0059] Figure 4 The figure shows the average completion scores of each stage in five real robot tasks according to the embodiments of the present invention. The results show that the proposed method (FPF) outperforms the behavior clone (Filtered BC) and the separated actor-critic baseline (IDQL) in each stage of most tasks, especially showing more sustained stage completion ability in long-term multi-stage tasks.

[0060] Figure 5 The diagram shows a comparative analysis of the value calibration effects under coupled and decoupled weighted methods in an embodiment of the present invention. On mixed quality rollback data, the successful potential prediction distribution of the coupled AWR baseline (red) clusters in the high-value range (peak value). This results in severe overestimation in both the success and failure phases; the prediction distribution of the decoupled AWR (blue, the method of this invention) is calibrated, and the prediction peak in the failure phase is concentrated in the low value range. The value remains high during the success phase. This comparison verifies the effectiveness of the decoupling strategy in suppressing value illusion.

[0061] Figure 6 The figure shows the efficiency and accuracy ablation experiment of the single-step value estimator in an embodiment of the present invention. Figure 6 (a) Demonstration of training efficiency comparison: NFE=1 (the method of this invention) achieves a 47% success rate in 25.2 hours, while NFE=20 (complete ODE integral) requires 32.9 hours to achieve the same performance, achieving an acceleration of approximately 1.31 times. Figure 6 (b) Demonstrates verification of approximate accuracy: plotting single-step estimates over 1024 states randomly sampled from the rollback trajectory. ) and the complete ODE integral value ( The scatter plot shows a high linear correlation between the two, with a Pearson correlation coefficient. Coefficient of determination The predicted points are closely clustered on the diagonal. The effectiveness of the single-step approximation on the full value spectrum (low and high value states) was verified in the vicinity.

[0062] In one specific embodiment, the hyperparameter configuration of the method of the present invention is as follows: action block length Batch size 128, learning rate Cosine learning rate scheduling is used, and the optimizer is AdamW( Training steps: 30,000 steps; stream sampling: 15; AWR temperature. AWR cutoff constant Fine-tuning of all parameters.

[0063] In another specific embodiment, the backbone network is based on a pre-trained multimodal Transformer architecture. The perception side includes three visual encoders (processing front, left, and right RGB images respectively) that encode the images into visual token sequences, a language encoder that encodes instructions into language token embeddings, and normalized ontology perception information projected as state tokens. These heterogeneous tokens are concatenated and input into the Transformer backbone network. The output side has two independent linear heads: an action head (…). ) and potential for success ( The spliced ​​output constitutes an augmented velocity field. .

[0064] Furthermore, those skilled in the art should understand that the embodiments involved in this application are merely one implementation of the present invention and should not be construed as limiting the present invention. The various technical features therein can be combined arbitrarily, and as long as the combination of these technical features does not contradict each other, they should all be considered to be within the scope of protection of this application.

Claims

1. A method for training a robot operation strategy model based on a flow potential field, characterized in that, include: Data Acquisition and Labeling: Collect a mixed-quality offline dataset of robot operations and perform phased binary labeling on each trajectory in the dataset; Concatenate the action sequence in the trajectory with binary success labels to construct the training data endpoints. Using the multimodal observation sequence in the trajectory as a condition ; Policy learning: Construct a unified generative backbone network based on conditional flow matching to sample noise samples from a Gaussian prior distribution. For data endpoints Interpolation is performed along the optimal transmission straight path to obtain the interpolation state. Configure the backbone network in interpolation mode. Interpolation time variables and conditions Input: Prediction speed At the noise endpoint At this point, a single forward propagation is performed using a stopping gradient approach to obtain the predicted endpoint of the augmented flow state. From the predicted endpoint Extract the success potential component and aggregate it into a scalar baseline value along the time step. ; the scalar baseline value As an approximate estimate of the value function of the behavioral strategy, the advantage estimate is calculated, and the backbone network parameters are updated by using a decoupled advantage-weighted regression objective to optimize the strategy of the backbone network. Robot self-guided reasoning: Obtain multimodal observations at the current moment as conditional input, generate candidates along the optimal flow path starting from Gaussian noise, and select the optimal trajectory based on prediction potential ranking.

2. The robot operation strategy model training method based on flow potential field according to claim 1, characterized in that, The data endpoint Interpolation is performed along the optimal transmission straight path to obtain the interpolation state. for: in, This is the interpolation time variable.

3. The robot operation strategy model training method based on flow potential field according to claim 2, characterized in that, Construct a single-step value estimator at the noisy endpoint. Perform a single forward propagation using the stopping gradient method to obtain the predicted endpoints of the augmented flow state: in, This indicates that the gradient operation is stopped. A snapshot of network parameters that is updated periodically; From the prediction endpoint Extracting potential components for success The scalar baseline values ​​aggregated along the time step are: in, Indicates the predicted endpoint Extract the potential for success from it; Indicates the temporal length of the action block; The calculated advantage estimate is: in, This indicates a binary success label.

4. The robot operation strategy model training method based on flow potential field according to claim 3, characterized in that, The specific process of using a decoupling advantage weighted regression objective to optimize and update the backbone network parameters is as follows: Calculate the indexed importance weights: in, For temperature parameters; This is the cutoff constant; Predicting the velocity field of the backbone network Decomposed into motion components And potential for success and target speed Decomposed into motion components And the potential for success The total loss function is defined as: 。 5. The robot operation strategy model training method based on flow potential field according to claim 4, characterized in that, The specific process of robot self-guided reasoning is as follows: Obtain multimodal observations at the current time And use this as a condition for generating candidate actions, setting the number of candidate samples to be... Independent sampling from a standard Gaussian distribution Initial noise state Each initial noise state corresponds to the starting point for generating a candidate action; During the candidate action generation process, the backbone network is based on the initial noise state. Continuous time variables and conditions It predicts and generates a continuous generation path from initial noise to candidate actions; The final result Each candidate endpoint is represented as , Candidate action sequences, This is the step-by-step success potential score vector corresponding to the action sequence; Based on the stepwise success potential scoring vector, the quality of different candidate action sequences can be compared, and the better action to be executed under the current observation conditions can be selected.

6. The robot operation strategy model training method based on flow potential field according to claim 1, characterized in that, The multimodal observation includes multi-view RGB images, language commands, and ontological perception information.

7. The robot operation strategy model training method based on flow potential field according to claim 6, characterized in that, The multi-view RGB images are processed by extracting image feature sequences through a visual encoder, and the RGB images from the front, left, and right perspectives are processed respectively, encoding each image into a visual token sequence; the language instructions are embedded by extracting language features through a language encoder. The ontology perception information includes the robot's joint angles and end effector poses, which, after normalization, are mapped into compact state vectors through a linear projection layer. The visual token sequence, embedded language features, and state vectors are concatenated into a heterogeneous token sequence, which is then input into a pre-trained multimodal Transformer backbone network. Cross-modal interactions are managed through attention masks, and the output fused representation serves as the conditional input for conditional flow matching. .

8. The robot operation strategy model training method based on flow potential field according to claim 7, characterized in that, The backbone network comprises two independent linear heads; wherein, the action head outputs the action component corresponding to the augmented velocity field, and the success potential outputs the success potential component corresponding to the augmented velocity field; the outputs of the two independent linear heads are concatenated along the feature dimension to form a complete velocity field prediction of the augmented flow state. .

Images