Post-training method and device of body motion model, equipment, medium and product

Abstract

The application provides a post-training method and device for a body motion model, equipment, a medium and a product. The method comprises the following steps: taking an initial body motion model as a current model and taking an initial motion state value function as a current function; iteratively performing the following update steps on the current model and the current function, and taking the current model after each update as a new current model and taking the current function after each update as a new current function until a preset termination condition is met: obtaining a first batch of data from a training data set, updating the current function based on the first batch of data; obtaining a second batch of data from the training data set, determining a first value of a state-motion pair in the second batch of data based on the optimized current function, and performing noise processing on the motion in the state-motion pair to obtain a noise motion; fusing the first value to a reference velocity field to obtain a target velocity field; and updating model parameters based on the noise motion and the target velocity field. The application can improve the stability of post-training of the model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, medium, and product for post-training of embodied motion models. Background Technology

[0002] Embodied action models are a type of robot control model based on deep neural networks. The input is a natural language description of the task and visual images read by the robot's sensors, and the output is the actions that the robot can perform. Embodied action models usually need to be post-trained through reinforcement learning after pre-training to improve their performance in specific downstream long-term tasks. However, the post-training methods related to large embodied action models generally use traditional policy gradients to adjust the direction, which can easily lead to a large deviation of the velocity field, causing gradient explosion and network collapse, resulting in unstable training.

[0003] Therefore, how to improve the stability of post-training of large-scale embodied motion models has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and product for post-training of embodied motion models, in order to solve the technical problem of how to improve the stability of post-training of large embodied motion models in related technologies.

[0005] In a first aspect, this application provides a post-training method for embodied action models, including: The initial embodied action model is used as the current model, and the initial action state value function is used as the current function; The following update steps are performed iteratively on the current model and the current function, with each updated current model and each updated current function being used as the new current function, until a preset termination condition is met: Obtain the first batch of data from the training dataset, and update the current function based on the first batch of data; A second batch of data is obtained from the training dataset, and a first value of the state-action pair in the second batch of data is determined based on the optimized current function. The first value represents the expected cumulative reward of performing the action corresponding to the given state in a given state. Noise-adding processing is performed on the actions in the state-action pair to obtain noisy actions; The first value is fused into the reference velocity field of the current model to obtain the target velocity field; The model parameters of the current model are updated based on the noise action and the target velocity field.

[0006] In some embodiments, updating the current function based on the first batch of data includes: The second value of the first batch of data is determined based on the current function; Obtain the multi-step time difference error and importance sampling truncation weight of the first batch of data; The third value is determined based on the second value, the multi-step time difference error, and the importance sampling truncation weight; The model parameters of the current function are updated based on the difference between the third value and the second value.

[0007] In some embodiments, obtaining the multi-step time difference error and importance sampling truncation weight of the first batch of data includes: Obtain the probability ratio of the current model and the behavior policy to select the same action in the state of the first batch of data, wherein the behavior policy is the policy that provides the first batch of data; The importance sampling truncation weight is determined based on the probability ratio; Obtain the instant reward for each time step in the first batch of data and the state-action pair for the next time step corresponding to each time step; The fourth value of the state-action pair at the next time step is determined based on the current function; The single-step time difference error for each time step is determined based on the instant reward and the fourth value; The multi-step time difference error is determined based on each of the single-step time difference errors.

[0008] In some embodiments, adding noise to the actions in the state-action pair to obtain noisy actions includes: Multiple diffusion time steps are sampled in parallel, and the actions in the state-action pair are obtained; The noise action is obtained by linearly weighting the action and the standard normal noise corresponding to each diffusion time step.

[0009] In some embodiments, fusing the first value into the reference velocity field of the current model to obtain the target velocity field includes: The velocity field predicted by the model parameters of the current model before the previous iteration update is used as the reference velocity field; The direction correction term is determined based on the noise action, the state action pair, and the velocity field predicted by the current model. The direction correction term is processed based on the first value, and the processed direction correction term is superimposed on the reference velocity field to obtain the target velocity field.

[0010] In some embodiments, updating the model parameters of the current model based on the noise action and the target velocity field includes: A gradient-stopping operation is performed on the target velocity field to isolate the backpropagation gradient update of the model parameters; With the goal of minimizing the mean square error between the velocity field predicted by the current model and the target velocity field after performing the stopping gradient operation, a consistency constraint loss function is constructed and the model parameters of the current model are updated by backpropagation.

[0011] In some embodiments, after updating the model parameters of the current model based on the noise action and the target velocity field, the method further includes: Deploy the updated current model to the robot; Control the robot to interact with the real or simulated environment to obtain interaction trajectory data; The interaction trajectory data is added to the training dataset to obtain a new training dataset. In the next update step, the first batch of data and the second batch of data are obtained from the new training dataset.

[0012] In some embodiments, the initial embodied action model includes a visual language action model constructed based on a stream matching network.

[0013] In some embodiments, the preset termination condition includes at least one of the following: The number of times the update step is performed is greater than or equal to a preset threshold number; The success rate of the robot controlling the current model to complete the preset task tends to converge. The average cumulative reward index obtained by the current model in the interactive evaluation is greater than or equal to the preset reward threshold.

[0014] Secondly, this application provides a post-training device for an embodied motion model, comprising: The determination module is used to take the initial embodied action model as the current model and the initial action state value function as the current function; The iteration module is used to iteratively perform the following update steps on the current model and the current function, taking the updated current model as the new current model and the updated current function as the new current function, until a preset termination condition is met: Obtain the first batch of data from the training dataset, and update the current function based on the first batch of data; A second batch of data is obtained from the training dataset, and a first value of the state-action pair in the second batch of data is determined based on the optimized current function. The first value represents the expected cumulative reward of performing the action corresponding to the given state in a given state. Noise-adding processing is performed on the actions in the state-action pair to obtain noisy actions; The first value is fused into the reference velocity field of the current model to obtain the target velocity field; The model parameters of the current model are updated based on the noise action and the target velocity field.

[0015] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the above-described method when executing the program through the computer program.

[0016] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0018] The embodied action model post-training method, apparatus, device, medium, and product provided in this application can provide high-quality guidance for the update direction of the target velocity field by evaluating the first value of the state-action pair using an optimized current function; by fusing the first value into the reference velocity field and adding noise to the action, it can prevent drastic changes and excessive deviations in the flow matching velocity field under policy consistency constraints; through the decoupling of function learning and model parameter update and the bidirectional feedback design, gradient explosion and policy collapse are effectively avoided, improving the overall stability of the embodied action model in the reinforcement learning post-training process. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts illustrating the post-training method for an embodied action model provided in an embodiment of this application.

[0021] Figure 2 This is the second flowchart illustrating the post-training method for the embodied action model provided in the embodiments of this application.

[0022] Figure 3 This is one of the structural schematic diagrams of the post-training device for the embodied motion model provided in the embodiments of this application.

[0023] Figure 4This is a second schematic diagram of the post-training device for the embodied motion model provided in the embodiments of this application.

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0027] The post-training method for embodied motion models provided in this application is applicable to terminals, which can be various electronic devices with displays and web browsing capabilities, including servers, smartphones, tablets, laptops, and desktop computers.

[0028] Figure 1 This is one of the flowcharts illustrating the post-training method for the embodied motion model provided in the embodiments of this application, such as... Figure 1 As shown, the method includes steps 110 and 120. These method steps are merely one possible implementation of this application.

[0029] Step 110: Use the initial embodied action model as the current model and the initial action state value function as the current function.

[0030] Specifically, the execution subject of the post-training method for embodied motion models provided in this application embodiment is a post-training device for embodied motion models. This device can be a hardware device independently set in the terminal, or it can be a software program running in the terminal.

[0031] The initial embodied action model refers to the basic robot control network to be trained and optimized. In this embodiment, the initial embodied action model includes a visual language action model built based on a flow matching network, which generates robot actions based on the velocity field distribution of the flow matching model.

[0032] The initial action state value function refers to the initial neural network used to evaluate the long-term value of performing a specific action in a given state.

[0033] A training dataset refers to a collection of data used to provide offline experience or interaction samples for the post-training of embodied action models. The training dataset includes multiple sets of four-tuples, each containing a state. ,action ,award and the next state This state refers to the environmental observation information and body pose data obtained by the robot through sensors at a specific time step, such as visual image sequences, robotic arm joint angles, and end effector positions.

[0034] Figure 2 This is the second flowchart illustrating the post-training method for the embodied motion model provided in the embodiments of this application, as shown below. Figure 2 In the initialization phase, the embodied action model training dataset is obtained, the action state value function is initialized to obtain the initial action state value function, and the embodied action model to be trained is initialized to obtain the initial embodied action model. The initial action state value function and the initial embodied action model are respectively used as the current function and the current model as the starting point for iterative optimization.

[0035] Step 120: Iteratively perform the following update steps on the current model and the current function, using the updated current model as the new current model and the updated current function as the new current function, until the preset termination condition is met: Obtain the first batch of data from the training dataset and update the current function based on the first batch of data; Obtain a second batch of data from the training dataset and determine the first value of the state-action pairs in the second batch of data based on the optimized current function. The first value represents the expected cumulative reward of performing the action corresponding to the given state. Noise is added to the actions in the state-action pair to obtain noisy actions; The target velocity field is obtained by integrating the first value into the reference velocity field of the current model; The model parameters of the current model are updated based on the noise action and the target velocity field.

[0036] Specifically, the first batch of data and the second batch of data can be different batches of empirical samples extracted from the training dataset during the iterative update process to independently update the current function and the current model.

[0037] The state-action pairs in the second batch of data refer to the combination of the robot's environmental state at a specific moment and the specific actions performed in that state, as recorded in the second batch of data. The second batch of data includes multiple sets of state-action pairs.

[0038] Noise addition refers to the mathematical process of introducing standard normal noise at a specific diffusion time step into the sampled action variables to construct noisy operations.

[0039] The reference velocity field refers to the velocity field predicted by the model parameters before the previous iteration update, and serves as the baseline component for stable model updates.

[0040] Each iteration includes the following specific operations: The first batch of data is sampled from the training dataset of the embodied action model. The current function is optimized based on multi-step temporal difference estimation and importance sampling truncation weights. The second batch of data is sampled from the training dataset to extract state-action pairs. The optimized current function is used to evaluate the state-action pairs to obtain the first value reflecting the expected cumulative reward. Multiple diffusion time steps are processed in parallel. Noisy actions are constructed by adding noise to the actions in the state-action pairs. The first value is used as a value guiding component and fused into the reference velocity field to construct a hybrid target velocity field. The model parameters of the current model are updated by minimizing the difference between the current model's predicted velocity field and the target velocity field.

[0041] Furthermore, after a parameter update is completed, the updated current model is used to control the robot to interact in a real or simulated environment, collect interaction trajectory data, and add the interaction trajectory data to the training dataset to obtain a new dataset. If the preset termination condition is not met, the updated current function and the updated current model are used as the starting point for the next round of iteration optimization. In the next round of iteration, the first batch of data and the second batch of data are collected through the new training dataset. If the preset termination condition is met, the training process ends.

[0042] The preset termination condition can be set by checking whether the preset number of iterations has been reached, or by evaluating whether the model performance has converged, such as the task success rate, cumulative reward, and other indicators. For example, the preset termination condition includes at least one of the following: the number of times the update step is executed is greater than or equal to the preset number threshold; the test success rate of the current model controlling the robot to complete the preset task tends to converge; the average cumulative reward indicator obtained by the current model in the interactive evaluation is greater than or equal to the preset reward threshold.

[0043] The post-training method for embodied action models provided in this application can provide high-quality guidance for the update direction of the target velocity field by evaluating the first value of the state-action pair using an optimized current function; by fusing the first value into the reference velocity field and adding noise to the action, it can prevent drastic changes and excessive deviations in the flow matching velocity field under policy consistency constraints; through the decoupling of function learning and model parameter update and the bidirectional feedback design, it effectively avoids gradient explosion and policy collapse, and improves the overall stability of the embodied action model in the post-training process of reinforcement learning.

[0044] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0045] In some embodiments, updating the current function based on the first batch of data includes: The second value of the first batch of data is determined based on the current function; Obtain the multi-step time difference error and importance sampling truncation weights for the first batch of data; The third value is determined based on the second value, multi-step time difference error, and importance sampling truncation weight; The model parameters of the current function are updated based on the difference between the third value and the second value.

[0046] The multi-step time difference error and importance sampling truncation weights for obtaining the first batch of data include: Obtain the probability ratio of the current model and the behavior policy to select the same action in the state of the first batch of data. The behavior policy is the policy that provides the first batch of data. Importance sampling truncation weights are determined based on probability ratios; Obtain the instant reward for each time step in the first batch of data, as well as the state-action pair for the next time step corresponding to each time step; The fourth value of the state-action pair for the next time step is determined based on the current function; The single-step time difference error for each time step is determined based on immediate rewards and fourth value. The multi-step time difference error is determined based on the single-step time difference error.

[0047] Specifically, the first batch of data is sampled from the training dataset of the embodied action model, and the predicted value of the current function, i.e., the second value, is calculated. Calculate the multi-step time difference target value of the first batch of data, i.e., the third value. Using loss function Based on third value With second value The difference between them updates the model parameters w of the current function. Here, E represents the expectation for the training samples. This is the mean squared error loss function for the current function.

[0048] The third value is calculated using a multi-step time difference estimation method, and the calculation formula is shown below: ; in, Let j be the single-step time difference error at the j-th time step. , The fourth value is the state-action pair determined based on the current function for the next time step; Weights are truncated for importance sampling. , Indicates the current state of the model. Select action The probability, Indicates the behavioral strategy in the state Select action The probability of importance sampling truncation weights is determined by obtaining the ratio of the probability of the current model and the behavior policy selecting the same action, and is used to correct the distribution deviation from the policy data. This serves as a discount factor, controlling the weight of future rewards. is the attenuation factor, which balances the bias and variance of multi-step estimation; h is the look-ahead step number, which determines how many steps to look ahead for the current function estimation. By accumulating the single-step time difference error, the multi-step time difference error is obtained, which accelerates the propagation of the reward signal in the long trajectory. Representing state and actions This set of state-action pairs corresponds to the third value. This indicates the current function with parameter w in state. and actions The second value of this set of state-action pairs; j represents the cumulative index of the time difference error; k represents the multiplicative index of the truncated weights; This represents the instantaneous reward at the j-th time step. This represents the expectation operator under the current model, where t is the index of the time step.

[0049] The post-training method for the embodied action model provided in this application accelerates the effective propagation of sparse reward signals in long trajectories by combining multi-step time difference estimation with importance sampling truncation weight update of the current function. It can correct deviations from policy data distribution and significantly improve sample utilization efficiency.

[0050] In some embodiments, adding noise to the actions in a state-action pair to obtain a noisy action includes: Multiple diffusion time steps are sampled in parallel, and the actions in the state-action pairs are obtained; The noise action is obtained by linearly weighting the action and the standard normal noise corresponding to each diffusion time step.

[0051] The target velocity field is obtained by fusing the first value into the reference velocity field of the current model, including: Use the velocity field predicted by the model parameters before the previous iteration update as the reference velocity field; The direction correction term is determined based on the noise action, state action pair, and the velocity field predicted by the current model. The direction correction term is processed based on the first value, and the processed direction correction term is superimposed on the reference velocity field to obtain the target velocity field.

[0052] The model parameters of the current model are updated based on the noise action and the target velocity field, including: Perform a stop gradient operation on the target velocity field to isolate the backpropagation gradient update of the model parameters; With the goal of minimizing the mean square error between the velocity field predicted by the current model and the target velocity field after performing the stopping gradient operation, a consistency constraint loss function is constructed and backpropagated to update the model parameters of the current model.

[0053] Specifically, stopping gradient operations refers to a mechanism that blocks gradient backpropagation in the computation graph to prevent the gradient of the current function from being incorrectly backpropagated into the velocity field network; backpropagation update refers to a deep learning optimization process that uses the error gradient calculated by the loss function to adjust the network model parameters layer by layer from back to front.

[0054] The post-training phase primarily optimizes the velocity field of the flow matching network for the embodied action model, obtaining state-action pairs from the second batch of data sampled from the training samples. The first value is obtained by evaluating the current function using the optimized function. ,right Multiple diffusion time steps are processed in parallel. N The number of diffusion time steps for parallel sampling; the sampling diffusion time. and standard normal noise , For the identity matrix; actions Constructing noise action by linear weighted interpolation with standard normal noise , For noise-related actions, the first value By fusing the velocity field with the reference velocity field, the mixed velocity field target, i.e., the target velocity field, is obtained, as shown in the following formula: ; in, These are the model parameters of the current model before the last iteration update. The first term represents the standard normal noise sampled at the nth diffusion time step. The velocity field predicted by the current model parameters before the previous iteration update, i.e., the reference velocity field, is used to preserve the velocity field direction of the old parameters; For noise-based actions State-action pair and the velocity field predicted by the current model The definite directional correction term; the second term utilizes the first value. The direction correction term is processed and superimposed onto the reference velocity field to obtain the target velocity field. .

[0055] The consistency-constrained loss function is constructed with the optimization objective of minimizing the mean squared error between the velocity field predicted by the current model and the target velocity field after performing the stopping gradient operation. The loss function is shown below: ; Where sg indicates stopping gradient operations and updating model parameters through backpropagation using the loss function. To prevent the gradient of the current function from backpropagating to the velocity field, This is the consistency constraint loss function for the current model.

[0056] This policy consistency constraint mechanism ensures that velocity field updates maintain consistency with the old policy while moving towards high-value action regions based on the current function score, effectively preventing policy collapse during training.

[0057] The post-training method for the embodied action model provided in this application integrates the first value into the reference velocity field and constructs consistency constraints by stopping gradient operations. This enables the model to maintain consistency with the old policy while exploring high-value action regions, and can prevent gradient explosion and policy collapse in the flow matching model during reinforcement learning training.

[0058] In some embodiments, after updating the model parameters of the current model based on the noise action and the target velocity field, the method further includes: Deploy the updated current model to the robot; Control the robot to interact with the real or simulated environment to obtain interaction trajectory data; The interaction trajectory data is added to the training dataset to obtain a new training dataset. In the next update step, the first batch of data and the second batch of data are obtained from the new training dataset.

[0059] Specifically, the current model with updated model parameters is deployed to the robot and controlled to interact with the real or simulated environment, perform tasks, and record complete interaction trajectory data. The interaction trajectory data may include state sequences, action sequences, and reward sequences.

[0060] The newly collected interaction trajectory data is processed into a quadruple format and added to the training dataset to obtain a new training dataset, which expands the sample diversity and provides richer first-batch and second-batch data to support iterative optimization in the next round of update steps.

[0061] The post-training method for embodied action models provided in this application realizes a closed-loop feedback between model execution, data acquisition, and model training by controlling the embodied robot to continuously interact with the environment and supplementing the training dataset with newly collected trajectory increments. This achieves the goal of continuously improving sample diversity to support the model's continuous iterative optimization in long-term tasks.

[0062] In some embodiments, the parameters related to the current function learning may include a discount factor. Attenuation factor and forward steps Discount factor The value range is [0.9, 0.99], used to control the weight of future rewards; decay factor The value range can be [0.8, 0.95], used to balance the bias and variance of multi-step time difference error; the number of look-ahead steps The value range can be [3, 10], which is used to determine how many steps forward the multi-step time difference error should look.

[0063] In some embodiments, model optimization parameters may include the number of parallel diffusion time steps. and diffusion time Number of parallel diffusion time steps The value range of is [8, 32], which determines the number of diffusion time steps processed in parallel during each model update; diffusion time steps Sampling is performed in the [0,1] interval to control the noise level of the noise action, where =0 corresponds to the original action. =1 corresponds to pure noise. Training control parameters can include the learning rate, batch size, and number of iterations. The current function learning rate can be from 1e-4 to 5e-4, the model learning rate can be from 1e-5 to 1e-4, the batch size can be from 64 to 256, and the number of iterations can be determined based on task complexity and performance convergence.

[0064] The following uses a robot grasping task as an example to illustrate the application of this application. The robot needs to control the end of the robotic arm to perform grasping operations based on natural language instructions and visual images. The task may include stages such as approaching the target, adjusting the posture, closing the gripper, and lifting the object.

[0065] In this example, the embodied motion model can be a flow-matching-based visual-language motion model, with RGB images and text commands as inputs and 7-dimensional motions at the end of the robotic arm as outputs, including position, posture, and gripper opening and closing. The initial training dataset can contain 1,000 expert demonstration trajectories. The training environment can be a simulation environment that supports rapid interaction and data acquisition.

[0066] The post-training process may include: loading a pre-trained visual language action model and an initial training dataset, initializing the action state value function network; training the current function using a multi-step temporal difference method; optimizing the velocity field using a policy consistency constraint method; performing a grasping task in a simulation environment using the trained current model to collect new trajectories; adding the new trajectories to the training dataset; and repeating the function training, model training, environment interaction, and data update.

[0067] In this example, parameter settings may include: discount factor =0.95, attenuation factor =0.9, look-ahead steps h=5, number of parallel diffusion time steps =16, the current function learning rate is 3e-4, the model learning rate is 1e-4, and the batch size is 128. The above parameter settings are for example only and are not intended to limit the scope of protection.

[0068] According to the method of this invention, by learning the multi-step time difference score function, each sample can be used for multiple steps of current function update, which can improve data utilization; by retaining the old policy components and adjusting the update direction by weighting with the current function, policy collapse and gradient explosion problems in the training of the flow matching model can be prevented; by decoupling the value function learning and model optimization training, mutual interference can be avoided and a feedback loop can be formed; by processing multiple diffusion time steps in parallel, computational efficiency can be improved; this method is applicable to embodied action models based on flow matching models, including visual language action models and world action models, and can be applied to scenarios such as robot operation and embodied navigation.

[0069] The post-training device for the embodied motion model provided in the embodiments of this application will be described below. The post-training device for the embodied motion model described below can be referred to in correspondence with the post-training method for the embodied motion model described above.

[0070] Figure 3This is one of the structural schematic diagrams of the post-training device for the embodied motion model provided in the embodiments of this application, such as... Figure 3 As shown, the device includes a determination module 310 and an iteration module 320.

[0071] The determination module is used to take the initial embodied action model as the current model and the initial action state value function as the current function; The iteration module is used to iteratively perform the following update steps on the current model and the current function, using the updated current model as the new current model and the updated current function as the new current function, until a preset termination condition is met: Obtain the first batch of data from the training dataset and update the current function based on the first batch of data; Obtain a second batch of data from the training dataset and determine the first value of the state-action pairs in the second batch of data based on the optimized current function. The first value represents the expected cumulative reward of performing the action corresponding to the given state. Noise is added to the actions in the state-action pair to obtain noisy actions; The target velocity field is obtained by integrating the first value into the reference velocity field of the current model; The model parameters of the current model are updated based on the noise action and the target velocity field.

[0072] Specifically, according to the embodiments of this application, any number of modules in the determination module and the iteration module can be merged into one module, or any one of the modules can be split into multiple modules.

[0073] Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in a single module.

[0074] According to embodiments of this application, at least one of the determining module and the iterative module can be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-a-Chip, a System-on-a-Substrate, a System-on-Package, an Application Specific Integrated Circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in hardware or firmware, or in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of them.

[0075] Alternatively, at least one of the determining module and the iterating module can be at least partially implemented as a computer program module that can perform the corresponding function when the computer program module is run.

[0076] In some embodiments, updating the current function based on the first batch of data includes: The second value of the first batch of data is determined based on the current function; Obtain the multi-step time difference error and importance sampling truncation weights for the first batch of data; The third value is determined based on the second value, multi-step time difference error, and importance sampling truncation weight; The model parameters of the current function are updated based on the difference between the third value and the second value.

[0077] In some embodiments, obtaining the multi-step time difference error and importance sampling truncation weight of the first batch of data includes: Obtain the probability ratio of the current model and the behavior policy to select the same action in the state of the first batch of data. The behavior policy is the policy that provides the first batch of data. Importance sampling truncation weights are determined based on probability ratios; Obtain the instant reward for each time step in the first batch of data, as well as the state-action pair for the next time step corresponding to each time step; The fourth value of the state-action pair for the next time step is determined based on the current function; The single-step time difference error for each time step is determined based on immediate rewards and fourth value. The multi-step time difference error is determined based on the single-step time difference error.

[0078] In some embodiments, adding noise to the actions in a state-action pair to obtain a noisy action includes: Multiple diffusion time steps are sampled in parallel, and the actions in the state-action pairs are obtained; The noise action is obtained by linearly weighting the action and the standard normal noise corresponding to each diffusion time step.

[0079] In some embodiments, fusing the first value into the reference velocity field of the current model to obtain the target velocity field includes: Use the velocity field predicted by the model parameters before the previous iteration update as the reference velocity field; The direction correction term is determined based on the noise action, state action pair, and the velocity field predicted by the current model. The direction correction term is processed based on the first value, and the processed direction correction term is superimposed on the reference velocity field to obtain the target velocity field.

[0080] In some embodiments, updating the model parameters of the current model based on the noise action and the target velocity field includes: Perform a stop gradient operation on the target velocity field to isolate the backpropagation gradient update of the model parameters; With the goal of minimizing the mean square error between the velocity field predicted by the current model and the target velocity field after performing the stopping gradient operation, a consistency constraint loss function is constructed and backpropagated to update the model parameters of the current model.

[0081] In some embodiments, after updating the model parameters of the current model based on the noise action and the target velocity field, the method further includes: Deploy the updated current model to the robot; Control the robot to interact with the real or simulated environment to obtain interaction trajectory data; The interaction trajectory data is added to the training dataset to obtain a new training dataset. In the next update step, the first batch of data and the second batch of data are obtained from the new training dataset.

[0082] In some embodiments, the preset termination condition includes at least one of the following: The number of times the update step is performed is greater than or equal to a preset threshold. The current model's success rate in controlling the robot to complete the preset task is converging. The average cumulative reward metric obtained by the current model in interactive evaluation is greater than or equal to the preset reward threshold.

[0083] Figure 4 This is a second schematic diagram of the post-training device for the embodied motion model provided in the embodiments of this application, as shown below. Figure 4 As shown, the system includes an initialization module, a value function learning module, a policy optimization module, and an environment interaction module. The initialization module stores or retrieves trajectory data generated by the robot's interaction with the environment. This trajectory data contains a quadruple of state, action, reward, and next state. It also initializes the action-state value function, which evaluates the long-term value of performing a specific action in a given state. The parameters are... ; and initialize the embodied action model, which is a policy network based on a flow matching model, generating action distributions through a velocity field, with parameters as follows. .

[0084] The value function learning module is used for multi-step time difference estimation, importance sampling truncation, and value function optimization. The multi-step time difference estimation utilizes… The third value is calculated by stepping future reward and value function estimation to accelerate the propagation of reward signals in long trajectories; importance sampling truncation corrects the distribution deviation of policy data by truncating weights to improve sample utilization; value function optimization updates parameters by minimizing the mean square error between the third value and the second value. .

[0085] The policy optimization module is used for velocity field evaluation, policy consistency constraints, and velocity field updates. Velocity field evaluation scores sampled state-action pairs using an optimized current function to identify high-value action directions. In velocity field updates, the old policy components are retained, and the update direction is adjusted using the optimized current function to prevent drastic model changes. Velocity field updates optimize velocity field parameters through a hybrid objective function, enabling the model to move towards high-value regions while maintaining stability.

[0086] The environment interaction module is used for policy execution, trajectory acquisition, and incremental data updates. Policy execution uses a trained embodied motion model to control the robot to perform tasks in a real or simulated environment; trajectory acquisition records the robot's state sequence, action sequence, and reward sequence during execution; and incremental data updates add newly acquired trajectory data to the training dataset, providing richer samples for the next round of training.

[0087] The value function learning module and the policy optimization module form a two-way feedback: the action state value function provides evaluation signals for the model, and the updated data is used to improve the action state value function; the environment interaction module and the initialization module form a closed loop; new data continuously expands the training set, supporting iterative optimization; the model is continuously improved through the cycle of value function learning, policy optimization, environment interaction and data updates.

[0088] Specifically, according to the embodiments of this application, any and multiple modules among the initialization module, value function learning module, policy optimization module, and environment interaction module can be merged into one module, or any one of these modules can be split into multiple modules.

[0089] Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in a single module.

[0090] According to embodiments of this application, at least one of the initialization module, value function learning module, policy optimization module, and environment interaction module can be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), System-on-a-Chip, System-on-a-Substrate, System-on-Package, Application Specific Integrated Circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in hardware or firmware, or in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of them.

[0091] Alternatively, at least one of the initialization module, value function learning module, policy optimization module, and environment interaction module can be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0092] It should be noted that the post-training device for embodied motion models provided in this application embodiment can implement all the method steps implemented in the above-mentioned post-training method embodiment for embodied motion models, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0093] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the computer program in the memory 530 to execute the above-described method.

[0094] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional modules and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the methods provided in the above embodiments.

[0096] On the other hand, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program for causing a processor to execute the methods provided in the above embodiments.

[0097] The non-transitory computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0098] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A post-training method for an embodied action model, characterized in that, include: The initial embodied action model is used as the current model, and the initial action state value function is used as the current function; The following update steps are performed iteratively on the current model and the current function, with each updated current model and each updated current function being used as the new current function, until a preset termination condition is met: Obtain the first batch of data from the training dataset, and update the current function based on the first batch of data; A second batch of data is obtained from the training dataset, and a first value of the state-action pair in the second batch of data is determined based on the optimized current function. The first value represents the expected cumulative reward of performing the action corresponding to the given state in a given state. Noise-adding processing is performed on the actions in the state-action pair to obtain noisy actions; The first value is fused into the reference velocity field of the current model to obtain the target velocity field; The model parameters of the current model are updated based on the noise action and the target velocity field.

2. The post-training method for the embodied motion model according to claim 1, characterized in that, The step of updating the current function based on the first batch of data includes: The second value of the first batch of data is determined based on the current function; Obtain the multi-step time difference error and importance sampling truncation weight of the first batch of data; The third value is determined based on the second value, the multi-step time difference error, and the importance sampling truncation weight; The model parameters of the current function are updated based on the difference between the third value and the second value.

3. The post-training method for the embodied motion model according to claim 2, characterized in that, The process of obtaining the multi-step time difference error and importance sampling truncation weight of the first batch of data includes: Obtain the probability ratio of the current model and the behavior policy to select the same action in the state of the first batch of data, wherein the behavior policy is the policy that provides the first batch of data; The importance sampling truncation weight is determined based on the probability ratio; Obtain the instant reward for each time step in the first batch of data and the state-action pair for the next time step corresponding to each time step; The fourth value of the state-action pair at the next time step is determined based on the current function; The single-step time difference error for each time step is determined based on the instant reward and the fourth value; The multi-step time difference error is determined based on each of the single-step time difference errors.

4. The post-training method for the embodied motion model according to claim 1, characterized in that, The step of adding noise to the actions in the state-action pair to obtain noisy actions includes: Multiple diffusion time steps are sampled in parallel, and the actions in the state-action pair are obtained; The noise action is obtained by linearly weighting the action and the standard normal noise corresponding to each diffusion time step.

5. The post-training method for the embodied motion model according to claim 1, characterized in that, The step of fusing the first value into the reference velocity field of the current model to obtain the target velocity field includes: The velocity field predicted by the model parameters of the current model before the previous iteration update is used as the reference velocity field; The direction correction term is determined based on the noise action, the state action pair, and the velocity field predicted by the current model. The direction correction term is processed based on the first value, and the processed direction correction term is superimposed on the reference velocity field to obtain the target velocity field.

6. The post-training method for the embodied motion model according to claim 1, characterized in that, The step of updating the model parameters of the current model based on the noise action and the target velocity field includes: A gradient-stopping operation is performed on the target velocity field to isolate the backpropagation gradient update of the model parameters; With the goal of minimizing the mean square error between the velocity field predicted by the current model and the target velocity field after performing the stopping gradient operation, a consistency constraint loss function is constructed and the model parameters of the current model are updated by backpropagation.

7. The post-training method for the embodied motion model according to claim 1, characterized in that, After updating the model parameters of the current model based on the noise action and the target velocity field, the method further includes: Deploy the updated current model to the robot; Control the robot to interact with the real or simulated environment to obtain interaction trajectory data; The interaction trajectory data is added to the training dataset to obtain a new training dataset. In the next update step, the first batch of data and the second batch of data are obtained from the new training dataset.

8. The post-training method for the embodied motion model according to claim 1, characterized in that, The initial embodied action model includes a visual language action model built based on a stream matching network.

9. The post-training method for the embodied motion model according to claim 1, characterized in that, The preset termination condition includes at least one of the following: The number of times the update step is performed is greater than or equal to a preset threshold number; The success rate of the robot controlling the current model to complete the preset task tends to converge. The average cumulative reward index obtained by the current model in the interactive evaluation is greater than or equal to the preset reward threshold.

10. A post-training device for an embodied motion model, characterized in that, include: The determination module is used to take the initial embodied action model as the current model and the initial action state value function as the current function; The iteration module is used to iteratively perform the following update steps on the current model and the current function, taking the updated current model as the new current model and the updated current function as the new current function, until a preset termination condition is met: Obtain the first batch of data from the training dataset, and update the current function based on the first batch of data; A second batch of data is obtained from the training dataset, and a first value of the state-action pair in the second batch of data is determined based on the optimized current function. The first value represents the expected cumulative reward of performing the action corresponding to the given state in a given state. Noise-adding processing is performed on the actions in the state-action pair to obtain noisy actions; The first value is fused into the reference velocity field of the current model to obtain the target velocity field; The model parameters of the current model are updated based on the noise action and the target velocity field.

11. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the post-training method for the embodied action model according to any one of claims 1 to 9 through the computer program.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the post-training method for the embodied action model as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the post-training method for the embodied action model as described in any one of claims 1 to 9.

Images