Robot operation strategy offline reinforcement optimization method based on diffusion world model

Abstract

The present application relates to the technical field of robots, and relates to a robot operation strategy offline reinforcement optimization method based on a diffusion world model. In response to a robot operation strategy optimization request for task execution, an initial state of the robot is sampled from an offline trajectory dataset. According to the robot operation strategy to be optimized and the initial state, an initial action of the robot is determined. According to the initial state, the initial action and the task, an imagined trajectory is determined based on a trained diffusion world model. According to a loss calculation method, the loss of the imagined trajectory is determined, and the robot operation strategy is optimized according to the loss. Based on the offline trajectory dataset, a new imagined trajectory is generated according to the optimized robot operation strategy, and an evaluation index of the new imagined trajectory is determined, and the robot operation strategy with the evaluation index reaching a preset condition is taken as a robot deployment strategy for task execution. Low-risk and low-cost data-driven strategy evolution is realized.

Description

Technical Field

[0001] This specification relates to the field of robotics technology, and in particular to an offline reinforcement optimization method for robot operation strategies based on a diffusion world model. Background Technology

[0002] Against the backdrop of rapid evolution in intelligent robot operation technology and the continued growth in demand for manufacturing automation, the learning and optimization of robot operation strategies has become a core issue in the field of intelligent manufacturing. In robot operation, acquiring efficient, safe, and generalizable strategies is crucial for improving operational success rates and production line adaptability.

[0003] Currently, robot operation strategies are typically pre-trained using human demonstration data, teleoperation data, or historical robot logs. Imitation learning can acquire executable initial strategies with limited data, but its strategy performance is severely limited by the quality and coverage of the demonstration data, making it difficult to proactively discover and surpass suboptimal behaviors from the demonstration trajectory during the training process.

[0004] Offline reinforcement learning based on world models offers a feasible solution to the aforementioned challenges. This type of method first learns an environmental state transition model using offline data, then generates imagined trajectories within the model, and calculates rewards and policy gradients based on these imagined trajectories, thereby significantly reducing or even completely avoiding interactions with the real environment. Diffusion models, with their powerful multimodal generation capabilities, are well-suited for modeling nonlinear state transitions and multiple action outcomes in robot operations, and are therefore often used as the core generator of world models. However, existing robot policy optimization methods based on diffusion world models still suffer from unclear offline optimization boundaries, and some methods still rely on supplementary data collection from the real environment or online trial and error during policy iteration, making it difficult to meet the stringent requirements of low-risk, low-cost offline training.

[0005] To this end, this specification provides an offline reinforcement and optimization method for robot operation strategies based on a diffusion world model. Summary of the Invention

[0006] This specification provides an offline reinforcement and optimization method for robot operation strategies based on a diffusion world model, in order to partially solve the aforementioned problems existing in the prior art.

[0007] The following technical solution is adopted in this specification: This manual provides an offline reinforcement and optimization method for robot operation strategies based on the diffusion world model, including: S1. In response to the request for optimization of the operation strategy for the robot to perform the task, sample the initial state of the robot from the offline trajectory dataset, the initial state including the observation state and the semantic state; S2. Determine the robot's initial action based on the robot operation strategy to be optimized and the initial state; S3. Based on the initial state, the initial action, and the task, determine the imagined trajectory for completing the task output by the diffused world model, which has been trained; S4. Determine the loss of the imagined trajectory according to the preset loss calculation method, and optimize the robot operation strategy to be optimized based on the loss; S5. Based on the offline trajectory dataset, generate a new imagined trajectory according to the optimized robot operation strategy, and determine the evaluation index of the new imagined trajectory according to the preset evaluation method. Use the robot operation strategy that meets the preset conditions as the robot deployment strategy to perform the task.

[0008] Based on the aforementioned technical methods, this solution relies solely on imagined trajectories generated from offline trajectory datasets and a diffuse world model. It avoids updating trajectories and model parameters through online trial and error with real robots, strictly enforcing the rule of "not submitting exploratory actions to the real environment." This fundamentally eliminates problems such as hardware wear and tear, safety risks, and difficulties in task resetting, achieving low-risk, low-cost, purely data-driven strategy evolution. It decouples strategy optimization from strategy selection. This prevents strategies overfitted to the imagined world from being deployed on real robots, ensuring a high success rate, strong generalization, and safe execution of the final deployed strategy in the real environment. This is the core guarantee for a successful transfer from simulation to reality.

[0009] Furthermore, S2 specifically includes: Based on the robot operation strategy to be optimized and the initial state, candidate actions for the robot are determined; Determine the prior behavioral policy obtained by training the offline trajectory dataset; The candidate actions are filtered using the aforementioned behavior prior strategy, and the filtered candidate actions are used as the robot's initial actions.

[0010] Furthermore, the diffusion world model includes a coarse-grained semantic diffusion module, a fine-grained observation diffusion module, and a shared latent space encoder; S3 specifically includes: S31. Input the current initial state, the initial action, and the task into the trained diffusion world model, and pass the initial state through the shared latent space encoder to obtain the shared latent variables output by the shared latent space encoder; S32. Input the semantic state of the initial state, the initial action, and the task into the coarse-grained semantic diffusion module to obtain the semantic state of the next moment output by the coarse-grained semantic diffusion module; S33. Input the semantic state of the next moment, the observation state of the initial state, the initial action, and the shared latent variable into the fine-grained observation diffusion module to obtain the observation state of the next moment output by the fine-grained observation diffusion module; S34. The semantic state and the observation state at the next moment are fused and used as the robot's state at the next moment as the output of the diffuse world model, and the robot's action at the next moment is determined based on the robot's state at the next moment. S35. Repeat steps S31 to S34 until the state and actions of the robot at the preset time are determined, and based on the state and actions of the robot at the preset time, determine the imagined trajectory to complete the task.

[0011] Based on the above technical means, the coarse-grained semantic module is responsible for high-level transfers such as task stages, contact relationships, and object poses, while the fine-grained observation module is responsible for pixel or point cloud detail changes. The two are generated collaboratively by sharing potential space and conditional constraints, reducing the illusion and long-term drift of a single-granularity model.

[0012] Furthermore, in S4, the loss of the imagined trajectory is determined according to a preset loss calculation method, specifically including: Based on the preset loss calculation method, the credibility-weighted loss term, behavioral prior loss term, and out-of-distribution penalty term of the imagined trajectory are determined; The loss of the imagined trajectory is determined based on the credibility-weighted loss term, the behavior prior loss term, and the out-of-distribution penalty term.

[0013] Furthermore, based on a preset loss calculation method, the credibility-weighted loss term for the imagined trajectory is determined, specifically including: For each moment in the imagined trajectory, determine the state and action at that moment, and output the logarithmic probability of the action at that moment according to the robot operation strategy to be optimized. Based on the trained success classifier and progress estimation network, determine the cumulative reward at each moment in the imagined trajectory; The confidence weights of the imagined trajectory are determined based on the trained sequence discriminator, the preset model uncertainty estimator, and the support domain estimator. The credibility-weighted loss term for the imagined trajectory is determined based on each logarithmic probability, each cumulative reward, and the credibility weight.

[0014] Based on the above technical means, the imagined trajectory is weighted or truncated by sequence discriminator, model uncertainty and behavior support domain evaluation, so that the contribution of high-quality imagined trajectory is greater and the impact of low-confidence trajectory on policy gradient is reduced.

[0015] Furthermore, based on the trained success classifier and progress estimation network, the cumulative reward at each moment in the imagined trajectory is determined, specifically including: Determine the state at a given moment from the imagined trajectory as the evaluation state; The evaluation state is input into the trained success classifier to obtain the success probability value output by the trained success classifier. For each moment in the imagined trajectory, the progress estimation network trained with the state at that moment, the action at that moment, the state at the next moment, and the task input is used to obtain the task progress increment output by the progress estimation network at that moment. The cumulative reward for that moment is determined based on the task progress increment and the preset constraint penalty at that moment; wherein, when that moment is the final moment in the imagined trajectory, the cumulative reward for that moment is determined based on the task progress increment, the preset constraint penalty, and the success probability value at that moment.

[0016] Based on the above technical means, by using a success classifier and a progress estimation network, the endpoint success signal is expanded into progress feedback at each step, enabling the imagined trajectory to provide more stable advantage estimation and faster policy convergence.

[0017] Furthermore, based on the trained sequence discriminator, the preset model uncertainty estimator, and the support domain estimator, the confidence weight of the imagined trajectory is determined, specifically including: Input the states, actions, and tasks of the imagined trajectory into the trained sequence discriminator to obtain the offline data distribution probability output by the trained sequence discriminator; The imagined trajectory is input into the preset model uncertainty estimator to obtain the uncertainty output by the model uncertainty estimator; For each moment in the imagined trajectory, the state and action at that moment are input into a preset support domain evaluator to obtain the deviation at that moment output by the support domain evaluator, and the overall deviation is determined based on each deviation. The credibility weight of the imagined trajectory is determined based on the offline data distribution probability, the uncertainty, and the overall deviation.

[0018] Furthermore, based on a preset loss calculation method, the behavioral prior loss term of the imagined trajectory is determined, specifically including: Determine the prior behavioral policy obtained by training the offline trajectory dataset; Based on the KL divergence between the robot operation strategy to be optimized and the behavioral prior strategy, the behavioral prior loss term of the imagined trajectory is determined.

[0019] Furthermore, based on a preset loss calculation method, the out-of-distribution penalty term of the imagined trajectory is determined, specifically including: The out-of-distribution penalty term of the imagined trajectory is determined based on the preset support domain function.

[0020] Furthermore, in S4, the robot operation strategy to be optimized is optimized based on the loss, specifically including: The robot operation strategy to be optimized is updated with the goal of maximizing the loss.

[0021] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: This solution relies solely on offline trajectory datasets and imagined trajectories generated by a diffuse world model. It avoids updating trajectories and model parameters through online trial and error with real robots, strictly enforcing the rule of "not submitting exploratory actions to the real environment." This fundamentally eliminates issues such as hardware wear and tear, safety risks, and difficulties in task resetting, achieving low-risk, low-cost, purely data-driven strategy evolution. It decouples strategy optimization from strategy selection. This prevents strategies overfitted to the imagined world from being deployed on real robots, ensuring a high success rate, strong generalization, and safe execution of the final deployed strategy in the real environment. This is the core guarantee for a successful transfer from simulation to reality. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 A flowchart illustrating the offline reinforcement and optimization method for robot operation strategies based on the diffusion world model provided in the embodiments of this specification; Figure 2 This is a schematic diagram of the offline enhancement and optimization device for robot operation strategies based on the diffusion world model provided in this specification. Figure 3 This specification provides a corresponding Figure 1 A schematic diagram of the structure of an electronic device. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0024] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0025] In the field of robotics, efficient, safe, and generalizable strategy acquisition is crucial for improving job success rates and production line adaptability. Robot operation strategies are typically pre-trained using human demonstration data, teleoperation data, or historical robot logs. Imitation learning can acquire executable initial strategies with limited data, but its strategy performance is severely limited by the quality and coverage of the demonstration data, making it difficult to proactively discover and surpass suboptimal behaviors from the demonstration trajectory during the training process. While online reinforcement learning can theoretically further improve strategy performance through real-world interaction, the interaction process with real robots presents significant challenges, including high sampling costs, significant hardware wear and tear, high safety risks, and difficulties in task resetting. Especially in contact-intensive tasks such as grasping, assembly, opening and closing doors, and plugging and unplugging connectors, blind trial-and-error exploration can easily damage grippers, tooling, target objects, or surrounding equipment.

[0026] However, offline reinforcement learning based on world models still has the following shortcomings: First, the offline optimization boundary is unclear. Some methods still rely on real-world data acquisition or online trial and error during policy iteration, making it difficult to meet the strict requirements of low-risk, low-cost offline training. Second, the granularity of state transition modeling is singular, predicting only at a certain level of pixels, point clouds, or semantic states, making it difficult to simultaneously ensure the consistency of macro-level task stage changes and micro-level contact details. Third, reward signals are sparse. Common practices only determine task success or failure at the trajectory endpoint, lacking continuous feedback in the intermediate process, making it difficult for offline imagined trajectories to effectively guide the continuous improvement of policies. Fourth, long-term imagined trajectories suffer from error accumulation. The diffuse world model may gradually deviate from the real dynamic distribution during multi-step inference. If all imagined samples are used equally for policy updates, it will amplify the model bias. Fifth, there is a lack of credibility assessment for imagined inference trajectories. The trajectory generated by the model contains both high-quality samples close to real dynamics and low-quality samples caused by occlusion, unstable contact, or off-distribution actions. Existing methods have difficulty distinguishing and utilizing the two during the optimization stage.

[0027] To this end, this specification provides an offline reinforcement and optimization method for robot operation strategies based on a diffusion world model.

[0028] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A flowchart illustrating the offline reinforcement optimization method for robot operation strategies based on a diffusion world model, provided in the embodiments of this specification, includes the following steps: S1: In response to the request for optimization of the operation strategy for the robot to perform the task, sample the robot's initial state from the offline trajectory dataset, the initial state including the observation state and the semantic state.

[0030] This specification describes the offline enhancement and optimization process of a robot operation strategy based on a diffusion world model. In the embodiments described herein, this offline enhancement and optimization process can be executed by a server. However, this specification does not limit the type of device or platform used to perform this offline enhancement and optimization process; for example, a personal computer, mobile terminal, or other such device or platform can also be used. For ease of description, the following description uses a server as the execution entity.

[0031] In one or more embodiments of this specification, the server may, in response to a request for optimizing the operation strategy of the robot performing a task, sample the robot's initial state from an offline trajectory dataset. The initial state includes the observation state and the semantic state.

[0032] In this specification, the offline trajectory dataset is denoted as... Offline trajectory dataset This includes multiple robot task trajectories, each trajectory They all include state sequences, action sequences, task phase annotations, and termination state annotations. , It is a positive integer. For offline trajectory datasets Number of tracks included. Each track The included state sequence is composed of the states of the trajectory at each moment in chronological order. The state at each moment can include RGB-D images, point clouds, robot joint states, end-effector pose, gripper opening / closing amount, target object pose, contact state, and task semantic labels (such as "grasping" or "inserted"). Furthermore, the state at each moment can be divided into observation states and semantic states. Observation states include RGB-D images and point clouds, while semantic states include robot joint states, end-effector pose, gripper opening / closing amount, target object pose, contact state, and task semantic labels. Similarly, the action sequence is composed of the actions of the trajectory at each moment in chronological order. The actions at each moment can include joint velocities, end-effector pose increments, gripper control variables, or combinations thereof. Task stage labels indicate which macroscopic stage the task has reached (e.g., "approaching," "grasping," "moving," "inserting"). Termination state labels indicate whether the trajectory ultimately succeeded, failed, or was prematurely terminated.

[0033] Therefore, a robot's state can be sampled from the offline trajectory dataset as the initial state for subsequent generation of imagined trajectories.

[0034] Of course, before sampling the robot's initial state from the offline trajectory dataset, the server can also perform trajectory execution time synchronization, sensor extrinsic parameter alignment, abnormal frame removal, motion smoothing, contact event annotation, and task endpoint calibration in the offline trajectory dataset. For trajectory segments with occlusion, frame loss, sensor saturation, or actuator jitter, interpolation repair, resampling, segment downweighting, or removal are used.

[0035] Furthermore, based on the offline trajectory dataset, a prior behavioral policy can be learned from the offline trajectory dataset. This behavior prior strategy The action support domain used to characterize the offline trajectory dataset is an action probability distribution learned from the offline trajectory dataset, reflecting "what action is typically taken under what conditions". This behavior prior policy It can be trained using Gaussian mixture models, conditional diffusion strategies, or Transformer strategy models, all of which can capture multimodal action distributions.

[0036] S2: Determine the robot's initial action based on the robot operation strategy to be optimized and the initial state.

[0037] In one or more embodiments of this specification, the server can determine the robot operation strategy to be optimized. Then, based on the robot operation strategy to be optimized Based on the initial state of the sample, determine the robot's initial actions.

[0038] It is worth noting that, due to the robot operation strategy that needs optimization Essentially, it's a mapping from state to action, typically represented by a neural network. It receives an initial state and outputs parameters of a probability distribution. If the output parameter's probability distribution is Gaussian, it outputs the mean μ and standard deviation σ. From this distribution, a specific action vector can be sampled as the robot's initial action. Of course, the robot's operational strategy to be optimized... It can receive the initial state or directly output a specific action as the initial action. This specification does not limit the robot operation strategy to be optimized. The presentation format can be set according to the actual situation.

[0039] Furthermore, the server can determine candidate actions for the robot based on the robot operation strategy to be optimized and the initial state. It then determines the prior behavior policy trained on the offline trajectory dataset. Subsequently, the candidate actions can be filtered using the prior behavior policy, and the filtered candidate actions are used as the robot's initial actions. This ensures that the actions output by the robot operation strategy to be optimized are constrained by the prior behavior policy, preventing the robot operation strategy from generating strange or dangerous actions that have never appeared in the offline trajectory dataset.

[0040] S3: Based on the initial state, the initial action, and the task, determine the imagined trajectory for completing the task output by the diffuse world model, which has been trained.

[0041] In one or more embodiments of this specification, the server can determine the imagined trajectory for completing the task output by the diffuse world model based on the initial state, initial action, and the task to be performed by the robot (i.e., the task conditions required for the robot to perform the task), according to the trained diffuse world model.

[0042] Specifically, S3 includes S31 to S35.

[0043] The diffusion world model includes a coarse-grained semantic diffusion module, a fine-grained observation diffusion module, and a shared latent space encoder.

[0044] S31. The server inputs the initial state, initial action, and task at the current moment into the trained diffusion world model. The initial state is processed by the shared latent space encoder to obtain the shared latent variables output by the shared latent space encoder.

[0045] S32. The server inputs the initial semantic state, initial action, and task into the coarse-grained semantic diffusion module to obtain the semantic state of the next moment output by the coarse-grained semantic diffusion module.

[0046] S33. The server then inputs the semantic state of the next time step, the observation state of the initial state, the initial action, and the shared latent variables into the fine-grained observation diffusion module to obtain the observation state of the next time step output by the fine-grained observation diffusion module.

[0047] S34. The server fuses the semantic state and the observation state of the next moment into the robot's state at the next moment, which is the output of the diffuse world model. Based on the robot's state at the next moment, the server determines the robot's action at the next moment according to the robot operation strategy to be optimized.

[0048] S35. The server repeats steps S31-S34, inputting the robot's state, actions, and task for the next moment into the diffusion world model again, until the robot's state and actions for the preset moment are determined. Thus, based on the robot's state and actions at the preset moment, the server can determine the imagined trajectory for the robot to complete the task.

[0049] In this specification, the server may train the diffusion world model using the following method, wherein: The diffusion world model to be trained may include a coarse-grained semantic diffusion module. Fine-grained diffusion observation module Shared latent space encoder .

[0050] Determine the state at a given moment from an offline trajectory dataset. Actions at that moment The task stage label or termination status label of the trajectory to which that moment belongs serves as the task condition. And the actual state and actions at the next moment.

[0051] The state at that moment Actions at that moment and mission conditions Input the training diffusion world model to be trained. The state at this moment in the input training diffusion world model. The shared latent space encoder is obtained by training the diffuse world model to be trained, and then obtaining the output of the shared latent space encoder. semantic state Corresponding semantic latent variables and Observation status The corresponding latent variables observed.

[0052] In this specification, observed latent variables may be used as shared latent variables, or shared latent variables may be determined by fusing observed latent variables and semantic latent variables.

[0053] semantic state Actions at that moment and mission conditions Input coarse-grained semantic diffusion module This yields a coarse-grained semantic diffusion module. Output the semantic state of the next time step .

[0054] Observation status Actions at that moment Shared latent variables and the semantic state of the next time step Input fine-grained observation diffusion module To obtain the observed state at the next moment. .

[0055] Then, the semantic state of the true state at the next moment can be obtained through the following: Output the semantic state of the next time step The difference between the observed state and the real state at the next time step is used to determine the semantic diffusion denoising loss. Output the observation state at the next time step. The difference between them is used to determine the observation diffusion denoising loss. The contrast loss between the shared latent variables and the shared latent variables obtained from other time-series states is calculated. The semantic state of the next time-series true state is used as a reference. The differences between them are used to determine the true variable. Through... Output the semantic state of the next time step and semantic state The differences between the actual and predicted variables are used to determine the predictor variables. Based on the differences between the actual and predicted variables, the neighboring state dynamic consistency loss is determined.

[0056] The comprehensive loss is determined by summing the semantic diffusion denoising loss, observation diffusion denoising loss, contrast loss, and neighboring state dynamic consistency loss. The diffusion world model to be trained is then trained with the goal of minimizing the comprehensive loss.

[0057] Of course, the diffusion world model can also include coarse-to-fine conditional constraint units. This is used to calculate the physical constraint loss of the diffuse world model output within preset physical constraints. The comprehensive loss is determined by summing the semantic diffusion denoising loss, observation diffusion denoising loss, contrast loss, neighboring state dynamic consistency loss, and physical constraint loss. The model is then trained with the goal of minimizing this comprehensive loss.

[0058] The overall loss can be expressed as:

[0059] in, For semantic diffusion denoising loss, To observe the diffusion denoising loss, The shared potential spatial alignment loss is the contrast loss. The loss is due to the dynamic consistency between adjacent states. Losses due to physical constraints such as contact, speed, collision, or joint limitation. These are the preset weight parameters.

[0060] It is worth noting that, Conditional U-Net (U-Net is a convolutional neural network architecture for image segmentation. The structure of U-Net consists of two parts, an encoder and a decoder, which are figuratively represented as a U-shape, hence the name.), conditional DiT (Conditional Diffusion Transformers), or point cloud diffusion networks can be preferred. The diffusion Transformer architecture is preferred.

[0061] S4: Determine the loss of the imagined trajectory according to the preset loss calculation method, and optimize the robot operation strategy to be optimized based on the loss.

[0062] In one or more embodiments of this specification, the server can determine the loss of the imagined trajectory according to a preset loss calculation method, and optimize the robot operation strategy to be optimized based on the loss of the imagined trajectory.

[0063] Specifically, the server can determine the credibility-weighted loss term, behavioral prior loss term, and out-of-distribution penalty term for the imagined trajectory based on a preset loss calculation method. Then, based on the credibility-weighted loss term, behavioral prior loss term, and out-of-distribution penalty term, the loss of the imagined trajectory is determined.

[0064] The loss of the imagined trajectory can be expressed as:

[0065] In the formula, For credibility-weighted loss terms, For the prior loss term of the behavior, This is an out-of-distribution penalty term.

[0066] In this specification, the confidence-weighted loss term is determined as follows: for each moment in the imagined trajectory, the server determines the logarithmic probability of the action at that moment based on the state at that moment and the robot's operation strategy to be optimized. The expression for the logarithmic probability is: , indicating the state Next, according to the strategy Determined actions The logarithmic probability.

[0067] Next, based on the trained success classifier and progress estimation network, the cumulative reward at each time step in the imagined trajectory is determined. Then, based on the trained sequence discriminator, the pre-defined model uncertainty estimator, and the support domain estimator, the confidence weights of the imagined trajectory are determined. Finally, based on the log probabilities, the cumulative rewards, and the confidence weights, the confidence-weighted loss term for the imagined trajectory is determined.

[0068] The method for determining the cumulative reward at each moment in the imagined trajectory is as follows: A state at a given moment in the imagined trajectory is determined as the evaluation state. This evaluation state can be the state at the final moment of the imagined trajectory or the state at any moment in the imagined trajectory. The evaluation state is then input into the trained success classifier. To obtain a successfully trained classifier Output success probability value .

[0069] Then, for each time t in the imagined trajectory, the progress estimation network trained with the state at that time, the action at that time, the state at the next time step, and the task input is trained. The progress estimate network output is obtained. The task progress increment at that moment Finally, based on the task progress increment at that moment and the preset constraints and penalties... Determine the cumulative reward at time t. When this moment is the final moment in the imagined trajectory, the cumulative reward for that moment is determined based on the task progress increment, the preset constraint penalty, and the success probability value. The expression for the cumulative reward at moment t is:

[0070] In the formula, , , These are the preset weights. Indicates only at the final moment in the imagined trajectory Just activated For the length of the imagined trajectory, For indicator functions, For successful classifiers Output success probability value . The constraint penalty at each moment represents the penalty for collision, boundary violation, contact anomaly, task phase reversal, or deviation from the behavior support domain. A reward calibration unit can also be preset. Based on the reward distribution on the offline validation set Normalization is performed to prevent a single reward item from dominating strategy updates.

[0071] The method for determining the credibility weights of the imagined trajectory is to input the states, actions, and tasks of the imagined trajectory into the trained sequence discriminator. The trained sequence discriminator is obtained. The output is the probability distribution of offline data. Then, the imagined trajectory is input into the preset model uncertainty estimator. This yields the uncertainty output by the model uncertainty estimator.

[0072] Furthermore, for each moment in the imagined trajectory, the state and action at that moment are input into a preset support domain evaluator. The support domain evaluator is obtained. The output shows the deviation at that moment, and the overall deviation is determined based on the sum or average of the individual deviations. Finally, the credibility weight of the imagined trajectory is determined based on the offline data distribution probability, uncertainty, and overall deviation. The expression for the credibility weight is:

[0073] In the formula, For offline data distribution probability, For uncertainty, For overall deviation, This indicates that the credibility weight will be trimmed to... Interval. , The preset adjustment coefficient, The minimum confidence weight is preset. If the confidence of a certain time step is lower than the threshold δ for M consecutive steps, the imagined trajectory is truncated in advance. If the state or action violates the hard constraint, the segment is marked as unusable for policy gradient update.

[0074] Therefore, based on each cumulative reward It can calculate the total accumulated reward, and then subtract the preset average return baseline from the total accumulated return to obtain the advantage function. The reward.

[0075] Then, based on the log probabilities and the dominance function The reward and credibility weights are used to determine the credibility-weighted loss term for the imagined trajectory. .

[0076] In this specification, the method for determining the behavioral prior loss term is to determine the behavioral prior policy obtained by training on an offline trajectory dataset. Then, based on the robot operation strategy to be optimized... and behavioral prior strategies The KL divergence between the two values ​​determines the behavioral prior loss term for the imagined trajectory. , which is the preset KL constraint strength.

[0077] In this specification, the method for determining the out-of-distribution penalty term is as follows: based on a preset support domain function, the out-of-distribution penalty term of the imagined trajectory is determined. . To measure the impact of each state and action in the imagined trajectory Whether a function is outside the domain supported by offline data can be represented by its overall deviation.

[0078] The pre-defined out-of-distribution penalty intensity.

[0079] Therefore, after determining the loss of the imagined trajectory, the robot operation strategy to be optimized is updated with the goal of maximizing the loss.

[0080] S5: Based on the offline trajectory dataset, generate a new imagined trajectory according to the optimized robot operation strategy, and determine the evaluation index of the new imagined trajectory according to the preset evaluation method. Use the robot operation strategy that meets the preset conditions as the robot deployment strategy to perform the task.

[0081] In one or more embodiments of this specification, after each round of policy iteration and update, the policy needs to be evaluated based on the frozen offline trajectory dataset. Specifically, the server can generate a new imagined trajectory based on the offline trajectory dataset and the optimized robot operation policy. The process of generating a new imagined trajectory is the same as in S3 above and will not be repeated here. Furthermore, according to a preset evaluation method, the evaluation index of the new imagined trajectory is determined, and the robot operation policy that meets the preset conditions is used as the robot deployment policy for executing the task. The policy update can adopt PPO (Proximal Policy Optimization) or other offline reinforcement learning update forms.

[0082] The evaluation indicators may include success probability, task progress increment, cumulative reward, offline data distribution probability, uncertainty, and overall deviation. Preset conditions may be that one or more of the success probability, task progress increment, cumulative reward, offline data distribution probability, uncertainty, and overall deviation reach their respective preset thresholds.

[0083] Furthermore, in one or more embodiments of this specification, the training method for the success classifier involves determining trajectory samples and their task completion labels from an offline trajectory dataset. The trajectory samples are then input into the success classifier to be trained, and the success probability value output by the classifier is determined. Based on the difference between the success probability value and the label, a loss is determined, and the success classifier is trained with the minimum loss as the optimization objective.

[0084] During the training of the progress estimation network, neighboring state pairs are input. ,action and mission conditions Output task progress increment .Label This refers to the progress monitoring signal obtained by fusing time progress and task phase in the offline trajectory. Its loss... It can be represented as:

[0085] This is a preset monotonic loss used to constrain the cumulative progress along the same successful trajectory. These are the preset weighting coefficients.

[0086] During training, the sequence discriminator takes a trajectory's state sequence, action sequence, and task conditions as input, and outputs the probability that the sequence originates from an offline trajectory dataset. Positive samples are taken from... Negative samples are generated by the diffusion world model under different policy actions. These negative samples include imagined trajectories with different trajectory lengths, initial states, and deviations from the target action. This is used for binary adversarial training, enabling the sequence discriminator to learn to identify subtle differences between real dynamic trajectories and imagined trajectories—such as unnatural jitters or physical inconsistencies that may exist in the imagined trajectories.

[0087] The uncertainty output by the model uncertainty estimator is determined by training multiple diffuse world models and generating multiple imagined trajectories for the same input. The variance among these imagined trajectories is the uncertainty; the larger the variance, the higher the uncertainty.

[0088] The method to support the deviation of the domain evaluator output is to compare the policy action with the prior policy behavior. The degree of deviation, for example, using KL distance, Mahalanobis distance, or behavioral clonal negative log-likelihood. If it can be calculated directly... .

[0089] In this specification, a robot is used as a six-axis robotic arm with a two-finger gripper and an RGB-D camera mounted at the end effector. An example is provided where the task is to pick up a target object from a table and place it in a designated area. The offline trajectory dataset contains human remote control operation trajectories, historical successful trajectories, and a small number of failed trajectories. Each trajectory records the image, depth, point cloud, joint angles, end effector pose, gripper state, motion commands, and task endpoint label.

[0090] First, the data in the offline trajectory dataset is preprocessed. Camera timestamps are aligned with robot control cycles, the trajectory is resampled to a fixed frequency, and segments with gripper collision anomalies, complete target object occlusion, and abrupt changes in control signals are removed. The semantic state of the target object is extracted using the target detection and pose estimation modules, and the task stage is labeled based on gripping contact, lifting height, and placement area entry. Prior behavioral strategies are then trained. , used to estimate the action support domain of offline data.

[0091] Secondly, a hierarchical diffusion world model is trained. The coarse-grained semantic diffusion Transformer uses the current target object pose, gripper relative pose, contact state, and action as input to predict the semantic state at the next moment. The fine-grained observation diffusion U-Net uses the current RGB-D observation, action, and coarse-grained prediction results as conditions to predict local observation changes in the next frame. During training, the semantic denoising loss, observation denoising loss, latent spatial alignment loss, and physical constraint loss are optimized simultaneously. The physical constraint loss penalizes unreasonable predictions such as objects penetrating the table, grippers closing without contact, or instantaneous object displacement exceeding a threshold.

[0092] Next, a dense reward generation mechanism is trained. The success classifier takes the endpoint semantic state as input and determines whether the target object is located in the designated area and the gripper has been released. The progress estimation network takes the states of two adjacent frames as input and outputs the task progress increment. For example, approaching the target, forming stable contact, successfully lifting, moving to the placement area, and releasing the object correspond to different stages of progress improvement. The reward calibration unit normalizes the progress reward based on the offline verification trajectory, making the reward scale consistent for different objects and different initial positions.

[0093] Next, the rollout credibility assessment module is trained. A sequence discriminator is trained using real offline trajectories as positive samples and imagined trajectories generated by the world model as negative samples. To cover long-term error accumulation, the negative sample length is set to 5, 10, 20, and 30 steps. Model uncertainty is calculated from the prediction variance of multiple diffusion world models, and behavioral support domain deviation is calculated from the negative log-likelihood of the action to be optimized under the behavioral prior. The credibility weight of each imagined rollout is obtained by fusing these three scores.

[0094] Subsequently, policy optimization is performed in the offline imagination space. Initial states are sampled from the offline dataset, and the current policy generates actions. Actions are filtered using a behavior support domain filter. An imagination rollout (i.e., the imagination trajectory) is recursively generated using a hierarchical diffuse world model, and the dense reward, endpoint success reward, and constraint penalty are calculated for each step. If the rollout's confidence level continuously falls below a threshold, the trajectory is immediately truncated; if the action exceeds the joint velocity range or the predicted state violates physical constraints, the segment is not included in the policy gradient update.

[0095] Finally, a conservative PPO (Proof-of-Performance) update strategy parameter is adopted using credibility weighting. For each rollout, the advantage function is calculated using a comprehensive reward, the credibility weight is multiplied into the strategy loss, and a KL (Knowledge, Language, and Liskov) penalty relative to the prior behavior is added. After each update, the candidate strategy is evaluated in a frozen offline validation scenario. If the credibility rollout ratio of the strategy decreases or the out-of-distribution action rate increases, the strategy parameters of the previous round are reverted. After reaching the preset number of iterations or the validation metric converges, the candidate deployment strategy is output.

[0096] In the above embodiments, the offline trajectory dataset can contain 100 to 10,000 trajectories, and the rollout length H can be 10 to 50. These parameters can be adjusted according to task duration, world model error, and computational resources. For tasks such as assembly, insertion / removal, door opening, and drawer operations, only the task semantic state, success classification label, and stage progress annotation need to be replaced to implement the same technical process.

[0097] based on Figure 1 The offline reinforcement optimization of robot operation strategies based on the diffused world model, as shown, relies solely on offline trajectory datasets and imagined trajectories generated by the diffused world model. It avoids updating trajectories and model parameters through online trial and error with real robots, strictly enforcing the rule of "not submitting exploratory actions to the real environment." This fundamentally eliminates problems such as hardware wear and tear, safety risks, and difficulties in task resetting, achieving low-risk, low-cost, purely data-driven strategy evolution. It decouples strategy optimization from strategy selection. This prevents strategies overfitted to the imagined world from being deployed on real robots, ensuring a high success rate, strong generalization, and safe execution of the final deployed strategy in the real environment—a core guarantee for successful transfer from simulation to reality.

[0098] Furthermore, the coarse-grained semantic module is responsible for high-level transfers such as task phases, contact relationships, and object poses, while the fine-grained observation module is responsible for pixel or point cloud detail changes. The two are generated collaboratively by sharing the latent space and conditional constraints, reducing the illusion and long-term drift of a single-granularity model.

[0099] By weighting or truncating imagined trajectories through sequence discriminators, model uncertainty, and behavioral support domain evaluation, the contribution of high-quality imagined trajectories can be increased, while the impact of low-confidence trajectories on policy gradients can be reduced.

[0100] Based on the offline reinforcement and optimization method for robot operation strategies based on the diffusion world model provided in one or more embodiments of this specification, following the same approach, this specification also provides a corresponding offline reinforcement and optimization device for robot operation strategies based on the diffusion world model, such as... Figure 2 As shown.

[0101] Figure 2 This is a schematic diagram of the offline reinforcement and optimization device for robot operation strategies based on the diffusion world model provided in this specification, specifically including: The sampling module 200 is used to sample the robot's initial state from the offline trajectory dataset in response to the robot's operation strategy optimization request for performing the task. The initial state includes the observation state and the semantic state. The motion determination module 202 is used to determine the initial motion of the robot based on the robot operation strategy to be optimized and the initial state. The trajectory generation module 204 is used to determine the imagined trajectory for completing the task output by the diffuse world model based on the initial state, the initial action, and the task, according to the trained diffuse world model. The strategy update module 206 is used to determine the loss of the imagined trajectory according to a preset loss calculation method, and optimize the robot operation strategy to be optimized according to the loss. The strategy deployment module 208 is used to generate a new imagined trajectory based on the offline trajectory dataset and according to the optimized robot operation strategy, and to determine the evaluation index of the new imagined trajectory according to the preset evaluation method, and to use the robot operation strategy that meets the preset conditions as the robot deployment strategy for executing the task.

[0102] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 An offline reinforcement optimization method for robot operation strategies based on a diffusion world model is provided.

[0103] This instruction manual also provides Figure 3 The diagram shows a schematic structural representation of the electronic device. Figure 3 As shown, at the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above. Figure 3An offline reinforcement optimization method for robot operation strategies based on a diffusion world model is provided.

[0104] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0105] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0106] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0107] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0108] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0115] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic or disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0119] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0120] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. An offline reinforcement optimization method for robot operation strategies based on a diffusion world model, characterized in that, include: S1. In response to the request for optimization of the operation strategy for the robot to perform the task, sample the initial state of the robot from the offline trajectory dataset, the initial state including the observation state and the semantic state; S2. Determine the robot's initial action based on the robot operation strategy to be optimized and the initial state; S3. Based on the initial state, the initial action, and the task, determine the imagined trajectory for completing the task output by the diffused world model, which has been trained; S4. Determine the loss of the imagined trajectory according to the preset loss calculation method, and optimize the robot operation strategy to be optimized based on the loss; S5. Based on the offline trajectory dataset, generate a new imagined trajectory according to the optimized robot operation strategy, and determine the evaluation index of the new imagined trajectory according to the preset evaluation method. Use the robot operation strategy that meets the preset conditions as the robot deployment strategy to perform the task.

2. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 1, characterized in that, S2 specifically includes: Based on the robot operation strategy to be optimized and the initial state, candidate actions for the robot are determined; Determine the prior behavioral policy obtained by training the offline trajectory dataset; The candidate actions are filtered using the aforementioned behavior prior strategy, and the filtered candidate actions are used as the robot's initial actions.

3. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 1, characterized in that, The diffusion world model includes a coarse-grained semantic diffusion module, a fine-grained observation diffusion module, and a shared latent space encoder; S3 specifically includes: S31. Input the current initial state, the initial action, and the task into the trained diffusion world model, and pass the initial state through the shared latent space encoder to obtain the shared latent variables output by the shared latent space encoder; S32. Input the semantic state of the initial state, the initial action, and the task into the coarse-grained semantic diffusion module to obtain the semantic state of the next moment output by the coarse-grained semantic diffusion module; S33. Input the semantic state of the next moment, the observation state of the initial state, the initial action, and the shared latent variable into the fine-grained observation diffusion module to obtain the observation state of the next moment output by the fine-grained observation diffusion module; S34. The semantic state and the observation state at the next moment are fused and used as the robot's state at the next moment as the output of the diffuse world model, and the robot's action at the next moment is determined based on the robot's state at the next moment. S35. Repeat steps S31 to S34 until the state and actions of the robot at the preset time are determined, and based on the state and actions of the robot at the preset time, determine the imagined trajectory to complete the task.

4. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 1, characterized in that, In S4, the loss of the imagined trajectory is determined according to a preset loss calculation method, specifically including: Based on the preset loss calculation method, the credibility-weighted loss term, behavioral prior loss term, and out-of-distribution penalty term of the imagined trajectory are determined; The loss of the imagined trajectory is determined based on the credibility-weighted loss term, the behavior prior loss term, and the out-of-distribution penalty term.

5. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 4, characterized in that, Based on a preset loss calculation method, the credibility-weighted loss term for the imagined trajectory is determined, specifically including: For each moment in the imagined trajectory, determine the state and action at that moment, and output the logarithmic probability of the action at that moment according to the robot operation strategy to be optimized. Based on the trained success classifier and progress estimation network, determine the cumulative reward at each moment in the imagined trajectory; The confidence weights of the imagined trajectory are determined based on the trained sequence discriminator, the preset model uncertainty estimator, and the support domain estimator. The credibility-weighted loss term for the imagined trajectory is determined based on each logarithmic probability, each cumulative reward, and the credibility weight.

6. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 5, characterized in that, Based on the trained success classifier and progress estimation network, the cumulative reward at each moment in the imagined trajectory is determined, specifically including: Determine the state at a given moment from the imagined trajectory as the evaluation state; The evaluation state is input into the trained success classifier to obtain the success probability value output by the trained success classifier. For each moment in the imagined trajectory, the progress estimation network trained with the state at that moment, the action at that moment, the state at the next moment, and the task input is used to obtain the task progress increment output by the progress estimation network at that moment. The cumulative reward for that moment is determined based on the task progress increment and the preset constraint penalty at that moment; wherein, when that moment is the final moment in the imagined trajectory, the cumulative reward for that moment is determined based on the task progress increment, the preset constraint penalty, and the success probability value at that moment.

7. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 5, characterized in that, Based on the trained sequence discriminator, the pre-defined model uncertainty estimator, and the support domain estimator, the confidence weights of the imagined trajectory are determined, specifically including: Input the states, actions, and tasks of the imagined trajectory into the trained sequence discriminator to obtain the offline data distribution probability output by the trained sequence discriminator; The imagined trajectory is input into the preset model uncertainty estimator to obtain the uncertainty output by the model uncertainty estimator; For each moment in the imagined trajectory, the state and action at that moment are input into a preset support domain evaluator to obtain the deviation at that moment output by the support domain evaluator, and the overall deviation is determined based on each deviation. The credibility weight of the imagined trajectory is determined based on the offline data distribution probability, the uncertainty, and the overall deviation.

8. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 4, characterized in that, Based on a preset loss calculation method, the behavioral prior loss term of the imagined trajectory is determined, specifically including: Determine the prior behavioral policy obtained by training the offline trajectory dataset; Based on the KL divergence between the robot operation strategy to be optimized and the behavioral prior strategy, the behavioral prior loss term of the imagined trajectory is determined.

9. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 4, characterized in that, Based on a preset loss calculation method, the out-of-distribution penalty term of the imagined trajectory is determined, specifically including: The out-of-distribution penalty term of the imagined trajectory is determined based on the preset support domain function.

10. The offline reinforcement optimization method for robot operation strategies based on the diffusion world model as described in claim 4, characterized in that, S4 optimizes the robot operation strategy to be optimized based on the loss, specifically including: The robot operation strategy to be optimized is updated with the goal of maximizing the loss.

Images