Desktop operation task-oriented intelligent trajectory planning method and device

Abstract

The invention provides an intelligent trajectory planning method and device for a desktop operation task, and belongs to the field of agent behavior planning, and the method comprises the steps: obtaining a mixed data set composed of suboptimal state-action pairs of an agent under a plurality of different desktop operation tasks; performing minimum noise prediction error pre-training on the conditional diffusion generation model based on the mixed data set, and obtaining a reward signal returned by the environment through the pre-training model; based on the reward signal, optimizing a multi-step decision problem of the pre-training multi-task operation model through a strategy gradient algorithm, and adjusting parameters of the pre-training model; calculating the difference between the high reward action sequence and the model output in the optimization process, and carrying out regularization constraint; and generating an optimal agent action sequence plan suitable for a specific desktop operation task based on the optimized model. In this way, the training cost and time overhead are greatly reduced, and the problems of data dependence and generalization existing in a traditional reinforcement learning method are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of agent behavior planning, and particularly relates to a somatic intelligent trajectory planning method and device for desktop operation tasks. BACKGROUND

[0002] In daily life, desktop operation tasks are very common, such as arranging desktop items, assembling small parts, opening packaging, writing, using tools, and carrying containers. These seemingly simple operations actually involve multiple links such as perception, motion control, task planning, and object interaction. With the rapid development of service robots, home assistants, and human-machine collaboration systems, how to enable agents to have the ability to complete such tasks in a desktop environment has become an important part of realizing general intelligent systems. Desktop operation tasks have the characteristics of limited operation space, high precision requirements, diverse task types, and fast dynamic changes in the environment, which bring many challenges, such as object recognition difficulties, multi-object occlusion, force control and contact control complexity, high-dimensional action planning space, and weak cross-scene generalization ability. These all pose high requirements on traditional robot systems. Somatic intelligence, as an intelligent system that integrates perception, control, and cognition, provides an effective path to solve desktop operation tasks. Under this framework, the automatic execution of desktop operation tasks can be regarded as a problem of somatic agent policy trajectory planning in a complex environment: the system needs to plan a series of continuous actions according to the current perception state and target task, so that the somatic agent can complete the multi-stage, multi-contact operation process in a limited space. This process not only requires high-precision motion control, but also relies on the coordinated operation of task understanding, environment modeling, and action result prediction capabilities.

[0003] Traditional somatic agent policy trajectory planning often relies on expert systems, which use pre-set logic and rules to guide the behavior of somatic agents. For example, to make a robot arm open a drawer, the human expert needs to manually specify the robot arm's action path, making the robot arm approach the drawer, grab the handle, and pull open the drawer. This method requires human experts to manually design and code specific motion control schemes, and requires human experts to extract features of the specific somatic agent operating environment and accurately describe the actions that the corresponding agent should perform. This direct human control method not only requires human experts to have deep relevant agent expertise, but also lacks generalization ability. Once the somatic agent or its operating environment changes, the corresponding control scheme needs to be redesigned and adjusted.

[0004] Most of the existing automatic control methods of embodied agents adopt reinforcement learning algorithms. Reinforcement learning models the interaction process between the embodied agent and the desktop environment as a Markov process, and the goal of the embodied agent is to maximize the expected cumulative reward obtained in the multi-round interaction process. To achieve this goal, existing reinforcement learning is mostly based on value iteration method and policy iteration method. The value iteration method is a method of approximating the optimal policy by iteratively updating the value function, which does not need to explicitly perform policy evaluation and improvement, but directly updates the value function of the state until convergence. The policy iteration method is to gradually approximate the optimal policy by alternately performing policy evaluation (calculating the state value under the current policy) and policy improvement (improving the policy according to the state value under the current policy). The existing reinforcement learning method often models the policy as a Gaussian distribution, and the output of the policy network is the mean and variance of the Gaussian distribution. When selecting actions through the policy, actions close to the mean region are selected with a high probability. However, the policy network learned by these methods is often a high-dimensional unimodal Gaussian distribution, which cannot model complex multi-modal control behaviors. The diffusion model is a method that can effectively model complex multi-modal distributions.

[0005] The reinforcement learning method based on the diffusion model uses the expected return (i.e., the expected cumulative reward value) as a guide for the diffusion model to denoise the generated policy, with the goal of making the generated policy perform better and obtain higher rewards. However, this method requires a large dataset with reward labels for training, so that the model can distinguish which policies are better and can correspond to higher rewards. This relies on human prior to design the reward function, and the reward function is often different in different environments and different tasks, which is difficult to generalize. Moreover, this reward-based reinforcement learning method often requires a large amount of labeled data for training, which is time-consuming and costly to obtain. SUMMARY

[0006] To solve the above problems, the present application provides a embodied intelligent trajectory planning method, device, equipment and medium for desktop operation tasks.

[0007] To achieve the above purpose, the present application provides the following technical solutions: An embodied intelligent trajectory planning method for desktop operation tasks, the method comprising: Obtaining a mixed dataset composed of suboptimal state-action pairs of an embodied agent under multiple different desktop operation tasks; the desktop operation task includes interaction actions of the embodied agent with desktop objects and / or desktop states; minimizing noise prediction error pre-training a conditional diffusion generative model based on the mixed dataset, obtaining a desktop operation model and deploying it to a single desktop operation task environment for environmental interaction, and obtaining a reward signal returned by the environment; the reward signal is used to evaluate the embodied agent's actions in the single desktop operation task; An iterative denoising inference process of the desktop operation model is constructed as a multi-step decision problem, and based on the reward signal fed back by the task environment, a policy gradient algorithm is used to optimize the multi-step decision problem, and the parameters of the desktop operation model are adjusted; and in the optimization process, the optimal historical action sequence is determined based on the reward signal, and the difference between the optimal historical action sequence and the current output of the desktop operation model is calculated as a regularization term to regularize the desktop operation model; the optimal historical action sequence is a historical action sequence in which the reward signal meets a preset requirement; Based on the optimized desktop operation model, an optimal agent action sequence planning for a single target desktop operation task is generated.

[0008] Optionally, the loss function of the minimizing noise prediction error pre-training is: ; Wherein, K is the total number of diffusion steps, is the real noise added to the original action sequence , is the noise prediction network, is the action sequence after adding noise at the kth step, is the historical state sequence, and D is the pre-training dataset, is the mathematical expectation, which represents the average of all possible diffusion steps, data samples of the dataset, and real noise.

[0009] Optionally, the iterative denoising inference process of the desktop operation model is constructed as a multi-step decision problem, which includes: Each step of the denoising process of the desktop operation model is defined as a time step of a Markov decision process (MDP); The input state of the kth step denoising process is defined as ; wherein, is the current desktop environment state, is the current noisy action sequence; The action output by the kth step denoising process is defined as ; The reward function is defined as the reward obtained from the task environment only when the final action sequence is generated, i.e. When k≠K-1, is 0, is the action at the K-1th step.

[0010] ​Optionally, the multi-step decision problem is optimized by a policy gradient algorithm, and an objective function is to maximize the expected reward of the action sequence generated under a given single desktop operation task: ; The gradient is calculated by a proximal policy optimization method, and a loss function corresponding to the proximal policy optimization method is: ; Wherein, is the action sequence The reward obtained after executing the desktop task environment, is a mathematical expectation, indicating that the denoising process is averaged from the old policy Sampling, , is the current model Given and The conditional probability distribution of denoising one step to get , is the old model Given and The conditional probability distribution of denoising one step to get , is a hyperparameter.

[0011] Optionally, the regularization constraint term is specifically: A high-quality action sequence buffer is maintained to store historical action sequences obtained in the optimization process whose reward value exceeds a set threshold; The regularization constraint term is a behavior cloning-based loss function, and its calculation formula is: ; Wherein, is the distribution of action sequences whose reward value exceeds the set threshold, is the action sequence generated in the optimization process whose reward value exceeds the set threshold.

[0012] Optionally, the total loss function of joint optimization is: ; Wherein, is a regularization coefficient for balancing the loss weight of the two items.

[0013] Optionally, the generation of the optimal intelligent agent action sequence plan of the desktop operation task comprises: After generating a complete action sequence, the first preset number of steps in the action sequence are executed, and the environment state of the current desktop is determined, and the action sequence is re-planned according to the change of the desktop environment state.

[0014] A body-aware intelligent trajectory planning device for a desktop operation task, the device comprising: An acquisition module configured to acquire a hybrid dataset composed of suboptimal state-action pairs of a body-aware agent in a plurality of different desktop operation tasks; the desktop operation tasks comprising interaction actions of the body-aware agent with desktop objects and / or desktop states; An interaction module configured to pre-train a conditional diffusion generation model based on the hybrid dataset to minimize noise prediction error, obtain a desktop operation model, and deploy the desktop operation model to a single desktop operation task environment for environmental interaction to obtain a reward signal returned by the environment; the reward signal being used to evaluate actions of the body-aware agent in the single desktop operation task; An optimization module configured to construct an iterative denoising inference process of the desktop operation model as a multi-step decision problem, optimize the multi-step decision problem based on the reward signal fed back by the task environment, adjust parameters of the desktop operation model through a policy gradient algorithm, determine an optimal historical action sequence based on the reward signal during the optimization process, calculate a difference between the optimal historical action sequence and a current output of the desktop operation model as a regularization term, and regularize the desktop operation model; the optimal historical action sequence being a historical action sequence in which the reward signal meets a preset requirement; A generation module configured to generate an optimal agent action sequence plan for a single target desktop operation task based on the optimized desktop operation model.

[0015] A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the body-aware intelligent trajectory planning method for a desktop operation task.

[0016] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the body-aware intelligent trajectory planning method for a desktop operation task when executing the program.

[0017] The body-aware intelligent trajectory planning method for a desktop operation task provided by the present application has the following beneficial effects: The present application faces a desktop operation type embodied intelligent agent, through pre-training by using a suboptimal mixed desktop operation data set, makes the diffusion model learn a wide range of desktop operation motion prior, effectively reduces the dependence on a large amount of artificial annotation reward data. Subsequently, under the guidance of a small amount of real desktop environment feedback signal, the diffusion model denoising process is constructed as an optimizable sequence decision problem, and the efficient optimization is realized through the strategy gradient algorithm, which significantly reduces the dependence of the reward function design of the desktop operation type task and the demand amount of the reward label, enhances the autonomous learning ability of the algorithm, enables the intelligent agent to learn from the environment more independently, and can self-optimize even in the absence of perfect demonstration, improves the adaptability and flexibility of the intelligent agent in various desktop operation tasks. At the same time, the regularization constraint with high reward action as the target is introduced, which ensures that the model does not forget the pre-training knowledge in the optimization process, and improves the stability and generalization performance of the strategy. Finally, the method can quickly adapt to different desktop operation tasks under a small amount of online interaction and reward feedback, generate high-performance desktop operation action sequences, greatly reduce the training cost and time overhead, and effectively solve the data dependence and generalization problems of the traditional reinforcement learning method based on the diffusion model in the desktop operation task. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application and the design scheme thereof, the drawings required by the present embodiments will be briefly introduced as follows. The drawings in the following description are only part of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 An overall framework structure diagram is provided according to an exemplary embodiment of the present application.

[0020] Figure 2 A flowchart of an embodied intelligent trajectory planning method for a desktop operation task is provided according to an exemplary embodiment of the present application.

[0021] Figure 3 A pre-training stage flowchart is provided according to an exemplary embodiment of the present application.

[0022] Figure 4 An optimization stage flowchart is provided according to an exemplary embodiment of the present application.

[0023] Figure 5 A performance comparison diagram of different reinforcement learning methods in different tasks is provided according to an exemplary embodiment of the present application.

[0024] Figure 6 A performance comparison diagram of different tasks and existing multi-task strategy models is provided according to an exemplary embodiment of the present application.

[0025] Figure 7 This is a schematic diagram comparing the performance of different regularization constraint methods provided by the present invention according to an exemplary embodiment.

[0026] Figure 8 This is a visual analysis diagram of different regularization constraint methods provided by the present invention according to an exemplary embodiment.

[0027] Figure 9 This is a block diagram of an embodied intelligent trajectory planning device for desktop operation tasks provided by the present invention according to an exemplary embodiment. Detailed Implementation

[0028] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0029] The overall process of this invention is as follows: Figure 1 As shown, different colors represent different desktop operation tasks. The diffusion policy model is first pre-trained on a mixed operation dataset from multiple desktop operation tasks (such as opening a drawer, getting coffee, and placing a plate), and then fine-tuned for specific downstream tasks using the reward signal corresponding to each task. This invention aims to learn a general diffusion policy planner from a large amount of low-quality trajectory desktop operation data without reward labels, and to have the ability to quickly adapt to different downstream tasks. This invention only requires that these poor data consist of suboptimal state-action (SA) pairs from different desktop operation tasks, where state S describes the current environmental state of the desktop task, such as the location of the target object, and action A is the behavior taken by the agent. This type of data is readily available in the real world. This invention proposes a two-stage training framework, including pre-training and fine-tuning stages. In the pre-training stage, the goal of the diffusion policy planner is to model a diverse distribution of desktop operation action trajectories with broad coverage, thereby learning a basic desktop task operation policy with generalization capabilities. Subsequently, during the fine-tuning phase, the planner can use a rapid adaptation mechanism to further focus on high-reward areas in specific downstream desktop tasks, which require only a small number of reward tags for guidance.

[0030] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0031] First, this invention provides an embodied intelligent trajectory planning method for desktop operation tasks, specifically as follows: Figure 2 As shown, it includes the following steps: S101. Obtain a hybrid dataset consisting of suboptimal state-action pairs of an embodied agent under multiple different desktop operation tasks.

[0032] The desktop operation task includes interactive actions with desktop items and / or desktop status; for example, tidying up desktop items, assembling small parts, opening packaging, writing, using tools, and moving containers.

[0033] In this step, various service task experimental environments are first set up on a desktop. For example, a drawer is placed on the desktop, and the embodied intelligent agent is a robotic arm. The goal is to find the drawer handle, control the gripper to grasp the handle, and then smoothly pull the drawer open. Another example is a cup and a coffee machine placed on the desktop. The robotic arm's goal is to find the cup, grasp it, find the coffee machine, and place the cup under it. These tasks reconstruct real-world smart living service scenarios. The collection of suboptimal data is diverse. This invention considers using the experience buffer during the reinforcement learning agent's learning process as suboptimal data. Specifically, for each task, a reinforcement learning agent is trained from scratch until convergence. The experience buffer from the start of training to halfway through is collected as suboptimal data. This experience buffer contains the initial random exploration trajectory of the robotic arm. The average reward of the entire buffer is low, containing a large number of poor trajectories from early trial and error. Therefore, learning from this dataset is difficult and also beneficial for verifying the effectiveness of the proposed solution.

[0034] The suboptimal state-action pairs collected from various tasks are organized and standardized in data format, including specifying the representation of states and the encoding of actions. Then, these suboptimal state-action pairs from different tasks are combined to construct a comprehensive hybrid offline dataset encompassing all tasks, providing a rich data foundation for subsequent model pre-training.

[0035] S102. Based on the mixed dataset, the conditional diffusion generation model is pre-trained to minimize the noise prediction error, resulting in a desktop operation model. This model is then deployed to a single desktop operation task environment for environmental interaction, and the reward signal returned by the environment is obtained.

[0036] In this step, the conditional diffusion generation model is pre-trained based on the desktop operation dataset to minimize noise prediction error. The model is trained to predict noise added to the original action sequence, thereby giving the model the ability to generate action sequences from pure noise. The pre-trained multi-task desktop operation model is then deployed to a specific single desktop operation environment (e.g., opening a drawer) to interact with target objects in the desktop environment and obtain reward signals returned by the desktop environment. These reward signals are used to evaluate the embodied agent's actions in a single desktop operation task, for example, to evaluate the degree of completion and / or speed of completion of the action, etc., which are not limited in this invention.

[0037] The pre-training of the conditional diffusion generation model first requires performing multi-step noise addition on the action sequence in the suboptimal state-action pair. In each step of the noise addition process, the action sequence of the current step is transformed into Gaussian noise according to the preset noise scheduling parameters. Then, the denoising network is trained to predict the added noise. Its training objective is to minimize the mean square error between the noise prediction value and the real noise, so that the model can learn to recover a reasonable action sequence from the noise.

[0038] In one embodiment, the present invention relates to a general embodied policy learning method capable of performing multiple desktop operation tasks, modeled as including A reinforcement learning problem for each desktop task (such as opening a drawer, getting coffee, etc.), where each task... Modeled as task-dependent Markov processes, each Markov process is defined as a tuple. ,in For desktop tasks The state space, It is a universal motion space. This is the state transition function. Let t be the state at time t. Let t be the action taken at time t. For desktop tasks The reward function, The initial state distribution, This is the discount factor. All desktop tasks share the same action space because they all use the same embodied agent to complete different tasks. Different desktop tasks have their own state spaces, reward functions, and state transition functions. At each time step... The embodied intelligent agent observes the state of the desktop environment. According to the strategy Take an action The environment returns a reward value. The goal of an embodied intelligent agent is to learn an optimal strategy. Maximize the expected cumulative discount reward (i.e., expected return) across all tasks:

[0039] ; in, To obtain the mathematical expectation, we take the average over all tasks. For the mathematical expectation, the task at time t is used Calculate the expected value of all actions generated by the above strategy. Let be the discount factor at time t. For time t in the task The rewards obtained.

[0040] The pre-training phase is given an offline dataset containing multiple desktop operation tasks. ,in, For the offline dataset of the i-th desktop task, Given the number of desktop tasks, existing diffusion-based work typically models the embodied agent policy planning problem as a conditional generation problem: ; in The generation condition is usually a reward tag. , To give desktop tasks The planned strategy trajectory Conditions for the diffusion model generate The conditional probability distribution, To achieve mathematical expectation, the average is calculated across all possible desktop tasks. However, reward labels are often sparse and difficult to obtain. To eliminate the dependence on reward labels, this invention aims to learn from data containing only state-action transition pairs, i.e., data consisting only of... ,in, Let t be the desktop environment state at time t. The actions taken by the embodied intelligent agent at time t. This represents the state of the desktop environment at time t+1. It does not include a reward label. The goal of the pre-training phase is to train a conditional diffusion generative model. Based on the input historical observation information, generate future action sequences. Therefore, the corresponding condition generation problem is modeled as follows: ,in For action sequences, This is the sequence of future actions of the embodied intelligent agent, starting from time t. To obtain the mathematical expectation, average the values ​​over all possible state-action sequences in the mixed task dataset. For the planned length, For desktop datasets The time step of the sampled trajectory. For history Step environment status information, i.e. In this way, during the pre-training phase, the diffusion strategy can learn a wide range of action sequence distributions based solely on historical observation information, without the need for additional guidance.

[0041] During training, following the training paradigm of traditional diffusion models, noise is progressively added to the original data, transforming the original action sequences into pure Gaussian noise. The model's goal is to learn the noise at a given time step, so that the original action sequences can be inferred from the pure Gaussian noise during subsequent inference. Specifically, the diffusion model transforms the action sequences in the dataset... conduct Noise is added step by step, and the noise addition method for each step is as follows:

[0042] ; in These are preset step parameters. It is the identity matrix. Let be the noisy action sequence obtained by adding k steps of noise to the action sequence at time t. Its corresponding inverse denoising definition is:

[0043] ; Here It is the posterior mean of the denoising process, which is the noise. The function is called the diffusion model. Therefore, the training objective of the diffusion model is to predict the value at a given time step. Above, added to the sequence The loss function for pre-training, which minimizes the noise prediction error, is:

[0044] ; Where K is the total number of diffusion steps, This is real noise. For noise prediction networks, For the noisy action sequence at step k, Let D be a sequence of historical states, and D be a pre-trained desktop dataset from which samples are taken. To conduct training, Let be the mathematical expectation, representing the average over all possible diffusion steps, the dataset samples, and the real noise. Here, the number of all possible diffusion steps follows a certain pattern. Uniform distribution on ( The dataset consists of state-action pairs, sampled from a pre-trained dataset. Real noise follows a standard normal distribution. After training, sampling is performed as follows to iteratively transform the original noise into a sequence of executable future actions:

[0045] ; in, , , .

[0046] The goal of the pre-training phase is to train an action planning model, taking historical environment state information as input and outputting a sequence of future actions. The flowchart for the pre-training phase is as follows: Figure 3 As shown.

[0047] S103. The iterative denoising inference process of the desktop operation model is constructed as a multi-step decision problem. Based on the reward signal fed back by the task environment, the multi-step decision problem is optimized by the policy gradient algorithm to adjust the parameters of the desktop operation model. During the optimization process, the optimal historical action sequence is determined based on the reward signal. The difference between the optimal historical action sequence and the current output of the desktop operation model is calculated as a regularization term to regularize the desktop operation model.

[0048] The optimal historical action sequence is the historical action sequence in which the reward signal meets a preset requirement. For example, the reward signal can be an evaluation score obtained from a reward function used to evaluate the completion of actions of an embodied agent. If the evaluation score is greater than or equal to a high reward score threshold, the action sequence corresponding to the reward signal is determined to be a high-reward action sequence. The reward signal can be set according to actual conditions, and this invention does not impose any restrictions on it.

[0049] In this step, the parameters of the pre-trained model are further optimized. This step employs a proximal policy optimization method, calculating the policy update ratio through importance sampling; using a shearing function to constrain the magnitude of policy updates to prevent policy mutations during training; and utilizing task rewards from environmental feedback to evaluate the quality of the generated action sequences, and using this to calculate policy gradients to update the model parameters.

[0050] In one embodiment, the optimization phase comprises two MDPs: the first, consistent with traditional reinforcement learning decision-making processes, models the interaction and transitions with the desktop environment as an MDP; the second models the denoising process of the diffusion model as an MDP. This includes two types of time steps: the desktop environment time step. The subscript indicates the time step for denoising the diffusion model. Use superscript to indicate.

[0051] For example, each step of the pre-trained model denoising process is defined as a time step of a Markov Decision Process (MDP); the input state of the k-th denoising step is defined as... ;in, The current environment reflects information such as the location of the target object being manipulated. Given the current noisy action sequence; define the action output at step k as... , for The denoising results; the reward function Defined as a reward that is only received in response to feedback from the task environment when generating the final action sequence, i.e. When k=K-1 =0, This is the action at step K-1. Specifically, the formal representation of the diffusion model denoising process MDP is:

[0052] ; ; ; ; ; It is a piecewise function, when hour, Otherwise, it is 0. and The diffusion model MDP is represented in the first... The state and movement of the steps; and Represent the initial state distribution and the state transition equation. Indicates in The Dirac distribution of points; It is a reward function. This is the action diffusion model obtained during the pre-training phase. After this modeling, the diffusion MDP can correspond to the denoising process of the original diffusion model: the diffusion MDP samples an initial state. Initialization is performed at the start of the inverse denoising process of the diffusion model. ; at each time step , Based on the current state Take an action This corresponds to the diffusion model based on the reverse process. Denoising generation in one step The overall diffusion MDP reward is sparse because the noisy action sequences in the middle are difficult to evaluate, and therefore only the final executable action sequences are evaluated. Rewards are only obtained after the event. Traditional image generation model research typically uses pre-trained reward models to evaluate the final generated results. For example, an image aesthetic reward model is used to score the generated images, and the model is optimized based on the scores. However, this approach is difficult to apply to reinforcement learning because training an accurate reward model is very challenging. The goal of this invention is to improve the performance of pre-trained policies in downstream tasks. Therefore, this invention employs an online evaluation method, whereby the generated action sequences are evaluated online. Execute in the downstream desktop task environment and directly obtain the reward from the desktop environment's feedback. This allows for the construction of a dynamic "generate-feedback-update" fine-tuning paradigm, iteratively learning and optimizing from the environment. Specifically, given any time step... The diffusion model is used to reverse the process and plan the future action sequence. and before execution The action involves receiving rewards from environmental feedback as an evaluation of the generated action. Evaluation indicators, namely .

[0053] By modeling the diffusion model as an MDP, the objective of the optimization phase aligns with that of traditional reinforcement learning: maximizing the performance of the pre-trained diffusion policy model on the downstream target task. The expected reward of the action sequence generated is calculated using the policy gradient algorithm to optimize a multi-step decision problem. The objective function is to maximize the expected reward for generating a given action sequence for a desktop task. .

[0054] However, direct optimization It is very difficult because the model cannot directly traverse all possible action sequences. Therefore, this invention uses a policy gradient algorithm for optimization. The policy gradient algorithm directly optimizes the policy by calculating the gradient of the policy parameters and performing gradient ascent. This method does not require calculating the Q-value or value function; it directly learns how to select the best action given a state through sampling and gradient updates. Specifically, the gradient of the objective function of the diffusion MDP is:

[0055] .

[0056] However, directly updating this policy gradient consumes significant computational resources because new training samples need to be regenerated after each update. To improve sample utilization and leverage previously generated sequences, this invention employs an importance sampling method and further introduces the PPO-clip method, which updates the gradient using a proximal policy optimization method. The loss function corresponding to the proximal policy optimization method is:

[0057] ; in, Action sequence In the mission The reward obtained after performing an action in an environment (such as an environment where you open a drawer or receive coffee). Let be the mathematical expectation, representing the expectation from the old strategy. The denoising process obtained from the mid-sample is averaged. , For the current model Given and Noise reduction in one step The conditional probability distribution, For the old model Given and Noise reduction in one step The conditional probability distribution, This is a hyperparameter.

[0058] Furthermore, relying solely on the aforementioned reward maximization loss function Simply optimizing the model is insufficient because the model can shift excessively during updates, forgetting knowledge acquired in the pre-training phase, leading to performance crashes and instability when maximizing rewards. Therefore, this invention introduces a novel behavior-clone (BC)-based regularization method during the optimization process to constrain the model's distribution changes. Specifically, the goal of BC regularization is to regulate the model's distribution changes during optimization. Constrained on a target distribution Above, ensure that the model does not deviate from its intended path during the optimization and update process. Too far. This constraint process can be formally described using negative logarithmic probability (NLL):

[0059] .

[0060] Similarly, directly optimizing NLL is very difficult because it is impossible to obtain the necessary parameters from the NLL implementation. Sample all possible action sequences .

[0061] In one embodiment, the present invention simultaneously applies a behavior cloning-based regularization constraint during model optimization. First, a high-quality action sequence buffer is maintained to store historical action sequences whose reward values ​​exceed a set threshold obtained during optimization. Then, action sequences are sampled from this buffer, and the noise prediction difference between the current policy and these high-quality sequences is calculated. Finally, this difference is added as a regularization term to the overall loss function to constrain the policy model from excessively deviating from the action distribution learned during the pre-training phase. The regularization constraint term is a behavior cloning-based loss function. The present invention derives an equivalent loss function form using the Evidence Lower Bound (ELBO), and its calculation formula is as follows:

[0062] ; in, The distribution of action sequences whose reward values ​​exceed a set threshold is an unknown distribution. This distribution is approximated by continuously sampling during the optimization process. Action sequences whose reward values ​​exceed a set threshold during the optimization process are considered to originate from a distribution. .

[0063] The final overall loss function obtained from the joint optimization is: ; in, This is the regularization coefficient used to balance the weights of the two loss terms.

[0064] In this way, the pre-trained diffusion model can be optimized in an end-to-end manner.

[0065] Intuitively, the most ideal target distribution The optimal strategy It can generate those that satisfy samples ,here Indicates sample The performance or quality metric is the cumulative reward of an action sequence. However, in actual optimization, the optimal distribution is unknown. Therefore, this invention uses a sample set... To approximate the target distribution That is, by collecting several high-quality action sequences and computing on these action sequences. The goal is to achieve the optimization process constrained by the optimal target distribution. Specifically, during the optimization process, this invention designs and maintains a target distribution sampling buffer. After sampling and executing an action sequence, it is determined whether the action sequence is a high-quality action sequence. If so, it is added to the target distribution sampling buffer. Through continuous sampling and interaction, the target distribution sampling buffer will contain more and more high-quality action sequences. When updating, sampling action sequences from the buffer can be approximated as actions from the target distribution. Sampling is performed during this phase. The optimization phase flowchart is as follows: Figure 4 As shown.

[0066] S104. Based on the optimized desktop operation model, generate the optimal agent action sequence planning for a single-target desktop operation task.

[0067] After generating a complete action sequence, the system executes a pre-set number of steps and determines the current environmental state. Based on changes in the environmental state, the action sequence is replanned to improve the accuracy of desktop operation tasks performed by the intelligent agent.

[0068] This invention improves the performance of embodied agents in multiple downstream desktop tasks compared to traditional reinforcement learning strategies. In the widely used desktop robotic arm operation environment MetaWorld, this invention is compared with other existing, high-performance pre-training-optimization methods based on traditional Gaussian policies, such as… Figure 5 As shown in the figure. These methods all model the policy model as a Gaussian policy, using imitation learning or traditional offline reinforcement learning in the pre-training stage and traditional online reinforcement learning in the optimization stage. Compared with previous methods, the diffusion policy model used in this invention shows better performance for complex representations and multi-modal distributions. Since the pre-training stage uses mixed data from multiple tasks with diverse action distributions, traditional Gaussian policies are difficult to model accurately, making it difficult to extract useful action priors during pre-training. This leads to a decline in model performance during reward optimization on specific tasks, and even results in a persistently low completion rate for many tasks. At the same time, using traditional online reinforcement learning for optimization does not improve the performance of the pre-trained model in downstream tasks, while the reward optimization method proposed in this invention can significantly improve the performance of the pre-trained policy model in downstream tasks.

[0069] Compared to existing multi-task policy models based on diffusion models, this invention improves the performance of embodied agents across multiple downstream tasks. In the widely used desktop robotic arm operation environment MetaWorld, this invention is compared with other existing high-performance multi-task policy models, including diffusion-based and Transformer-based methods, such as... Figure 6As shown. These methods all require a large amount of reward-labeled data for training, and need to set explicit expected rewards as generation conditions when generating action sequences. Compared with other existing methods, this invention fully leverages the powerful representational capabilities of the diffusion model. Through the proposed two-stage training framework, it endows the policy model with a wide range of action sequence distribution priors, enabling the model to quickly adapt to and converge to the high-reward region in downstream tasks using the optimization scheme proposed in this invention.

[0070] Some existing works have also proposed using reward optimization methods to improve the performance of diffusion models and have adopted other constraint schemes, such as using KL divergence or using loss functions from the pre-training stage. However, in the field of general embodied policy optimization, none of the above schemes can improve the capabilities of pre-trained policy models. Figure 7 As shown in the previous analysis, directly using reward optimization without additional constraints can easily lead to the model forgetting pre-trained knowledge, resulting in a decline in model performance. Existing constraint methods restrict the model to the vicinity of the pre-trained model, making the model prone to getting trapped in local optima and unable to escape the local optimum to find a higher reward region, thus resulting in essentially unchanged model performance. Figure 8 The policy trajectories obtained after optimization of different models were visualized. After dimensionality reduction using t-SNE, each point represents a trajectory, and the points are colored according to the reward magnitude. Figure 8 The trajectory distribution is as follows. Existing constraint methods yield policy distributions that are largely consistent with pre-trained models. However, after applying the regularized constraint reward optimization scheme proposed in this invention, the overall distribution shifts towards the high-reward region.

[0071] By employing the aforementioned method and pre-training with a suboptimal mixed desktop operation dataset, the diffusion model learns a broad range of prior desktop operation actions, effectively reducing its reliance on large amounts of manually labeled reward data. Subsequently, guided by a small amount of feedback signals from real desktop environments, the denoising process of the diffusion model is constructed as an optimizable sequence decision problem. Efficient optimization is achieved through a policy gradient algorithm, significantly reducing the dependence on the design of reward functions for desktop operation tasks and the need for reward labels. This enhances the algorithm's autonomous learning ability, allowing the agent to learn more independently from the environment and self-optimize even in the absence of perfect examples, improving its adaptability and flexibility across various desktop operation tasks. Simultaneously, a regularization constraint targeting high-reward actions is introduced to ensure the model retains pre-trained knowledge during optimization, improving policy stability and generalization performance. Ultimately, this method can quickly adapt to different desktop operation tasks with limited online interaction and reward feedback, generating high-performance desktop operation action sequences. This significantly reduces training costs and time overhead, effectively addressing the data dependency and generalization problems of traditional diffusion model-based reinforcement learning methods in desktop operation tasks.

[0072] Secondly, the present invention also provides an embodied intelligent trajectory planning device for desktop operation tasks, such as... Figure 9 As shown, it includes: The acquisition module 201 is used to acquire a mixed dataset consisting of suboptimal state-action pairs of an embodied agent under multiple different desktop operation tasks; the desktop operation task includes the interaction actions of the embodied agent with desktop objects and / or desktop states.

[0073] The interaction module 202 is used to pre-train the conditional diffusion generative model based on the mixed dataset to minimize the noise prediction error, obtain the desktop operation model, and deploy it to a single desktop operation task environment for environmental interaction, and obtain the reward signal returned by the environment; the reward signal is used to evaluate the actions of the embodied agent in the single desktop operation.

[0074] The optimization module 203 is used to construct the iterative denoising inference process of the desktop operation model into a multi-step decision problem, and optimize the multi-step decision problem by means of the policy gradient algorithm based on the reward signal fed back by the task environment, and adjust the parameters of the desktop operation model. In the optimization process, the optimal historical action sequence is determined based on the reward signal, and the difference between the optimal historical action sequence and the current output of the desktop operation model is calculated as a regularization term to regularize the desktop operation model. The optimal historical action sequence is the historical action sequence in which the reward signal reaches the preset requirements.

[0075] The generation module 204 is used to generate the optimal agent action sequence planning for a single-objective desktop operation task based on the optimized desktop operation model.

[0076] Using the aforementioned apparatus, the diffusion model learns a broad range of prior desktop operation actions through pre-training on a suboptimal mixed desktop operation dataset, effectively reducing its reliance on large amounts of manually labeled reward data. Subsequently, guided by a small amount of feedback signals from real desktop environments, the denoising process of the diffusion model is constructed as an optimizable sequence decision problem. Efficient optimization is achieved through a policy gradient algorithm, significantly reducing the dependence on the design of reward functions for desktop operation tasks and the need for reward labels. This enhances the algorithm's autonomous learning ability, allowing the agent to learn more independently from the environment and self-optimize even in the absence of perfect examples, improving its adaptability and flexibility across various desktop operation tasks. Simultaneously, a regularization constraint targeting high-reward actions is introduced to ensure the model retains pre-trained knowledge during optimization, improving policy stability and generalization performance. Ultimately, this method can quickly adapt to different desktop operation tasks with limited online interaction and reward feedback, generating high-performance desktop operation action sequences. This significantly reduces training costs and time overhead, effectively addressing the data dependency and generalization problems of traditional diffusion model-based reinforcement learning methods in desktop operation tasks.

[0077] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 2 The steps of the provided embodied intelligent trajectory planning method for desktop operation tasks.

[0078] This invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for various operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above-mentioned functions. Figure 2 The steps of the provided embodied intelligent trajectory planning method for desktop operation tasks.

[0079] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the patent of the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A body-aware trajectory planning method for tabletop operation tasks, characterized in that, The method comprises: acquiring a mixed data set composed of suboptimal state-action pairs of a embodied agent under multiple different desktop operation tasks; the desktop operation tasks include interaction actions of the embodied agent with desktop articles and / or desktop states; based on the mixed data set, pre-training a conditional diffusion generation model with a minimized noise prediction error to obtain a desktop operation model and deploy the desktop operation model to a single desktop operation task environment for environment interaction to acquire a reward signal returned by the environment; the reward signal is used to evaluate actions of the embodied agent in the single desktop operation task; constructing an iterative denoising inference process of the desktop operation model as a multi-step decision problem, and based on the reward signal fed back by the task environment, optimizing the multi-step decision problem through a policy gradient algorithm to adjust parameters of the desktop operation model; and in the optimization process, determining an optimal historical action sequence based on the reward signal, and calculating a difference between the optimal historical action sequence and a current output of the desktop operation model as a regularization term to regularize the desktop operation model; the optimal historical action sequence is a historical action sequence in which the reward signal meets a preset requirement; based on the optimized desktop operation model, generating an optimal embodied agent action sequence plan for a single target desktop operation task.

2. The method of claim 1, wherein, A loss function for the pre-training of the minimized noise prediction error is: ； where K is the total number of diffusion steps, is the real noise added to the original action sequence , is the noise prediction network, is the action sequence after adding noise at the k-th step, is the history state sequence, and D is the pre-training dataset, is the mathematical expectation, which means averaging over all possible diffusion steps, data samples of the dataset, and real noise.

3. The method of claim 2, wherein, the construction of the iterative denoising inference process of the desktop operation model as a multi-step decision problem comprises: defining each step of the denoising process of the desktop operation model as a time step of a Markov decision process (MDP); Let the input state of the kth step of the denoising process be defined as ; where, is the current state of the environment of the tabletop, is the current action sequence with noise. defining the action of the kth step denoising output as ; The reward function is defined as the reward obtained from the task environment only at the generation of the final action sequence, i.e. when k≠K-1 , is 0, is the action at the K-1th step.

4. The method of claim 3, wherein, the optimization of the multi-step decision problem through the policy gradient algorithm has an objective function of maximizing an expected reward of an action sequence generated under a given single desktop operation task: ； the gradient is updated through a proximal policy optimization method, and a loss function corresponding to the proximal policy optimization method is: ； where, is the sequence of actions is the reward obtained after performing the sequence of actions in the environment, is the mathematical expectation denoting the average over the denoising processes sampled from the old policy , , is the current model given and is the conditional probability distribution of obtaining after one denoising step, is the old model given and is the conditional probability distribution of obtaining after one denoising step, is a hyperparameter.

5. The method of claim 4, wherein, the construction of the regularization constraint term is specifically: maintaining a high-quality action sequence buffer for storing historical action sequences obtained in the optimization process and having reward values exceeding a set threshold; the regularization constraint term is a loss function based on behavior cloning, and a calculation formula thereof is: ； wherein, are action sequences whose reward values exceed a set threshold, are action sequences whose reward values generated in the optimization process exceed a set threshold.

6. The method of claim 1, wherein, a total loss function for the joint optimization is: ； wherein, is a regularization coefficient for balancing the two loss weights.

7. The method of claim 1, wherein, the generation of the optimal embodied agent action sequence plan for the desktop operation task comprises: after generating a complete action sequence, executing a preset number of steps in the action sequence, determining an environment state of the current desktop, and re-planning the action sequence according to a change in the desktop environment state.

8. A body-aware trajectory planning device for a tabletop operation task, characterized by, The device comprises: an acquisition module configured to acquire a mixed data set composed of suboptimal state-action pairs of a embodied agent under multiple different desktop operation tasks; the desktop operation tasks include interaction actions of the embodied agent with desktop articles and / or desktop states; an interaction module configured to pre-train a conditional diffusion generation model with a minimized noise prediction error based on the mixed data set to obtain a desktop operation model and deploy the desktop operation model to a single desktop operation task environment for environment interaction to acquire a reward signal returned by the environment; the reward signal is used to evaluate actions of the embodied agent in the single desktop operation task; The optimization module is configured to construct an iterative denoising inference process of the desktop operation model as a multi-step decision problem, and optimize the multi-step decision problem by a policy gradient algorithm based on a reward signal of task environment feedback, and adjust parameters of the desktop operation model; and determine an optimal historical action sequence based on the reward signal in the optimization process, calculate a difference between the optimal historical action sequence and a current output of the desktop operation model as a regularization term, and perform regularization constraint on the desktop operation model; the optimal historical action sequence is a historical action sequence in which the reward signal reaches a preset requirement. The generation module is configured to generate an optimal agent action sequence plan of a single target desktop operation task based on the optimized desktop operation model.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-7.

10. A computer device, comprising: The computer program is stored in the memory and executable on the processor, and the processor implements the method in any one of claims 1-7 when executing the program.

Images