Strategy model training method and device, computer device, medium and product

Abstract

Embodiments of the present application provide a policy model training method and device, computer equipment, medium and product. The method comprises: receiving task input information corresponding to a robot operation task; performing search tree construction processing on the robot operation task based on the task input information, a policy model and a world model, to obtain target search tree data; performing in-group relative evaluation on subsequent search result information of a plurality of child search nodes corresponding to a same parent search node, to obtain advantage level labels of action data associated with each of the child search nodes; and training the policy model based on action data in the target search tree data and the advantage level labels corresponding to the action data, to obtain a trained policy model. Embodiments of the present application can convert subsequent results obtained by searching into fine-grained training signals that can be used for conditional training of a policy model, thereby improving the training effect of the policy model.

Description

Technical Field

[0001] This application relates to the fields of robotics and model training technology, and more particularly to strategy model training methods and apparatus, computer equipment, media, and products. Background Technology

[0002] With the development of robotics, artificial intelligence, and automatic control technologies, robots are increasingly being applied in industrial manufacturing, warehousing and logistics, home services, medical assistance, and laboratory automation. Robot strategy models can generate actions based on environmental conditions and task objectives, enabling the robot to complete operational tasks. However, in related technologies, the training performance of robot strategy models is unsatisfactory, making it difficult to meet the needs of practical applications.

[0003] Based on this, embodiments of this application provide a strategy model training method and apparatus, computer equipment, medium, and product to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a strategy model training method and apparatus, computer equipment, medium, and product to improve the training effect of strategy models.

[0005] In a first aspect, embodiments of this application provide a strategy model training method, the method comprising: receiving task input information corresponding to a robot operation task; wherein the task input information includes initial state information and task information; performing search tree construction processing on the robot operation task based on the task input information, a strategy model, and a world model to obtain target search tree data; wherein the target search tree data includes multiple search nodes, connection relationships between different search nodes, and action data corresponding to the connection relationships, the action data being used to characterize actions transferred from a corresponding parent search node to a corresponding child search node, and associated with the corresponding child search node; performing intra-group relative evaluation on the subsequent search result information of multiple child search nodes corresponding to the same parent search node based on the target search tree data to obtain an advantage level label for the action data associated with each child search node; wherein the advantage level label is used to characterize the relative quality level between action data associated with different child search nodes under the same parent search node; training the strategy model based on the action data in the target search tree data and the advantage level label corresponding to the action data to obtain a trained strategy model; wherein, during the training of the strategy model, the advantage level label is used to determine the training condition information of the strategy model.

[0006] Secondly, embodiments of this application provide a strategy model training device, the device comprising: an information receiving module for receiving task input information corresponding to a robot operation task; wherein the task input information includes initial state information and task information; and a search tree construction module for performing search tree construction processing on the robot operation task based on the task input information, the strategy model, and the world model to obtain target search tree data; wherein the target search tree data includes multiple search nodes, connection relationships between different search nodes, and action data corresponding to the connection relationships, the action data being used to characterize actions transferred from the corresponding parent search node to the corresponding child search node, and associated with the corresponding child search node; The relative evaluation module is used to perform intra-group relative evaluation of the subsequent search results information of multiple child search nodes corresponding to the same parent search node based on the target search tree data, and obtain the dominance level label of the action data associated with each child search node; wherein, the dominance level label is used to characterize the relative quality level between action data associated with different child search nodes under the same parent search node; the model training module is used to train the strategy model based on the action data in the target search tree data and the dominance level label corresponding to the action data, and obtain the trained strategy model; wherein, during the training of the strategy model, the dominance level label is used to determine the training condition information of the strategy model.

[0007] Thirdly, embodiments of this application provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the above methods.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0010] This application provides a strategy model training method and apparatus, computer equipment, medium, and product. It adopts the concept of relative evaluation within groups based on target search tree data and using advantage level labels as training condition information, thereby transforming the subsequent search results into fine-grained training signals that can be used for conditional training of the strategy model, thus improving the training effect of the strategy model. Attached Figure Description

[0011] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0012] Figure 1 This is a flowchart illustrating a strategy model training method provided in an embodiment of this application.

[0013] Figure 2 This is a structural block diagram of a strategy model training device provided in an embodiment of this application.

[0014] Figure 3 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] See Figure 1 , Figure 1 This is a flowchart illustrating a strategy model training method provided in an embodiment of this application.

[0016] To improve the training effect of policy models, this application provides a policy model training method, which includes steps S101 to S104.

[0017] Step S101: Receive task input information corresponding to the robot operation task; wherein, the task input information includes initial state information and task information.

[0018] Step S102: Based on the task input information, policy model, and world model, perform search tree construction processing on the robot operation task to obtain target search tree data; wherein, the target search tree data includes multiple search nodes, connection relationships between different search nodes, and action data corresponding to the connection relationships, the action data is used to characterize the action transferred from the corresponding parent search node to the corresponding child search node, and is associated with the corresponding child search node.

[0019] Step S103: Based on the target search tree data, perform intra-group relative evaluation on the subsequent search results information of multiple child search nodes corresponding to the same parent search node to obtain the advantage level label of the action data associated with each child search node; wherein, the advantage level label is used to characterize the relative quality level between the action data associated with different child search nodes under the same parent search node.

[0020] Step S104: Based on the action data in the target search tree data and the dominance level labels corresponding to the action data, train the policy model to obtain the trained policy model; wherein, during the training of the policy model, the dominance level labels are used to determine the training condition information of the policy model.

[0021] The strategy model training method can refer to the method of updating the parameters of the strategy model used to generate robot motion data.

[0022] Robot operation tasks refer to target tasks that require robots to perform actions to complete. For example, robot operation tasks may include grasping a target object and placing it in a target position, inserting a part into an assembly hole, pushing an object over an obstacle area, opening a drawer, pressing a button, and tidying up items on a desktop.

[0023] Task input information refers to input data used to describe the initial conditions and objectives of a robot operation task. Initial state information refers to the state description information at the start of a robot operation task or at the start of the search tree construction process. Initial state information may include multi-view image information, depth image information, robot joint angle information, robot joint velocity information, end effector pose information, gripper opening and closing state information, target object position information, environmental obstacle information, etc. Task information refers to information used to indicate the robot's operation objectives or operation requirements. Task information may include a task description, such as "put the red square into the blue box"; it may also include task identifiers, target object identifiers, target area identifiers, target pose information, task completion condition information, etc.

[0024] A policy model can refer to a model used to generate action data based on task input information or the corresponding state of a search node. Policy models can include action generation models, conditional action generation models, neural network-based policy models, generative structure-based policy models, etc.

[0025] A world model can refer to a model used to predict the state changes and task performance quality of a robot after it executes action data. World models can include state prediction models, visual prediction models, robot state transition models, reward prediction models, task completion prediction models, etc.

[0026] Search tree construction refers to the process of building search tree data based on task input information, policy models, and world models. Target search tree data refers to the structured search data obtained after the search tree construction process is completed.

[0027] A search node can refer to a state node in the target search tree data. A search node can represent the state of the robot's operation task at a certain search position, such as the state where the robot has not yet contacted the target object, the state where the robot has approached the target object, the state where the robot has grasped the target object, the state where the robot has moved to the vicinity of the target area, etc.

[0028] A parent search node refers to a search node located at the starting point of a connection relationship in the target search tree data. A parent search node corresponds to a decision state, which can contain multiple candidate action data points. Different candidate action data points can form different child search nodes. Parent search nodes can be used to construct a group of comparison objects, i.e., multiple child search nodes under the same parent search node.

[0029] A child search node can be a search node in the target search tree data that is formed by the transfer of action data from the parent search node. The action data associated with a child search node can be the action data corresponding to the connection relationship transferred from the parent search node to the child search node. Evaluating the subsequent search results information of a child search node can reflect the quality of subsequent task execution after selecting the corresponding action data at the parent search node.

[0030] A connection relationship refers to a directed association between different search nodes in the target search tree data. A connection relationship can point from a parent search node to a child search node, representing a search state transition caused by an action.

[0031] Motion data refers to information used to characterize the actions performed by a robot. Motion data can correspond to the connections in a target search tree and be associated with the corresponding sub-search nodes pointed to by those connections. Motion data can include end-effector displacement increments, end-effector posture increments, gripper opening / closing control quantities, joint angle control quantities, joint velocity control quantities, motion sequences, motion segments, etc.

[0032] Subsequent search result information refers to the results obtained by continuing the search from a certain sub-search node. Subsequent search result information may include the number of subsequent leaf search nodes, the number of successful leaf search nodes, the number of failed leaf search nodes, task completion statistics, cumulative reward information, and success probability estimation information, etc.

[0033] Intragroup relative evaluation can refer to the evaluation process of comparing multiple child search nodes corresponding to the same parent search node.

[0034] Advantage level labels can be level labels obtained based on intra-group relative evaluation, used to characterize the relative quality level between action data associated with different child search nodes under the same parent search node. Advantage level labels can be discrete levels, such as level 0 to level 4, or other preset level ranges. A higher advantage level label indicates that the corresponding action data is relatively more conducive to the robot's task completion under the same parent search node.

[0035] Training condition information refers to the information used as conditional input during the training of a policy model. When determining training condition information based on dominance level labels, the policy model can learn to generate action data of different quality levels under different dominance level conditions. Training condition information can directly use dominance level labels, or it can use feature representations encoded from the dominance level labels, such as dominance level conditional feature information.

[0036] The trained policy model can refer to the policy model obtained after training with action data and advantage level labels in the target search tree data.

[0037] This embodiment constructs target search tree data based on task input information, a policy model, and a world model. This organizes multiple candidate actions and their subsequent search results in a robot operation task into structured data with parent-child relationships, providing a data foundation for comparing the relative quality of action data. By performing intra-group relative evaluation of the subsequent search results of multiple child search nodes corresponding to the same parent search node, the subsequent performance of different action data can be compared under the same decision state, resulting in more granular dominance level labels than a single success or failure result. By using the dominance level labels as training condition information for the policy model, the policy model can learn the correspondence between action data and relative quality levels, thereby improving the efficiency of training data utilization and enhancing the ability of the trained policy model to generate high-quality action data. This embodiment addresses the problem that action exploration data in robot operation tasks cannot fully reflect the differences in the quality of different actions under the same decision state. It adopts the concept of intra-group relative evaluation based on target search tree data and using dominance level labels as training condition information, thereby transforming the subsequent search results into fine-grained training signals that can be used for conditional training of the policy model, improving the training effect of the policy model.

[0038] In some embodiments, the step of performing search tree construction processing on the robot operation task to obtain target search tree data may include: performing a first search process, wherein the first search process is used to determine a first search node based on the formed search nodes, and to perform node expansion processing on the first search node until a first search termination condition is met, thereby obtaining first-stage search tree data; and, if the first search termination condition is met, performing a second search process, wherein the second search process is used to determine at least one second search node based on the internal search nodes in the first-stage search tree data that do not meet the child node quantity condition, and to perform node expansion processing on each second search node so that the number of child search nodes corresponding to each second search node meets the child node quantity condition, thereby obtaining second-stage search tree data, which serves as the target search tree data.

[0039] The first search process can refer to the first type of search process in the search tree construction process. A formed search node can refer to a search node that already exists in the search tree before the current moment of the first search process. A first search node can refer to a search node that is determined to require node expansion processing during the first search process. The first search termination condition can refer to the conditions used to determine whether the first search process has ended. The first search termination condition may include reaching a set number of searches, reaching a set number of expanded nodes, reaching a set number of successful leaf search nodes, reaching a set budget, reaching a set budget, or reaching a set number of successful task paths. The first-stage search tree data can refer to the staged search tree data obtained after the first search process has ended.

[0040] The second search process can refer to a supplementary search process executed after the first search process has concluded. An internal search node can be a search node in the target search tree data or the first-stage search tree data that has subsequent branches. An internal search node can act as a parent search node and can have one or more child search nodes. A second search node can be a search node identified in the second search process as requiring node expansion processing. There can be one or more second search nodes.

[0041] The child node quantity condition can refer to the condition set for the number of child search nodes corresponding to an internal search node. The child node quantity condition can be a fixed quantity condition, such as each internal search node having 4 child search nodes; it can also be a range condition, such as each internal search node having no less than 3 child search nodes; or it can be a dynamic quantity condition related to search depth, task type, and search budget.

[0042] The second-stage search tree data refers to the search tree data obtained after the second search process. This second-stage search tree data serves as the target search tree data and can be used for subsequent intra-group relative evaluation and policy model training.

[0043] Node expansion processing refers to the process of creating one or more child search nodes for a given search node. Node expansion processing can be used for adaptive expansion in the first search process or for supplementary expansion in the second search process.

[0044] This embodiment enables the search tree construction process to prioritize expanding more valuable search branches under limited search resources, thereby improving the efficiency of forming effective search paths in the target search tree data. Performing node expansion processing on internal search nodes that do not meet the required number of child nodes increases the consistency of the number of child search nodes under each internal search node, thus reducing subsequent evaluation bias caused by differences in the number of expansions between different parent search nodes. Addressing the problem that adaptive search during search tree construction can easily lead to an imbalance in the number of branches among different internal search nodes, thus affecting the reliability of subsequent intra-group evaluations, this embodiment adopts a two-stage search processing approach: adaptive expansion followed by supplementary expansion. This balances search efficiency and the fairness of subsequent relative evaluations, improving the effectiveness of the target search tree data for policy model training.

[0045] In some embodiments, determining the first search node based on the established search nodes may include: determining node selection score information corresponding to the established search node based on access statistics, node value information, and prior probability information corresponding to the established search node; determining the first search node from the established search nodes based on the node selection score information; wherein, the first search process may further include: after performing node expansion processing on the first search node, updating the access statistics and node value information of at least some search nodes on the node path where the first search node is located based on the task execution result corresponding to the corresponding inference result obtained from the node expansion processing.

[0046] Access statistics refer to statistical data used to characterize whether a search node is accessed or selected during the search process. Access statistics can include the number of times a search node is accessed, the number of times each child search node under a parent search node is accessed, the cumulative number of times the path of the search node is located, and the access frequency of the search node within a preset search round.

[0047] Node value information refers to information used to characterize the task value or success potential of a search branch corresponding to a search node. Node value information may include the cumulative value, average value, successful feedback value, cumulative reward value, task completion estimate, and success probability estimate obtained based on the inference results of the search node.

[0048] Prior probability information refers to information used to characterize the prior tendency for search nodes to be preferentially selected or corresponding action data to be preferentially attempted before the calculation of node selection scoring information. Prior probability information can be determined by candidate action probabilities, action generation confidence, task relevance scores, action quality guidance information, or preset exploration allocation rules output by the policy model.

[0049] Node selection scoring information can refer to a node selection score determined based on access statistics, node value information, and prior probability information, used to determine the first search node among the existing search nodes. Node selection scoring information can be a combination of node value items, exploration items, and prior items, or it can be a comprehensive score obtained by weighting based on access frequency, value estimation, and prior probability.

[0050] The simulation results refer to the information obtained during the node expansion process after predicting or evaluating the robot's task execution based on candidate action data. Simulation results may include predicted state information, task evaluation information, reward information, task success results, task failure results, and task completion results.

[0051] Task execution results can refer to task-level results further determined by the deduction results. Task execution results can include successful results, failed results, incomplete results, partially completed results, reward values, completion percentages, etc. Task execution results can serve as the basis for reverse updating of access statistics and node value information, enabling the first search process to feed back the task performance of the newly expanded branch to the relevant search nodes on the node path.

[0052] A node path can refer to the path formed by the search nodes and connections from the root search node to the first search node. A node path can correspond to a set of path action data, or it can correspond to a search history that gradually leads from the initial state of the robot's operation task to the state of the first search node.

[0053] Updates to access statistics may include incrementing the access count of relevant search nodes, increasing the path access count, or updating the access frequency; updates to node value information may include increasing the cumulative value, recalculating the average value, updating the success probability estimate, or updating the cumulative reward value.

[0054] This embodiment unifies the historical access data, acquired task value, and prior selection bias of search nodes for determining the first search node, thereby improving the rationality of the first search process in selecting search nodes to be expanded. After node expansion processing, by updating the access statistics and node value information of at least some search nodes on the node path based on the task execution results corresponding to the deduction results, the task performance of the newly expanded branch can be fed back to the relevant search nodes in the search tree, thus dynamically adjusting the selection of subsequent search nodes according to the search results. This embodiment addresses the problem of difficulty in rationally allocating search resources between high-value branches and branches to be explored during robot operation task search. It adopts the concept of determining node selection scoring information based on access statistics, node value information, and prior probability information, and updating node path information in reverse based on the deduction results, thereby improving the adaptive search capability of the first search process and the effectiveness of the target search tree data.

[0055] In some embodiments, the node expansion process may include: using the first search node or any of the second search nodes as the target search node, and placing the world model in an initial model state corresponding to the initial state information; determining path action data from the root search node to the target search node based on the node path corresponding to the target search node, and performing replay processing corresponding to the path action data on the world model to obtain the node model state corresponding to the target search node; determining node state information corresponding to the target search node based on the node model state corresponding to the target search node, and generating at least one candidate action data corresponding to the target search node based on the node state information, the task information, and the policy model; performing deduction on each of the candidate action data based on the world model in the node model state to obtain deduction results corresponding to each of the candidate action data, and forming a sub-search node corresponding to each of the candidate action data under the target search node based on the deduction results corresponding to each of the candidate action data.

[0056] The initial model state can refer to the model state corresponding to the world model and the initial state information. The initial model state can be obtained by initializing the world model based on the initial state information, or it can be obtained by loading the model snapshot corresponding to the initial state information, copying the model instance, setting the model's hidden state, etc.

[0057] The root search node can be the starting search node in the search tree that corresponds to the initial state of the robot's operation task. The root search node can correspond to the state of the robot's operation task before any actions are performed. The path from the root search node to the target search node can represent the sequence of actions taken from the initial state to the state of the target search node.

[0058] Path action data refers to the combination of action data corresponding to each connection relationship along the node path from the root search node to the target search node. Path action data can include one or more action data, such as multiple action blocks, action sequences, or control segments. Performing replay processing on the world model corresponding to the path action data can advance the world model from the initial model state to the model state corresponding to the target search node.

[0059] Replay processing refers to inputting path action data sequentially according to the node path order, starting from the initial model state of the world model, so that the world model can reproduce the state evolution process from the root search node to the target search node. The purpose of replay processing is to utilize the replayability of the world model to allow the world model to enter the node model state corresponding to the target search node, thereby supporting the continued attempt of new candidate action data at the target search node.

[0060] Node model state can refer to the internal state or predicted state of the world model corresponding to the target search node after path action data replay processing. Node state information can refer to the state information determined based on the node model state corresponding to the target search node, which can be used by the policy model to generate actions.

[0061] Candidate action data can refer to action data generated by the policy model at the target search node and awaiting deduction by the world model.

[0062] Inference can refer to the process by which a world model predicts or simulates the subsequent execution of a robot's task based on the state of the node model and candidate action data.

[0063] This embodiment allows the adaptive expansion in the first search process and the supplementary expansion in the second search process to reuse a unified node expansion process, thereby reducing the process differences between different search processes. Performing replay processing on the world model corresponding to the path action data from the root search node to the target search node allows the world model to enter the node model state corresponding to the target search node without requiring the real robot to return to a historical state, thus supporting continued branch expansion at any target search node. It can match candidate action data with the state corresponding to the target search node and the robot's operation task requirements. Multiple subsequent action branches can be constructed under the target search node, thereby enriching the target search tree data. This embodiment addresses the problem that nodes within the search tree in robot operation tasks are difficult to return to their corresponding historical states and continue branching attempts. It adopts the concept of restoring the node model state based on the initial model state and path action data replay, regenerating and deducing candidate action data, thereby improving the feasibility of expanding nodes within the search tree and enhancing the ability to construct target search tree data.

[0064] In some embodiments, generating at least one candidate action data corresponding to the target search node based on the node state information, the task information, and the strategy model may include: determining multiple action generation condition information, wherein different action generation condition information corresponds to different action quality guidance information and / or different action perturbation information; and generating multiple candidate action data respectively through the strategy model based on the node state information, the task information, and each of the action generation condition information.

[0065] Action generation condition information refers to the conditional information used by the control policy model when generating candidate action data. This information can influence the quality tendency, randomness, action diversity, or exploration direction of the action data generated by the policy model.

[0066] Action quality guidance information refers to information used to guide the strategy model in generating actions that meet certain action quality preferences. Action quality guidance information may include information on setting advantage levels, setting action quality levels, task success tendency, high completion rate tendency, and high reward tendency.

[0067] Action perturbation information refers to information used to guide the policy model in generating candidate action data with differences. Action perturbation information may include random seeds, noise vectors, sampling temperature, perturbation amplitude, action sampling offset, etc.

[0068] Multiple action generation condition information can differ in action quality guidance information, action disturbance information, or both action quality guidance information and action disturbance information.

[0069] This embodiment enables candidate action data generated by the strategy model to simultaneously match the state conditions at the target search node and the robot's operational task requirements, thereby improving the relevance of candidate action data to the current search context. It can generate action generation conditions with different quality preferences or exploration directions at the same target search node, thus increasing the discriminability and diversity of candidate action data. It provides a data foundation for forming multiple comparable action branches under the target search node, thereby enhancing the search tree construction process's ability to explore different candidate actions. Addressing the problem that a single action generation method at the target search node easily leads to insufficient candidate action branches and unclear action quality gradients, this embodiment adopts the concept of using multiple action generation condition information to drive the strategy model to generate multiple candidate action data, thus balancing action quality guidance and action diversity exploration, and improving the effectiveness of search tree expansion.

[0070] In some embodiments, determining multiple action generation condition information may include: determining first action generation condition information and multiple second action generation condition information; wherein, the first action generation condition information may include setting a maximum advantage level information to guide the strategy model to generate candidate action data that conforms to a set action quality preference; the multiple second action generation condition information may include other set advantage level information and / or different action perturbation information, wherein the other set advantage level information may be used to characterize an advantage level lower than the level corresponding to the set maximum advantage level information, and the multiple second action generation condition information may be used to guide the strategy model to generate candidate action data that differs from the candidate action data generated based on the first action generation condition information.

[0071] The first action generation condition information can refer to the condition information used to generate higher-quality candidate action data among multiple action generation condition information.

[0072] The second action generation condition information can refer to action generation condition information that is different from the first action generation condition information. There can be multiple second action generation condition information, and each second action generation condition information can include other set advantage level information and / or different action perturbation information.

[0073] Setting the highest dominance level information can refer to the highest level condition information pre-set during the action generation phase. This information can be a highest level value, a highest level identifier, highest level embedded information, or a condition vector representing the highest quality preference. For example, when the dominance level ranges from 0 to 4, setting the highest dominance level information could correspond to level 4.

[0074] Setting action quality preferences refers to the quality orientation requirements that the action generation condition information places on the action data generated by the policy model. Setting action quality preferences can indicate that the policy model tends to generate candidate action data that is more conducive to task completion, better aligns with task progress, is closer to the success path, offers higher rewards, or achieves a higher degree of completion. Setting action quality preferences does not require candidate action data to necessarily succeed; rather, it is used to provide guidance in the quality direction during the action generation process.

[0075] Other set advantage level information can refer to advantage level information that is lower than the level corresponding to the highest set advantage level information. Other set advantage level information can include medium-high level information, medium level information, low level information, or multiple decreasing level information. For example, when the advantage level value is 0 to 4, other set advantage level information can correspond to level 3, level 2, or level 1.

[0076] This embodiment guides the strategy model to generate candidate action data that meets set action quality preferences, thereby providing high-quality candidate branches for search tree expansion. It enables the strategy model to generate candidate action data that differs from the candidate action data corresponding to the first action generation condition information, thus increasing the diversity of candidate action branches under the same target search node. It can also create a quality gradient among candidate action data, facilitating subsequent comparisons of different candidate action data. Addressing the problem that a single action generation condition can easily lead to a single quality preference in candidate action data and insufficient difference in action branches, this embodiment employs the concept of anchoring high-quality candidate actions with the highest dominance level information and combining it with other set dominance level information and action perturbation information to generate differentiated candidate actions. This balances action quality guidance and action exploration diversity, improving the effectiveness of candidate action data for search expansion.

[0077] In some embodiments, the step of extrapolating each candidate action data based on the world model under the node model state to obtain the extrapolation result corresponding to each candidate action data may include: inputting the node model state and each candidate action data into the world model separately or in batches, and predicting the task execution process corresponding to each candidate action data based on the world model to obtain the predicted state information and task evaluation information corresponding to each candidate action data; determining the extrapolation result corresponding to each candidate action data based on the predicted state information and task evaluation information corresponding to each candidate action data; wherein, the extrapolation result can be used to characterize the state transition result and task execution quality corresponding to the corresponding candidate action data.

[0078] The task execution process can refer to the state evolution process formed when a robot continues to perform its operation task based on candidate action data. Predicted state information can refer to the state information after the candidate action data is executed, as predicted by the world model. Task evaluation information can refer to the evaluation information output by the world model for the task execution process corresponding to the candidate action data. Task evaluation information may include reward information, task completion information, success judgment information, failure judgment information, collision risk information, action stability information, and target object contact quality information, etc.

[0079] A state transition result refers to the outcome where candidate action data causes the robot's task to transition from the state corresponding to the target search node to a subsequent state. State transition results can manifest as changes in the robot's end effector pose, the target object's position, the gripper's opening and closing state, the robot's joint states, and changes in environmental observations.

[0080] Task execution quality refers to the performance of candidate action data in terms of task objective achievement, action stability, task progress, and safety during task execution. Task execution quality can be represented by task evaluation information or determined jointly by predicted state information and task evaluation information. Task execution quality can include high task completion rate, low collision risk, high grasping stability, high reward value, and high success probability.

[0081] This embodiment can predict multiple candidate action data under the corresponding state of the target search node, thereby obtaining the subsequent state and task performance corresponding to different candidate action data. It reduces the reliance on trial-and-error testing of actions by the real robot, thus lowering the actual execution cost and trial-and-error risk. The deduction results can simultaneously reflect state transition results and task execution quality, providing a basis for subsequent formation of sub-search nodes and evaluation of candidate action data. Addressing the problem of inefficiently comparing multiple candidate action data under the same target search node state in robot operation tasks, this embodiment adopts a concept of deducing candidate action data separately or in batches based on a world model, thereby improving the efficiency of candidate action data evaluation and enhancing the ability of search tree construction to express action consequences and task quality.

[0082] In some embodiments, the step of performing intra-group relative evaluation of the subsequent search results information of multiple child search nodes corresponding to the same parent search node based on the target search tree data to obtain the advantage level label of the action data associated with each child search node may include: for each child search node corresponding to the same parent search node, determining the subsequent leaf search node information corresponding to the child search node based on the target search tree data, and obtaining the subsequent search results information corresponding to the child search node based on the subsequent leaf search node information; determining the search evaluation information corresponding to the child search node based on the subsequent search results information; and performing relative evaluation of the multiple child search nodes based on the search evaluation information of each of the multiple child search nodes corresponding to the same parent search node to obtain the advantage level label of the action data associated with each child search node.

[0083] Subsequent leaf search node information refers to information about leaf search nodes formed after continuing the search from a certain sub-search node. This information may include the number of subsequent leaf search nodes, the number of successful leaf search nodes, the number of failed leaf search nodes, the number of completed leaf search nodes, the number of incomplete leaf search nodes, and the task results corresponding to each leaf search node.

[0084] Subsequent search result information refers to information obtained based on subsequent leaf search node information, used to describe the subsequent search performance of the sub-search node. Subsequent search result information may include the number of successful subsequent searches, the number of failed subsequent searches, the completion rate of subsequent tasks, subsequent reward statistics, and subsequent task completion statistics, etc.

[0085] Search evaluation information refers to the evaluation value or result determined based on subsequent search results, used to characterize the search quality of subsequent branches corresponding to a certain sub-search node. Search evaluation information may include success rate, average reward value, task completion evaluation value, percentage of successful leaf search nodes, task quality score, etc.

[0086] Relative evaluation can refer to the process of comparing, sorting, or classifying multiple search evaluation information after obtaining the search evaluation information of multiple sub-search nodes.

[0087] This embodiment utilizes the subsequent search results already formed in the target search tree data to characterize the subsequent task performance of each sub-search node. It transforms the scattered results of subsequent leaf search nodes into easily comparable evaluation criteria. It allows comparison of the subsequent effects of different action data under the same decision-making starting point, thereby obtaining an advantage level label that characterizes the relative quality level of the action data. Addressing the problem that the success or failure of a single action data point in a robot operation task cannot reflect the relative superiority or inferiority of different action data under the same decision state, this embodiment adopts a concept of intra-group relative evaluation based on subsequent search result information under the same parent search node. This transforms the subsequent search performance in the target search tree data into fine-grained advantage level labels, improving the accuracy and usability of action data quality labeling.

[0088] In some embodiments, the step of relatively evaluating the multiple sub-search nodes based on their respective search evaluation information corresponding to the same parent search node to obtain the advantage level label of the action data associated with each sub-search node may include: determining intra-group evaluation range information based on the search evaluation information of the multiple sub-search nodes corresponding to the same parent search node; wherein the intra-group evaluation range information is used to characterize the value range between the search evaluation information of the multiple sub-search nodes corresponding to the same parent search node; normalizing the search evaluation information corresponding to each sub-search node based on the intra-group evaluation range information to obtain normalized evaluation information corresponding to each sub-search node; and determining the advantage level label of the action data associated with each sub-search node based on the normalized evaluation information corresponding to each sub-search node.

[0089] Intra-group evaluation range information refers to information that characterizes the range of values ​​for the evaluation information of multiple child search nodes corresponding to the same parent search node. Intra-group evaluation range information may include the maximum and minimum values ​​of the search evaluation information, the difference between the maximum and minimum values, and the distribution interval of the evaluation values.

[0090] Normalization refers to the process of converting the search evaluation information corresponding to each sub-search node to a uniform numerical range based on the evaluation range information within the group. Normalization can be performed using max-min normalization, or it can be performed using normalization methods based on intra-group ranking, intra-group proportion, or a preset mapping function.

[0091] Normalized evaluation information refers to the evaluation information obtained after normalizing the search evaluation information. Normalized evaluation information can be a continuous value between 0 and 1, or a value within other preset ranges.

[0092] This embodiment can obtain the distribution range of subsequent search performance of multiple action data under the same decision state, thus providing a scale basis for intra-group comparison. Search evaluation information at different numerical scales can be converted into unified normalized evaluation information, thereby improving the stability of relative quality comparisons between different action data. The search evaluation information can be further converted into discrete quality labels suitable for policy model training, thereby improving the training usability of search tree data. This embodiment addresses the problem that the absolute numerical range of search evaluation information differs under different parent search nodes, and direct comparison easily affects the stability of action quality labeling. It adopts the concept of determining the intra-group evaluation range information within the same parent search node and performing normalization processing, thereby obtaining an advantage level label that can characterize the relative quality within the group, improving the precision and consistency of action data quality labeling.

[0093] In some embodiments, training the policy model based on action data in the target search tree data and the dominance level label corresponding to the action data to obtain the trained policy model may include: constructing multiple training samples based on the target search tree data, wherein each training sample includes node input information corresponding to a parent search node, action data associated with a child search node of the parent search node, and dominance level label corresponding to the action data; performing conditional encoding processing on the dominance level label to obtain dominance level conditional feature information; and training the policy model based on the node input information, the action data, and the dominance level conditional feature information to obtain the trained policy model.

[0094] Training samples can refer to data units used to train policy models.

[0095] Node input information can refer to the information corresponding to the parent search node used to input the strategy model.

[0096] Conditional encoding refers to the process of converting advantage level labels into conditional features that can be used by the policy model. Conditional encoding can employ methods such as numerical normalization, one-hot encoding, level embedding encoding, lookup table mapping, and neural network mapping.

[0097] Dominance level conditional feature information refers to the feature information obtained after conditionally encoding the dominance level label. Dominance level conditional feature information can be numerical features, vector features, embedded features, or conditional features that can be fused with node input information.

[0098] Training a policy model based on node input information, action data, and dominance level conditional features allows the model to learn what action data should be generated given the node input information corresponding to the parent search node and a specified dominance level. This training process can be implemented using supervised action fitting, generative action learning, or conditional action generation training.

[0099] This embodiment improves the training utilization of target search tree data by constructing multiple training samples based on the target search tree data. By conditionally encoding the dominance level labels to obtain dominance level conditional feature information, discrete relative quality levels can be converted into conditional features that the policy model can recognize and utilize. Through training, the policy model can learn the action generation rules under different node input conditions and different dominance level conditions, thereby improving the ability of the trained policy model to generate action data that conforms to specified quality preferences.

[0100] In one example, the robot's task is "to put the red cube into the blue storage box." The received task input information includes initial state information and task information. The initial state information includes 224×224 images acquired by 3 cameras, 7 robotic arm joint angles, gripper opening and closing width of 60 mm, and end effector pose; the task information is the text "put the red cube into the blue storage box."

[0101] The search tree construction process is performed based on the task input information, policy model, and world model. The initial search tree includes a root search node N0. The first search termination condition for the first search process is set as follows: the number of node expansion processes reaches 200, or the number of successfully searched leaf nodes reaches 20. During the first search process, node selection score information is determined based on the access statistics, node value information, and prior probability information of the formed search nodes. For example, search node N1 has 8 accesses, a node value of 0.52, a prior probability of 0.30, and a node selection score of 0.74; search node N2 has 3 accesses, a node value of 0.60, a prior probability of 0.42, and a node selection score of 0.83; search node N3 has 10 accesses, a node value of 0.68, a prior probability of 0.20, and a node selection score of 0.76. The first search process determines the search node N2 with the highest node selection score as the first search node and performs node expansion processing on the first search node N2.

[0102] When performing node expansion processing on the first search node N2, N2 is used as the target search node, and the world model is placed in the initial model state S0 corresponding to the initial state information. The node path corresponding to the target search node N2 is root search node N0 → search node N1 → search node N2. Based on this node path, path action data A01 and A12 are determined, and the replay processing corresponding to path action data A01 and A12 is performed on the world model in sequence to obtain the node model state S2 corresponding to the target search node N2. Based on the node model state S2, the node state information is determined, indicating that the end effector is located about 4 cm above the red square, the gripper opening width is 60 mm, and the red square is about 25 cm away from the center of the blue storage box.

[0103] Four action generation conditions are determined for the strategy model. These conditions include one first action generation condition and three second action generation conditions. The first action generation condition includes setting the highest dominance level, corresponding to level 4. The three second action generation conditions include other set dominance levels corresponding to levels 3, 2, and 1, respectively, and are configured with random perturbation seeds of 21, 35, and 48. The strategy model generates four candidate action data based on node state information, task information, and the four action generation conditions. Candidate action data B1 indicates that the end effector moves 1 cm towards the center of the target object, descends 3 cm, and closes the gripper; candidate action data B2 indicates that the end effector shifts 1 cm to the right, descends 3 cm, and closes the gripper; candidate action data B3 indicates that the end effector descends directly 2 cm and closes the gripper with a smaller amplitude; and candidate action data B4 indicates that the end effector descends 5 cm and closes the gripper with a delay.

[0104] The node model state S2 and four candidate action data are batch input into the world model, which predicts the task execution process corresponding to each candidate action data. The predicted state information for candidate action data B1 indicates that the gripper stably grasps the red square, with a task evaluation of 0.86; the predicted state information for candidate action data B2 indicates that the gripper contacts the edge of the red square, with a task evaluation of 0.60; the predicted state information for candidate action data B3 indicates that the red square does not completely enter the center of the gripper, with a task evaluation of 0.42; and the predicted state information for candidate action data B4 indicates that the end effector is close to the table and there is a risk of collision, with a task evaluation of 0.18. Based on the above predicted state information and task evaluation information, the deduction results corresponding to each candidate action data are determined, and four sub-search nodes N21, N22, N23, and N24 are formed under the target search node N2. These four sub-search nodes are associated with candidate action data B1, B2, B3, and B4, respectively. After the node expansion process is completed, based on the task execution results corresponding to the inference results, the access statistics and node value information of the root search node N0, search node N1 and search node N2 on the node path where the target search node N2 is located are updated.

[0105] The first search process continues until the number of leaf search nodes reaches 20, resulting in the first-stage search tree data. This first-stage search tree data includes 260 search nodes. Subsequently, the second search process is executed, identifying internal search nodes with fewer than four child search nodes based on the first-stage search tree data. For example, if 30 internal search nodes in the first-stage search tree data do not meet the child node requirement, these 30 internal search nodes are identified as second-stage search nodes, and each second-stage search node undergoes node expansion processing to ensure that each second-stage search node has at least four child search nodes. After the second search process is completed, the second-stage search tree data is obtained, containing 340 search nodes, and this second-stage search tree data serves as the target search tree data.

[0106] For the parent search node P in the target search tree data, parent search node P corresponds to 4 child search nodes C1, C2, C3, and C4. Based on the target search tree data, the subsequent leaf search node information corresponding to each child search node is determined as follows: child search node C1 corresponds to 10 subsequent leaf search nodes, of which 6 subsequent leaf search nodes correspond to successful tasks; child search node C2 corresponds to 10 subsequent leaf search nodes, of which 8 subsequent leaf search nodes correspond to successful tasks; child search node C3 corresponds to 10 subsequent leaf search nodes, of which 5 subsequent leaf search nodes correspond to successful tasks; child search node C4 corresponds to 10 subsequent leaf search nodes, of which 1 subsequent leaf search node corresponds to successful tasks. The ratio of the number of successful leaf search nodes to the total number of subsequent leaf search nodes is used as search evaluation information, and the search evaluation information for C1, C2, C3, and C4 is 0.60, 0.80, 0.50, and 0.10, respectively.

[0107] Based on the four search evaluation information corresponding to the same parent search node P, the evaluation range information within the group is determined. The evaluation range information within the group includes a maximum value of 0.80, a minimum value of 0.10, and a value span of 0.70. The search evaluation information is normalized, resulting in normalized evaluation information of 0.71, 1.00, 0.57, and 0.00 for C1, C2, C3, and C4, respectively. Given that the advantage level label ranges from 0 to 4, the action data associated with C2 corresponds to advantage level label 4, the action data associated with C1 corresponds to advantage level label 3, the action data associated with C3 corresponds to advantage level label 2, and the action data associated with C4 corresponds to advantage level label 0.

[0108] Multiple training samples are constructed based on the target search tree data. Taking the parent search node P as an example, four training samples can be constructed: the first training sample includes the node input information corresponding to the parent search node P, the action data associated with the child search node C1, and the dominance level label 3; the second training sample includes the node input information corresponding to the parent search node P, the action data associated with the child search node C2, and the dominance level label 4; the third training sample includes the node input information corresponding to the parent search node P, the action data associated with the child search node C3, and the dominance level label 2; the fourth training sample includes the node input information corresponding to the parent search node P, the action data associated with the child search node C4, and the dominance level label 0. Conditional encoding is performed on the dominance level labels, for example, encoding levels 0 to 4 into 5-dimensional vectors respectively, and encoding the dominance level label 4 as [0, 0, 0, 0, 1]. The policy model is trained based on the node input information, action data, and dominance level conditional feature information to obtain the trained policy model.

[0109] As seen in the examples above, search tree construction and policy model training can form an iterative optimization relationship. During the search process, search tree construction can be performed within the world model. When it's necessary to return to a historical search node for further expansion, the world model can be placed in the initial model state corresponding to the initial state information, and the path action data from the root search node to that historical search node can be replayed, thus obtaining the node model state corresponding to that historical search node. Based on this node model state, new candidate action data can be generated and deduced to expand the search tree and obtain target search tree data containing successful paths, failed paths, and subsequent search results. During training, intra-group relative evaluation can be performed based on the subsequent search result information of multiple child search nodes under the same parent search node in the target search tree data, and the obtained advantage level labels can be used as training condition information to train the policy model. The trained policy model can continue to be used in the next search tree construction process, thereby generating more effective training data and enabling the policy model to gradually improve its action generation capability during iteration.

[0110] Building upon the world model's state backtracking capability, this embodiment can further employ a two-stage search strategy, which may include a first search process and a second search process. The first search process can utilize an adaptive search method based on PUCT (Predictor + Upper Confidence Bounds applied to Trees), while the second search process can employ an equal-width backfilling method to supplement the number of sub-search nodes corresponding to internal search nodes.

[0111] The first search process can start from the root search node and repeatedly perform node selection, node expansion, and backpropagation. For example, for the i-th child search node under the parent search node, the node selection score information can be determined using the following PUCT scoring formula.

[0112] Where Score(i) represents the node selection score information corresponding to the i-th sub-search node; Q(i) represents the average value estimate corresponding to the i-th sub-search node; P(i) represents the prior probability information corresponding to the i-th sub-search node; and N(i) represents the number of visits corresponding to the i-th sub-search node. This indicates the number of visits to the corresponding parent search node, or the sum of the number of visits to all child search nodes under the corresponding parent search node; Let Q(i) represent the PUCT exploration coefficient. In the above formula, Q(i) can be used to characterize the obtained task value. It can be used to characterize the exploratory tendency obtained by combining prior probability information and access statistics.

[0113] In some implementations, the average value estimate Q(i) can be determined based on the cumulative value information W(i) and the number of visits N(i), as shown below.

[0114] in, This represents the cumulative value information corresponding to the i-th sub-search node. If the value is 0, a preset smoothing value can be added to the denominator, or unvisited sub-search nodes can be included in the node selection according to a preset initial score, in order to reduce the impact of division by zero.

[0115] It should be noted that the advantage level information used in the node expansion phase is a pre-defined action generation condition, used to introduce action quality gradients and action diversity during the search process. The subsequent advantage level labels obtained based on the target search tree data are training labels statistically derived from the subsequent search results of multiple child search nodes under the same parent search node after the search is completed. These two have different meanings, but a correspondence can be gradually established through the iterative process of search and training.

[0116] After the world model extrapolates the candidate action data, it can obtain the extrapolation results corresponding to each candidate action data. The first search process can update the access statistics and node value information of at least some search nodes on the node path where the first search node is located based on the task execution results corresponding to the extrapolation results. For example, for the i-th search node on the node path, the access count, cumulative value information, and average value estimate can be updated in the following way.

[0117] Here, v represents the value information corresponding to the task execution result obtained from this node expansion process. As an example, v can be 1 if the task execution result is successful; v can be 0 if the task execution result is unsuccessful; and v can be a continuous value between 0 and 1 if the task execution result is a continuous task evaluation value.

[0118] After the target search tree data is constructed, for each internal search node, the K child search nodes corresponding to that internal search node can be considered as a sibling group, where K represents the preset number of child nodes. For the i-th child search node in the sibling group, let S(i) be the number of successful leaf search nodes in the subtree corresponding to that child search node, and let L(i) be the total number of leaf search nodes. Then the success rate of the subtree corresponding to the i-th child search node can be expressed as follows.

[0119] Where r(i) represents the subtree success rate corresponding to the i-th child search node. The subtree success rate can characterize the proportion of subsequent searches reaching the successful leaf search node after selecting the i-th child search node to associate action data under the decision state corresponding to the current parent search node. It can also be used as a Monte Carlo (random sampling statistics) estimate of the long-term task value of the corresponding action data.

[0120] Within the same sibling group, the success rate of the subtree corresponding to each child search node can be processed using a maximum-minimum normalization method. Let the success rate of the minimum subtree within the same sibling group be... The success rate of the largest subtree is Then the normalized evaluation information corresponding to the i-th sub-search node can be represented as follows.

[0121] Where norm(i) represents the normalized evaluation information corresponding to the i-th sub-search node. If the success rates of the subtrees corresponding to the sub-search nodes within the same sibling group are exactly the same, that is: Then, the advantage level label of the action data associated with each sub-search node in the same sibling group can be uniformly set to the highest level to reduce the calculation anomaly caused by the denominator being zero.

[0122] After obtaining the normalized evaluation information, the continuous normalized evaluation information can be discretized into... There are several levels of advantage, among which... This indicates the number of preset advantage levels. The advantage level label corresponding to the i-th sub-search node can be represented as follows.

[0123] Where bin(i) represents the advantage level label corresponding to the action data associated with the i-th sub-search node; This represents a clipping function used to restrict input values ​​to between 1 and... Within the range of values; This indicates rounding down. This discretization process converts continuous intra-group relative evaluation results into discrete dominance level labels.

[0124] After obtaining the dominance level labels, these labels can be used as training conditional information input into the policy model. Discrete dominance level labels can be mapped to vector representations using learnable embedding layers. Let the dominance level label be b, then the dominance level conditional feature information can be represented as follows.

[0125] in, This indicates the characteristic information of the dominance level conditions; 'b' represents the embedding layer; 'b' represents the advantage level label, and: .

[0126] The policy model can be either a conditional diffusion model or a conditional flow matching model. The training objective can be expressed as maximizing the conditional log-likelihood, as shown below.

[0127] in, Represent the training objective function; This represents the expectation operation; The parameter is The strategy model; a represents action data; o represents the current observation; l represents the task language description; b represents the advantage level label.

[0128] During the inference and deployment phase, the dominance level condition can be fixed at the highest level. Given the current observations and the task description, the policy model can sample target action data based on the highest dominance level condition.

[0129] in, This represents the target action data sampled during the inference phase; Indicates the source of the sample; This represents the action data to be sampled. By fixing the highest advantage level condition during the inference phase, the action data generated by the policy model can be more biased towards action patterns corresponding to the success rates of high subtrees in the target search tree data.

[0130] In some implementations, the policy model may include a state encoding unit, a task encoding unit, a condition encoding unit, and an action generation unit. The state encoding unit encodes state data from node state information or task input information to obtain state features; the task encoding unit encodes task information to obtain task features; the condition encoding unit encodes action generation condition information or advantage level condition feature information to obtain condition features; and the action generation unit generates action data based on the state features, task features, and condition features. Therefore, the input to the policy model may include node state information, task information, and training condition information, and the output of the policy model may include candidate action data or action data corresponding to the training phase.

[0131] In some implementations, the world model may include a state input unit, an action input unit, a state transition prediction unit, and a task evaluation unit. The state input unit receives initial state information, node model states, or node state information; the action input unit receives path action data or candidate action data; the state transition prediction unit predicts the subsequent states of the robot's operation task based on the input states and action data, obtaining predicted state information; and the task evaluation unit determines task evaluation information based on the predicted state information. Thus, the input to the world model may include node model states and candidate action data, and the output of the world model may include predicted state information and task evaluation information.

[0132] In some implementations, during policy model training, the node input information corresponding to the parent search node can be input into the state encoding unit and the task encoding unit, the advantage level condition feature information corresponding to the advantage level label can be input into the condition encoding unit, and the action data associated with a child search node of the parent search node can be used as the training target to update the parameters of the action generation unit, so that the policy model learns to generate corresponding action data under different node input information and different advantage level conditions.

[0133] See Figure 2 , Figure 2 This is a structural block diagram of a strategy model training device provided in an embodiment of this application.

[0134] This application also provides a strategy model training device, which includes an information receiving module, a search tree construction module, a relative evaluation module, and a model training module.

[0135] The information receiving module is used to receive task input information corresponding to robot operation tasks; wherein, the task input information includes initial state information and task information.

[0136] The search tree construction module is used to perform search tree construction processing on the robot operation task based on the task input information, policy model, and world model to obtain target search tree data. The target search tree data includes multiple search nodes, connection relationships between different search nodes, and action data corresponding to the connection relationships. The action data is used to represent the action transferred from the corresponding parent search node to the corresponding child search node and is associated with the corresponding child search node.

[0137] The relative evaluation module is used to perform intra-group relative evaluation of the subsequent search results information of multiple child search nodes corresponding to the same parent search node based on the target search tree data, and obtain the advantage level label of the action data associated with each child search node; wherein, the advantage level label is used to characterize the relative quality level between the action data associated with different child search nodes under the same parent search node.

[0138] The model training module is used to train the policy model based on the action data in the target search tree data and the dominance level labels corresponding to the action data, so as to obtain the trained policy model; wherein, during the training of the policy model, the dominance level labels are used to determine the training condition information of the policy model.

[0139] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0140] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0141] See Figure 3 , Figure 3 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0142] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0143] The embodiments of this application do not limit the computer device, such as a local computer device, a cloud computer device, a distributed computer device, etc.

[0144] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0145] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0146] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data.

[0147] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0148] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0149] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as would be understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “an,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0150] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0151] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0152] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and other division methods may exist in actual implementation. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections can be through some interfaces; indirect couplings or communication connections between apparatuses or units can be electrical, mechanical, or other forms.

[0153] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a policy model, characterized in that, The method includes: Receive task input information corresponding to robot operation tasks; wherein, the task input information includes initial state information and task information; Based on the task input information, policy model, and world model, a search tree construction process is performed on the robot operation task to obtain target search tree data. The target search tree data includes multiple search nodes, connection relationships between different search nodes, and action data corresponding to the connection relationships. The action data is used to represent the action transferred from the corresponding parent search node to the corresponding child search node and is associated with the corresponding child search node. Based on the target search tree data, the subsequent search results information of multiple child search nodes corresponding to the same parent search node are evaluated within the group to obtain the advantage level label of the action data associated with each child search node; wherein, the advantage level label is used to characterize the relative quality level between the action data associated with different child search nodes under the same parent search node. Based on the action data in the target search tree data and the dominance level labels corresponding to the action data, the policy model is trained to obtain the trained policy model; wherein, during the training of the policy model, the dominance level labels are used to determine the training condition information of the policy model. The search tree construction process includes node expansion processing for at least some search nodes. The node expansion processing includes: taking any search node requiring node expansion as a target search node and placing the world model in an initial model state corresponding to the initial state information; determining path action data from the root search node to the target search node based on the node path corresponding to the target search node, and performing replay processing corresponding to the path action data on the world model to obtain the node model state corresponding to the target search node; determining node state information corresponding to the target search node based on the node model state, and generating at least one candidate action data corresponding to the target search node based on the node state information, the task information, and the strategy model; performing deduction on each candidate action data based on the world model in the node model state to obtain deduction results corresponding to each candidate action data, and forming sub-search nodes corresponding to each candidate action data under the target search node based on the deduction results corresponding to each candidate action data. The step of generating at least one candidate action data corresponding to the target search node based on the node state information, the task information, and the strategy model includes: determining multiple action generation condition information, wherein different action generation condition information corresponds to different action quality guidance information and / or different action disturbance information; and generating multiple candidate action data respectively through the strategy model based on the node state information, the task information, and each of the action generation condition information.

2. The strategy model training method according to claim 1, characterized in that, The process of performing search tree construction on the robot operation task to obtain target search tree data includes: Execute the first search process, which is used to determine the first search node based on the formed search node, perform node expansion processing on the first search node, until the first search termination condition is met, and obtain the first stage search tree data. If the first search termination condition is met, a second search process is executed. The second search process is used to determine at least one second search node based on the internal search nodes in the first stage search tree data that do not meet the child node quantity condition, and to perform node expansion processing on each second search node so that the number of child search nodes corresponding to each second search node meets the child node quantity condition, thereby obtaining the second stage search tree data as the target search tree data.

3. The strategy model training method according to claim 2, characterized in that, The process of determining the first search node based on the established search nodes includes: Based on the access statistics, node value information and prior probability information corresponding to the formed search nodes, determine the node selection score information corresponding to the formed search nodes. Based on the node selection scoring information, the first search node is determined from the formed search nodes; The first search process further includes: after performing node expansion processing on the first search node, updating the access statistics and node value information of at least some search nodes on the node path where the first search node is located based on the task execution results corresponding to the deduction results obtained from the node expansion processing.

4. The strategy model training method according to claim 1, characterized in that, The step of performing deductions on each candidate action data based on the world model in the state of the node model to obtain the deduction results corresponding to each candidate action data includes: The node model state and each candidate action data are input into the world model separately or in batches, and the task execution process corresponding to each candidate action data is predicted based on the world model to obtain the predicted state information and task evaluation information corresponding to each candidate action data. Based on the predicted state information and task evaluation information corresponding to each candidate action data, the deduction result corresponding to each candidate action data is determined; The inference results are used to characterize the state transition results and task execution quality corresponding to the candidate action data.

5. The strategy model training method according to claim 1, characterized in that, Based on the target search tree data, the subsequent search results information of multiple child search nodes corresponding to the same parent search node are evaluated within a group to obtain the dominance level label of the action data associated with each child search node, including: For each child search node corresponding to the same parent search node, the subsequent leaf search node information corresponding to the child search node is determined based on the target search tree data, and the subsequent search result information corresponding to the child search node is obtained based on the subsequent leaf search node information. Based on the subsequent search results information, determine the search evaluation information corresponding to the sub-search node; Based on the search evaluation information of multiple child search nodes corresponding to the same parent search node, the multiple child search nodes are relatively evaluated to obtain the advantage level label of the action data associated with each child search node.

6. The policy model training method according to claim 5, characterized in that, The method involves relative evaluation of multiple child search nodes based on their respective search evaluation information corresponding to the same parent search node, to obtain an advantage level label for the action data associated with each child search node, including: Based on the search evaluation information of multiple child search nodes corresponding to the same parent search node, the evaluation range information within the group is determined; wherein, the evaluation range information within the group is used to characterize the value range among the search evaluation information of multiple child search nodes corresponding to the same parent search node. Based on the evaluation range information within the group, the search evaluation information corresponding to each sub-search node is normalized to obtain the normalized evaluation information corresponding to each sub-search node. Based on the normalized evaluation information corresponding to each sub-search node, the advantage level label of the action data associated with each sub-search node is determined.

7. The strategy model training method according to claim 1, characterized in that, The process of training the policy model based on action data in the target search tree data and the advantage level labels corresponding to the action data to obtain the trained policy model includes: Based on the target search tree data, multiple training samples are constructed, wherein each training sample includes node input information corresponding to the parent search node, action data associated with a child search node of the parent search node, and advantage level label corresponding to the action data. The dominance level label is conditionally encoded to obtain dominance level condition feature information; The strategy model is trained based on the node input information, the action data, and the advantage level conditional feature information to obtain the trained strategy model.

8. A strategy model training device, characterized in that, The device includes: The information receiving module is used to receive task input information corresponding to robot operation tasks; wherein, the task input information includes initial state information and task information; The search tree construction module is used to perform search tree construction processing on the robot operation task based on the task input information, strategy model, and world model to obtain target search tree data. The target search tree data includes multiple search nodes, connection relationships between different search nodes, and action data corresponding to the connection relationships. The action data is used to represent the action transferred from the corresponding parent search node to the corresponding child search node and is associated with the corresponding child search node. The relative evaluation module is used to perform intra-group relative evaluation of the subsequent search results information of multiple child search nodes corresponding to the same parent search node based on the target search tree data, and obtain the advantage level label of the action data associated with each child search node; wherein, the advantage level label is used to characterize the relative quality level between the action data associated with different child search nodes under the same parent search node. The model training module is used to train the policy model based on the action data in the target search tree data and the dominance level labels corresponding to the action data, so as to obtain the trained policy model; wherein, during the training of the policy model, the dominance level labels are used to determine the training condition information of the policy model. The search tree construction process includes node expansion processing for at least some search nodes. The node expansion processing includes: taking any search node requiring node expansion as a target search node and placing the world model in an initial model state corresponding to the initial state information; determining path action data from the root search node to the target search node based on the node path corresponding to the target search node, and performing replay processing corresponding to the path action data on the world model to obtain the node model state corresponding to the target search node; determining node state information corresponding to the target search node based on the node model state, and generating at least one candidate action data corresponding to the target search node based on the node state information, the task information, and the strategy model; performing deduction on each candidate action data based on the world model in the node model state to obtain deduction results corresponding to each candidate action data, and forming sub-search nodes corresponding to each candidate action data under the target search node based on the deduction results corresponding to each candidate action data. The step of generating at least one candidate action data corresponding to the target search node based on the node state information, the task information, and the strategy model includes: determining multiple action generation condition information, wherein different action generation condition information corresponds to different action quality guidance information and / or different action disturbance information; and generating multiple candidate action data respectively through the strategy model based on the node state information, the task information, and each of the action generation condition information.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images