Robot operation execution method, robot operation execution system, and robot

Abstract

The application discloses a robot operation execution method, a robot operation execution system and a robot. The method comprises the following steps: receiving a current operation task of the robot, collecting scene perception information corresponding to the current operation task, and constructing a task-related space state; generating one or more candidate actions based on the task-related space state, performing pre-space feasibility verification on each candidate action, and outputting structured verification information; filtering the candidate actions that pass the verification or can be executed after parameter adjustment, and issuing the candidate actions to the robot for execution; collecting post-execution space feedback after the action is completed; updating the task-related space state based on the post-execution space feedback, and binding the structured verification information and the post-execution space feedback to form structured space experience; and instructing the robot to execute a subsequent operation task based on the updated task-related space state and the structured space experience. According to the application, the execution strategy corresponding to each operation task can be effectively planned.

Description

Technical Field

[0001] This application relates to the field of robotics technology, specifically to a robot operation execution method, a robot operation execution system, and a robot. Background Technology

[0002] Robotic manipulation tasks are typically not static, one-time execution processes. Take object placement as an example: when the robot prepares to place an object into a target area, the target area may already be occupied by other objects, the object may be occluded and unable to be directly grasped, or the previously available temporary storage area may have become invalid after a certain step. If the robot's manipulation execution system relies solely on a one-time planning result or only maintains a broad state representation for the entire scene, it is prone to generating unsuitable candidate actions in subsequent steps, leading to repeated failures. Summary of the Invention

[0003] This application discloses a robot operation execution method, a robot operation execution system, and a robot, which can effectively plan the execution strategy for each operation task and avoid ineffective operation execution.

[0004] In a first aspect, this application discloses a robot operation execution method, which may include the following steps: Receive the robot's current operation task, collect the scene perception information corresponding to the current operation task, and extract the target object, related objects, target area, temporary storage area, obstacle area, operable area and spatial constraint relationship specific to the current operation task to construct the task-related spatial state; Based on the task-related spatial state, one or more candidate actions are generated, and a preliminary spatial feasibility check is performed on each candidate action, outputting structured check information. The structured check information includes check items, check pass / fail results, unmet spatial constraints, corresponding failure reasons, whether the current candidate action is executable, and action processing suggestions for the current candidate action. Candidate actions that pass the screening and verification or are executable after parameter adjustment are sent to the robot for execution, and post-execution spatial feedback is collected after the action is completed; wherein, the post-execution spatial feedback indicates the spatial feedback that can be used for subsequent operation tasks; The task-related spatial state is updated based on the post-execution spatial feedback, and the structured verification information is bound to the post-execution spatial feedback to form structured spatial experience. Based on the updated task-related spatial state and the structured spatial experience, the robot is instructed to perform subsequent operational tasks.

[0005] By implementing the embodiments of this application, in the scenario of multi-step operation tasks, the robot can be instructed to perform subsequent operation tasks based on the updated task-related spatial state and structured spatial experience. The updated task-related spatial state comprehensively considers the spatial feedback after execution, which is the spatial feedback that can be used by subsequent operation tasks. In this way, the generation of invalid actions repeatedly under similar spatial states can be reduced, the targeting of actions to continue execution, retry, and recovery can be improved, the stability of multi-step operation task execution can be improved, and the frequency of complete replanning after execution failure can be reduced.

[0006] In one possible implementation, the structured spatial experience includes positive example spatial experience and negative example spatial experience; wherein, the positive example spatial experience is used to characterize candidate actions that have passed the pre-existing spatial feasibility check and achieve the task objective after execution; the negative example spatial experience is used to characterize candidate actions that have not passed the pre-existing spatial feasibility check or that have not achieved the task objective after execution.

[0007] In one possible implementation, upon matching a positive example of spatial experience, instructing the robot to perform subsequent operational tasks based on the updated task-related spatial state and the structured spatial experience includes: Candidate actions corresponding to the positive example space experience are used first to perform subsequent operation tasks.

[0008] In one possible implementation, when a negative example space experience is matched, instructing the robot to perform subsequent operation tasks according to the target strategy based on the updated task-related space state and the structured space experience includes: not using or avoiding the use of candidate actions corresponding to the negative example space experience, and avoiding the robot from repeatedly executing invalid actions.

[0009] In one possible implementation, updating the task-related spatial state based on the post-execution spatial feedback includes: Update the spatial relationships between the target object, related objects, and regions; update the availability of the operable region, temporary region, and alternative region; update the current validity of the candidate action; update whether the current operation task stage has progressed; and update whether the spatial constraints have been met or not yet met.

[0010] In one possible implementation, the candidate action includes at least one of the following; Directly capture the target object; After adjusting the grab position, the target object is grabbed. Move the obstacle object first, then perform the grab; First, move the relevant objects to the temporary storage area; Place the target object into the target area; Place the target object into the alternative area; Perform the recovery action first, then re-execute the target action.

[0011] In one possible implementation, the pre-space feasibility check includes at least one of the following: The system includes verification of reachability, collision risk of moving path, capacity of target area, stability of placement support, attitude consistency, occlusion relationship, availability of temporary storage area, and consistency of target area state after action execution.

[0012] In one possible implementation, the post-execution spatial feedback includes at least one of the following: change in object pose, change in spatial relationship between objects, change in contact relationship between the robot actuator and the object, whether the target area is still occupied, whether the temporary storage area is still available, whether the occlusion relationship has changed, whether the current task objective has been achieved, and whether a new collision, slippage, offset or obstruction has occurred.

[0013] Secondly, this application discloses a robot operation execution system, comprising: The task input and scene perception unit is used to receive the robot's current operation task and collect the scene perception information corresponding to the current operation task. The task-related spatial state construction unit is used to extract the target object, related objects, target area, temporary storage area, obstacle area, operable area and spatial constraint relationship specific to the current operation task, so as to construct the task-related spatial state; The candidate action generation unit is used to generate one or more candidate actions based on the task-related spatial state, and to perform a preliminary spatial feasibility check on each candidate action, outputting structured verification information, wherein the structured verification information includes verification items, verification pass / fail results, unmet spatial constraints, corresponding failure reasons, whether the current candidate action is executable, and action processing suggestions for the current candidate action; The spatial feasibility screening unit is used to screen candidate actions that have passed verification or can be executed after parameter adjustment and send them to the robot for execution, and to collect post-execution spatial feedback after the action is completed; wherein, the post-execution spatial feedback indicates the spatial feedback that can be used for subsequent operation tasks; A spatial experience construction unit is used to update the task-related spatial state based on the post-execution spatial feedback, and bind the structured verification information with the post-execution spatial feedback to form structured spatial experience; The subsequent decision-making unit is used to instruct the robot to perform subsequent operation tasks based on the updated task-related spatial state and the structured spatial experience.

[0014] Thirdly, this application discloses a robot, which has a built-in robot operation execution system as described in the second aspect above, to perform the robot operation execution method as described in any of the first aspects above. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0016] Figure 1 A flowchart illustrating a robot operation execution method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a robot operation execution system provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] Robotic tasks are typically not static, one-time actions. In real-world scenarios, target areas may be occupied, operation paths may be obstructed, placement areas may fail during execution, and the intermediate state after a certain action can directly affect the feasibility of the next action. Therefore, when performing multi-step tasks, robots not only need to perceive the current scene but also need to continuously maintain states that can be used for subsequent decisions, forming a closed loop before and after the action execution.

[0019] The existing technologies mainly include the following types of methods for implementing robot operation tasks: (1) Task action generation and replanning technology based on scene graph or state abstraction These technologies typically construct scene graphs, state abstractions, or object relationship representations based on perception results, and generate task action sequences based on these representations. When an action becomes unexecutable or target conditions change, replanning is then performed. For example, a hierarchical scene graph and state abstraction are used to generate task action sequences, and the executability of actions is assessed before execution. Dynamic replanning is performed when execution fails or target conditions change. The basic framework of this type of technology, encompassing "state representation, action generation, action feasibility assessment, and feedback replanning," has been publicly disclosed. While this technology can support task action generation through state abstraction and perform replanning after execution failure, its state is more focused on scene description or task planning itself. It does not emphasize continuously maintaining task-related spatial states around the current task that can directly serve subsequent decisions, nor does it integrate pre-execution verification results with post-execution feedback into spatial experience.

[0020] (2) Feasibility assessment technology for candidate actions based on task and motion planning This type of technology typically utilizes geometric states, symbolic states, candidate sub-goals, or candidate actions to assess the reachability, preconditions, or feasibility of candidate actions before execution, and then re-perceives and updates the state after execution. For example, candidate sub-goals or candidate actions can be generated using geometric and symbolic scene graphs, and their executable status can be determined through precondition or reachability assessments. After execution, the scene graph is re-perceived and updated. This type of technology has a relatively tight integration between candidate action generation and pre-execution assessments, but the results of these pre-execution assessments are mostly used as the basis for passing or rejecting the current action. They are not further bound to post-execution feedback to form reusable spatial experience, nor are failure modes, spatial constraints, and candidate action handling methods used as unified decision-making resources in subsequent steps.

[0021] (3) Robot task planning technology based on initial state, target state and intermediate state This type of technology typically generates actionable paths around initial, target, and intermediate states, and determines whether the target state has been reached after execution. If not, it continues to observe and replan. For example, based on the initial, target, and intermediate states of an object, it can perform robot operations through actionable analysis and path generation. After execution, it determines whether the target has been achieved; if not, it re-observes and re-plans. This type of technology has the basic outline of closed-loop robot operation planning, but its post-execution processing is mostly limited to re-observation and re-planning. It does not structure and precipitate the pre-execution judgment results, post-execution spatial deviations, and failure reasons into reusable experience, nor does it directly use this experience for subsequent candidate action screening or action recovery selection.

[0022] (4) Follow-up decision-making techniques based on execution history, failure feedback, or skill memory These technologies typically preserve execution history, failure reasons, skill correction results, or long-term memories after a robot fails, and then recall these memories to assist planning in subsequent tasks. For example, execution history, failure reasons, and long-term memories can be used to assist in planning subsequent tasks; or, short-term and long-term memories can be formed based on execution feedback, and then retrieved to guide subsequent planning. While these technologies have disclosed the general idea of ​​using failure results or experience for subsequent decision-making, this experience is mostly general history, natural language experience, skill memory, or corrective knowledge. They do not explicitly bind the spatial feasibility verification results before execution with the spatial feedback after execution to form structured spatial experience, nor do they explicitly include spatial constraints, failure modes, applicable conditions, and candidate action handling methods as unified items in the generation, screening, continuation, retry, recovery, or replanning of subsequent candidate actions.

[0023] Based on this, this application aims to propose a novel robot operation execution method. In multi-step operation tasks, the robot can be instructed to execute subsequent operation tasks based on updated task-related spatial states and structured spatial experience. The updated task-related spatial states comprehensively consider post-execution spatial feedback; specifically, the post-execution spatial feedback refers to the spatial feedback that can be utilized by subsequent operation tasks. Subsequent embodiments will elaborate on the technical solutions described in this application.

[0024] As an optional implementation of the disclosure in this application, an embodiment of this application discloses a robot operation execution method, such as... Figure 1 As shown, Figure 1 This is a flowchart of a robot operation execution method disclosed in an embodiment of this application. The method may include, but is not limited to, the following steps: Step S101: Receive the robot's current operation task and collect the scene perception information corresponding to the current operation task. Extract the target object, related objects, target area, temporary storage area, obstacle area, operable area and spatial constraint relationship specific to the current operation task to construct the task-related spatial state.

[0025] In this embodiment, the task-related spatial state excludes redundant global environmental information unrelated to the current operation task. Specifically, scene information can be layered and irrelevant elements can be removed layer by layer, rather than maintaining a full environmental representation. For example, the filtering criteria can include at least: a task semantic layer, which identifies objects and regions related to the complete task objective based on the operation task, excluding non-operational objects such as actuators and backgrounds; a task relevance layer, which removes instances that are irrelevant or redundant to the current decision operation; a geometrically observable layer, which only includes instances with obtained poses, dimensions, and necessary spatial relationships, and can trigger re-observation for those with incomplete information; and a decision operability layer, which only generates operational elements with geometric evidence to support candidate actions. After filtering, the task-related spatial state at least represents the target and related objects, the target area and temporary storage area, spatial relationships, area availability, and spatial constraints at this stage. Unlike the static full-scene representation of a global scene graph or contextual memory, the task-related spatial state involved in this invention is dynamically determined according to the current operation task, and its fields directly serve spatial feasibility verification and structured spatial experience matching. Dynamic updates to the task-related state space are triggered when task-related perceptions are successfully written back or when the space feedback is successfully written back after an action is executed. When the version of the task-related state space changes, the verification conclusions obtained based on the old version become invalid. The update granularity is based on the entire state snapshot and version number, with the version bound to the decision round to avoid high-frequency full reconstruction of the entire environment.

[0026] Generally, task-related spatial states can include, but are not limited to: spatial relationships between the target object and related objects; spatial relationships between the target object and the target area; spatial relationships between the robot's actuator and the target object; occlusion, collision, or obstruction relationships caused by obstacles during grasping, moving, or placing; the current availability of operable or temporary storage areas; candidate action information that has passed or failed verification; and the pre- and post-action dependencies corresponding to the current operation task stage. In this implementation, task execution is not organized around a broad scene state or a complete environment reconstruction, but rather around the task-related spatial states corresponding to the current operation task. Understandably, task-related spatial states are more closely aligned with the operational decisions corresponding to the current operation task, reducing the interference of irrelevant information on subsequent action selection.

[0027] Step S102: Generate one or more candidate actions based on the task-related spatial state, and perform a preliminary spatial feasibility check on each candidate action, outputting structured check information. The structured check information includes check items, check pass / fail results, unmet spatial constraints, corresponding failure reasons, whether the current candidate action is executable, and action processing suggestions for the current candidate action.

[0028] In this implementation, this application does not simply treat the pre-emptive spatial feasibility verification as a one-time pass or rejection judgment, but further retains the verification items, verification results, unmet spatial constraints, corresponding failure reasons, whether the current candidate action is executable, and action processing suggestions for the current candidate action, so as to provide a structured basis for subsequent task-related spatial state updates and structured spatial experience.

[0029] In this application embodiment, the candidate action includes at least one of the following; Directly retrieve the target object; Adjust the grab position and then grab the target object; Move the obstacle object first, then perform the grab; First, move the relevant objects to the temporary storage area; Place the target object in the target area; Place the target object into the replacement area; Perform the recovery action first, then re-execute the target action.

[0030] Specifically, the candidate action can be a single-step atomic action or a sequence of candidate operations in the current task phase. For example, the candidate action could be to directly grab the target object, adjust the grabbing pose and then grab the target object, move the obstacle object first and then perform the grabbing, move the relevant object to the temporary area first, place the target object in the target area, place the target object in the alternative area, or perform the recovery action first and then re-execute the target action, etc. This application does not impose specific limitations on the comparison.

[0031] In this embodiment, a preliminary spatial feasibility check is performed on each candidate action. This not only determines whether the candidate action is successful, but also provides, as far as possible, the check items corresponding to the current candidate action, the pass / fail result, the unmet spatial constraints, the corresponding failure reason, whether the current candidate action is executable, and action processing suggestions for the current candidate action. For example, a candidate action may fail because the target area is still occupied, while another candidate action may pass because the temporary storage area is available.

[0032] In one embodiment, the spatial feasibility verification result of candidate action a under the task-related spatial state S is denoted as V(a,S)=(Q,R,C,M). Here, Q represents the set of executed spatial verification items and the local pass / fail results for each item; R represents the overall pass / block result and the main failure reason code; C represents the identifier, type, involved objects and regions, and location information of unmet spatial constraints; and M represents recovery or alternative strategy suggestions. The verification result is also bound to the candidate action identifier, the state version on which it is based, and time information. Unlike existing technologies that only output pass or rejection, this invention retains a structured output of verification dimensions, unmet constraints, corresponding failure reasons, and processing suggestions. This output can be machine-indexed and reused, and serves as standardized input for constructing spatial experience, i.e., E=Bind(V(a,S),F,S), where F is the spatial feedback after execution. Spatial constraints, failure modes, and processing methods are directly inherited from V and M, without requiring secondary parsing from natural language logs.

[0033] In one embodiment, V(a,S) can be represented using the following structured field format: {"candidate_action_id": " <string>","state_version": " <string>","result": "PASS |BLOCKED","primary_reason_code": "<enum | null>","check_items": [{ "check_type": " <string>", "check_result": "pass | fail"}],"unsatisfied_constraints": [{"constraint_type": " <string>","involved_objects": [{ "object_id": " <string>", "role": " <string>"}],"involved_regions": [{ "region_id": " <string>", "status": " <string>"}],"locus": { "action_index": 0, "action_type": "place"}}],"handling_suggestion": {"suggested_recovery": " <string>"}} In this dataset, `candidate_action_id` corresponds to the candidate action identifier; `state_version` corresponds to the task-related spatial state version; `result` and `primary_reason_code` correspond to `R`; `check_items` corresponds to `Q`, `check_type` indicates the type of check item (such as reachability, path collision, target area capacity, occlusion relationship, temporary area availability, etc.), and `check_result` indicates whether the item partially passes or fails; `unsatisfied_constraints` corresponds to `C`, `constraint_type` indicates the constraint type, `involved_objects` and `involved_regions` respectively indicate the involved objects and roles, and the involved areas and states, and `locus` indicates the constraint location information; `handling_suggestion` corresponds to `M`, and `suggested_recovery` indicates the suggested recovery or alternative strategy. The above field format is only used to illustrate the structured verification output method of this invention and does not constitute a limitation on the scope of protection of this invention.

[0034] For example, if the candidate action "directly place the target object" fails the validation because the target region is occupied, V(a,S) will have a result of BLOCKED, a primary_reason_code indicating that the target is occupied, and check_items will contain entries with check_type "target region containment" and check_result of failure. unsatisfied_constraints indicates that constraint_type is occupancy constraint, the target region status in involved_regions is occupied, and suggested_recovery in handling_suggestion indicates that the occupied object should be moved to the temporary region first. If the candidate action "remove the occupied object first and then place it" has a result of PASS when the temporary region is available, V(a,S) can be directly used as input for negative example space experience or positive example space experience. In this application embodiment, the pre-space feasibility verification includes at least one of the following: The system includes verification of reachability, collision risk of moving path, capacity of target area, stability of placement support, attitude consistency, occlusion relationship, availability of temporary storage area, and consistency of target area state after action execution.

[0035] Furthermore, the preliminary spatial feasibility verification result is synchronously written to or associated with the candidate validity partition of the task-related spatial state, and participates in the generation, screening and recovery selection of subsequent candidate actions.

[0036] In one implementation, during the t-th round of decision-making, the system constructs St, generates a candidate set At based on St, calculates V(a,St) for each candidate, and writes the pass or blockage result, main failure reason, unmet spatial constraints, and the state version on which each candidate is based into the candidate validity partition of St, obtaining St'; for candidates that pass the verification, actions are executed and the spatial feedback Ft after execution is obtained, and St+1 is updated accordingly, with the state version incremented; simultaneously, spatial experience Et is generated according to Bind(V,Ft,St) and included in the spatial experience set; when generating candidate actions in the next round, the robot operation execution system simultaneously reads St+1, the spatial experience set, and the still valid candidate validity records, downweights or filters blocking type candidates, suppresses corresponding candidates for matched negative example spatial experience, prioritizes the recommended verified processing method for matched positive example spatial experience, and feeds the decision results back to the candidate action generation stage, forming a closed loop. The verification result is bound to the state version, and the old verification conclusion becomes invalid when the state version changes.

[0037] Step S103: Select candidate actions that have passed the screening and verification or are executable after parameter adjustment and send them to the robot for execution, and collect the spatial feedback after the action is completed; wherein, the spatial feedback after execution indicates the spatial feedback that can be used for subsequent operation tasks.

[0038] In the embodiments of this application, the spatial feedback after execution may include, but is not limited to, at least one of the following: changes in object pose, changes in spatial relationships between objects, changes in contact relationships between the robot actuator and the object, whether the target area is still occupied, whether the temporary storage area is still available, whether the occlusion relationship has changed, whether the current task objective has been achieved, and whether new collisions, slips, offsets or obstructions have occurred.

[0039] In this embodiment, spatial feedback is not limited to success or failure indicators, but includes spatial change information sufficient to support subsequent decisions. For example, in a scenario where the obstacle object is moved first, the spatial feedback after execution can reflect whether the original target area has been released, whether the temporary storage area has been occupied, whether the obstacle relationship has been resolved, and whether new occlusion or collision risks have occurred.

[0040] Step S104: Update the task-related spatial state based on the post-execution spatial feedback, and bind the structured verification information with the post-execution spatial feedback to form a structured spatial experience.

[0041] In the embodiments of this application, updating the spatial state related to the task based on post-execution spatial feedback may include: updating the spatial relationship between the target object, related objects and regions, updating the availability of operable regions, temporary regions and alternative regions, updating the current validity of candidate actions, updating whether the current operation task stage has progressed, and updating whether certain spatial constraints have been met or not yet met, etc.

[0042] In the embodiments of this application, this application does not simply re-observe or re-plan after the action is executed, but directly writes the spatial feedback after execution back to the task-related spatial state, so that the subsequent action selection can be carried out based on the updated task-related spatial state (i.e., the real task spatial state).

[0043] In this embodiment, structured spatial experience emphasizes the structured recording of spatial constraints, failure modes, applicable conditions, and candidate action processing methods, rather than forming ordinary logs, natural language summaries, or generalized memory resources. Specifically, structured spatial experience may at least include spatial constraint identifiers, involved objects or regions, verification items, verification results, post-execution feedback results, failure modes, applicable conditions, and candidate action processing methods.

[0044] Generally, structured spatial experience includes positive example spatial experience and negative example spatial experience. Positive example spatial experience is used to characterize candidate actions that have passed the preliminary spatial feasibility check and achieve the task objective after execution. Negative example spatial experience is used to characterize candidate actions that have not passed the preliminary spatial feasibility check or that have not achieved the task objective after execution.

[0045] For example, a negative example space experience can be formed for the case of "direct placement fails because the target area is occupied and the verification fails"; a positive example space experience can be formed for the case of "removing the obstacle object first and then placing the target object" and successfully completing the task.

[0046] Step S105: Instruct the robot to perform subsequent operation tasks based on the updated task-related spatial state and structured spatial experience.

[0047] In this embodiment, the similarity between the updated task-related spatial state and one or more of the spatial constraints, failure modes, applicable conditions, and candidate action processing methods indicated by the structured spatial experience can be obtained. If the similarity between the updated task-related spatial state and one or more of the spatial constraints, failure modes, applicable conditions, and candidate action processing methods indicated by the structured spatial experience is greater than or equal to a first threshold, a matching positive example spatial experience is determined. In this case, the candidate action corresponding to the positive example spatial experience can be used preferentially to execute subsequent operation tasks. Here, the first threshold can be set according to actual usage requirements, and this application does not specifically limit it. For example, the execution strategy for executing subsequent operation tasks may include continuing execution, retrying the current action, adjusting action parameters and then executing, selecting an alternative candidate action, executing a recovery action first, regenerating candidate actions, performing local replanning, and performing global replanning.

[0048] In this embodiment, the similarity between the updated task-related spatial state and one or more of the spatial constraints, failure modes, applicable conditions, and candidate action processing methods indicated by the structured spatial experience can be obtained. If the similarity between the updated task-related spatial state and one or more of the spatial constraints, failure modes, applicable conditions, and candidate action processing methods indicated by the structured spatial experience is less than a first threshold, a negative example spatial experience is determined to be matched. In this case, the candidate actions corresponding to the negative example spatial experience are not used or are avoided to prevent the robot from repeatedly executing invalid actions. Alternatively, when a negative example spatial experience is matched, the robot operation execution system can reduce the priority of the candidate actions corresponding to the negative example spatial experience, adjust parameters, trigger recovery actions, or directly execute replanning. Here, the first threshold can be set according to actual usage requirements, and this application does not specifically limit it.

[0049] In one embodiment, the conditions for writing negative example spatial experience include: the pre-execution verification result is blocked; or the goal is not achieved after execution and the position deviation or attitude deviation exceeds the allowable threshold; or a new collision, occlusion, or occupation occurs after execution. The conditions for writing positive example spatial experience include: the pre-execution verification passes and the goal or key sub-goal is achieved after execution. Formatting errors unrelated to space are not written to the structured spatial experience. The position deviation threshold, attitude deviation threshold, minimum confidence level for writing, and similarity threshold for merging duplicate entries are configurable; if the confidence level is reached, it is written; if it is highly similar to an existing entry, it is merged and the confidence level is updated. Positive examples can cover negative examples in the same scenario, and high-confidence entries can be persisted across tasks. Unlike existing execution histories, which are mostly time-series logs or natural language descriptions, the structured spatial experience described in this invention consists of decision entries with a fixed structure. The verification result V, the post-execution feedback F, and the state S are bound together in a ternary manner, and they directly participate in candidate filtering and recovery recommendation through matching.

[0050] In one embodiment, the robot operation execution system performs a comprehensive scoring retrieval of each entry in the historical spatial experience set with the current decision context. The scoring dimensions include two parts: task semantic similarity and task-related spatial state similarity. The task semantic similarity (Sem_task) is obtained by comparing the current operation task type, current task stage, and the type of the sub-objective to be decided with the applicable task semantic information recorded in the experience entry. The task-related spatial state similarity (Sim_state) is obtained by comparing the current task-related spatial state summary with the state summary bound by Bind(V,F,S) in the experience entry in dimensions such as object category, region occupancy pattern, and whether key spatial constraints are violated, and is not limited to isomorphic matching of the complete spatial relationship graph. The robot operation execution system calculates the score of each experience entry according to Score(E) = α·Sem_task + β·Sim_state, where the weights α and β and the number of searches k are configurable. The top-k experiences with the highest scores are used as matching results. When k=1, the robot operation execution system selects the experience with the highest score and priority for subsequent decision-making.

[0051] For negative examples in the top-k matching results, the robot operation execution system can reduce the priority of the corresponding candidate actions, adjust parameters, trigger recovery actions, or directly perform replanning; for positive examples, the robot operation execution system can prioritize the selection of verified and effective candidate actions.

[0052] In summary, when structured spatial experience includes both positive and negative example spatial experience, positive example spatial experience can be used to improve the success rate of subsequent operations, while negative example spatial experience can be used to limit repeated failures. This can improve the stability, continuity, and efficiency of the robot in multi-step operation tasks.

[0053] In this embodiment, in a multi-step operation task scenario, the task-related spatial state corresponding to the previous operation task and the task-related spatial state corresponding to the next operation task can be obtained separately. The similarity between these two spatial states is also calculated. If the similarity is greater than a second threshold, the action executed in the previous operation task is determined as the action executed in the next operation task. Here, the second threshold can be set according to actual usage requirements, and this application does not impose specific limitations on it. In this way, structured spatial experience can be clearly used to generate subsequent candidate actions, filter candidate actions, continue execution, retry, resume action selection, or replanning. Therefore, it can suppress the robot operation execution system from repeatedly generating invalid actions in similar task-related spatial states.

[0054] It should be noted that the first threshold and the second threshold mentioned above can be set to the same value or different values, and this application does not make any specific restrictions on this.

[0055] It should also be noted that the structured verification information obtained from the pre-execution feasibility verification is bound with the spatial feedback after execution to form structured spatial experience. The resulting structured spatial experience is no longer a general history, skill set, or natural language summary, but a decision-making resource oriented towards spatial constraints and failure modes.

[0056] By implementing the embodiments of this application, in the scenario of multi-step operation tasks, the robot can be instructed to perform subsequent operation tasks based on the updated task-related spatial state and structured spatial experience. The updated task-related spatial state comprehensively considers the spatial feedback after execution, which is the spatial feedback that can be used by subsequent operation tasks. In this way, the generation of invalid actions repeatedly under similar spatial states can be reduced, the targeting of actions to continue execution, retry, and recovery can be improved, the stability of multi-step operation task execution can be improved, and the frequency of complete replanning after execution failure can be reduced.

[0057] It should be noted that the task-related spatial state update parameters involved in this application may include the state refresh cycle, target area occupancy determination threshold, temporary area availability determination threshold, and spatial relationship change confirmation conditions. Spatial feasibility verification parameters may include the reachability judgment threshold, collision risk allowable range, occlusion allowable range, support placement stability threshold, attitude deviation allowable range, and target achievement deviation allowable range. Structured spatial experience construction parameters may include positive example spatial experience writing conditions, negative example spatial experience writing conditions, failure mode classification rules, experience item retention duration, and experience item confidence update rules. Structured spatial experience retrieval parameters may include the experience matching similarity threshold, candidate action weight reduction magnitude, candidate action filtering conditions, recovery action trigger threshold, and local replanning and global replanning switching conditions. Generally, the above parameters can be set according to scene complexity, object scale, and robot execution progress; this application does not impose specific limitations on these parameters.

[0058] To facilitate a better understanding of the technical solutions described in this application, the following description is provided in conjunction with specific application scenarios: Application Scenario 1: Placement task execution when the target area is occupied The robot receives a task to place a target object into a target area. The robot's operation execution system first acquires scene information including the target object, target area, occupied object, and temporary storage area, and constructs the task-related spatial state. Subsequently, the robot's operation execution system generates multiple candidate actions, including directly placing the target object, removing the occupied object before placing the target object, and adjusting the target object's posture before placing it.

[0059] Before executing the aforementioned candidate actions, the robot's operation and execution system performs a preliminary spatial feasibility check. It finds that directly placing the target object fails the check because the target area is occupied. However, it finds that removing the occupying object before placing the target object, provided the current temporary storage area is available, passes the check. The robot's operation and execution system executes the latter action and obtains post-execution spatial feedback afterward, confirming that the target area has been released, the temporary storage area has been occupied, and the obstacle relationship has been removed.

[0060] The robot's operation execution system forms negative example space experience from the information that "directly placing the target object failed because the target area was occupied," and positive example space experience from the information that "removing the occupied object first and then placing the target object can successfully complete the task." Subsequently, in similar task-related space states, the robot's operation execution system can prioritize calling the corresponding positive example space experience, or directly suppress candidate actions that match the negative example space experience.

[0061] Application Scenario 2: Resuming execution when the capture path is obscured.

[0062] The robot receives a task to grasp a target object. After constructing the task-related spatial state, the robot's operation and execution system generates multiple candidate actions, including direct grasping, grasping after changing the grasping posture, and grasping after removing the occluded object. Before executing the candidate actions, the robot's operation and execution system performs a preliminary spatial feasibility check. It finds that direct grasping fails due to occlusion, changing the grasping posture still cannot meet the accessibility requirements, while grasping after removing the occluded object passes the check.

[0063] After the robot's manipulation and execution system performs the obstacle removal action, it acquires post-execution spatial feedback and updates the task-related spatial state. Subsequently, the system generates new candidate actions and completes the grasping task. During this process, the system incorporates occlusion-induced failure modes, unmet spatial constraints, effective recovery actions, and applicable conditions into its structured spatial experience. In subsequent scenarios with similar occlusion relationships (i.e., under similar task-related spatial states), the system can directly prioritize selecting the same or similar recovery actions.

[0064] The applicant adopted the above-mentioned technical solution in practical applications. On the FR5 (Fao Collaborative Robot) desktop operating platform, a real-machine comparative experiment was conducted for two types of multi-step operation scenarios: dynamic occlusion and target area occupancy. Each scenario was performed 30 independent tasks, for a total of 60. All solutions were tested under the same object set, the same motion interface, and the same motion budget. Control group A used narrative-based contextual memory-assisted replanning, without pre-execution structured verification or spatial experience binding; control group B included task-related spatial states and pre-verification, but did not enable structured spatial experience or experience-driven decision-making; control group C was the complete closed-loop solution of this invention. The summary results of 60 tasks show that: Group C's success rate was 76.7% (46 / 60), an improvement of 21.7 percentage points compared to Group A's 55.0%, and an improvement of 18.4 percentage points compared to Group B's 58.3%; the average number of decision steps was 5.1, a decrease of 26.1% compared to Group A's 6.9 steps, and a decrease of 20.3% compared to Group B's 6.4 steps; the rate of repeated invalid candidates was 10.8%, a decrease of 20.4 percentage points compared to Group A's 31.2%, and a decrease of 17.7 percentage points compared to Group B's 28.5%; the global replanning trigger rate was 0.5 times / task, a decrease of 66.7% compared to Group A's 1.5 times / task, and a decrease of 58.3% compared to Group B's 1.2 times / task. The above data shows that the present invention binds the pre-execution verification and post-execution feedback fields at the field level into structured spatial experience and directly participates in candidate filtering and recovery recommendation, which can effectively reduce the occurrence rate of invalid actions and reduce the number of global replanning triggers in scenarios with dynamic occlusion and target area occupancy.

[0065] By implementing the embodiments of this application, it is possible to reduce the repeated generation of invalid actions in similar task-related spatial states, improve the targeting of actions to continue execution, retry, and recover, improve the stability of multi-step operation task execution, reduce the frequency of complete replanning after execution failure, and enable the robot to more stably complete tasks such as grasping, moving, temporarily storing, and placing target objects in dynamic scenes.

[0066] Figure 2 This application provides a robot operation execution system as an embodiment. For example... Figure 2 As shown, the robot operation execution system may include: The task input and scene perception unit 201 is used to receive the robot's current operation task and collect the scene perception information corresponding to the current operation task; The task-related spatial state construction unit 202 is used to extract the target object, related objects, target area, temporary storage area, obstacle area, operable area and spatial constraint relationship specific to the current operation task, so as to construct the task-related spatial state; The candidate action generation unit 203 is used to generate one or more candidate actions based on the task-related spatial state, and to perform a preliminary spatial feasibility check on each candidate action, outputting structured verification information, wherein the structured verification information includes verification items, verification pass / fail results, unmet spatial constraints, corresponding failure reasons, whether the current candidate action is executable, and action processing suggestions for the current candidate action; The spatial feasibility screening unit 204 is used to screen candidate actions that have passed verification or are executable after parameter adjustment and send them to the robot for execution, and to collect post-execution spatial feedback after the action is completed; wherein, the post-execution spatial feedback indicates the spatial feedback that can be used for subsequent operation tasks; The spatial experience construction unit 205 is used to update the task-related spatial state based on the post-execution spatial feedback, and bind the structured verification information with the post-execution spatial feedback to form structured spatial experience; The subsequent decision selection unit 206 is used to instruct the robot to perform subsequent operation tasks based on the updated task-related spatial state and the structured spatial experience.

[0067] By implementing the embodiments of this application, in the scenario of multi-step operation tasks, the robot can be instructed to perform subsequent operation tasks based on the updated task-related spatial state and structured spatial experience. The updated task-related spatial state comprehensively considers the spatial feedback after execution, which is the spatial feedback that can be used by subsequent operation tasks. In this way, the generation of invalid actions repeatedly under similar spatial states can be reduced, the targeting of actions to continue execution, retry, and recovery can be improved, the stability of multi-step operation task execution can be improved, and the frequency of complete replanning after execution failure can be reduced.

[0068] On the other hand, embodiments of this application also disclose a robot, which has built-in features such as... Figure 2 The robot operation execution system shown is used to execute the robot operation execution method described in this application.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above embodiments are merely illustrative of several implementation methods described in detail, but they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this specification, and these all fall within the scope of protection of this specification. Therefore, the scope of protection of this patent should be determined by the appended claims.< / string> < / string> < / string> < / string> < / string> < / string> < / string> < / string> < / string>

Claims

1. A method for performing robot operations, characterized in that, include Receive the robot's current operation task, collect the scene perception information corresponding to the current operation task, and extract the target object, related objects, target area, temporary storage area, obstacle area, operable area and spatial constraint relationship specific to the current operation task to construct the task-related spatial state; Based on the task-related spatial state, one or more candidate actions are generated, and a preliminary spatial feasibility check is performed on each candidate action, outputting structured check information. The structured check information includes check items, check pass / fail results, unmet spatial constraints, corresponding failure reasons, whether the current candidate action is executable, and action processing suggestions for the current candidate action. Candidate actions that pass the screening and verification or are executable after parameter adjustment are sent to the robot for execution, and post-execution spatial feedback is collected after the action is completed; wherein, the post-execution spatial feedback indicates the spatial feedback that can be used for subsequent operation tasks; The task-related spatial state is updated based on the post-execution spatial feedback, and the structured verification information is bound to the post-execution spatial feedback to form structured spatial experience. Based on the updated task-related spatial state and the structured spatial experience, the robot is instructed to perform subsequent operational tasks.

2. The method according to claim 1, characterized in that, The structured spatial experience includes positive example spatial experience and negative example spatial experience; wherein, the positive example spatial experience is used to characterize candidate actions that have passed the pre-existing spatial feasibility check and have achieved the task objective after execution; the negative example spatial experience is used to characterize candidate actions that have not passed the pre-existing spatial feasibility check or have not achieved the task objective after execution.

3. The method according to claim 2, characterized in that, Upon finding a positive example of spatial experience, instructing the robot to perform subsequent operational tasks based on the updated task-related spatial state and the structured spatial experience includes: Candidate actions corresponding to the positive example space experience are used first to perform subsequent operation tasks.

4. The method according to claim 2, characterized in that, In the event of a negative example space experience being matched, the robot is instructed to perform subsequent operation tasks according to the target strategy based on the updated task-related space state and the structured space experience, including: not using or avoiding the use of candidate actions corresponding to the negative example space experience, and avoiding the robot from repeatedly executing invalid actions.

5. The method according to claim 1, characterized in that, The step of updating the task-related spatial state based on the post-execution spatial feedback includes: Update the spatial relationships between the target object, related objects, and regions; update the availability of the operable region, temporary region, and alternative region; update the current validity of the candidate action; update whether the current operation task stage has progressed; and update whether the spatial constraints have been met or not yet met.

6. The method according to any one of claims 1-5, characterized in that, The candidate action includes at least one of the following: Directly capture the target object; After adjusting the grab position, the target object is grabbed. Move the obstacle object first, then perform the grab; First, move the relevant objects to the temporary storage area; Place the target object into the target area; Place the target object into the alternative area; Perform the recovery action first, then re-execute the target action.

7. The method according to any one of claims 1-5, characterized in that, The preliminary space feasibility verification includes at least one of the following: The system includes verification of reachability, collision risk of moving path, capacity of target area, stability of placement support, attitude consistency, occlusion relationship, availability of temporary storage area, and consistency of target area state after action execution.

8. The method according to any one of claims 1-5, characterized in that, The post-execution spatial feedback includes at least the following: changes in object pose, changes in spatial relationships between objects, changes in contact relationships between the robot actuator and the object, whether the target area is still occupied, whether the temporary storage area is still available, whether the occlusion relationship has changed, whether the current task objective has been achieved, and whether a new collision, slippage, offset, or obstruction has occurred.

9. A robot operation execution system, characterized in that, include: The task input and scene perception unit is used to receive the robot's current operation task and collect the scene perception information corresponding to the current operation task. The task-related spatial state construction unit is used to extract the target object, related objects, target area, temporary storage area, obstacle area, operable area and spatial constraint relationship specific to the current operation task, so as to construct the task-related spatial state; The candidate action generation unit is used to generate one or more candidate actions based on the task-related spatial state, and to perform a preliminary spatial feasibility check on each candidate action, outputting structured verification information, wherein the structured verification information includes verification items, verification pass / fail results, unmet spatial constraints, corresponding failure reasons, whether the current candidate action is executable, and action processing suggestions for the current candidate action; The spatial feasibility screening unit is used to screen candidate actions that have passed verification or can be executed after parameter adjustment and send them to the robot for execution, and to collect post-execution spatial feedback after the action is completed; wherein, the post-execution spatial feedback indicates the spatial feedback that can be used for subsequent operation tasks; A spatial experience construction unit is used to update the task-related spatial state based on the post-execution spatial feedback, and bind the structured verification information with the post-execution spatial feedback to form structured spatial experience; The subsequent decision-making unit is used to instruct the robot to perform subsequent operation tasks based on the updated task-related spatial state and the structured spatial experience.

10. A robot, characterized in that, The robot has a built-in robot operation execution system as described in claim 9 to execute the robot operation execution method as described in any one of claims 1-7.

Images