Robot operation method, robot, medium, apparatus and program product

Abstract

The invention provides a robot operation method, a robot, a medium, equipment and a program product, and the method comprises the steps: carrying out task planning through employing target task information and visual input information, so as to determine a subtask and a target object of at least one stage, and the target object of the stage comprises an active object and a passive object; for at least one stage, performing spatial constraint planning according to the subtasks of the stage and the standard interaction primitive of the target object to obtain spatial constraint information based on the standard interaction primitive; and controlling operation of an end effector of the robot according to the subtasks of the stages and the spatial constraint information. Spatial constraint planning can be carried out by using the standard interaction primitive of the object, and the universality of robot operation control is improved.

Description

Technical Field

[0001] The present application relates to the field of robot operation technology, and in particular to a robot operation method, a robot, a medium, a device and a program product. Background Art

[0002] Due to the complexity and diversity of real-world environments, developing a general-purpose robot operating system has always been a challenging task. Inspired by the rapid development of Large Language Models (LLM) and Vision Language Models (VLM), researchers have used massive Internet data to obtain rich common sense knowledge and explored its application in the field of robotics. Most current research focuses on using this knowledge for high-level task planning, but lacks the ability to understand space, which is crucial for fine-grained, low-level manipulation tasks, resulting in low versatility in robot manipulation control.

[0003] Based on this, the present application provides a robot operation method, a robot, a medium, a device and a program product to improve the relevant technology. Summary of the invention

[0004] The purpose of this application is to provide a robot operation method, a robot, a medium, a device and a program product to enhance the versatility of robot operation control.

[0005] The purpose of this application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a robot operation method, the method comprising: performing task planning using target task information and visual input information to determine subtasks and target objects of at least one stage, wherein the target objects of the stage include active objects and passive objects; for at least one stage, performing spatial constraint planning according to the canonical interaction primitives of the subtasks and target objects of the stage to obtain spatial constraint information based on the canonical interaction primitives; and controlling the operation of the robot's end effector according to the subtasks and spatial constraint information of the stage.

[0007] In some embodiments, the active object and the passive object corresponding to the same stage meet the following conditions: the passive object is one of at least one task object; the active object is the end effector, or the active object is one of the at least one task object and is different from the passive object.

[0008] In some embodiments, the process of acquiring the at least one task object includes: processing the visual input information to mark at least one input object; and screening out at least one task object corresponding to the target task information from the at least one input object.

[0009] In some embodiments, the process of acquiring the canonical interaction primitives of the target object includes: extracting the canonical interaction primitives using the subtasks of the stage and the canonical representation information of the target object to obtain the canonical interaction primitives of the target object; the canonical representation information includes a three-dimensional model and posture.

[0010] In some embodiments, the three-dimensional model of the target object is obtained by reconstructing the three-dimensional structure of the target object using the visual input information, and / or the posture of the target object is obtained by estimating the posture of the target object using the visual input information.

[0011] In some embodiments, the canonical interaction primitives of the target object include an interaction point and / or an interaction direction.

[0012] In some embodiments, the interaction point includes at least one of: a first interaction point that is visible and touchable; and a second interaction point that is invisible or untouchable.

[0013] In some embodiments, the process of extracting the interaction point includes: superimposing a Cartesian network on a target image containing the target object to obtain a superimposed image; the target image is determined based on the visual input information; and according to the subtasks of the stage, the interaction point is located using the superimposed image.

[0014] In some embodiments, the target image corresponding to the first interaction point includes an input image, which is obtained by processing the visual input information; and / or, the target image corresponding to the first interaction point includes at least one view of the three-dimensional model of the target object, and the at least one view includes at least one of the six views.

[0015] In some embodiments, locating the interaction point using the superimposed image according to the subtask of the stage includes: locating the interaction point using the superimposed image according to the task type of the subtask of the stage and the object type of the target object; the object types include active and passive.

[0016] In some embodiments, the interaction point extraction process includes: extracting standardized interaction primitives based on the standardized representation information of the target object to obtain multiple candidate interaction points; processing the multiple candidate interaction points according to the subtasks of the stage to generate an interaction point heat map; and using the interaction point heat map to determine at least one interaction point from the multiple candidate interaction points.

[0017] In some embodiments, the interaction direction extraction process includes: extracting at least one candidate interaction direction of the target object based on the normalized representation information of the target object; generating semantic description information corresponding to the candidate interaction direction for one or more candidate interaction directions, and calculating the relevance score between the semantic description information and the subtasks of the stage; and sorting the corresponding candidate interaction directions according to the relevance score to determine at least one interaction direction.

[0018] In some embodiments, the spatial constraint planning is performed based on the canonical interaction primitives of the subtasks of the stage and the target object to obtain spatial constraint information based on the canonical interaction primitives, including: performing spatial constraint planning based on the canonical interaction primitives of the subtasks of the stage and the target object to obtain at least one unverified constraint information, the at least one unverified constraint information including distance constraint information and / or angle constraint information; for one or more unverified constraint information, verifying the unverified constraint information to obtain verification results corresponding to the unverified constraint information; when the verification results corresponding to the unverified constraint information are successful, including the unverified constraint information in the spatial constraint information.

[0019] In some embodiments, verifying the unverified constraint information to obtain a verification result corresponding to the unverified constraint information includes: rendering an interactive image corresponding to the unverified constraint information; and verifying the interactive image to obtain a verification result corresponding to the unverified constraint information.

[0020] In some embodiments, the method further includes: when the verification result corresponding to the unverified constraint information is failure, verifying the next unverified constraint information; or, when the verification result corresponding to the unverified constraint information is optimization, resampling based on the current standard interaction primitive to achieve adjustment of the standard interaction primitive, and re-performing spatial constraint planning based on the adjusted standard interaction primitive to obtain at least one unverified constraint information.

[0021] In some embodiments, controlling the operation of the robot's end effector according to the subtasks and spatial constraint information of the stage includes: calculating the loss value of the target loss function according to the subtasks and spatial constraint information of the stage to determine the target posture of the end effector that meets the target optimization condition; the target loss function includes one or more loss terms among constraint loss terms, collision loss terms and path loss terms, and the loss value of at least one loss term is calculated according to the posture of the end effector; performing trajectory planning on the end effector using the target posture to obtain the trajectory information of the end effector; the trajectory information is used to control the operation of the end effector.

[0022] In some embodiments, the constraint loss term is calculated based on the posture of the target object at the stage, and the posture of the active object is determined based on the posture of the end effector. The method also includes: performing posture tracking on the target object at the stage to update the posture of the target object; updating the loss value of at least one loss term in the target loss function based on the updated posture of the target object to achieve the update of the loss value of the target loss function; and re-determining the target posture of the end effector that meets the target optimization condition based on the updated loss value of the target loss function.

[0023] In a second aspect, the present application provides a robot operating system, the system comprising: a task planning module, used to perform task planning using target task information and visual input information to determine subtasks and target objects of at least one stage, wherein the target objects of the stage include active objects and passive objects; a constraint planning module, used to perform spatial constraint planning for at least one stage according to the subtasks and target objects of the stage according to the canonical interaction primitives to obtain spatial constraint information based on the canonical interaction primitives; an operation control module, used to control the operation of the robot's end effector according to the subtasks and spatial constraint information of the stage for at least one stage.

[0024] In a third aspect, the present application provides a robot, comprising a robot operating system and an end effector, wherein the robot operating system is used to execute any one of the above methods to control the operation of the end effector.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements any of the above methods when executed by a processor.

[0026] In a fifth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0027] In a sixth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program implements any one of the above methods when executed by a processor.

[0028] The present application provides a robot operation method, a robot, a medium, a device and a program product, the method comprising: performing task planning using target task information and visual input information to determine at least one stage of subtasks and target objects, the target objects of the stage including active objects and passive objects; for at least one stage, performing spatial constraint planning according to the standard interaction primitives of the subtasks and target objects of the stage to obtain spatial constraint information based on the standard interaction primitives; and controlling the operation of the robot's end effector according to the subtasks and spatial constraint information of the stage. The present application can use the standard interaction primitives of objects to perform spatial constraint planning, thereby improving the versatility of robot operation control. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present application is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0030] Figure 1a It is a schematic diagram of the framework of a robot operating system provided in an embodiment of the present application.

[0031] Figure 1b It is a flow chart of a robot operation method provided in an embodiment of the present application.

[0032] Figure 2 This is a schematic diagram of a process for generating an interaction point provided in an embodiment of the present application.

[0033] Figure 3 This is a schematic diagram of an interaction direction extraction process provided in an embodiment of the present application.

[0034] Figure 4 It is a schematic diagram of stability analysis of an interaction primitive provided in an embodiment of the present application.

[0035] Figure 5 It is a qualitative analysis diagram of the impact of viewing angle on performance provided in an embodiment of the present application.

[0036] Figure 6 It is a closed-loop planning schematic diagram provided in an embodiment of the present application.

[0037] Figure 7 These are two typical failure cases without closed-loop execution provided in the embodiments of the present application.

[0038] Figure 8 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0039] Fig. 9 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the implementation mode of this application. Obviously, the described implementation mode is only a part of the implementation mode of this application, not all the implementation modes. Based on the implementation mode in this application, all other implementation modes obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0041] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0042] Developing a general-purpose robot operating system has always been a challenging task, mainly due to the complexity and diversity of real-world environments. Inspired by the rapid development of large language models (LLMs) and visual language models (VLMs), researchers have recently begun to explore their applications in robotics by leveraging rich commonsense knowledge from massive Internet data. Most current research focuses on leveraging this knowledge for high-level task planning, such as semantic reasoning. Despite these advances, current VLMs are mainly trained on large-scale 2D visual data and lack the ability to understand precise 3D spaces, which is critical for fine-grained, low-level manipulation tasks. As a result, they still perform poorly in fine-grained manipulation tasks in unstructured environments.

[0043] One way to overcome this limitation is to convert the VLM into a VLA (Vision-Language-Action) model by fine-tuning it on large-scale robotics datasets. However, this approach faces two major challenges. First, it is costly and time-consuming to obtain diverse, high-quality robotics data. Second, fine-tuning a VLM to a VLA generates representations that are specific to certain robots and have limited generalizability. A promising alternative is to abstract robot actions into interaction primitives (e.g., points or vectors) and use VLMs for reasoning to define the spatial constraints of these primitives, while the associated planning algorithm is responsible for execution. However, related methods have several limitations in defining and using primitives: the process of generating primitive proposals is task-independent, which may lead to a lack of suitable proposals. In addition, relying on hand-designed rules to post-process the proposals can also introduce instabilities.

[0044] In the application of visual language models (VLMs), the basic models in the field of robotics have performed well in environmental understanding and high-level common sense reasoning, demonstrating the potential for controlling robots to perform general tasks in novel and unstructured environments. In some embodiments, a visual language action (VLA) model capable of outputting robot trajectories is created by fine-tuning the VLM on a robotics dataset, but the visual language action model in these embodiments is limited by the high cost of data acquisition and insufficient generalization. In some embodiments, operation primitives can be extracted through a visual base model, and then these primitives are used as visual or language cues for the VLM for high-level common sense reasoning and combined with a motion planner for low-level control. However, in these embodiments, there are two main limitations: one is the ambiguity of compressing 3D primitives into 2D images or 1D texts required by the VLM; the other is the hallucination tendency of the VLM itself. These limitations make it difficult to ensure that the high-level plans generated by the VLM are accurate.

[0045] Some related techniques reveal the unique advantages of open vocabulary manipulation models (e.g., OmniManip model) in addressing these challenges, especially in fine-grained 3D (Three Dimensions) understanding and mitigating large model hallucinations. The structured representation used for manipulation determines the power and effectiveness of the manipulation method. Among various representation methods, keypoint representation methods have become a popular choice due to their flexibility, generalization ability, and ability to model diversity. However, these keypoint-based methods require manual task-specific annotations to generate actions. To achieve zero-shot open-world manipulation, keypoints can be converted into visual cues for visual language models (VLMs), thereby facilitating the automatic generation of high-level planning results. However, despite their advantages, keypoints are also susceptible to occlusions and there are challenges in the extraction and selection of specific keypoints.

[0046] Some related techniques use a rigid pose representation, which can efficiently define long-range dependencies between objects and provide a certain degree of robustness in the case of occlusion. However, these related techniques require pre-modeling of geometric relationships and are difficult to provide fine-grained geometric information due to the sparsity of poses. This limitation may cause the operation strategy across different objects to fail, especially when there is large intra-class variation.

[0047] See also Figure 1a and Figure 1b , Figure 1a is a schematic diagram of a robot operating system framework provided in an embodiment of the present application. Figure 1b It is a flow chart of a robot operation method provided in an embodiment of the present application.

[0048] In order to improve the relevant technology, an embodiment of the present application provides a robot operation method, which includes steps S101 to S102.

[0049] Step S101: performing task planning using target task information and visual input information to determine subtasks and target objects of at least one stage, wherein the target objects of the stage include active objects and passive objects.

[0050] Step S102: For at least one stage, perform spatial constraint planning according to the subtasks of the stage and the standard interaction primitives of the target object to obtain spatial constraint information based on the standard interaction primitives; and control the operation of the end effector of the robot according to the subtasks of the stage and the spatial constraint information.

[0051] In some embodiments, the method can be performed on a robot operating system. As an example, the robot operating system can be integrated on a robot.

[0052] For each stage, in addition to determining the subtasks and target objects, actions can also be determined. In some embodiments, performing task planning using the target task information and visual input information to determine the subtasks and target objects of at least one stage can include: performing task planning using the target task information and visual input information to determine the subtasks, actions, and target objects of at least one stage.

[0053] In the above embodiment, the target object corresponding to the same stage includes two objects, namely, an active object and a passive object. In some embodiments, the active object and the passive object corresponding to the same stage can meet the following conditions: the passive object is one of at least one task object; the active object is the end effector, or the active object is one of the at least one task object and is different from the passive object. In other words, the active object may be an end effector or a task object, but the passive object can only be a task object. Among them, the task object is a collection of objects related to the task.

[0054] The above embodiments do not limit the visual input information. In some embodiments, the visual input information may include one or more input images. The input images may include, for example, RGB color images, grayscale images, depth images, and the like.

[0055] In some embodiments, the acquisition process of the at least one task object may include: processing the visual input information to mark at least one input object; and filtering at least one task object corresponding to the target task information from the at least one input object. Since the task object is obtained by filtering the input object, the task object can be regarded as a subset of the input object.

[0056] In some embodiments, the processing of the visual input information to mark at least one input object may include: processing the visual input information using a visual basis model to obtain visual prompt information, wherein the visual prompt information is used to mark at least one input object.

[0057] The above embodiment does not limit the visual basic model, which may include GroundingDINO (a target detection model), SAM (Segment Anything Model, a segmentation model), etc. As an example, GroundingDINO can mark the foreground objects and their positions in the scene, and SAM can mark the corresponding masks (which can be regarded as segmentation results) of the foreground objects.

[0058] In some embodiments, the filtering out at least one task object corresponding to the target task information from the at least one input object may include: using a visual language model to process the visual prompt information and RGB-D observation information of the at least one input object to filter out at least one task object corresponding to the target task information. Among them, RGB (Red-Green-Blue) in RGB-D represents color information, and D (Depth) in RGB-D represents depth information. The RGB-D observation information of the input object can be observed using an RGB-D camera.

[0059] For example, visual input information can be input into a visual base model, and the visual base model is used to process the visual input information to obtain visual prompt information. As an example, the visual prompt information may include a labeling instruction, and the labeling instruction is used to label one or more foreground objects (these labeled foreground objects can be used as input objects). The labeling instruction and RGB-D observation information of the visual base model (VFM) are input into the VLM visual language model, and the visual language model is used to screen out task objects related to the task, and the task is divided into one or more stages.

[0060] For example, Figure 1a As shown, assume that an operation task T is given (for example, pouring tea into a cup), and assume that the visual input information includes an input image containing a teapot and a cup. First, after receiving the target task information corresponding to the operation task, the operation task can be decomposed. As an example, the two visual basis models (VFMs) GroundingDINO and SAM can be used to process the input image, annotate all foreground objects in the scene, and obtain visual cue information. Then, the target task information and visual cue information are input into the VLM, and the VLM is used to screen objects related to the task (ie, task objects), and the task is decomposed into one or more stages S = {S1, S2, ..., S n}, n is a positive integer. Each stage S i It can be formalized as Where i represents the stage number, i is a positive integer, A i indicates an action to be performed (e.g., grabbing, pouring liquid), while and They represent the object that initiates the interaction (called the active object) and the object that is manipulated (called the passive object). It can be seen that in the first stage of grasping the teapot, the subtask is, for example, grasping the handle of the teapot with a gripper, the action is, for example, grasping, the active object is, for example, the end effector of the robot (for example, the gripper), and the passive object is, for example, the teapot. In the second stage of pouring tea into a cup, the subtask is, for example, pouring tea from a teapot into a cup, the action is, for example, pouring, the active object is, for example, the teapot, and the passive object is, for example, the cup. It can be seen that the teapot is a passive object in the first stage and an active object in the second stage. In other words, in the first stage, the object type of the teapot is passive; in the second stage, the object type of the teapot is active.

[0061] In some embodiments, the process of acquiring the canonical interaction primitives of the target object may include: extracting the canonical interaction primitives using the subtasks of the stage and the canonical representation information of the target object to obtain the canonical interaction primitives of the target object; the canonical representation information includes a three-dimensional model and posture.

[0062] In some embodiments, the three-dimensional model of the target object can be obtained by reconstructing the three-dimensional structure of the target object using the visual input information. As an example, a 3D generation model such as a single-view 3D generation network can be used to process the visual input information to generate an object mesh (e.g., a three-dimensional model) of the target object. The three-dimensional model can, for example, adopt a 3D mesh model. Among them, the 3D generation model can adopt a 3D AIGC model. AIGC (Artificial Intelligence Generated Content) refers to generative artificial intelligence.

[0063] In some embodiments, the pose of the target object can be obtained by estimating the pose of the target object using the visual input information. As an example, a 6D pose estimation model can be used to process the visual input information to estimate the pose of the target object. Among them, 6D refers to six dimensions, and the 6D pose can, for example, include three-dimensional position information and three-dimensional pose information. In other words, a general 6D pose estimation model can be used to normalize the object and describe the rigid body transformation of the object during the interaction process.

[0064] In some embodiments, the standard interaction primitives of the target object may include interaction points and / or interaction directions. The standard interaction primitives, for example, refer to interaction primitives in a standard space.

[0065] The above embodiments implement object-centric canonical interaction primitives. Specifically, the above embodiments propose a novel object-centric representation method, which uses canonical interaction primitives to describe the interaction mode of objects in operation tasks. For example, the interaction primitives of an object (e.g., a target object, including active objects and passive objects) can be represented by the interaction points and interaction directions in its canonical space. As an example, the interaction point p∈R 3 For example, it can represent the key location where the interaction occurs on the object, and the interaction direction v∈R 3 For example, it can represent the main axis related to the task. Together, they form the interaction primitive O = {p, v}, which encapsulates the basic geometric and functional properties required to meet the task constraints. 3 Refers to a three-dimensional space. Since these canonical interaction primitives are defined relative to the canonical space of objects, they remain consistent across different scenarios, making the operation strategies more general and reusable.

[0066] In some embodiments, the interaction point may include at least one of the following: a first interaction point that is visible and touchable; and a second interaction point that is invisible or untouchable. Wherein, touchable may be regarded as physically touchable, and untouchable may be regarded as physically untouchable.

[0067] In some embodiments, the process of extracting the interaction points may include: superimposing a Cartesian network on a target image containing the target object to obtain a superimposed image; according to the subtasks of the stage, locating the interaction points using the superimposed image; and determining the target image based on the visual input information.

[0068] In some embodiments, locating the interaction point using the superimposed image according to the subtask of the stage may include: locating the interaction point using the superimposed image according to the task type of the subtask of the stage and the object type of the target object; the object type includes, for example, active and passive.

[0069] For example, assume that the target object is a teapot, and assume that the task type includes one or more of a grasping task, a pouring task, and a placing task. When the task type is a grasping task and the object type is passive (i.e., the teapot is a passive object), the handle of the teapot can be located as the interaction point; when the task type is a pouring task and the object type is active (i.e., the teapot is an active object), the spout can be located as the interaction point; when the task type is a pouring task or a placing task, and the object type is passive (i.e., the teapot is a passive object), the center of the teapot opening can be located as the interaction point. It can be seen that for the same object (e.g., a teapot), even if the task type is the same (e.g., a pouring task), if the object type is different, the extracted interaction points may be different. In the above example, when the teapot is an active object, the located interaction point is the spout; when the teapot is a passive object, the located interaction point is the center of the teapot opening.

[0070] In some embodiments, the target image corresponding to the first interaction point may include an input image; the input image is obtained by processing the visual input information. Assuming that the visual input information is in the form of an image, as an example, the image corresponding to the visual input information may be cropped to obtain the input image. Next, a Cartesian network may be superimposed on the input image to obtain a superimposed image; according to the subtask of the stage, the first interaction point is located in the image plane corresponding to the superimposed image.

[0071] In some embodiments, the target image corresponding to the second interaction point includes, for example, at least one view of the three-dimensional model of the target object. In some embodiments, the at least one view may include at least one of six views. Among them, the six views include, for example, a front view, a rear view, a left view, a right view, a bottom view, and the like. As an example, views of the three-dimensional model from different perspectives can be obtained by rendering the three-dimensional model. Next, a Cartesian network can be superimposed on at least one view to obtain a superimposed image; according to the subtasks of the stage, the second interaction point is located in the image plane corresponding to the superimposed image.

[0072] The following example illustrates the extraction process of the first interaction point and the second interaction point. Figure 1a As shown, first, we can use a single view Figure 3 The 3D generative network (as an example of a 3D generative model) obtains the 3D mesh models of the active and passive objects related to the task, and then uses Omni6DPose (as an example of a 6D pose estimation model) to estimate the pose of the object and achieve a canonical representation. Next, the task-related canonical interaction primitives can be extracted.

[0073] See also Figure 2 , Figure 2 is a schematic diagram of a generation process of an interaction point provided in an embodiment of the present application. In order to realize the positioning of the interaction point, Figure 2 As shown, the interaction points can be divided into two categories: a first interaction point that is visible and touchable (e.g., the handle of a teapot); and a second interaction point that is invisible or untouchable (e.g., the center of the opening of a teapot). The above embodiment can use the SCAFFOLD visual cue mechanism to enhance the VLM's ability to locate the interaction points. As an example, the mechanism can overlay a Cartesian grid on the input image, and the corresponding two-dimensional coordinates (e.g., (x1, y1), (x1, y2), (x2, y1), (x2, y2)) are as follows Figure 2 As shown. The first interaction point can, for example, be located directly in the image plane, while the second interaction point can, for example, be inferred through multi-view reasoning based on a canonical object representation. As an example, the reasoning can start from the main perspective, and if there is ambiguity, switch to the orthogonal perspective to deal with the problem. The above embodiment uses different positioning methods to locate two types of interaction points, the first interaction point and the second interaction point, which can make the positioning of the interaction points more flexible and reliable.

[0074] In some embodiments, the interaction point extraction process may include: extracting a standardized interaction primitive based on the standardized representation information of the target object to obtain a plurality of candidate interaction points; processing the plurality of candidate interaction points according to the subtasks of the stage to generate an interaction point heat map; and using the interaction point heat map to determine at least one interaction point from the plurality of candidate interaction points. The advantage of this is that by generating a heat map through a plurality of candidate interaction points and then using the interaction point heat map to determine the interaction point, the robustness of interaction point positioning can be improved.

[0075] In practical applications, the interaction point heat map can be drawn in combination with the task type and the object type to achieve accurate interaction point positioning. In some embodiments, the processing of multiple candidate interaction points according to the subtasks of the stage to generate the interaction point heat map may include: processing multiple candidate interaction points according to the task type of the subtask and the object type of the target object to generate the interaction point heat map. As an example, assuming that the target object is a teapot, when the task type is grasping and the object type is passive, the interaction point heat map can be as follows: Figure 2 As shown in the figure, the handle is highlighted in the interaction point heat map.

[0076] In the canonical space, the main axis of an object is mostly related to its function. In some embodiments, the process of extracting the interaction direction may include: extracting at least one candidate interaction direction of the target object based on the normalized representation information of the target object; for one or more candidate interaction directions, generating semantic description information corresponding to the candidate interaction direction, and calculating the relevance score between the semantic description information and the subtasks of the stage; sorting the corresponding candidate interaction directions according to the relevance score to determine at least one interaction direction. Among them, the relevance score can be expressed in the form of a ten-point system, a hundred-point system or a percentage, for example, which is not limited to the above embodiment. In addition to the relevance score, the evaluation and sorting can also be carried out in the form of relevance level, which is not limited to the above embodiment.

[0077] In order to efficiently obtain semantic description information, in some embodiments, generating the semantic description information corresponding to the candidate interaction direction may include: using a visual language model to process the candidate interaction direction to generate the corresponding semantic description information.

[0078] In some embodiments, the calculating of the relevance score between the semantic description information and the subtasks of the stage may include: using an LLM large language model to process the semantic description information and the subtask description information of the stage to obtain the relevance score between the semantic description information and the subtasks of the stage. The subtask description information of the stage is, for example, information used to describe the subtasks of the stage, which may be obtained when determining the subtasks of the stage.

[0079] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an extraction process of an interaction direction provided in an embodiment of the present application. In the figure, Stage is a stage. Figure 3 As shown, for the teapot, the three main axes in the figure can be regarded as candidate interaction directions (at this time, the three main axes can be called candidate axes). Due to the limitations of the current VLM's ability to understand space, it is still challenging to evaluate the relevance of these directions to the task. To this end, the above embodiment proposes a mechanism that combines VLM to generate descriptions and LLM scoring. First, VLM is used to generate semantic description information for each candidate interaction direction. As an example, the semantic description information of the first candidate axis is, for example: the axis passes through the center of the teapot, extends outward from the side, and is perpendicular to the spout outlet; the semantic description information of the second candidate axis is, for example: the axis is arranged vertically through the teapot, from the bottom to the top, and is perpendicular to the spout outlet; the semantic description information of the third candidate axis is, for example: the axis is horizontally aligned, passes through the center of the teapot, and extends outward from the spout outlet. Then, LLM is used to infer and evaluate the relevance of these semantic description information to the task, and a relevance score for each semantic description information is obtained. Through this process, a set of candidate interaction directions sorted by task requirements can be generated. In Figure 3 In the example, the ranking results from high to low according to the relevance score are: the third candidate axis, the second candidate axis, and the first candidate axis. According to the ranking results, at least one interaction direction may be determined, for example, the direction where the third candidate axis is located.

[0080] In some embodiments, the spatial constraint planning is performed based on the canonical interaction primitives of the subtasks of the stage and the target object to obtain spatial constraint information based on the canonical interaction primitives, which may include: performing spatial constraint planning based on the canonical interaction primitives of the subtasks of the stage and the target object to obtain at least one unverified constraint information, wherein the at least one unverified constraint information includes distance constraint information and / or angle constraint information; for one or more unverified constraint information, verifying the unverified constraint information to obtain verification results corresponding to the unverified constraint information; and when the verification results corresponding to the unverified constraint information are successful, including the unverified constraint information in the spatial constraint information.

[0081] In practical applications, assuming that verification results can be divided into three categories: success, failure, and optimization, targeted treatment can be carried out for the two types of verification results other than success.

[0082] In some embodiments, the method may further include: verifying the next unverified constraint information when the verification result corresponding to the unverified constraint information is failure.

[0083] In some embodiments, the method may further include: when the verification result corresponding to the unverified constraint information is optimized, resampling is performed based on the current standard interaction primitive to adjust the standard interaction primitive, and re-planning the spatial constraints based on the adjusted standard interaction primitive to obtain at least one unverified constraint information. For the unverified constraint information whose verification result is optimized, the standard interaction primitives may be resampled and the spatial constraints may be replanned, and after obtaining the new unverified constraint information, the unverified constraint information may be verified.

[0084] In some embodiments, the verifying the unverified constraint information to obtain the verification result corresponding to the unverified constraint information may include: rendering an interactive image corresponding to the unverified constraint information; and verifying the interactive image to obtain the verification result corresponding to the unverified constraint information. In some embodiments, the verifying the interactive image to obtain the verification result corresponding to the unverified constraint information includes: verifying the interactive image using a visual language model to obtain the verification result corresponding to the unverified constraint information.

[0085] In the above embodiment, the space constraint information is defined using the specification interaction primitive. i In this case, we can use a set of spatial constraints C i To regulate the spatial relationship between the active object and the passive object. As an example, the spatial constraint may include a distance constraint and / or an angle constraint. i Used to adjust the distance between interaction points. Angle constraint θ i These spatial constraints together define the geometric rules required for precise spatial alignment and task execution. i The overall spatial constraint of can be expressed as formula (1), for example.

[0086]

[0087] In the above embodiment, the space constraint information is defined using the specification interaction primitive. As an example, for each stage S i Generate an ordered list of constraint interaction primitives, denoted as Wherein, N is a positive integer. The above embodiment extracts the standard interaction primitives of the active object and the passive object, which are respectively denoted as O active and O passive , and the spatial constraints C that define their spatial relationship. However, this process belongs to open-loop reasoning, which inherently limits the robustness and adaptability of the system. Its main limitations come from the hallucination effects of large models and the dynamic characteristics of the real environment.

[0088] To overcome these challenges, Figure 1a As shown, the embodiment of the present application proposes a dual closed-loop system. The dual closed-loop includes, for example, closed-loop planning and closed-loop execution. Closed-loop planning corresponds to the process of constructing spatial constraint information, and closed-loop execution corresponds to the process of controlling the robot operation. The following first introduces the closed-loop planning process, and then the closed-loop execution process.

[0089] The above embodiments can realize closed-loop planning. Specifically, in order to improve the accuracy of the canonical interaction primitives and alleviate the illusion problem of VLM, the above embodiments propose a self-correction mechanism (RRC) based on resampling, rendering and checking. The mechanism uses timely feedback from the visual language model (VLM) to detect and correct interaction errors, thereby ensuring the accurate execution of the task. The RRC process can be divided into two stages: the initial stage and the optimization stage.

[0090] In the initial stage, the interaction constraints K defined above can be i is regarded as unverified constraint information, which specifies the spatial relationship between active objects and passive objects. For each unverified constraint C (k) , can render interactive images according to the current configuration I i , and transform it (i.e., the interaction image I i ) is submitted to VLM for verification. VLM returns one of the following three results: success, failure, or optimization. As an example, if it returns success, the constraint is accepted and the next constraint is evaluated (until all unverified constraint information is verified); if it returns failure, the next constraint is evaluated (until all unverified constraint information is verified); if it returns optimization, it enters the optimization phase for further adjustment. Taking the interaction direction in the specification interaction primitive as an example, in the optimization phase, the predicted interaction direction v i Fine resampling is performed around the object to correct the misalignment between the functional axis and the geometric axis. For example, in v i Resample one or more optimization directions around v (j) Wherein, j is a positive integer, and the number of optimization directions can be 1, 2, 4, 6, etc., which is not limited in the above embodiment. As an example, when the number of optimization directions is greater than 1, a uniform sampling method can be used.

[0091] The above introduces the closed-loop planning process, and the following introduces the closed-loop execution process.

[0092] In some embodiments, controlling the operation of the robot's end effector according to the subtasks and spatial constraint information of the stage may include: calculating the loss value of the target loss function according to the subtasks and spatial constraint information of the stage to determine the target posture of the end effector that meets the target optimization conditions; the target loss function includes one or more loss terms among constraint loss terms, collision loss terms and path loss terms, and the loss value of at least one loss term is calculated according to the posture of the end effector; performing trajectory planning on the end effector using the target posture to obtain the trajectory information of the end effector; the trajectory information is used to control the operation of the end effector.

[0093] In some embodiments, the constraint loss term can be calculated based on the pose of the target object at the stage, the pose of the active object can be determined based on the pose of the end effector, and the method can also include: tracking the pose of the target object at the stage to update the pose of the target object; updating the loss value of at least one loss term in the target loss function based on the updated pose of the target object to update the loss value of the target loss function; and re-determining the target pose of the end effector that meets the target optimization condition based on the updated loss value of the target loss function. Next, the re-determined target pose can be used to perform trajectory planning on the end effector to update the trajectory information of the end effector; the trajectory information is used to control the operation of the end effector, which will not be described in detail herein.

[0094] In some embodiments, tracking the pose of the target object at the stage to update the pose of the target object may include: using a 6D pose tracker to track the pose of the target object at the stage to update the pose of the target object.

[0095] In the above-mentioned embodiment, during the planning process of at least one stage, the object-centered canonical interaction primitives are extracted as spatial constraints in a closed-loop manner, and during the execution process of at least one stage, the trajectory of the end effector is optimized by the constraints, and the target pose is updated in the closed loop. As an example, in the closed-loop planning process of each stage, the VLM visual language model can be used to extract the object-centered canonical interaction primitives as spatial constraints in a closed-loop manner. During the closed-loop execution process of each stage, the trajectory of the end effector is optimized by the constraints and updated in the closed loop using the 6D pose tracker.

[0096] The above-mentioned embodiment combines the fine-grained geometric information of key points and the stability of the rigid body 6D posture. As an example, the interaction points and interaction directions in the object's canonical coordinate system (corresponding to the canonical space of the object) can be automatically extracted by VLM, so as to achieve precise operation. In the framework provided by the above-mentioned embodiment, complex robot tasks can be decomposed into multiple stages, each of which is defined by a canonical interaction primitive with a spatial constraint. Since these canonical interaction primitives are based on the canonical space of the object, they can be referred to as canonical interaction primitives. This structured approach can accurately define task requirements and promote the execution of complex operation tasks. The following will illustrate how canonical interaction primitives are used as the basis of spatial constraints to achieve robust operations.

[0097] Once the canonical interaction primitives and their corresponding spatial constraints are defined for each stage, task execution can be formalized as an optimization problem. As an example, the optimization objective can be to determine the target pose P of the end effector by minimizing the objective loss function ee* , the optimization problem can be expressed as formula (2).

[0098]

[0099] Among them, arg is the abbreviation of variable (i.e., independent variable argument), arg min is the value of the independent variable when the following expression reaches the minimum value, and j is the serial number of the loss term. It can be seen that the objective loss function L, for example, includes a constraint loss term (i.e., constraint loss L C ), collision loss term (i.e., collision loss L collision ) and the path loss term (i.e., path loss L path ), in which case N in formula (2) is, for example, 3. The loss value of at least one loss term can be calculated based on the posture P of the end effector. ee Calculated.

[0100] Among them, the constraint loss L C Ensure that the action complies with the spatial constraint C of the task, which can be defined as shown in formula (3).

[0101]

[0102] Among them, ρ(·) measures the active object P active and passive object P passive The deviation between the current spatial relation and the desired constraint C, Φ(·) maps the pose of the end effector to the pose of the active object, and the subscript t is the time.

[0103] Collision loss L collision Reduce the collision between the end effector and obstacles in the environment. Its definition can be shown in formula (4).

[0104]

[0105] Among them, d(p ee , O j ) represents the distance between the end effector and the obstacle O j The distance between min is the minimum safety distance allowed. min For example, it can be selected or set according to the needs of actual application.

[0106] Path loss L p To ensure smooth motion, its definition can be shown in formula (5).

[0107]

[0108] Among them, d trans (·) and d rot (·) represent the translation and rotation displacement of the end effector, respectively, and λ1 and λ2 are weight factors to balance the translation and rotation effects.

[0109] By minimizing the objective loss function, the target pose P of the end effector can be dynamically adjusted ee* , while reducing collisions and maintaining smooth motion to ensure successful execution of the task. Although formula (3) outlines how to optimize the target pose P of the end effector using the canonical interaction primitives and their corresponding spatial constraint information ee* , but real-world task execution often involves significant dynamic factors. For example, in grasping tasks, deviations in grasping posture may cause unexpected movement of the object during grasping. In addition, in some dynamic environments, the target object may be displaced. These challenges highlight the key role of closed-loop execution in coping with uncertainty.

[0110] In order to cope with the uncertainty in the real world, the above embodiments not only use object-centric canonical interaction primitives, but also adopt a pose tracking algorithm (e.g., a 6D pose tracking algorithm) to timely update the pose P of the active object. active and the pose P of the passive object passive This timely feedback allows dynamic adjustment of the target pose of the end-effector, resulting in robust and precise closed-loop execution.

[0111] In a specific application scenario, the relevant experiments of the above embodiment are carried out. The following first introduces the experimental settings.

[0112] As an example, the hardware configuration is as follows. The experimental platform is based on a Franka Emika Panda robot arm, and the fingers of its parallel gripper are replaced with UMI fingers. In terms of perception, two Intel RealSense D415 depth cameras are used. One camera is installed on the gripper to provide a first-person perspective of the manipulation area; the other camera is placed opposite the robot to provide a third-person perspective of the workspace.

[0113] As an example, 12 tasks are designed to evaluate the model's manipulation capabilities in real scenes. Six of the tasks involve the manipulation of rigid objects (e.g., pouring tea), and the other six tasks focus on the manipulation of articulated objects (e.g., opening a drawer). These tasks cover a diverse set of objects and are designed to evaluate the model's ability to generalize and adapt in complex environments. For each task, 10 experiments are performed for each method, and the success rate is recorded. After each experiment, the object layout can be reconfigured to ensure the robustness of the evaluation.

[0114] As an example, the method provided in an embodiment of the present application (i.e., the OmniManip method) is compared with the following three baseline (track) methods. The first method is the VoxPoser method, which uses a large language model (LLM) and a visual language model (VLM) to generate a three-dimensional value map to synthesize robot trajectories, and is good at zero-sample learning and closed-loop control. The second method is the CoPa method, which introduces spatial constraints on object parts and combines VLM to achieve open vocabulary manipulation. The third method is the ReKep method, which uses relational key point constraints and hierarchical optimization to generate real-time actions from natural language instructions.

[0115] As an example, we can use GPT-4o from the OpenAI API as a visual language model, and use a set of interaction examples as prompts to guide the model's reasoning in manipulation tasks. As an example, we use the related model to implement 6-DOF general grasping, and use GenPose++ (as an example of a pose estimation model) to implement general 6D pose estimation.

[0116] In the above experiments, OmniManip was comprehensively evaluated on 12 open-vocabulary manipulation tasks, ranging from simple grasping and placing operations to more complex tasks such as object-object interaction with orientation constraints and articulated object manipulation. As shown in Table 1, the OmniManip method demonstrates strong zero-shot generalization and excellent overall performance without the need for task-specific training. This generalization capability benefits from the common sense knowledge embedded in the VLM. In addition, the efficient, object-centric canonical interaction primitives promote accurate 3D perception and execution. OmniManip shows significant performance advantages over baseline methods, which is mainly attributed to the following two factors. The first factor is the efficiency and stability of the object-centric canonical interaction primitives, which is verified by extensive experiments. The second factor is the advanced dual closed-loop system for planning and execution. By introducing a novel self-correction mechanism based on RRC, the dual closed-loop system effectively alleviates the hallucination problem of large models.

[0117] Table 1

[0118]

[0119]

[0120] Table 1 shows the quantitative results of 12 real-world manipulation tasks. The first six tasks focus on the manipulation of rigid objects, while the last six involve the manipulation of articulated objects. In the table, "-" indicates that the task is difficult to handle due to the inherent principles of the method. OmniManip (Ours) is a method provided in an embodiment of the present application, in which Closed-loop refers to disabling closed-loop planning, and Open-loop refers to enabling closed-loop planning. As shown in Table 1, this closed-loop planning achieved a performance improvement of more than 15% in both rigid and articulated object manipulation tasks.

[0121] The following is an illustration of the reliability of OmniManip. Figure 4 and Figure 5 , Figure 4 is a schematic diagram of stability analysis of an interactive primitive provided in an embodiment of the present application (visualization of planning based on different methods and corresponding execution results, taking the "pouring tea" task as an example), Figure 5 This is a qualitative analysis diagram of the impact of a perspective on performance provided by an embodiment of the present application (taking the "battery recycling" task as an example). Figure 4 In the above, Planning refers to planning and Execution refers to execution.

[0122] In order to effectively connect the visual language model (VLM) with low-level manipulation, reliable interaction primitives are very important. During the experiment, the reliability of OmniManip was evaluated from two dimensions: stability and view consistency. Stability, for example, refers to the ability to reliably extract task-related canonical interaction primitives. Figure 4 As shown, ReKep extracts key point proposals through semantic clustering, but it lacks sensitivity to spatial geometry and tasks, making it difficult to generate enough task-relevant key points. CoPa extracts parts through explicit pixel segmentation and is highly sensitive to image texture and part shape. In contrast, OmniManip's object-centric canonical interaction primitives sample interaction points in a canonical space aligned with the object's functional axes, thereby ensuring robustness and task-specific accuracy. Viewpoint consistency, for example, refers to the consistency of canonical interaction primitive extraction under different viewpoints, which is key to ensuring manipulation stability. Since ReKep and CoPa sample points directly from the object surface, they both perform poorly in terms of viewpoint consistency. Taking ReKep as an example, Figure 5 The planning results of ReKep and OmniManip for different perspectives in the task "Battery Recycling" are shown. Figure 5 As shown in Figure 2, ReKep successfully identifies the interaction point at a 90° top-down view, but fails at a 0° frontal view, when the ideal target point floats in the air. In contrast, OmniManip uses an object-centered canonical interaction primitive representation in a canonical space, ensuring view invariance.

[0123] Table 2

[0124]

[0125] Table 2 is a quantitative analysis of the effect of viewpoint (or angle) on performance, taking the task "battery recycling" as a case study. The quantitative comparison results provided in Table 2 show that the performance of OmniManip is almost unchanged under different viewpoints, while the performance of ReKep is significantly affected by the change of viewpoint.

[0126] The following is an illustration of the efficiency of OmniManip. The interaction direction planning in OmniManip is driven by a goal-oriented sampling strategy. Compared with uniform sampling in the rotation space SO(3), OmniManip samples along the principal axes of the object's canonical space. Since the object's canonical space is aligned with the functional axis, the sampling efficiency and effectiveness can be ensured.

[0127] To evaluate this efficiency, this experiment compares the sampling strategy of OmniManip with the uniform sampling strategy of SO(3) through two key indicators, namely, the number of iterations and the corresponding task success rate.

[0128] Table 3

[0129]

[0130] Table 3 shows a quantitative analysis of the sampling efficiency for canonical interaction primitives. As shown in Table 3, OmniManip not only requires fewer iterations but also achieves higher task performance, indicating that aligning the sampling process with the object function axis can reduce the sampling overhead while improving the overall performance.

[0131] See also Figure 6 , Figure 6 It is a closed-loop planning schematic diagram provided in an embodiment of the present application, which implements a self-correction mechanism through RRC.

[0132] The above-mentioned embodiments can achieve closed-loop planning. In related methods, the planning component of VLM runs in an open-loop manner, that is, it is difficult to verify the correctness of the plan before execution. Although ReKep achieves closed-loop control through point tracking, this is limited to the execution stage, and it is difficult to provide feedback on the planning results generated by VLM. In contrast, OmniManip introduces a self-correction mechanism through RRC to achieve closed-loop planning, which greatly reduces planning failures due to VLM hallucinations, thereby providing more reliable planning. As an example, the results of disabling closed-loop planning (corresponding to Closed-loop) are provided in Table 1, where the success rates of rigid body and articulated object manipulation tasks both dropped by more than 15%, which demonstrates the effectiveness of the closed-loop planning method. Figure 6 In the paper, the closed-loop planning results are qualitatively demonstrated by taking the task of "inserting the pen into the pen holder" as an example. Obviously, OmniManip can effectively pre-render the planning results and perform self-correction through the RRC mechanism to achieve closed-loop planning.

[0133] Even with perfect planning, open-loop execution can still lead to mission failure. Figure 7 , Figure 7 These are two typical failure cases without closed-loop execution provided in the embodiments of the present application.

[0134] Figure 7 Two typical examples are shown. As examples, the relative pose between the gripper and the object changes during the interaction; or, the pose of the target object (e.g., a cup) changes, such as when the object moves during the task. In these examples, open-loop execution leads to failure despite successful planning. To address these challenges, OmniManip employs pose tracking for real-time closed-loop execution. ReKep can use point tracking for closed-loop control, but it suffers from a high failure rate due to occlusions. In contrast, OmniManip is more robust to occlusions caused by object movement, thanks to object-centric pose tracking, which continues to track interaction primitives in the canonical space based on the object pose even when they are no longer visible.

[0135] In some embodiments, OmniManip can be used to generate automatic demonstration data. Unlike related methods that rely on task-specific information, OmniManip can collect demonstration trajectories for new tasks in a zero-shot manner without requiring task-specific details or prior object knowledge. To verify the effectiveness of OmniManip-generated data, 150 trajectories can be collected for each task for training the behavior cloning strategy.

[0136] Table 4

[0137]

[0138] Table 4 shows the results of behavior cloning using examples generated by OmniManip. As can be seen, OmniManip achieves a high success rate.

[0139] In order to develop a more efficient and versatile representation to connect the high-level reasoning capabilities of the VLM with precise low-level robotic operations, the above embodiments provide a novel object-centric intermediate representation (e.g., canonical interaction primitives) that integrates interaction points and interaction directions in the canonical space of the object. This representation bridges the gap between the high-level common sense reasoning of the VLM and precise 3D spatial understanding. The canonical space of an object can be defined based on its functional affordances. Therefore, the functionality of an object can be described in a more structured and semantically meaningful way in the canonical space. At the same time, technical advances in general object pose estimation make it possible to normalize a wide range of objects.

[0140] By way of example, the above embodiment may employ a general 6D pose estimation model to normalize objects and describe the rigid body transformation of objects during interaction. In addition, a single-view 3D generation network may be used to generate a detailed object mesh (e.g., a three-dimensional model). In the canonical space, initial sampling of the interaction directions along the principal axes of the object provides a rough set of interaction possibilities (e.g., candidate interaction directions). Afterwards, the VLM is used to predict the interaction points. Subsequently, the VLM is used to identify the canonical interaction primitives relevant to the task and estimate the spatial constraints between them. In order to improve the hallucination problem in VLM reasoning, a self-correction mechanism implemented by interaction rendering and primitive resampling may be introduced to achieve closed-loop reasoning. Once the final strategy is determined, the actions may be calculated by constraint optimization, and robust and timely control may be ensured by pose tracking during the closed-loop execution phase.

[0141] The above embodiments can achieve efficient and effective sampling of canonical interaction primitives. By utilizing the canonical space of objects, efficient and effective sampling of canonical interaction primitives is achieved, and the reasoning ability of the robot operating system is enhanced. In addition, based on the object-centric intermediate representation, a dual closed-loop, open-vocabulary robot operating system is provided. In addition, the rendering and resampling process drives the reasoning closed loop of decision-making, while posture tracking ensures the closed loop of action execution.

[0142] In summary, the embodiments of the present application propose a novel object-centric interaction representation (e.g., canonical interaction primitives), which bridges the gap between high-level common sense reasoning of VLM and low-level robot operation, and realizes an open vocabulary operation system with double closed loops of planning and execution without fine-tuning VLM. A large number of experiments show that the method provided by the above embodiment has strong zero-shot generalization ability in a variety of operation tasks, and also demonstrates its potential in automatically generating robot operation data. Specifically, the above embodiment proposes a novel object-centric intermediate representation method, which effectively bridges the gap between the visual language model (VLM) and the precise spatial reasoning required for robot manipulation. Secondly, the above embodiment constructs interaction primitives in the canonical space of objects, converting high-level semantic reasoning into operational three-dimensional spatial constraints. In addition, the double closed-loop system ensures the robustness of decision-making and execution without fine-tuning VLM. The method provided by the above embodiment shows strong zero-shot generalization ability in various manipulation tasks, highlighting its potential in automatic robot data generation and improving the efficiency of robot systems in unstructured environments, and provides a promising foundation for future research on scalable open vocabulary robot manipulation.

[0143] The embodiment of the present application also provides a robot operating system, the system includes a task planning module, a constraint planning module and an operation control module. The task planning module is used to perform task planning using target task information and visual input information to determine subtasks and target objects of at least one stage, and the target objects of the stage include active objects and passive objects. The constraint planning module is used to perform spatial constraint planning for at least one stage according to the canonical interaction primitives of the subtasks and target objects of the stage to obtain spatial constraint information based on the canonical interaction primitives. The operation control module is used to control the operation of the end effector of the robot according to the subtasks and spatial constraint information of the stage.

[0144] See also Figure 8 , Figure 8 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0145] An embodiment of the present application further provides a robot, comprising a robot operating system and an end effector, wherein the robot operating system is used to execute any one of the above methods to control the operation of the end effector.

[0146] In some embodiments, the robot is a humanoid robot (ie, a humanoid robot, a bipedal robot), a quadrupedal robot, a wheeled robot, a multi-jointed robotic arm, or other automated equipment.

[0147] In some embodiments, the robot may also include one or more of an odometer, an IMU (Inertial Measurement Unit), an image sensor, a laser sensor, an angle encoder, a torque sensor, and a PIR sensor (Passive Infrared Sensor).

[0148] In some embodiments, the image sensor may include a camera (or a camera module).

[0149] In some embodiments, the laser sensor may include a 2D laser sensor and / or a 3D laser sensor.

[0150] In some embodiments, the robot may be a multi-jointed robot.

[0151] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0152] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0153] The computer program product may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer readable media.

[0154] An embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0155] See also Fig. 9 , Fig. 9 It is a structural block diagram of a computer device provided in an embodiment of the present application.

[0156] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0157] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0158] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0159] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solution of the embodiment of the present application is implemented by software or firmware, the computer program for implementing the solution of the embodiment of the present application can be stored in the processor 120 and executed by the processor 120.

[0160] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0161] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor, etc.

[0162] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 120 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor 120 for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0163] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input output system (BIOS), application software, etc.

[0164] The operating system is used to manage the hardware and / or software resources of computer devices and is the core and cornerstone of computer devices. The operating system needs to handle basic tasks such as managing and configuring memory, determining the priority of system resource supply and demand, controlling input and output devices, operating networks, and managing file systems. In order to facilitate user operations, most operating systems will provide an operating interface for users to interact with the system.

[0165] BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and application programs. In some implementations, BIOS can also monitor and display processor temperature and execute functions such as adjusting temperature protection strategies.

[0166] Application software is also called application program, which can be understood as software written for a specific application purpose of the user. It is one of the main categories of computer software. For example, application software can be a program used to achieve power control, temperature management, etc.

[0167] It should be understood that the specific examples in this application are only intended to help those skilled in the art to better understand the implementation methods of this application, rather than to limit the scope of protection of this application.

[0168] It can be understood that in various implementations of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the present application.

[0169] It can be understood that the various implementation modes described in this application can be implemented individually or in combination, and this application is not limited to this.

[0170] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as those generally understood by those skilled in the art of the technical field of this application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the scope of this application. The term "and / or" used in this application includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0171] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each embodiment described above can refer to the corresponding process in other embodiments and will not be repeated here.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0176] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0177] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A robot operation method, characterized in that: The method comprises: Performing task planning using target task information and visual input information to determine subtasks and target objects of at least one stage, wherein the target objects of the stage include active objects and passive objects; For at least one stage, spatial constraint planning is performed according to the subtasks of the stage and the canonical interaction primitives of the target object to obtain spatial constraint information based on the canonical interaction primitives; and the operation of the end effector of the robot is controlled according to the subtasks of the stage and the spatial constraint information.

2. The robot operation method according to claim 1, characterized in that: The active objects and passive objects corresponding to the same stage meet the following conditions: The passive object is one of the at least one task object; The active object is the end effector, or the active object is one of the at least one task object and is different from the passive object.

3. The robot operation method according to claim 2, characterized in that: The process of acquiring the at least one task object includes: Processing the visual input information to mark at least one input object; At least one task object corresponding to the target task information is screened out from the at least one input object.

4. The robot operation method according to claim 1, characterized in that: The process of acquiring the standard interaction primitives of the target object includes: The canonical interaction primitives are extracted using the subtasks of the stage and the canonical representation information of the target object to obtain the canonical interaction primitives of the target object; the canonical representation information includes a three-dimensional model and a posture.

5. The robot operation method according to claim 4, characterized in that: The three-dimensional model of the target object is obtained by reconstructing the three-dimensional structure of the target object using the visual input information, and / or the position and posture of the target object is obtained by estimating the position and posture of the target object using the visual input information.

6. The robot operation method according to claim 5, characterized in that: The standard interaction primitives of the target object include an interaction point and / or an interaction direction.

7. The robot operation method according to claim 6, characterized in that: The interaction point includes at least one of the following: a first interaction point that is visible and touchable; and a second interaction point that is invisible or untouchable.

8. The robot operation method according to claim 7, characterized in that: The process of extracting the interaction points includes: superimposing a Cartesian network on a target image containing the target object to obtain a superimposed image; the target image is determined according to the visual input information; According to the subtasks of the stage, the interaction point is located using the superimposed image.

9. The robot operation method according to claim 8, characterized in that: The target image corresponding to the first interaction point includes an input image, which is obtained by processing the visual input information; and / or the target image corresponding to the first interaction point includes at least one view of the three-dimensional model of the target object, and the at least one view includes at least one of the six views.

10. The robot operation method according to claim 8, characterized in that: The method of obtaining the interaction point by locating the interaction point using the superimposed image according to the subtask of the stage includes: The interaction point is located by using the superimposed image according to the task type of the subtask of the stage and the object type of the target object; the object type includes active and passive.

11. The robot operation method according to claim 6, characterized in that: The process of extracting the interaction points includes: Extracting standardized interaction primitives based on the standardized representation information of the target object to obtain a plurality of candidate interaction points; Processing a plurality of candidate interaction points according to the subtasks of the stage to generate an interaction point heat map; At least one interaction point is determined from a plurality of candidate interaction points using the interaction point heat map.

12. The robot operation method according to claim 6, characterized in that: The process of extracting the interaction direction includes: Extracting at least one candidate interaction direction of the target object according to the normalized representation information of the target object; For one or more candidate interaction directions, generate semantic description information corresponding to the candidate interaction directions, and calculate a relevance score between the semantic description information and the subtasks of the stage; The corresponding candidate interaction directions are sorted according to the relevance scores to determine at least one interaction direction.

13. The robot operation method according to claim 1, characterized in that: The performing of spatial constraint planning according to the subtasks of the stage and the canonical interaction primitives of the target object to obtain spatial constraint information based on the canonical interaction primitives includes: Performing spatial constraint planning based on the subtasks of the stage and the canonical interaction primitives of the target object to obtain at least one unverified constraint information, wherein the at least one unverified constraint information includes distance constraint information and / or angle constraint information; For one or more unverified constraint information, the unverified constraint information is verified to obtain a verification result corresponding to the unverified constraint information; when the verification result corresponding to the unverified constraint information is successful, the unverified constraint information is included in the spatial constraint information.

14. The robot operation method according to claim 13, characterized in that: The verifying the unverified constraint information to obtain a verification result corresponding to the unverified constraint information includes: Rendering an interactive image corresponding to the unverified constraint information; The interactive image is verified to obtain a verification result corresponding to the unverified constraint information.

15. The robot operation method according to claim 13, characterized in that: The method further comprises: If the verification result corresponding to the unverified constraint information is a failure, verify the next unverified constraint information; or, When the verification result corresponding to the unverified constraint information is optimized, resampling is performed based on the current standard interaction primitive to achieve adjustment of the standard interaction primitive, and spatial constraint planning is re-performed based on the adjusted standard interaction primitive to obtain at least one unverified constraint information.

16. The robot operation method according to claim 1, characterized in that: The controlling the operation of the end effector of the robot according to the subtasks and space constraint information of the stage comprises: Calculating the loss value of the target loss function according to the subtasks and spatial constraint information of the stage to determine the target posture of the end effector that meets the target optimization condition; the target loss function includes one or more loss terms among a constraint loss term, a collision loss term and a path loss term, and the loss value of at least one loss term is calculated according to the posture of the end effector; The target posture is used to perform trajectory planning on the end effector to obtain trajectory information of the end effector; the trajectory information is used to control the operation of the end effector.

17. A robot, characterized in that: The robot comprises a robot operating system and an end effector, wherein the robot operating system is used to execute the method according to any one of claims 1 to 16 to control the operation of the end effector.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

19. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 16 when executing the computer program.

20. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images