Robot manipulation methods and systems based on spatiotemporal constraints

Abstract

This invention provides a robot operation method and system based on spatiotemporal constraint enhancement. It improves the accuracy and semantic consistency of spatiotemporal constraint expression through a key element extraction mechanism. The operation target in the visual image is transformed into three basic geometric elements: points, lines, and surfaces, which are used to express the task objective and spatial constraints. The constraints not only cover point-to-point distances, line-to-surface angles, and relative relationships between surfaces, but also describe complex structural dependencies and operation sequences. This application achieves structural logicalization of task verification, improving the accuracy of completion judgment. By using the satisfaction of structural constraints by the state of key elements as the criterion for task completion, each stage can clearly determine whether the current state conforms to the preset structural logic. This process constitutes a logic-driven adaptive recovery closed loop, supporting dynamic adjustment of the execution plan after structural changes, significantly improving the system's fault tolerance in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of robot manipulation technology, and in particular to robot manipulation methods and systems based on spatiotemporal constraint enhancement. Background Technology

[0002] In current robot operating systems, in order to complete complex tasks with time and space dependencies, researchers have gradually introduced constraint mechanisms to improve the controllability and safety of task execution, with the aim of ensuring that robot behavior conforms to the semantic and physical constraints of the task.

[0003] Within this framework, constraints primarily function in the following aspects: defining the range of motion of the end effector; limiting the accessibility or placement conditions of the target object; ensuring the rationality of changes in the operating path, speed, or attitude; and guaranteeing accuracy during critical stages of operation.

[0004] Constraint generation methods can be broadly categorized into two types: one is static constraint expression based on manually generated rules and templates, such as using fixed spatial thresholds, like "distance less than..." "The included angle is less than" The first approach involves setting conditions for task completion; the second is to use a visual language model to automatically infer the action target based on the input image and instructions, and generate action prompts. These prompts are then parsed and converted into corresponding geometric operation conditions. The latter approach is more versatile and semantically adaptable, and has become the mainstream trend.

[0005] To achieve the above functions, the system typically incorporates advanced visual segmentation models, such as SAM and Grounding DINO, to extract target object regions or semantic masks. These visual cues, along with task instructions, form the multimodal input of the visual language model, used for task decomposition, action reasoning, and sub-target recognition.

[0006] Action prompts obtained through language model reasoning, such as "move above the object," "grab the middle," and "place in the center of the platform," are further translated into operation parameter inputs by the "prompt parsing module" or "constraint converter," and then executed by the path planner or operator. In this process, constraint representation plays a crucial role in achieving the accuracy of path operations and target contact, and is one of the key guarantees for the accuracy and robustness of task execution.

[0007] In addition, some systems have also built preliminary execution result verification and task recovery mechanisms. When the execution result does not meet expectations, such as failing to enter the target area or the object not being placed stably, the system can interrupt the task, roll back the path, or reinitialize to achieve simple recovery.

[0008] In summary, existing robot operating systems have initially established a three-dimensional linkage operation execution mode of "vision-language-operation", which can generate task constraints and execute actions through image perception and language reasoning, and has a certain degree of intelligent operation capability and fault response capability.

[0009] However, the above process has the following problems:

[0010] 1) Lack of constraint expression ability driven by structural semantics

[0011] The current system mainly relies on coarse-grained visual cues such as mask regions and bounding boxes, as well as manually defined rule templates, to generate constraints. Due to the lack of fine-grained geometric modeling, the constraint expressions are mostly limited to simple semantics such as "near", "align", or "place within the region", and cannot support the modeling of structural relationships such as "alignment of the orientation line with the surface normal", "point must fall on the center of the stable support surface", and "multiple elements are coplanar".

[0012] The insufficient geometric granularity and physical consistency of this constraint expression directly affect the spatial reasoning ability of the language model, limiting the execution accuracy and adaptability of the robot system for structure-sensitive tasks (such as grasping direction, rotational movements, precise positioning and placement).

[0013] 2) Lack of error detection and recovery mechanisms based on structural logic

[0014] In the event of execution failure, the current system typically detects and recovers through the following methods: detection of abnormal operator status (such as trajectory termination, exceeding boundaries, or failure to converge); simple visual feedback judgment (such as target misalignment or deviation from the area); static path rollback or action retry.

[0015] This type of detection method heavily relies on low-level feedback and lacks logical verification related to the semantics of the task structure. For example, the system cannot determine whether the specified gripping point has been reached, whether the gripper is aligned with the normal of the support surface, or whether the object is stably attached to the placement surface, leading to frequent misjudgments, low recovery efficiency, and even task deadlocks. Summary of the Invention

[0016] In view of this, the purpose of this invention is to provide a robot operation method and system based on spatiotemporal constraint enhancement, which realizes the unification of the entire chain of task expression, constraint generation, execution control and anomaly recovery under the guidance of key elements, and has high precision, strong interpretability and good versatility.

[0017] In a first aspect, embodiments of the present invention provide a robot operation method based on spatiotemporal constraint enhancement, the method comprising:

[0018] The original image is acquired, and the original image is segmented using the first image encoder and the language encoder in the SEEM model to obtain the segmentation result;

[0019] The segmentation results are used to perform image annotation through the SAM model to obtain multiple masks and a confidence score corresponding to each mask;

[0020] Calculate the aspect ratio and area of ​​the segmented region by analyzing multiple masks.

[0021] The type of element is determined based on the aspect ratio and area of ​​the segmented region; wherein, the type of element includes point element, line element, and area element;

[0022] The DINOv2 model is used to extract the type features of the elements, resulting in a high-dimensional feature vector.

[0023] The high-dimensional feature vector is reduced in dimensionality to obtain the reduced high-dimensional feature vector.

[0024] The reduced high-dimensional feature vectors are clustered using the K-Means clustering algorithm to output an annotated image;

[0025] The key elements with color on the annotated image, the language instructions for the task, and the prompt word templates are input into the visual language big model to obtain the stage decomposition information and constraints of the task.

[0026] The final trajectory is obtained by performing constraint optimization on the stage decomposition information and the constraints of the task.

[0027] Obtain parameter information of the end effector and voxel information of the scene;

[0028] The configuration of the key element is verified using the parameter information of the end effector, the physical location of the key element, the voxel information of the scene, and the constraint function, and a verification result is obtained. The verification result includes whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

[0029] Furthermore, the SAM model includes a second image encoder, a cue encoder, and a mask decoder. The segmentation result is annotated using the SAM model to obtain multiple masks and a confidence score corresponding to each mask, including:

[0030] The segmentation result is encoded using the second image encoder to obtain the encoded image;

[0031] The prompts input by the user are converted by the prompt encoder to obtain prompts in vector form;

[0032] The encoded image and the vector-based cue are mapped by the mask decoder into a plurality of masks and a confidence score corresponding to each mask.

[0033] Furthermore, the high-dimensional feature vector is reduced in dimensionality to obtain a reduced high-dimensional feature vector, including:

[0034] The high-dimensional feature vector is centered using the PCA algorithm to obtain the centered data.

[0035] Singular value decomposition is performed on the centered data to obtain a set of orthogonal principal component directions and corresponding variances;

[0036] The PCA algorithm is used to select the k principal component directions with the largest variances from the corresponding variances.

[0037] The high-dimensional feature vector is projected onto the directions of the k principal components with the largest variance to obtain the dimensionality-reduced high-dimensional feature vector.

[0038] Furthermore, constraint optimization is performed on the stage decomposition information of the task and the constraints to obtain the final trajectory, including:

[0039] The task's phase decomposition information and constraints are used to perform a global search using a double annealing algorithm to obtain a preliminary solution.

[0040] The preliminary solution is optimized using the SLSQP algorithm to obtain the optimized solution;

[0041] Using the optimized solution as a new starting point, the SLSQP algorithm is called again for local optimization until the number of iterations reaches a preset number or the cost is less than a preset threshold, at which point the iteration ends, thus obtaining the final trajectory.

[0042] Furthermore, the end effector's parameter information includes three-dimensional pose and environmental collision points; the three-dimensional pose is provided by a robot encoder and kinematic model, the environmental collision points are generated by physical simulation collision detection between the end effector grasping the object and other objects in the environment, and the voxel information of the scene is represented by a three-dimensional occupancy mesh constructed by a depth camera or point cloud sensor.

[0043] Furthermore, the method also includes:

[0044] After each task phase is completed, the images captured by the robot and the phase breakdown information of the task are used as input.

[0045] The visual language model is used to determine whether the current stage has been successfully executed.

[0046] If the execution fails, you need to go back to the previous stage and solve the problem again.

[0047] Secondly, embodiments of the present invention provide a robot operating system based on spatiotemporal constraint enhancement, the system comprising:

[0048] The segmentation module is used to acquire the original image and segment the original image using the first image encoder and the language encoder in the SEEM model to obtain the segmentation result;

[0049] The annotation module is used to annotate the segmentation results using the SAM model to obtain multiple masks and a confidence score corresponding to each mask;

[0050] The calculation module is used to calculate the aspect ratio and area of ​​the segmented region from the multiple masks.

[0051] The determining module is used to determine the type of element based on the aspect ratio and area of ​​the segmented region; wherein the type of element includes point element, line element and area element;

[0052] The extraction module is used to extract the type features of the element through the DINOv2 model to obtain a high-dimensional feature vector;

[0053] The dimension reduction module is used to reduce the dimension of the high-dimensional feature vector to obtain the dimension-reduced high-dimensional feature vector.

[0054] The clustering module is used to cluster the reduced high-dimensional feature vectors using the K-Means clustering algorithm and output the labeled image.

[0055] The input module is used to input the key elements with color on the annotated image, the language instructions for the task, and the prompt word template into the visual language big model to obtain the stage decomposition information and constraints of the task.

[0056] The optimization solution module is used to perform constraint optimization on the stage decomposition information and constraints of the task to obtain the final trajectory;

[0057] The acquisition module is used to acquire parameter information of the end effector and voxel information of the scene;

[0058] The verification module is used to verify the configuration of the key element using the parameter information of the end effector, the physical location of the key element, the voxel information of the scene, and the constraint function, and to obtain the verification result; wherein, the verification result includes whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

[0059] Furthermore, the SAM model includes a second image encoder, a cue encoder, and a mask decoder, and the annotation module is specifically used for:

[0060] The segmentation result is encoded using the second image encoder to obtain the encoded image;

[0061] The prompts input by the user are converted by the prompt encoder to obtain prompts in vector form;

[0062] The encoded image and the vector-based cue are mapped by the mask decoder into a plurality of masks and a confidence score corresponding to each mask.

[0063] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described above.

[0064] Fourthly, embodiments of the present invention provide a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the method described above.

[0065] This invention provides a robot operation method and system based on spatiotemporal constraint enhancement, comprising: acquiring an original image; segmenting the original image using a first image encoder and a speech encoder in a SEEM model to obtain a segmentation result; annotating the segmentation result using a SAM model to obtain multiple masks and a confidence score corresponding to each mask; calculating the aspect ratio and area of ​​the segmented region using the multiple masks; determining the element type based on the aspect ratio and area of ​​the segmented region; wherein the element type includes point elements, line elements, and area elements; extracting the features of the element type using a DINOv2 model to obtain a high-dimensional feature vector; reducing the dimensionality of the high-dimensional feature vector to obtain a dimensionality-reduced high-dimensional feature vector; and performing dimensionality reduction using a K-Means clustering algorithm. The high-dimensional feature vectors are clustered to output an annotated image. The key elements with color, the language instructions for the task, and the prompt word templates from the annotated image are input into a large visual language model to obtain the task's stage decomposition information and constraints. Constraint optimization is performed on the task's stage decomposition information and constraints to obtain the final trajectory. The parameter information of the end effector and the voxel information of the scene are obtained. The configuration of the key elements is verified using the end effector's parameter information, the physical location of the key elements, the voxel information of the scene, and the constraint functions, yielding verification results. These results include whether the final trajectory conforms to the planned trajectory and whether it deviates from the planned trajectory. This achieves a unified end-to-end system for task expression, constraint generation, execution control, and anomaly recovery under the guidance of key elements, possessing high precision, strong interpretability, and good versatility.

[0066] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0067] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0068] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0069] Figure 1 This is a flowchart of a robot operation method based on spatiotemporal constraint enhancement provided in Embodiment 1 of the present invention;

[0070] Figure 2 This is a schematic diagram of the robot operation process based on spatiotemporal constraint enhancement provided in Embodiment 1 of the present invention;

[0071] Figure 3 This is a schematic diagram of the key element extraction process provided in Embodiment 1 of the present invention;

[0072] Figure 4 A flowchart of the two-layer error detection and recovery mechanism provided in Embodiment 1 of the present invention;

[0073] Figure 5 This is a schematic diagram of a robot operating system based on spatiotemporal constraint enhancement provided in Embodiment 2 of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] The key element extraction mechanism proposed in this application abstracts task-related structures in images into points, lines, and surfaces, endowing the system with the ability to understand and express scene structures. By constructing constraint representation and state verification logic centered on key elements, the following improvements can be achieved:

[0076] More structured constraint representation: Supports the representation of geometric relationships in tasks based on point, line, and surface constraints;

[0077] Error detection is more logical: by judging whether the state of key elements satisfies the constraint logic, it can be clearly determined whether the task is successful;

[0078] The path reconstruction is more directional: key elements can be re-extracted based on the current image, new sub-targets and constraints can be dynamically generated, and efficient adaptive recovery can be supported.

[0079] Therefore, introducing key element structural modeling not only makes up for the shortcomings of existing systems in constraint expression and task verification, but also provides a technical foundation for building a truly universal, high-precision, and robust robot task execution system.

[0080] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0081] Example 1:

[0082] Figure 1This is a flowchart of a robot operation method based on spatiotemporal constraint enhancement provided in Embodiment 1 of the present invention.

[0083] Reference Figure 1 The method includes the following steps:

[0084] Step S101: Obtain the original image and segment it using the first image encoder and language encoder in the SEEM model to obtain the segmentation result;

[0085] Step S102: The segmentation results are annotated using the SAM model to obtain multiple masks and the confidence score corresponding to each mask;

[0086] Step S103: Calculate the aspect ratio and area of ​​the segmented region by performing calculations on multiple masks.

[0087] Step S104: Determine the type of element based on the aspect ratio and area of ​​the segmented region; the type of element includes point element, line element, and area element.

[0088] Specifically, the key element extraction process starts from the original image and is segmented using Set-of-Mark (SoM). SoM integrates several state-of-the-art segmentation models, including SEEM, SAM, and Semantic-SAM, and supports automatic segmentation and annotation of images at various granularities.

[0089] For low-granularity segmentation, such as identifying objects in the original image (e.g., "a cup"), SoM chooses the SEEM model for coarser segmentation. Segment Everything Everywhere All at Once (SEEM) is a multimodal image segmentation model proposed by Meta AI. It can segment images using various cues, including natural language, points, and bounding boxes, and requires no additional training for unseen images. Using a multimodal encoder composed of a first image encoder and a language encoder, it performs cross-feature extraction on the image and text respectively, ultimately extracting segmentation results with semantic details.

[0090] For higher-granularity segmentation, such as identifying a part of an object in the original image, like "the handle of a cup", SoM chooses the Semantic-SAM model and the SAM model for more refined image annotation. Segment Anything Model (SAM) is a basic vision model proposed by Meta AI.

[0091] Step S105: Extract the type features of the elements using the DINOv2 model to obtain a high-dimensional feature vector;

[0092] Step S106: Reduce the dimensionality of the high-dimensional feature vector to obtain the dimensionality-reduced high-dimensional feature vector.

[0093] Step S107: Cluster the reduced high-dimensional feature vectors using the K-Means clustering algorithm and output the labeled image;

[0094] Step S108: Input the key elements with color on the labeled image, the language instructions for the task, and the prompt word template into the visual language big model to obtain the stage decomposition information and constraints of the task.

[0095] Step S109: Perform constraint optimization on the stage decomposition information and constraints of the task to obtain the final trajectory;

[0096] Step S110: Obtain the parameter information of the end effector and the voxel information of the scene;

[0097] Step S111: The configuration of the key elements is verified by using the parameter information of the end effector, the physical location of the key elements, the voxel information of the scene, and the constraint function to obtain the verification result; wherein, the verification result includes whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

[0098] This application introduces a structure-driven key element extraction and expression mechanism, and based on this, constructs a closed loop for spatiotemporal constraint modeling, motion planning, state verification, and error recovery throughout the entire task process. This mechanism breaks the fragmented pattern between "semantic cues - rule constraints - low-level control" in traditional robot operating systems, and establishes a unified, high-precision, and interpretable robot operation and control system.

[0099] Furthermore, the SAM model includes a second image encoder, a cue encoder, and a mask decoder, and step S102 includes the following steps:

[0100] Step S201: Encode the segmentation result using a second image encoder to obtain the encoded image;

[0101] Step S202: The user-input prompt is converted by a prompt encoder to obtain a prompt in vector form;

[0102] Step S203: The encoded image and the vector-form cue are mapped into multiple masks and a confidence score corresponding to each mask by a mask decoder.

[0103] Specifically, the SAM model comprises a second image encoder, a cue encoder, and a mask decoder. The second image encoder uses a Vision Transformer (ViT) pre-trained with MAE, running once for each image. The cue encoder considers both discrete and continuous cues, converting user-input cues into vector forms that the model can understand. The mask decoder's main function is to map image embeddings, cue embeddings, and output words to masks. The model ultimately outputs multiple valid masks and their confidence scores.

[0104] Compared to the SAM model, Semantic-SAM is an enhanced version of the original SAM model in terms of semantics. It adds semantic information labels to the masks segmented by SAM, compensating for SAM's inability to identify object categories. This model can extract region image patches from instance masks obtained by SAM, and then use an image-text alignment model and a category vocabulary to calculate their similarity to each category to complete semantic classification. This achieves semantic segmentation while also imbuing the masks with semantic meaning.

[0105] SoM divides the image into multiple regions. All processed masks, by calculating the aspect ratio and area of ​​each region, can classify them into three types of elements: points, lines, and surfaces. These two geometric features directly reflect the shape of the object's structure: small, nearly square regions typically represent point elements, such as buttons or fingertips; long, thin regions with extremely large aspect ratios represent line elements, such as edges or wires; and larger regions with moderate aspect ratios represent surface elements, such as desktops or the main body of an object. This classification helps subsequent tasks to process and understand elements more effectively based on their structure.

[0106] Furthermore, step S106 includes the following steps:

[0107] Step S301: The high-dimensional feature vector is centered using the PCA algorithm to obtain the centered data;

[0108] Step S302: Perform singular value decomposition on the centered data to obtain a set of orthogonal principal component directions and corresponding variances;

[0109] Step S303: Select the k principal component directions with the largest variance from the corresponding variances using the PCA algorithm;

[0110] Step S304: Project the high-dimensional feature vector onto the directions of the k principal components with the largest variance to obtain the dimensionality-reduced high-dimensional feature vector.

[0111] Specifically, the DINOv2 model is then used to extract key features of points, lines, and surfaces in the image, including high-dimensional vector features such as semantic representation, geometric shape, spatial location, and contextual relationships. Then, the PCA algorithm first centers the high-dimensional feature vectors, subtracting the mean from each feature to calculate the covariance matrix, which reflects the linear relationship between the features.

[0112] Next, singular value decomposition is performed on the centered data to obtain a set of orthogonal principal component directions and their corresponding variances. The PCA algorithm selects the k principal component directions with the largest variances and projects the high-dimensional feature vectors onto these k directions, thereby achieving dimensionality reduction while retaining as much original information as possible.

[0113] In image feature processing, the high-dimensional feature vectors extracted using the DINOv2 model can be effectively compressed using the PCA algorithm to remove redundant features, providing a more compact representation for subsequent clustering.

[0114] The K-Means clustering algorithm is then used to cluster these features, and the centroid of each cluster is projected onto the original image and assigned a corresponding numerical code. Different types of elements are distinguished by different colored keypoints. If the number of keypoints for a line or surface element is insufficient, keypoint extraction for that type needs to be performed again. The flowchart for key element extraction is shown below. Figure 3 As shown.

[0115] Specifically, this application uses the extracted key elements as the basis for the direct constraint expression of task execution and designs a standard structural logic constraint function system.

[0116] Within the framework of this application, a constraint is defined as a mapping function. It is a collection of key elements of a point, line, and plane. Mapped to a real number, representing the cost of the set. This cost is unbounded, but when When this condition is met, it indicates that the constraint has been satisfied. The constraint function corresponding to each key element. The key element's sequence number.

[0117] The function f is stateless and calculates costs by performing a series of operations on key elements. These operations are typically non-linear and may be non-convex. The function f is used to represent the desired spatial relationships between key elements and is suitable for robotic arms and object parts.

[0118] Each task is broken down into N stages, and for each stage... Specify two types of constraints:

[0119] 1) Sub-path constraints: Constraints that need to be satisfied during the execution of sub-goals, expressed as... express.

[0120] 2) Sub-goal constraints: Constraints that need to be satisfied when a sub-goal is completed, expressed as... express.

[0121] The set of all constraints can be represented as: .

[0122] Constraint set for each stage It can be represented by a combination of sub-path constraints and sub-goal constraints, i.e. .

[0123] In this part where constraints are generated, the original image observed by the robot's camera is O, and the visual cues after key element extraction are... Visual cues The language instructions L and the prompt word template P of the task are input into the visual language big model to obtain the stage decomposition information I of the task and the relevant constraints C. Among them, the visual prompts are the key elements with color on the labeled image. This process can be referred to formula (1):

[0124] (1)

[0125] in, It abstracts the process of using a visual language model into a function, where VLM stands for Visual Language Large Model.

[0126] For example, the task of "reorienting the white pen and inserting it vertically into the black pen holder" can be broken down into three stages by the visual language big data model: "grabbing the white pen," "reorienting the white pen to a vertical position," and "releasing the white pen and inserting it vertically into the black pen holder." The constraints for each stage can be represented using Python functions. For instance, in stage 2, the sub-path constraint function requires the robot to grasp the white pen while moving its robotic arm, while the sub-objective constraint function requires the vector composed of two key points to be aligned with the z-axis to ensure the white pen is vertical.

[0127] Furthermore, step S109 includes the following steps:

[0128] Step S401: The task's phase decomposition information and constraints are used to perform a global search using the double annealing algorithm to obtain a preliminary solution;

[0129] Step S402: Optimize the preliminary solution using the SLSQP algorithm to obtain the optimized solution;

[0130] Step S403: Using the optimized solution as a new starting point, continue to call the SLSQP algorithm for local optimization until the number of iterations reaches the preset number or the cost is less than the preset threshold, and then the iteration ends, thus obtaining the final trajectory.

[0131] Specifically, in the constraint optimization and solution section, we transform the robot's operation task into a constraint optimization problem that includes multi-stage sub-path constraints and sub-objective constraints. The robot's end effector pose is represented as... During the robot's operation task, the objective function we solve for constraint optimization is the overall discrete-time trajectory composed of the robot's end effector poses at different times. , refer to formula (2):

[0132] (2)

[0133] in, This represents the auxiliary cost function for each sub-path problem in the i-th stage. Let represent the auxiliary cost function for each sub-objective problem in the i-th stage. Auxiliary cost functions include, for example, collision cost, path length cost, and reachability cost. In each stage, any element in the set of elements must satisfy both the sub-path constraint and the sub-objective constraint.

[0134] All constrained optimization problems were implemented and solved using SciPy, with decision variables normalized to [0, 1]. In the initial stage, a dual annealing algorithm was used for global search. This algorithm combines simulated annealing and interval decomposition strategies, enabling effective exploration of multiple possible solutions in complex non-convex spaces.

[0135] After obtaining the initial solution, the SLSQP (Sequential Least Squares Programming) algorithm is used as the starting point for local fine optimization. The SLSQP algorithm can efficiently handle constrained nonlinear problems.

[0136] Subsequently, in each iteration, the solution obtained from the previous optimization is used as a new starting point to continue calling the SLSQP algorithm for local optimization, thereby gradually approaching the optimal solution in the solution space until the number of iterations reaches a preset number or the cost is less than a preset threshold, at which point the iteration ends, thus obtaining the final trajectory; thus realizing a collaborative optimization process that combines global exploration with local convergence. The entire process is uniformly implemented and scheduled by the SciPy optimization module.

[0137] Furthermore, the end effector's parameter information includes 3D pose and environmental collision points; the 3D pose is provided by the robot encoder and kinematic model, the environmental collision points are generated by physical simulation collision detection between the end effector grasping the object and other objects in the environment, and the voxel information of the scene is represented by a 3D occupancy mesh constructed by a depth camera or point cloud sensor.

[0138] Specifically, this application also introduces an error detection and recovery mechanism for key elements. To enhance the robustness of the system in actual task execution, this study introduces an automatic detection and recovery mechanism for dynamic disturbances in the main task execution process, consisting of two layers: constraint verification and visual language large model verification. The constraint verification mechanism handles error detection and recovery during each stage of execution, while the visual language large model verification mechanism handles error detection and recovery after each stage of execution. The flowchart of the error detection and recovery mechanism is as follows: Figure 4 As shown.

[0139] For the constraint verification mechanism, during the execution of each task phase, the system acquires the 3D pose of the end effector (provided by the robot encoder and kinematic model), the physical position of key elements (derived from image coordinates by combining image segmentation results with camera parameters), environmental collision points (generated by physical simulation collision detection between the end effector, the grasped object, and other objects in the environment), and scene voxel information (a 3D occupancy mesh representation constructed by a depth camera or point cloud sensor) through the perception module, and uses the currently loaded constraint functions to verify the current configuration of key elements. These constraint functions include sub-path constraint functions and sub-target constraint functions. For example, constraint functions can check whether the end effector is within the allowed operating area and whether key elements are in the expected spatial relationship. If any constraint violates the pre-set tolerance threshold, it indicates that the execution process has deviated from the planned route due to external forces such as target object movement or grasping failure. In this case, backtracking logic needs to be initiated to perform the corresponding rollback operation.

[0140] Once a constraint violation is detected during execution, the system will start from the current stage and check in reverse order whether all constraints from previous stages are still satisfied. If all constraints of a certain historical stage are satisfied, the system will backtrack to that stage, re-register the states of key elements, and replan sub-goals and sub-paths to restore a stable execution state. This process requires no human intervention and is highly automated.

[0141] Furthermore, the method also includes the following steps:

[0142] Step S501: After each task stage is completed, the images captured by the robot and the stage decomposition information of the task are used as input.

[0143] Step S502: Determine whether the current stage has been successfully executed using the visual language big data model;

[0144] If step S503 fails to execute successfully, you need to return to the previous stage and solve it again.

[0145] Specifically, for the visual language large model verification mechanism, after the execution of each task stage, the images captured by the robot at the current time and the decomposition information of the task in the constraint generation stage are used as image input and text input into the visual language large model. The visual language large model determines whether the current stage has been successfully executed based on the images and text. If it has not been successfully executed, it needs to return to the previous stage and solve it again.

[0146] Reference Figure 2 This application mainly includes four modules: key element extraction, constraint generation, constraint optimization solution, and error detection and recovery.

[0147] The key element extraction module takes the image containing the extracted key elements as input, combines it with text input to query the visual language model, and the constraint generation module generates corresponding constraints. The constraint optimization and solution module optimizes and solves the generated constraints to enable the robot to perform its tasks. The error detection and recovery module detects errors during the robot's task execution, backtracks when errors occur, and causes the constraint optimization and solution module to replan.

[0148] For the key element extraction module, this part employs a large-scale pre-trained visual model to automatically extract key elements from scene images. The visual model analyzes different objects in the scene and identifies important feature regions, determining the element type (such as points, lines, and surfaces) based on the spatial geometric features of the objects. These elements represent important locations in the scene, such as object grab points or orientations, thus aiding in the overall understanding of the environment. Furthermore, if the number of extracted key points is insufficient to represent the type of key element after one extraction, the key elements need to be extracted again, thereby better representing the spatial geometric characteristics of the task.

[0149] For the constraint generation module, this stage first marks the key elements extracted in the previous stage on the original image as visual cues. Then, based on the natural language description of the task and the cues in the image, a large visual language model is used to generate the stage objectives and constraints required by the task. The key to this process is that the generated constraints must be able to adjust accordingly as the robot and its environment change, enabling the robot to adapt to different situations and successfully complete the task.

[0150] For the constraint optimization and solution module, the main task is to model the robot operation task as a constraint optimization problem. The task is decomposed into multiple different stages, and the optimization objective of each stage is to solve the pose of the robot's end effector, ensuring that the continuous trajectory from the initial position to task completion meets all constraints during the optimization process. Key elements are updated in real time and visualized during the robot's operation task execution, enabling the system to quickly respond to dynamic environmental changes and obtain the final trajectory.

[0151] For the error detection and recovery module, the system detects whether there are any behaviors that violate constraints at each stage of robot operation. When a stage of the task fails to be completed successfully, the system queries the visual language model in conjunction with the current scene image and task information to determine whether the stage has been completed. If the current stage of the task has not been completed, the system will backtrack to a previous stage and recover the task execution through constraint optimization and solving, ensuring the robustness of the robot system and completing closed-loop operation control.

[0152] This application has the following beneficial effects:

[0153] This invention improves the accuracy and semantic consistency of spatiotemporal constraint expression through a key element extraction mechanism. For the first time, it transforms operational targets in visual images into structured basic geometric elements of points, lines, and surfaces, and uses these as the basis for expressing task objectives and spatial constraints. The constraints not only cover point-to-point distances, line-to-surface angles, and relative relationships between surfaces, but also describe complex structural dependencies and operational sequences. Therefore, this application significantly outperforms existing methods in terms of expression granularity and geometric accuracy, ensuring consistency between task description, path planning, and execution control, and reducing execution failures caused by semantic ambiguity or geometric mismatch.

[0154] This application implements a structured logical approach to task verification, improving the accuracy of completion judgment. By using "key element states satisfying structural constraints" as the task completion criterion, each stage can clearly determine whether the current state conforms to the preset structural logic. This process constitutes a logic-driven adaptive recovery closed loop, supporting dynamic adjustment of the execution plan after structural changes, significantly improving the system's fault tolerance in complex dynamic environments. This invention significantly improves the accuracy, interpretability, and stability of task verification, and is particularly suitable for industrial and service robot scenarios with high execution precision requirements and complex operation processes.

[0155] Example 2:

[0156] Figure 5 This is a schematic diagram of a robot operating system based on spatiotemporal constraint enhancement provided in Embodiment 2 of the present invention.

[0157] Reference Figure 5 The system includes:

[0158] The segmentation module is used to acquire the original image and segment it using the first image encoder and the language encoder in the SEEM model to obtain the segmentation result.

[0159] The annotation module is used to annotate the segmentation results using the SAM model, obtaining multiple masks and a confidence score for each mask;

[0160] The calculation module is used to calculate multiple masks to obtain the aspect ratio and area of ​​the segmented region;

[0161] The determination module is used to determine the type of element based on the aspect ratio of the segmented region and the area size of the region; wherein, the type of element includes point element, line element and area element;

[0162] The extraction module is used to extract the type features of elements through the DINOv2 model to obtain high-dimensional feature vectors.

[0163] The dimensionality reduction module is used to reduce the dimensionality of high-dimensional feature vectors to obtain the dimensionality-reduced high-dimensional feature vectors.

[0164] The clustering module is used to cluster the reduced high-dimensional feature vectors using the K-Means clustering algorithm and output the labeled image.

[0165] The input module is used to input the key elements with color on the labeled image, the language instructions for the task, and the prompt word templates into the visual language big model to obtain the stage decomposition information and constraints of the task.

[0166] The optimization and solution module is used to perform constraint optimization on the stage decomposition information and constraints of the task to obtain the final trajectory;

[0167] The acquisition module is used to acquire parameter information of the end effector and voxel information of the scene;

[0168] The verification module is used to verify the configuration of key elements by using the parameter information of the end effector, the physical location of key elements, the voxel information of the scene, and the constraint functions, and to obtain the verification results. The verification results include whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

[0169] Furthermore, the SAM model includes a second image encoder, a cue encoder, and a mask decoder, and the annotation module is specifically used for:

[0170] The segmentation results are encoded using a second image encoder to obtain the encoded image;

[0171] The user-input prompts are converted into vector form using a prompt encoder.

[0172] The encoded image and the vector-based cue are mapped by a mask decoder into multiple masks and a confidence score for each mask.

[0173] Specifically, this application proposes a robot operation method and system based on spatiotemporal constraint enhancement, aiming to improve the robot's operational accuracy, robustness, and task adaptability in complex task scenarios. The method revolves around "key element extraction" and constructs a structure-driven closed-loop system of perception-understanding-execution-verification-recovery.

[0174] This application first integrates an image segmentation model and a feature encoding network to automatically extract key structural elements with physical semantics from environmental images, including point elements, line elements, and surface elements, and uses these to construct a unified spatial structure representation. Based on this, the system jointly inputs the natural language task and the structural information of the key elements into a multimodal language model to achieve phased semantic parsing and corresponding structural target generation for the task.

[0175] Subsequently, the system generates formalized spatiotemporal constraint functions based on the spatial relationships between key elements, and performs path planning and action control accordingly to ensure that the task execution process always satisfies the structural logic. Simultaneously, during task execution and after each stage, the system determines the success of the task by checking the state of key elements and structural constraints. If the constraints are not met, the system re-extracts key elements from the current image and automatically reconstructs the path, achieving intelligent error detection and recovery.

[0176] This application deeply integrates image perception and spatial structure modeling, and for the first time achieves a unified end-to-end process for task expression, constraint generation, execution control and anomaly recovery under the guidance of key elements. It has high precision, strong interpretability and good versatility, and is applicable to various intelligent operation platforms such as service robots and industrial manipulators.

[0177] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the robot operation method based on spatiotemporal constraint enhancement provided in the above embodiments.

[0178] This invention also provides a computer-readable medium having processor-executable non-volatile program code, on which a computer program is stored, and which, when run by a processor, executes the steps of the robot operation method based on spatiotemporal constraint enhancement described above.

[0179] The computer program product provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0180] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0181] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0182] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0184] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot manipulation method based on spatiotemporal constraint enhancement, characterized in that, The method includes: The original image is acquired, and the original image is segmented using the first image encoder and the language encoder in the SEEM model to obtain the segmentation result; The segmentation results are used to perform image annotation through the SAM model to obtain multiple masks and a confidence score corresponding to each mask; Calculate the aspect ratio and area of ​​the segmented region by analyzing multiple masks. The type of element is determined based on the aspect ratio and area of ​​the segmented region; wherein, the type of element includes point element, line element, and area element; The DINOv2 model is used to extract the type features of the elements, resulting in a high-dimensional feature vector. The high-dimensional feature vector is reduced in dimensionality to obtain the reduced high-dimensional feature vector. The reduced high-dimensional feature vectors are clustered using the K-Means clustering algorithm to output an annotated image; The key elements with color on the annotated image, the language instructions for the task, and the prompt word templates are input into the visual language big model to obtain the stage decomposition information and constraints of the task. The final trajectory is obtained by performing constraint optimization on the stage decomposition information and the constraints of the task. Obtain parameter information of the end effector and voxel information of the scene; The configuration of the key element is verified using the parameter information of the end effector, the physical location of the key element, the voxel information of the scene, and the constraint function, to obtain a verification result; wherein, the verification result includes whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

2. The robot operation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The SAM model includes a second image encoder, a cue encoder, and a mask decoder. The segmentation results are annotated using the SAM model to obtain multiple masks and a confidence score for each mask, including: The segmentation result is encoded using the second image encoder to obtain the encoded image; The prompts input by the user are converted by the prompt encoder to obtain prompts in vector form; The encoded image and the vector-based cue are mapped by the mask decoder into a plurality of masks and a confidence score corresponding to each mask.

3. The robot operation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The high-dimensional feature vector is reduced in dimensionality to obtain a reduced high-dimensional feature vector, including: The high-dimensional feature vector is centered using the PCA algorithm to obtain the centered data. Singular value decomposition is performed on the centered data to obtain a set of orthogonal principal component directions and corresponding variances; The PCA algorithm is used to select the k principal component directions with the largest variances from the corresponding variances. The high-dimensional feature vector is projected onto the directions of the k principal components with the largest variance to obtain the dimensionality-reduced high-dimensional feature vector.

4. The robot operation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The final trajectory is obtained by performing constraint optimization on the stage decomposition information and constraints of the task, including: The task's phase decomposition information and constraints are used to perform a global search using a double annealing algorithm to obtain a preliminary solution. The preliminary solution is optimized using the SLSQP algorithm to obtain the optimized solution; Using the optimized solution as a new starting point, the SLSQP algorithm is called again for local optimization until the number of iterations reaches a preset number or the cost is less than a preset threshold, at which point the iteration ends, thus obtaining the final trajectory.

5. The robot manipulation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The end effector's parameter information includes three-dimensional pose and environmental collision points; the three-dimensional pose is provided by a robot encoder and kinematic model, the environmental collision points are generated by physical simulation collision detection between the end effector grasping the object and other objects in the environment, and the voxel information of the scene is represented by a three-dimensional occupancy mesh constructed by a depth camera or point cloud sensor.

6. The robot manipulation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The method further includes: After each task phase is completed, the images captured by the robot and the phase breakdown information of the task are used as input. The visual language model is used to determine whether the current stage has been successfully executed. If the execution fails, you need to go back to the previous stage and solve the problem again.

7. A robot operating system based on spatiotemporal constraint enhancement, characterized in that, The system includes: The segmentation module is used to acquire the original image and segment the original image using the first image encoder and the language encoder in the SEEM model to obtain the segmentation result; The annotation module is used to annotate the segmentation results using the SAM model to obtain multiple masks and a confidence score corresponding to each mask; The calculation module is used to calculate the aspect ratio and area of ​​the segmented region from the multiple masks. The determining module is used to determine the type of element based on the aspect ratio and area of ​​the segmented region; wherein the type of element includes point element, line element and area element; The extraction module is used to extract the type features of the element through the DINOv2 model to obtain a high-dimensional feature vector; The dimension reduction module is used to reduce the dimension of the high-dimensional feature vector to obtain the dimension-reduced high-dimensional feature vector. The clustering module is used to cluster the reduced high-dimensional feature vectors using the K-Means clustering algorithm and output the labeled image. The input module is used to input the key elements with color on the annotated image, the language instructions for the task, and the prompt word template into the visual language big model to obtain the stage decomposition information and constraints of the task. The optimization solution module is used to perform constraint optimization on the stage decomposition information and constraints of the task to obtain the final trajectory; The acquisition module is used to acquire parameter information of the end effector and voxel information of the scene; The verification module is used to verify the configuration of the key element using the parameter information of the end effector, the physical location of the key element, the voxel information of the scene, and the constraint function, and to obtain the verification result; wherein, the verification result includes whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

8. The robot operating system based on spatiotemporal constraint enhancement according to claim 7, characterized in that, The SAM model includes a second image encoder, a cue encoder, and a mask decoder. The annotation module is specifically used for: The segmentation result is encoded using the second image encoder to obtain the encoded image; The prompts input by the user are converted by the prompt encoder to obtain prompts in vector form; The encoded image and the vector-based cue are mapped by the mask decoder into a plurality of masks and a confidence score corresponding to each mask.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 6.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method according to any one of claims 1 to 6.

Images