Robot operation method and system based on space-time constraint enhancement

Abstract

The invention provides a robot operation method and system based on space-time constraint enhancement. The precision and semantic consistency of space-time constraint expression are improved through a key element extraction mechanism; converting an operation target in the visual image into three types of basic geometric elements of structured points, lines and surfaces, and expressing a task target and spatial constraints on the basis of the basic geometric elements; the constraint condition not only covers the point-to-point distance, the angle between the line and the surface and the relative relation between the surfaces, but also can describe the complex structure dependence and the operation sequence. Structural logic of task verification is realized, and the accuracy of completeness judgment is improved; whether the current state conforms to the preset structure logic or not can be clearly judged in each stage by taking the condition that the key element state meets the structure constraint as a task completion criterion; the process forms a logic-driven self-adaptive recovery closed loop, supports dynamic adjustment of an execution plan after structure change, and greatly improves the fault-tolerant capability of the system in a complex dynamic environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot operation, in particular to a robot operation method and system based on spatiotemporal constraint enhancement. BACKGROUND

[0002] In current robot operation systems, in order to complete complex tasks with time-dependent and space-dependent relationships, researchers gradually introduce constraint mechanisms to improve the controllability and safety of task execution, with the aim of ensuring that the robot behavior meets the task semantics and physical constraint requirements.

[0003] Under this framework, constraint conditions mainly act on the following aspects: defining the movement range of the end effector; limiting the accessibility or placement conditions of the target object; rationalizing the operation path, speed or attitude change; and ensuring the accuracy of key stages in the operation.

[0004] Constraint generation methods can be roughly divided into two categories: one is static constraint expression based on artificial rules and templates, such as using fixed spatial thresholds, such as "distance less than ", "angle less than ", to set task completion conditions; the other is to automatically infer action goals and generate action prompts through visual language models based on input images and instructions, and to convert these prompts into corresponding geometric operation conditions. The latter is more versatile and semantically adaptable, and has become the mainstream trend.

[0005] To achieve the above functions, the system usually combines advanced visual segmentation models such as SAM, Grounding DINO, etc., to extract target object regions or semantic masks. These visual cues, together with task instructions, are used as multi-modal inputs for visual language models to perform task decomposition, action inference and sub-goal recognition.

[0006] Action prompts obtained through language model inference, such as "move above the object", "grab the middle", "place in the center of the platform", will be further translated into operation parameter inputs by "prompt word parsing modules" or "constraint converters" and executed by path planners or operators. In this process, constraint expression plays an important role in achieving path operation and target contact accuracy, and is one of the key guarantees for task execution accuracy and robustness.

[0007] In addition, some systems also build preliminary execution result verification and task recovery mechanisms. When the execution result does not meet the expectations, such as not entering the target area or the object not being placed stably, the system can interrupt the task, back up the path or reinitialize to achieve simple recovery.

[0008] In summary, the existing robot operating system has preliminarily established a "vision-language-operation" three-way linkage operation execution mode, can generate task constraints and execute actions through image perception and language reasoning, and has certain intelligent operation ability and fault response ability.

[0009] However, the above process has the following problems: 1) Lack of constraint expression ability driven by structural semantics The current system mainly relies on coarse-grained visual cues such as mask regions and bounding boxes, and artificially defined rule templates to generate constraint conditions. Due to the lack of fine geometric structure modeling, constraint expression is limited to simple semantics such as "close", "align" or "placed in the region", which cannot support modeling of structural relationships such as "alignment of line with face normal", "point must fall on stable support surface center", and "multiple elements coplanar".

[0010] The lack of geometric granularity and physical consistency of such constraint expression directly affects the spatial reasoning ability of the language model, limiting the execution precision and adaptability of the robot system for structure-sensitive tasks such as grasping direction, rotation action, and precise positioning and placement.

[0011] 2) Lack of error detection and recovery mechanism based on structural logic In the case of execution failure, the current system usually detects and recovers by the following methods: operation state abnormality detection (such as trajectory termination, boundary exceeding, and inability to converge); simple visual feedback judgment (such as target misalignment and deviation from the region); static path backtracking or action retry.

[0012] Such detection methods rely heavily on low-level feedback and lack logical verification related to task structure semantics. For example, the system cannot determine whether "the current position has reached the specified grasping point", "the gripper is aligned with the support surface normal", or "the object is stably placed on the surface", resulting in frequent misjudgment, low recovery efficiency, and even task dead loops. SUMMARY

[0013] Therefore, the purpose of the present application is to provide a robot operation method and system based on spatiotemporal constraint enhancement, which realizes the unification of task expression, constraint generation, execution control and abnormal recovery under the guidance of key elements, and has high precision, strong interpretability and good versatility.

[0014] In a first aspect, the embodiments of the present application provide a robot operation method based on spatiotemporal constraint enhancement, which comprises: obtaining an original image, and segmenting the original image through a first image encoder and a language encoder in a SEEM model to obtain a segmentation result; The segmentation result is subjected to image labeling by the SAM model to obtain a plurality of masks and a confidence score corresponding to each mask; The plurality of masks are calculated to obtain a length-width ratio of a segmentation region and an area size of the region; According to the length-width ratio of the segmentation region and the area size of the region, the type of the element is determined; wherein the type of the element includes a point-like element, a line-like element and a surface-like element; The features of the type of the element are extracted by the DINOv2 model to obtain a high-dimensional feature vector; The high-dimensional feature vector is reduced in dimension to obtain a reduced high-dimensional feature vector; The reduced high-dimensional feature vector is clustered by the K-Means clustering algorithm to output a labeled image; The colored key elements on the labeled image, the language instructions of the task and the prompt word templates are input into a visual language large model to obtain stage decomposition information and constraint conditions of the task; The stage decomposition information and the constraint conditions of the task are subjected to constraint optimization solution to obtain a final trajectory; Parameter information of an end effector and voxel information of a scene are obtained; The configuration of the key elements is verified by the parameter information of the end effector, the physical position of the key elements, the voxel information of the scene and a constraint function to obtain a verification result; wherein the verification result includes that the final trajectory conforms to a planned trajectory and the final trajectory deviates from the planned trajectory.

[0015] Further, the SAM model includes a second image encoder, a prompt encoder and a mask decoder, the segmentation result is subjected to image labeling by the SAM model to obtain a plurality of masks and a confidence score corresponding to each mask, including: The segmentation result is encoded by the second image encoder to obtain an encoded image; The prompt input by the user is converted by the prompt encoder to obtain a vector form prompt; The encoded image and the vector form prompt are mapped into a plurality of masks and a confidence score corresponding to each mask by the mask decoder.

[0016] Further, the high-dimensional feature vector is reduced in dimension to obtain a reduced high-dimensional feature vector, including: The high-dimensional feature vector is subjected to centering processing by the PCA algorithm to obtain centered data; The centered data is subjected to singular value decomposition to obtain a set of orthogonal principal component directions and corresponding variances; selecting, by the PCA algorithm, the first k principal component directions with the largest variances from the corresponding variances; projecting the high-dimensional feature vector onto the first k principal component directions with the largest variances to obtain the high-dimensional feature vector after dimension reduction.

[0017] Further, the stage decomposition information of the task and the constraint condition are subjected to constraint optimization solution to obtain a final trajectory, including: The stage decomposition information of the task and the constraint condition are subjected to global search by a double annealing algorithm to obtain a preliminary solution; The preliminary solution is subjected to optimization by an SLSQP algorithm to obtain an optimized solution; The optimized solution is taken as a new starting point, and the SLSQP algorithm is continuously called for local optimization until the number of iterations reaches a preset number or the cost is less than a preset threshold, so that the final trajectory is obtained.

[0018] Further, the parameter information of the end effector includes a three-dimensional pose and an environmental collision point; the three-dimensional pose is provided by a robot encoder and a kinematics model, the environmental collision point is generated by physical simulation collision detection of the end effector grabbing an object and other objects in the environment, and the voxel information of the scene is represented by a three-dimensional occupancy grid constructed by a depth camera or a point cloud sensor.

[0019] Further, the method further includes: When each task stage execution ends, the current robot shooting image and the stage decomposition information of the task are taken as inputs; Whether the current stage is successfully executed is determined by the visual language large model; If not, it needs to return to the previous stage to re-solve.

[0020] In a second aspect, an embodiment of the present application provides a robot operating system based on spatiotemporal constraint enhancement, which includes: A segmentation module is configured to obtain an original image, segment the original image by a first image encoder and a language encoder in a SEEM model, and obtain a segmentation result. A labeling module is configured to label the segmentation result by a SAM model to obtain a plurality of masks and a confidence score corresponding to each mask. A calculation module is configured to calculate the plurality of masks to obtain an aspect ratio of a segmented region and an area size of the region. A determination module is configured to determine a type of an element according to the aspect ratio of the segmented region and the area size of the region, wherein the type of the element includes a point-like element, a line-like element, and a surface-like element. extracting a feature of the type of the element through a DINOv2 model to obtain a high-dimensional feature vector; dimension reduction module, configured to reduce the dimension of the high-dimensional feature vector to obtain a reduced high-dimensional feature vector; The clustering module is configured to cluster the reduced high-dimensional feature vector through a K-Means clustering algorithm, and output an annotated image. The input module is configured to input the key elements with colors on the annotated image, language instructions of the task and prompt templates into a visual language large model to obtain stage decomposition information and constraint conditions of the task. The optimization solving module is configured to perform constraint optimization solving on the stage decomposition information and the constraint conditions of the task to obtain a final trajectory. The acquisition module is configured to acquire parameter information of an end effector and voxel information of a scene. The verification module is configured to verify a configuration of the key element through the parameter information of the end effector, the physical position of the key element, the voxel information of the scene and a constraint function to obtain a verification result, wherein the verification result includes that the final trajectory conforms to a planned trajectory and that the final trajectory deviates from the planned trajectory.

[0021] Further, the SAM model includes a second image encoder, a prompt encoder and a mask decoder, and the annotation module is specifically configured to: Encode the segmentation result through the second image encoder to obtain an encoded image; Convert the prompt input by a user through the prompt encoder to obtain a prompt in vector form; Map the encoded image and the prompt in vector form to a plurality of masks and a confidence score corresponding to each mask through the mask decoder.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory stores a computer program executable on the processor, and the processor implements the method described above when executing the computer program.

[0023] In a fourth aspect, an embodiment of the present application provides a computer readable medium having non-volatile program code executable by a processor, and the program code causes the processor to execute the method described above.

[0024] The embodiment of the application provides a robot operation method and system based on spatiotemporal constraint enhancement, which comprises the following steps: acquiring an original image, segmenting the original image through a first image encoder and a language encoder in a SEEM model to obtain a segmentation result; performing image labeling on the segmentation result through a SAM model to obtain a plurality of masks and a confidence score corresponding to each mask; calculating the plurality of masks to obtain the length-width ratio of a segmented area and the area size of the area; determining the type of an element according to the length-width ratio of the segmented area and the area size of the area; wherein the type of the element comprises a point-shaped element, a line-shaped element and a surface-shaped element; extracting the features of the type of the element through a DINOv2 model to obtain a high-dimensional feature vector; reducing the dimension of the high-dimensional feature vector to obtain a reduced high-dimensional feature vector; clustering the reduced high-dimensional feature vector through a K-Means clustering algorithm to output a labeled image; inputting the key elements with colors on the labeled image, language instructions of a task and a prompt word template into a visual language large model to obtain stage decomposition information and constraint conditions of the task; performing constraint optimization solving on the stage decomposition information and the constraint conditions of the task to obtain a final trajectory; acquiring parameter information of an end effector and voxel information of a scene; verifying the configuration of the key elements through the parameter information of the end effector, the physical position of the key elements, the voxel information of the scene and a constraint function to obtain a verification result; wherein the verification result comprises that the final trajectory conforms to a planned trajectory and the final trajectory deviates from the planned trajectory; realizing the whole-link unification of task expression, constraint generation, execution control and abnormal recovery under the guidance of key elements, and having high precision, strong interpretability and good universality.

[0025] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the description, claims and drawings.

[0026] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0028] Figure 1 The robot operation method based on spatiotemporal constraint enhancement provided by the embodiment of the present application is shown in the flowchart; Figure 2 A robot operation process diagram based on spatiotemporal constraint enhancement provided for the first embodiment of the present application; Figure 3 A key element extraction process diagram provided for the first embodiment of the present application; Figure 4 A flowchart of a two-layer error detection and recovery mechanism provided for the first embodiment of the present application; Figure 5 A robot operation system diagram based on spatiotemporal constraint enhancement provided for the second embodiment of the present application. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in connection with the drawings, which apparently are only some of the embodiments of the present application but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort should fall within the protection scope of the present application.

[0030] The key element extraction mechanism proposed in the present application abstracts the task-related structures in an image in the form of points, lines, and surfaces, endows the system with the ability to understand and express scene structures, and constructs constraint expression and state verification logic centered on key elements, so as to achieve the following improvements: Constraint expression is more structured: supporting point-line-surface-based constraint expression of task geometric relationships; Error detection is more logical: by judging whether the state of a key element satisfies constraint logic, it is clear to judge whether a task is successful; Recovery path reconstruction is more directional: based on the current image, key elements can be re-extracted, new sub-targets and constraints can be dynamically generated, and efficient adaptive recovery is supported.

[0031] Therefore, the introduction of key element structure modeling not only makes up for the shortcomings of existing systems in constraint expression and task verification, but also provides a technical foundation for building a truly universal, high-precision, and strong-robustness robot task execution system.

[0032] To facilitate the understanding of the present embodiment, the present embodiment will be described in detail below.

[0033] Embodiment One: Figure 1 A robot operation method flowchart based on spatiotemporal constraint enhancement provided for the first embodiment of the present application.

[0034] With reference to Figure 1 The method comprises the following steps: Step S101, an original image is obtained, and the original image is segmented by a first image encoder and a language encoder in a SEEM model to obtain a segmentation result; Step S102, the segmentation result is subjected to image labeling by a SAM model to obtain a plurality of masks and a confidence score corresponding to each mask; Step S103, the plurality of masks are calculated to obtain a length-width ratio of a segmented region and an area size of the region; Step S104, the type of an element is determined according to the length-width ratio of the segmented region and the area size of the region; wherein the type of the element includes a point-shaped element, a line-shaped element and a surface-shaped element; Specifically, the process of key element extraction starts from an original image, and is segmented by a Set-of-Mark (SoM). The SoM integrates a plurality of currently advanced segmentation models, including SEEM, SAM and Semantic-SAM, and supports automatic segmentation and labeling of images of multiple granularities.

[0035] For low-granularity segmentation such as recognizing an object in an original image, for example, “a cup”, the SoM selects the SEEM model for relatively rough segmentation. Segment Everything Everywhere All at Once (SEEM) is a multi-modal image segmentation model proposed by Meta AI. It can use natural language, points, frames and other forms of prompts to segment images, and does not require additional training for images that have not been seen before. A multi-modal encoder composed of a first image encoder and a language encoder is used to cross-extract features of images and texts, and finally extract segmentation results with semantic details.

[0036] For higher-granularity segmentation such as recognizing a part of an object in an original image, for example, “the handle of the cup”, the SoM selects the Semantic-SAM model and the SAM model for more detailed image labeling. Segment Anything Model (SAM) is a visual basic model proposed by Meta AI.

[0037] Step S105, features of the type of the element are extracted by a DINOv2 model to obtain a high-dimensional feature vector; Step S106, the high-dimensional feature vector is reduced in dimension to obtain a reduced high-dimensional feature vector; Step S107, the reduced high-dimensional feature vector is clustered by a K-Means clustering algorithm to output a labeled image; Step S108, input the key elements with color, language instructions of the task and prompt templates on the labeled image into the visual language large model to obtain stage decomposition information and constraint conditions of the task; Step S109, constraint optimization is performed on the stage decomposition information and constraint conditions of the task to obtain a final trajectory; Step S110, obtain parameter information of the end effector and voxel information of the scene; Step S111, verify the configuration of the key elements by using the parameter information of the end effector, the physical position of the key elements, the voxel information of the scene and the constraint function to obtain a verification result; wherein the verification result includes that the final trajectory conforms to the planned trajectory and the final trajectory deviates from the planned trajectory.

[0038] The application introduces a structure-driven key element extraction and expression mechanism, and builds a time and space constraint modeling, action planning, state verification and error recovery closed loop in the whole process of the task based on the mechanism. The mechanism breaks the fragmented mode between "semantic prompt, rule constraint and low-level control" in the traditional robot operating system, and establishes a unified, high-precision and interpretable robot operation control system.

[0039] Further, the SAM model includes a second image encoder, a prompt encoder and a mask decoder, and step S102 includes the following steps: Step S201, encode the segmentation result through the second image encoder to obtain an encoded image; Step S202, convert the prompt input by the user through the prompt encoder to obtain a vector form prompt; Step S203, map the encoded image and the vector form prompt to a plurality of masks and a confidence score corresponding to each mask through the mask decoder.

[0040] Specifically, the SAM model includes a second image encoder, a prompt encoder and a mask decoder. The second image encoder uses a Vision Transformer (ViT) pre-trained by MAE and runs once for each image. The prompt encoder considers both discrete and continuous prompts, and is responsible for converting the user input prompt into a vector form that the model can understand. The main function of the mask decoder is to map image embedding, prompt embedding and output tokens into masks, and the model finally outputs a plurality of valid masks and their confidence scores.

[0041] Compared with the SAM model, Semantic-SAM is an enhanced version of the original SAM model in terms of semantics, which can add semantic information tags to the mask segmented by SAM, making up for the defect that SAM cannot recognize object categories. This model can extract regional image blocks from the instance mask obtained by SAM, and then calculate the similarity of each category to complete semantic classification using a text-image alignment model and a category word table. It achieves semantic segmentation while also giving the mask certain semantic meaning.

[0042] SoM divides the image into multiple regions, and all processed masks can be classified into three types of elements: points, lines, and surfaces, by calculating the aspect ratio of the segmented region and the area size of the region. These two geometric features directly reflect the morphology of the object structure: regions with small area and approximate square shape usually represent point elements such as buttons or fingertips; elongated regions with extremely large aspect ratios represent line elements such as edges or wires; and regions with larger area and moderate aspect ratio represent surface elements such as desktops or object bodies. Such division helps subsequent tasks to process and understand more targeted according to the element structure.

[0043] Further, step S106 includes the following steps: Step S301, centering the high-dimensional feature vector by the PCA algorithm to obtain the centered data; Step S302, singular value decomposition is performed on the centered data to obtain a set of orthogonal principal component directions and corresponding variances; Step S303, the PCA algorithm selects the first k principal component directions with the largest variance from the corresponding variances; Step S304, project the high-dimensional feature vector onto the first k principal component directions with the largest variance to obtain the reduced high-dimensional feature vector.

[0044] Specifically, the DINOv2 model is used to extract important features of the three elements of points, lines, and surfaces in the image, including semantic representation, geometric shape, spatial position, and context relationship, etc. high-dimensional vector features. Then, the PCA algorithm first centers the high-dimensional feature vector, i.e. each feature is subtracted from its mean value, and the covariance matrix is calculated to reflect the linear relationship between the features.

[0045] Then, singular value decomposition is performed on the centered data to obtain a set of orthogonal principal component directions and corresponding variance sizes. The PCA algorithm selects the first k principal component directions with the largest variance, projects the high-dimensional feature vector onto these k directions, thereby realizing dimension reduction while retaining as much original information as possible.

[0046] In image feature processing, the high-dimensional feature vectors extracted using the DINOv2 model can be effectively compressed using the PCA algorithm to remove redundant features, providing a more compact representation for subsequent clustering.

[0047] The K-Means clustering algorithm is then used to cluster these features, and the centroid of each cluster is projected onto the original image and assigned a corresponding numerical code. Different types of elements are distinguished by different colored keypoints. If the number of keypoints for a line or surface element is insufficient, keypoint extraction for that type needs to be performed again. The flowchart for key element extraction is shown below. Figure 3 As shown.

[0048] Specifically, this application uses the extracted key elements as the basis for the direct constraint expression of task execution and designs a standard structural logic constraint function system.

[0049] Within the framework of this application, a constraint is defined as a mapping function. It is a collection of key elements of a point, line, and surface. Mapped to a real number, representing the cost of the set. This cost is unbounded, but when When this condition is met, it indicates that the constraint has been satisfied. The constraint function corresponding to each key element. The key element's sequence number.

[0050] The function f is stateless and calculates costs by performing a series of operations on key elements. These operations are typically non-linear and may be non-convex. The function f is used to represent the desired spatial relationships between key elements and is suitable for robotic arms and object parts.

[0051] Each task is broken down into N stages, and for each stage... Specify two types of constraints: 1) Sub-path constraints: Constraints that need to be satisfied during the execution of sub-goals, expressed as... express.

[0052] 2) Sub-goal constraints: Constraints that need to be satisfied when a sub-goal is completed, expressed as... express.

[0053] The set of all constraints can be represented as: .

[0054] Constraint set for each stage It can be represented by a combination of sub-path constraints and sub-goal constraints, i.e. .

[0055] In this part of the constraint generation, the original image observed by the robot camera is O, and the visual cues after key element extraction are The visual cues , the language instructions L of the task and the cue template P are input into the visual language large model to obtain the stage decomposition information I of the task and the related constraints C, wherein the visual cues are the key elements with colors on the labeled images. This process can refer to formula (1): (1) Wherein, is the process of abstracting the use of the visual language model into a function, and VLM represents the visual language large model.

[0056] For example, the task of "re-adjust the direction of the white pen and insert it vertically into the black pen container", the visual language large model decomposes the task into 3 stages, which are "grab the white pen", "re-adjust the direction of the white pen to the vertical direction", and "release the white pen and insert it vertically into the black pen container". The constraint conditions of each stage can be expressed in the form of Python functions. For example, in the second stage of the task, the sub-path constraint function requires the robot to grab the white pen during the movement of the robot arm, and the sub-goal constraint function requires the vector composed of two key points to be aligned with the z-axis to ensure that the white pen is in the vertical direction.

[0057] Further, step S109 includes the following steps: Step S401, the stage decomposition information and the constraints of the task are searched globally by a double annealing algorithm to obtain a preliminary solution; Step S402, the preliminary solution is optimized by the SLSQP algorithm to obtain an optimized solution; Step S403, the optimized solution is taken as a new starting point, and the SLSQP algorithm is continuously called for local optimization until the iteration number reaches a preset number, or the iteration ends when the cost is less than a preset threshold, thereby obtaining a final trajectory.

[0058] Specifically, in the constraint optimization and solving part, we convert the operation task of the robot into a constraint optimization and solving problem containing multi-stage sub-path constraints and sub-goal constraints. The pose of the robot end effector is represented as In the process of the robot performing the operation task, the objective function of the constraint optimization and solving is the overall discrete-time trajectory composed of the poses of the robot end effector at different times , which can refer to formula (2): (2) Wherein, represents the auxiliary cost function of each sub-path problem in the i-th stage, The auxiliary cost function represents the ith stage of each sub-target problem, including collision cost, path length cost, accessibility cost, etc. In each stage, any element in the element set needs to meet the sub-path constraint and the sub-target constraint.

[0059] All constraint optimization problems are implemented and solved using SciPy, and the decision variables are normalized to [0, 1]. The initial stage uses a dual annealing algorithm for global search, which combines simulated annealing and interval decomposition strategy, and can effectively explore multiple possible solutions in complex non-convex space.

[0060] After obtaining the preliminary solution, the SLSQP (Sequential Least Squares Programming) algorithm is used for local fine optimization, which can efficiently handle constrained nonlinear problems.

[0061] Subsequently, in subsequent iterations, each time the solution obtained by the previous optimization is taken as the new starting point, the SLSQP algorithm is called for local optimization, so as to gradually approach the optimal solution in the solution space until the iteration number reaches the preset number, or the cost is less than the preset threshold, and the iteration is ended, so as to obtain the final trajectory; thereby realizing the cooperative optimization process of global exploration combined with local convergence. The whole process is realized and scheduled by the SciPy optimization module.

[0062] Further, the parameter information of the end effector includes three-dimensional pose and environmental collision points; the three-dimensional pose is provided by the robot encoder and the kinematics model, and the environmental collision points are generated by physical simulation collision detection of the end effector grabbing objects and other objects in the environment, and the voxel information of the scene is represented by a three-dimensional occupancy grid constructed by a depth camera or a point cloud sensor.

[0063] Specifically, the application also introduces an error detection and recovery mechanism for key elements. In order to enhance the robustness of the system in actual task execution, this research introduces an automatic detection and recovery mechanism for dynamic interference in the main task execution process, which is divided into two layers of constraint verification and visual language large model verification. The constraint verification mechanism is for error detection and recovery during each stage of execution, and the visual language large model verification mechanism is for error detection and recovery after each stage of execution. The flow chart of the error detection and recovery mechanism is shown in Figure 4 .

[0064] For the constraint verification mechanism, during the execution of each task stage, the system obtains the three-dimensional pose of the end effector (provided by the robot encoder and kinematics model), the physical position of the key elements (converted from image coordinates by image segmentation results combined with camera parameters), the environment collision points (generated by physical simulation collision detection of the end effector, the grasped object and other objects in the environment), and the voxel information of the scene (a three-dimensional occupancy grid representation constructed by a depth camera or a point cloud sensor), and uses the currently loaded constraint functions to check the current key element configuration. Among them, the constraint functions include sub-path constraint functions and sub-goal constraint functions. For example, the constraint functions can check whether the end effector is within the allowed operating area and whether the key elements are in the expected spatial relationship. If any constraint violates the pre-set tolerance threshold, it means that the execution process has deviated from the planned route due to external forces such as target object movement or failed grasping, and the backtracking logic needs to be started to perform the corresponding rollback operation.

[0065] Once a violated constraint is detected during execution, the system will check in reverse order whether all constraints of the previous stage are still satisfied from the current stage. If all constraint conditions of a historical stage are satisfied, the system will backtrack to that stage and re-register the key element state, re-plan the sub-goal and sub-path to restore to a stable execution state. This process does not require human intervention and has high automation.

[0066] Further, the method further includes the following steps: Step S501, when each task stage execution ends, the current robot's shooting image and the task's stage decomposition information are taken as inputs; Step S502, determine whether the current stage is successfully executed by the visual language large model; Step S503, if not, return to the previous stage to solve again.

[0067] Specifically, for the visual language large model verification mechanism, after each task stage execution ends, the current robot's shooting image and the task's decomposition information in the constraint generation stage are taken as image input and text input into the visual language large model. The visual language large model determines whether the current stage is successfully executed according to the image and the text, and if not, returns to the previous stage to solve again.

[0068] Reference Figure 2 The present application mainly includes four modules: key element extraction, constraint condition generation, constraint optimization solving and error detection and recovery.

[0069] The key element extraction module is responsible for extracting key elements from the image as image input, combining with the text input to query the visual language large model, generating corresponding constraint conditions by the constraint condition generation module, and optimizing and solving the generated constraint conditions by the constraint optimization and solving module to make the robot execute the operation task. The error detection and recovery module detects errors that occur during the execution of the robot operation task, backtracks and makes the constraint optimization and solving module re-plan.

[0070] For the key element extraction module, this part adopts a large-scale pre-trained visual model to automatically extract key elements from the scene image. The visual model analyzes different objects in the scene and identifies important feature regions, determines the type of elements (such as points, lines, and surfaces) according to the spatial geometric characteristics of the objects. These elements represent important positions in the scene, such as object grasping points or orientations, to assist in the overall understanding of the environment. At the same time, after the extraction of key elements once, if the number of key points extracted is insufficient to represent the type of key elements, the key elements need to be extracted again, thereby better representing the spatial geometric characteristics of the task.

[0071] For the constraint condition generation module, the key elements extracted in the previous stage are first marked on the original image as visual cues. Then, according to the natural language description of the task and the cue information on the image, the visual language large model is used to generate the stage goal and constraint conditions required by the task. The key to this process is that the generated constraint conditions can be adjusted accordingly with the changes of the robot itself and the environment, so that the robot can adapt to different situations and successfully complete the task.

[0072] For the constraint optimization and solving module, the main task of this part is to model the robot operation task as a constraint optimization problem. The task is divided into multiple stages, and the optimization goal of each stage is to solve the pose of the robot end effector, ensuring that the continuous trajectory from the initial position to the completion of the task meets all the constraint conditions in the optimization process. The key elements are updated in real time during the execution of the robot operation task and are visually displayed, and the system can quickly respond to dynamic environmental changes to obtain the final trajectory.

[0073] For the error detection and recovery module, at each stage of the robot operation, the system detects whether there is a behavior that violates the constraint condition. When the task of a stage is not successfully completed, the system queries the visual language large model combining the current scene image and task information to determine whether the stage is completed. If the current stage task is not completed, the system will backtrack to the historical stage and re-plan the execution of the task through constraint optimization and solving to ensure the robustness of the robot system and complete the closed-loop operation control.

[0074] The application has the following beneficial effects: The application improves the accuracy and semantic consistency of space-time constraint expression through a key element extraction mechanism. For the first time, the operation target in a visual image is converted into three basic geometric elements of points, lines, and surfaces, and the task target and spatial constraints are expressed based on this. The constraint conditions not only cover the distance between points, the angle between lines and surfaces, and the relative relationship between surfaces, but also can describe complex structural dependencies and operation sequences. Therefore, the application is significantly superior to existing methods in terms of expression granularity and geometric accuracy, making the task description, path planning, and execution control consistent and reducing execution failures caused by semantic ambiguity or geometric mismatch.

[0075] The application realizes the structural logic of task verification and improves the accuracy of completion discrimination. By using "key element state satisfying structural constraints" as the task completion criterion, it can be determined whether the current state is consistent with the preset structural logic at each stage. This process constitutes a logically driven adaptive recovery closed loop, which supports dynamic adjustment of the execution plan after structural changes, significantly improving the fault tolerance of the system in complex dynamic environments. The application significantly improves the accuracy, interpretability, and stability of task verification, and is particularly suitable for industrial and service robot scenarios with high execution accuracy requirements and complex operation processes.

[0076] Embodiment two Figure 5 A robot operation system based on space-time constraint enhancement is provided for the second embodiment of the application.

[0077] Reference Figure 5 The system comprises: A segmentation module is configured to obtain an original image, segment the original image through a first image encoder and a language encoder in a SEEM model, and obtain a segmentation result. A labeling module is configured to label the segmentation result through a SAM model to obtain a plurality of masks and a confidence score corresponding to each mask. A calculation module is configured to calculate the plurality of masks to obtain the aspect ratio of a segmented region and the area size of the region. A determination module is configured to determine the type of an element according to the aspect ratio of the segmented region and the area size of the region, wherein the type of the element includes a point-like element, a line-like element, and a surface-like element. An extraction module is configured to extract features of the type of the element through a DINOv2 model to obtain a high-dimensional feature vector. A dimension reduction module is configured to reduce the dimension of the high-dimensional feature vector to obtain a reduced high-dimensional feature vector. A clustering module is configured to cluster the reduced high-dimensional feature vector through a K-Means clustering algorithm to output a labeled image. The input module is used to input the key elements with color on the labeled image, the language instructions for the task, and the prompt word templates into the visual language big model to obtain the stage decomposition information and constraints of the task. The optimization and solution module is used to perform constraint optimization on the stage decomposition information and constraints of the task to obtain the final trajectory; The acquisition module is used to acquire parameter information of the end effector and voxel information of the scene; The verification module is used to verify the configuration of key elements by using the parameter information of the end effector, the physical location of key elements, the voxel information of the scene, and the constraint functions, and to obtain the verification results. The verification results include whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

[0078] Furthermore, the SAM model includes a second image encoder, a cue encoder, and a mask decoder, and the annotation module is specifically used for: The segmentation results are encoded using a second image encoder to obtain the encoded image; The user-input prompts are converted into vector form using a prompt encoder. The encoded image and the vector-based cue are mapped by a mask decoder into multiple masks and a confidence score for each mask.

[0079] Specifically, this application proposes a robot operation method and system based on spatiotemporal constraint enhancement, aiming to improve the robot's operational accuracy, robustness, and task adaptability in complex task scenarios. The method revolves around "key element extraction" and constructs a structure-driven closed-loop system of perception-understanding-execution-verification-recovery.

[0080] This application first integrates an image segmentation model and a feature encoding network to automatically extract key structural elements with physical semantics from environmental images, including point elements, line elements, and surface elements, and uses these to construct a unified spatial structure representation. Based on this, the system jointly inputs the natural language task and the structural information of the key elements into a multimodal language model to achieve phased semantic parsing and corresponding structural target generation for the task.

[0081] Subsequently, the system generates formalized spatiotemporal constraint functions based on the spatial relationships between key elements, and performs path planning and action control accordingly to ensure that the task execution process always satisfies the structural logic. Simultaneously, during task execution and after each stage, the system determines the success of the task by checking the state of key elements and structural constraints. If the constraints are not met, the system re-extracts key elements from the current image and automatically reconstructs the path, achieving intelligent error detection and recovery.

[0082] The application deeply integrates image perception and spatial structure modeling, and for the first time realizes the whole-link unification of task expression, constraint generation, execution control and abnormal recovery under the guidance of key elements, has high precision, strong interpretability and good versatility, and is suitable for various intelligent operation platforms such as service robots and industrial operation arms.

[0083] The embodiment of the application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the steps of the robot operation method based on the spatiotemporal constraint enhancement provided by the above embodiment.

[0084] The embodiment of the application further provides a computer readable medium with non-volatile program codes executable by a processor, and the computer readable medium stores a computer program, and the computer program is executed by the processor to execute the steps of the robot operation method based on the spatiotemporal constraint enhancement of the above embodiment.

[0085] The computer program product provided by the embodiment of the application includes a computer readable storage medium storing program codes, and the program codes include instructions for executing the method described in the foregoing method embodiment, and specific implementation can be referred to the method embodiment, and will not be repeated here.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0087] In addition, in the description of the embodiment of the application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood in a broad sense, for example, can be fixedly connected, can also be detachably connected, or integrally connected, can be mechanically connected, can also be electrically connected, can be directly connected, can be indirectly connected through an intermediate medium, and can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0088] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0089] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0090] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art within the technical range disclosed by the present application can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A robot manipulation method based on spatiotemporal constraint enhancement, characterized in that, The method includes: The original image is acquired, and the original image is segmented using the first image encoder and the language encoder in the SEEM model to obtain the segmentation result; The segmentation results are used to perform image annotation through the SAM model to obtain multiple masks and a confidence score corresponding to each mask; Calculate the aspect ratio and area of ​​the segmented region by analyzing multiple masks. The type of element is determined based on the aspect ratio and area of ​​the segmented region; wherein, the type of element includes point element, line element, and area element; The DINOv2 model is used to extract the type features of the elements, resulting in a high-dimensional feature vector. The high-dimensional feature vector is reduced in dimensionality to obtain the reduced high-dimensional feature vector. The reduced high-dimensional feature vectors are clustered using the K-Means clustering algorithm to output an annotated image; The key elements with color on the annotated image, the language instructions for the task, and the prompt word templates are input into the visual language big model to obtain the stage decomposition information and constraints of the task. The final trajectory is obtained by performing constraint optimization on the stage decomposition information and the constraints of the task. Obtain parameter information of the end effector and voxel information of the scene; The configuration of the key element is verified using the parameter information of the end effector, the physical location of the key element, the voxel information of the scene, and the constraint function, to obtain a verification result; wherein, the verification result includes whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

2. The robot operation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The SAM model includes a second image encoder, a cue encoder, and a mask decoder. The segmentation results are annotated using the SAM model to obtain multiple masks and a confidence score for each mask, including: The segmentation result is encoded using the second image encoder to obtain the encoded image; The prompts input by the user are converted by the prompt encoder to obtain prompts in vector form; The encoded image and the vector-based cue are mapped by the mask decoder into a plurality of masks and a confidence score corresponding to each mask.

3. The robot operation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The high-dimensional feature vector is reduced in dimensionality to obtain a reduced high-dimensional feature vector, including: The high-dimensional feature vector is centered using the PCA algorithm to obtain the centered data. Singular value decomposition is performed on the centered data to obtain a set of orthogonal principal component directions and corresponding variances; The PCA algorithm is used to select the k principal component directions with the largest variances from the corresponding variances. The high-dimensional feature vector is projected onto the directions of the k principal components with the largest variance to obtain the dimensionality-reduced high-dimensional feature vector.

4. The robot operation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The final trajectory is obtained by performing constraint optimization on the stage decomposition information and constraints of the task, including: The task's phase decomposition information and constraints are used to perform a global search using a double annealing algorithm to obtain a preliminary solution. The preliminary solution is optimized using the SLSQP algorithm to obtain the optimized solution; Using the optimized solution as a new starting point, the SLSQP algorithm is called again for local optimization until the number of iterations reaches a preset number or the cost is less than a preset threshold, at which point the iteration ends, thus obtaining the final trajectory.

5. The robot manipulation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The end effector's parameter information includes three-dimensional pose and environmental collision points; the three-dimensional pose is provided by a robot encoder and kinematic model, the environmental collision points are generated by physical simulation collision detection between the end effector grasping the object and other objects in the environment, and the voxel information of the scene is represented by a three-dimensional occupancy mesh constructed by a depth camera or point cloud sensor.

6. The robot manipulation method based on spatiotemporal constraint enhancement according to claim 1, characterized in that, The method further includes: After each task phase is completed, the images captured by the robot and the phase breakdown information of the task are used as input. The visual language model is used to determine whether the current stage has been successfully executed. If the execution fails, you need to go back to the previous stage and solve the problem again.

7. A robot operating system based on spatiotemporal constraint enhancement, characterized in that, The system includes: The segmentation module is used to acquire the original image and segment the original image using the first image encoder and the language encoder in the SEEM model to obtain the segmentation result; The annotation module is used to annotate the segmentation results using the SAM model to obtain multiple masks and a confidence score corresponding to each mask; The calculation module is used to calculate the aspect ratio and area of ​​the segmented region from the multiple masks. The determining module is used to determine the type of element based on the aspect ratio and area of ​​the segmented region; wherein the type of element includes point element, line element and area element; The extraction module is used to extract the type features of the element through the DINOv2 model to obtain a high-dimensional feature vector; The dimension reduction module is used to reduce the dimension of the high-dimensional feature vector to obtain the dimension-reduced high-dimensional feature vector. The clustering module is used to cluster the reduced high-dimensional feature vectors using the K-Means clustering algorithm and output the labeled image. The input module is used to input the key elements with color on the annotated image, the language instructions for the task, and the prompt word template into the visual language big model to obtain the stage decomposition information and constraints of the task. The optimization solution module is used to perform constraint optimization on the stage decomposition information and constraints of the task to obtain the final trajectory; The acquisition module is used to acquire parameter information of the end effector and voxel information of the scene; The verification module is used to verify the configuration of the key element using the parameter information of the end effector, the physical location of the key element, the voxel information of the scene, and the constraint function, and to obtain the verification result; wherein, the verification result includes whether the final trajectory conforms to the planned trajectory and whether the final trajectory deviates from the planned trajectory.

8. The robot operating system based on spatiotemporal constraint enhancement according to claim 7, characterized in that, The SAM model includes a second image encoder, a cue encoder, and a mask decoder. The annotation module is specifically used for: The segmentation result is encoded using the second image encoder to obtain the encoded image; The prompts input by the user are converted by the prompt encoder to obtain prompts in vector form; The encoded image and the vector-based cue are mapped by the mask decoder into a plurality of masks and a confidence score corresponding to each mask.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 6.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method according to any one of claims 1 to 6.

Images