Robot operation planning method and system based on self-adaptive visual prompt

Abstract

The invention relates to a robot operation planning method and system based on self-adaptive visual prompt. The method comprises the following steps: firstly, acquiring a first visual angle image and a task text instruction of a robot, and extracting a task related object list and a preliminary positioning area by using a pre-trained visual language model; visual key point detection is carried out by combining the data, and an operation candidate key point set and corresponding pixel coordinate marks are generated; according to a task instruction, candidate key point marks and the like, optimal grabbing points, path points and target points are selected from the set, pixel coordinates of the optimal grabbing points, the path points and the target points and the target direction of an end effector are output, and three-dimensional coordinates under a coordinate system of a coordinate robot are connected to generate a planned path of the end effector; and finally, in combination with the target direction and the planned path, solving the joint position and generating a continuous trajectory to complete operation planning. Compared with the prior art, the method has the advantages that the reasoning time is shortened, and the method can be efficiently and widely applied to various scenes.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence, and in particular to a robot operation planning method and system based on adaptive visual cues. Background Technology

[0002] In the application of modern robotics, with the diversification of tasks and the increasing complexity of environments, traditional robot operation planning methods face numerous challenges. Existing robot control methods based on imitation learning or reinforcement learning often require training on large-scale datasets to cope with different tasks and scenarios. Moreover, whenever the task or environment changes, the robot system needs to re-collect data and retrain. This reliance on data acquisition and training is not only time-consuming and labor-intensive but also makes it difficult to effectively plan operations in rapidly changing or highly uncertain environments.

[0003] Visual language large models (VLMs), by integrating visual perception and natural language understanding capabilities, promise to enhance the adaptability and flexibility of robots in diverse tasks. By learning large-scale prior knowledge, VLMs can quickly reason and generate appropriate instructions when faced with new tasks and environmental changes, demonstrating stronger generalization capabilities. Patent CN120472173A discloses a method and system for enhancing perception using VLMs based on visual cues, but it does not explore integration with a robot operating system. Visual language motion large models further integrate robot joint movements into multimodal data training, directly outputting robot joint movements based on task text descriptions and visual observation inputs. However, they still have the following drawbacks:

[0004] 1) Although the visual-language large model can handle cross-modal input of vision and language, it lacks the display utilization of visual information and cannot fully and effectively extract the key features required for the task from the visual information, resulting in a low task execution success rate of real robot systems.

[0005] 2) Processing large amounts of high-dimensional, multimodal input information consumes enormous resources and results in slow model inference speed. In real-time task execution by robots, latency can affect the robot's response time, thereby impacting task efficiency.

[0006] Therefore, current robot training methods generally suffer from long inference times, low efficiency, and low applicability to various scenarios. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a robot operation planning method and system based on adaptive visual cues.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] According to one aspect of the present invention, a robot operation planning method based on adaptive visual cues is provided, characterized in that the method steps include:

[0010] S1. Obtain robot first-person view image data and task text instruction data. Based on the above data, use a pre-trained visual language model to extract a list of task-related objects and their preliminary localization areas.

[0011] S2. Based on the robot's first-view image data, the list of task-related objects and their preliminary localization areas, perform visual key point detection, generate and output a set of candidate key points for the operation of task-related objects and pixel coordinate labels of the candidate key points in the first-view image.

[0012] S3. Based on the task text instruction data and the pixel coordinate annotation of candidate key points in the first-view image, the optimal gripping point, path point and target point are predicted and selected from the set of operation candidate key points, and the pixel coordinates of the optimal gripping point, path point and target point and the target direction of the robot end effector are output.

[0013] S4. Using the coordinate transformation of the hand-eye calibration algorithm, the pixel coordinates of the optimal grasping point, path point and target point are transformed into three-dimensional coordinates in the robot coordinate system;

[0014] S5. Connect the optimal gripping point, path point and target point in the robot coordinate system in sequence, and perform smoothing to generate the robot end effector planning path;

[0015] S6. Based on the target direction of the robot end effector and the planned path of the robot end effector, solve the position of each joint of the robot, generate the continuous execution trajectory of each joint, and complete the robot operation planning.

[0016] As a preferred technical solution, the first-view image data includes RGB images and depth images from the robot's first-view perspective.

[0017] As a preferred technical solution, the specific steps in S2 for generating a set of candidate key points for task-related objects and labeling the pixel coordinates of the candidate key points in the first-view image include:

[0018] The robot's first-person RGB image and the task-related objects are input into the DINO-X model for encoding, and the mask of the task-related objects is extracted.

[0019] For each task-related object's mask, use the K-means algorithm for cluster analysis to obtain the cluster center points as candidate key points for that object;

[0020] Candidate keypoints are color-coded at the two-dimensional pixel coordinates of the cluster center to ensure that candidate keypoints of different objects use different colors, while the candidate keypoints of the same object have the same color. The RGB image after labeling serves as visual cue information.

[0021] As a preferred technical solution, the specific steps of S3 include:

[0022] The system prompts for optimal selection are constructed. The VLM accepts RGB images of visual cue tags and outputs two-dimensional pixel coordinates of the grab point, waypoint, and target point.

[0023] The optimal grasping point, path point, and target point are labeled on the RGB image of the robot's first-person view to serve as a visual cue for predicting the grasping direction of the end effector.

[0024] The system provides cue words for predicting the direction of the end effector and defines multiple predefined grasping directions. The VLM receives new visual cues and selects the optimal grasping direction from the predefined grasping directions.

[0025] As a preferred technical solution, the specific process of coordinate transformation in S4 includes:

[0026] Obtain the depth information of pixels in the RGB image in the depth image, and convert the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system;

[0027] Through coordinate transformation matrix Transform the 3D coordinates in the camera coordinate system into the 3D coordinates in the robot coordinate system;

[0028] For the grasping point and the target point: keep their three-dimensional coordinates unchanged to ensure that the grasping action is accurately aligned with the target object and the placement action is accurately placed in the target area;

[0029] For path points: Adjust their z-axis coordinates based on depth parameters, including the preset safe height of the operation scenario and the object placement height.

[0030] Adjust the z-axis coordinate of the path point r Furthermore, the three-dimensional coordinates of the grab point and the target point remain unchanged, and the target depth is calculated using the above formula.

[0031] As a preferred technical solution, the specific formula for coordinate transformation in S4 is as follows:

[0032]

[0033] Where u,v are two-dimensional pixel coordinates, x c ,y c ,z c The coordinates are in the camera coordinate system (3D). Let x be the intrinsic parameter matrix of the camera. r ,y r ,z r The coordinates are in the robot's three-dimensional coordinate system. This is the transformation matrix from the camera coordinate system to the robot coordinate system.

[0034] As a preferred technical solution, the inverse kinematics method is specifically used in S6 to solve the position of each joint of the robot.

[0035] According to another aspect of the present invention, a robot operation planning system based on adaptive visual cues is provided. The system operates using the above-described method and includes a visual language feature extraction module, a visual key point detection module, an optimal point prediction module, a coordinate transformation module, an end effector path planning module, and a joint trajectory generation module.

[0036] The visual language feature extraction module is used to acquire robot first-person view image data and task text instruction data. Based on the above data, a pre-trained visual language model is used to extract a list of task-related objects and their preliminary localization areas.

[0037] The visual key point detection module performs visual key point detection based on the robot's first-view image data, the list of task-related objects and their preliminary positioning areas, generates and outputs a set of candidate key points for the operation of task-related objects, and pixel coordinate labels of the candidate key points in the first-view image.

[0038] The optimal point prediction module predicts and selects the optimal gripping point, path point, and target point from the set of candidate key points based on the task text instruction data and the pixel coordinate annotation of candidate key points in the first-view image and the candidate key point annotation image, and outputs the pixel coordinates of the optimal gripping point, path point, and target point and the target direction of the robot end effector.

[0039] The coordinate transformation module uses the coordinate transformation of the hand-eye calibration algorithm to transform the pixel coordinates of the optimal gripping point, path point and target point into three-dimensional coordinates in the robot coordinate system;

[0040] The end effector path planning module is used to sequentially connect the optimal gripping point, path point and target point in the robot coordinate system, and perform smoothing to generate the robot end effector planned path;

[0041] The joint trajectory generation module calculates the position of each joint of the robot based on the target direction and the planned path of the robot end effector, generates the continuous execution trajectory of each joint, and completes the robot operation planning.

[0042] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the above-described method.

[0043] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. This invention enhances the scene perception capability of the visual language large model through adaptive visual cues. By utilizing the prior knowledge of the large model, it infers and generates continuous operation trajectories, thereby enhancing the generalization ability of the robot system in different tasks and scenarios and improving the success rate of the robot system in real-world environments. As a result, this method reduces inference time and can be efficiently and widely applied to various scenarios.

[0046] 2. This invention utilizes the rich prior knowledge of a large visual language model to understand tasks, perceive scenes, plan skills and robot operation trajectories, without the need for additional manually collected data to train the model, thus enhancing the model's generalization ability in multiple tasks and scenarios.

[0047] 3. This invention generates continuous trajectories for robot operations and calculates the continuous movements of each joint of the robot through inverse kinematics, thereby reducing model inference time and improving the efficiency of the robot operation planning system. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating the steps of a robot operation planning method based on adaptive visual cues in this invention.

[0049] Figure 2 The following is a flowchart illustrating the robot operation planning method in this embodiment. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] This paper proposes a robot operation planning method and system based on adaptive visual cues. It aims to address the problems of poor generalization, high computational resource consumption, and slow inference speed in existing robot control methods. Applicable to autonomous operation planning in desktop scenarios, it constructs a robust adaptive visual cues method to enhance the perception and understanding capabilities of large models, leveraging their rich prior knowledge to autonomously execute operation plans. This solution provides an effective training method for robots to autonomously execute operation tasks, and is expected to promote the application and promotion of robots in actual production.

[0052] Example 1

[0053] In this embodiment, a robot operation planning method based on adaptive visual cues is adopted, and the method steps are as follows: Figure 1 As shown, it specifically includes:

[0054] S1. Obtain robot first-person view image data and task text instruction data. Based on the above data, use a pre-trained visual language model to extract a list of task-related objects and their preliminary localization areas.

[0055] S2. Based on the robot's first-view image data, the list of task-related objects and their preliminary localization areas, perform visual key point detection, generate and output a set of candidate key points for the operation of task-related objects and pixel coordinate labels of the candidate key points in the first-view image.

[0056] S3. Based on the task text instruction data and the pixel coordinate annotation of candidate key points in the first-view image, the optimal gripping point, path point and target point are predicted and selected from the set of operation candidate key points, and the pixel coordinates of the optimal gripping point, path point and target point and the target direction of the robot end effector are output.

[0057] S4. Using the coordinate transformation of the hand-eye calibration algorithm, the pixel coordinates of the optimal grasping point, path point and target point are transformed into three-dimensional coordinates in the robot coordinate system;

[0058] S5. Connect the optimal gripping point, path point and target point in the robot coordinate system in sequence, and perform smoothing to generate the robot end effector planning path;

[0059] S6. Based on the target direction of the robot end effector and the planned path of the robot end effector, solve the position of each joint of the robot, generate the continuous execution trajectory of each joint, and complete the robot operation planning.

[0060] In this embodiment, the specific implementation process of the method is as follows: Figure 2 As shown below:

[0061] Taking a desktop tidying task as an example, this task requires a robotic arm to pick up bananas randomly placed on the desktop and place them on a plate. The task text instructions are things like "place the bananas on the plate" or "return the fruit from the table to the plate." Because the dataset lacks this specific task scenario, traditional learning-based algorithms struggle to achieve the desired accuracy. Therefore, the method provided in this invention is adopted according to... Figure 1 The process shown involves planning robot operations, including:

[0062] S1: Acquire RGB and depth images and task text instructions from the robot's first-person perspective, and extract task-related objects based on the VLM model.

[0063] S2: The visual model generates actionable candidate key point prompts for task-related objects, corresponding to the relevant objects labeled in the RGB image from the robot's first-person perspective.

[0064] S3: Based on the understanding of the task and scenario using the VLM model, predict the pixel positions of the optimal grasping point, path point, and target point, as well as the direction of the robot's end effector, from images labeled with candidate key points.

[0065] S4: Using the coordinate transformation of the hand-eye calibration algorithm, the two-dimensional pixel points are transformed into three-dimensional coordinate points in the robot coordinate system.

[0066] S5: Sequentially connect the gripping point, path point, and target point in the robot coordinate system to smoothly generate the robot end effector planning path.

[0067] S6: Solve the position of each joint of the robot by using the inverse kinematics (IK) method to generate the execution trajectory. Step S1 extracts the task-related objects, including the grasping object: banana and the target object: plate.

[0068] The candidate key point suggestions in step S2 are as follows:

[0069] 1. Input the RGB image from the robot's first-person perspective and the task-related objects into the DINO-X model for encoding, and extract the mask of the task-related objects.

[0070] 2. For the mask of each object related to the task, use the K-means algorithm for cluster analysis to obtain the cluster center points as candidate key points of the object.

[0071] 3. At the two-dimensional pixel coordinates of the cluster center, candidate key points are color-coded to ensure that candidate key points of different objects are coded with different colors, while the candidate key points of the same object are coded with the same color. The RGB image after marking serves as visual cue information.

[0072] Step S3 involves finding the optimal point and predicting the robot's end effector orientation. The specific steps are as follows:

[0073] 1. Construct system prompts for optimal selection. VLM accepts RGB images of visual cue tags and outputs two-dimensional pixel coordinates of the grab point, waypoint, and target point.

[0074] 2. The optimal gripping point, path point, and target point are labeled on the RGB image of the robot's first-person view, serving as visual cues for predicting the gripping direction of the end effector.

[0075] 3. Construct system prompts for end effector direction prediction, defining five predefined grasping directions: "down", "forward", "backward", "left", and "right". The VLM receives new visual cues and selects the optimal grasping direction from the predefined directions, i.e., grasping from top to bottom.

[0076] Step S3: Optimal gripping point selection for candidate key points of the self-grabbing object.

[0077] Step S3: Optimal target point selection is performed on candidate key points of the target object.

[0078] The path points in step S3 are directly calculated and generated by VLM, and there are a total of three path points.

[0079] The coordinate transformation in step S4 is as follows:

[0080] 1. Obtain the depth information z-axis of a pixel in an RGB image within a depth image. c This converts two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system.

[0081]

[0082] Where u,v are two-dimensional pixel coordinates, x c ,y c ,z c The coordinates are in the camera coordinate system (3D). This is the intrinsic parameter matrix of the camera.

[0083] 2. Through coordinate transformation matrix Transform the 3D coordinates in the camera coordinate system into 3D coordinates in the robot coordinate system.

[0084]

[0085] Where, x r ,y r ,z r The coordinates are in the robot's three-dimensional coordinate system. This is the transformation matrix from the camera coordinate system to the robot coordinate system.

[0086] 3. Adjust the z-axis coordinate of the path point. rThe three-dimensional coordinates of the grab point and the target point remain unchanged at a distance of 10cm above the work surface, and are calculated based on the target depth using the formula above.

[0087] Example 2

[0088] In this embodiment, a robot operation planning system based on adaptive visual cues is adopted. The system works by applying the above method and includes a visual language feature extraction module, a visual key point detection module, an optimal point prediction module, a coordinate transformation module, an end effector path planning module, and a joint trajectory generation module.

[0089] The visual language feature extraction module is used to acquire robot first-person view image data and task text instruction data. Based on the above data, a pre-trained visual language model is used to extract a list of task-related objects and their preliminary localization areas.

[0090] The visual key point detection module performs visual key point detection based on the robot's first-view image data, the list of task-related objects and their preliminary positioning areas, generates and outputs a set of candidate key points for the operation of task-related objects, and pixel coordinate labels of the candidate key points in the first-view image.

[0091] The optimal point prediction module predicts and selects the optimal gripping point, path point, and target point from the set of candidate key points based on the task text instruction data and the pixel coordinate annotation of candidate key points in the first-view image and the candidate key point annotation image, and outputs the pixel coordinates of the optimal gripping point, path point, and target point and the target direction of the robot end effector.

[0092] The coordinate transformation module uses the coordinate transformation of the hand-eye calibration algorithm to transform the pixel coordinates of the optimal gripping point, path point and target point into three-dimensional coordinates in the robot coordinate system;

[0093] The end effector path planning module is used to sequentially connect the optimal gripping point, path point and target point in the robot coordinate system, and perform smoothing to generate the robot end effector planned path;

[0094] The joint trajectory generation module calculates the position of each joint of the robot based on the target direction and the planned path of the robot end effector, generates the continuous execution trajectory of each joint, and completes the robot operation planning.

[0095] The specific implementation of this system is the same as in Example 1.

[0096] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot operation planning method based on adaptive visual cues, characterized in that, The method steps include: S1. Obtain robot first-person view image data and task text instruction data. Based on the above data, use a pre-trained visual language model to extract a list of task-related objects and their preliminary localization areas. S2. Based on the robot's first-view image data, the list of task-related objects and their preliminary localization areas, perform visual key point detection, generate and output a set of candidate key points for the operation of task-related objects and pixel coordinate labels of the candidate key points in the first-view image. S3. Based on the task text instruction data and the pixel coordinate annotation of candidate key points in the first-view image, the optimal gripping point, path point and target point are predicted and selected from the set of operation candidate key points, and the pixel coordinates of the optimal gripping point, path point and target point and the target direction of the robot end effector are output. S4. Using the coordinate transformation of the hand-eye calibration algorithm, the pixel coordinates of the optimal grasping point, path point and target point are transformed into three-dimensional coordinates in the robot coordinate system; S5. Connect the optimal gripping point, path point and target point in the robot coordinate system in sequence, and perform smoothing to generate the robot end effector planning path; S6. Based on the target direction of the robot end effector and the planned path of the robot end effector, solve the position of each joint of the robot, generate the continuous execution trajectory of each joint, and complete the robot operation planning.

2. The robot operation planning method based on adaptive visual cues according to claim 1, characterized in that, The first-view image data includes RGB images and depth images from the robot's first-view perspective.

3. The robot operation planning method based on adaptive visual cues according to claim 2, characterized in that, The specific steps in S2 for generating the set of candidate key points for task-related objects and the pixel coordinate annotation of the candidate key points in the first-view image include: The robot's first-person RGB image and the task-related objects are input into the DINO-X model for encoding, and the mask of the task-related objects is extracted. For each task-related object's mask, use the K-means algorithm for cluster analysis to obtain the cluster center points as candidate key points for that object; Candidate keypoints are color-coded at the two-dimensional pixel coordinates of the cluster center to ensure that candidate keypoints of different objects are coded with different colors, while the candidate keypoints of the same object are coded with the same color. The RGB image after coding serves as a visual cue.

4. The robot operation planning method based on adaptive visual cues according to claim 2, characterized in that, The specific steps of S3 include: The system prompts for optimal selection are constructed. The VLM accepts RGB images of visual cue tags and outputs two-dimensional pixel coordinates of the grab point, waypoint, and target point. The optimal gripping point, path point, and target point are labeled on the RGB image of the robot's first-person view to serve as a visual cue for predicting the gripping direction of the end effector. The system provides cue words for predicting the direction of the end effector and defines a variety of predefined grasping directions. The VLM receives new visual cues and selects the optimal grasping direction from the predefined grasping directions.

5. The robot operation planning method based on adaptive visual cues according to claim 2, characterized in that, The specific process of coordinate transformation in S4 includes: Obtain the depth information of pixels in the RGB image in the depth image, and convert the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system; Through coordinate transformation matrix Transform the 3D coordinates in the camera coordinate system into the 3D coordinates in the robot coordinate system; For the grasping point and the target point: keep their three-dimensional coordinates unchanged to ensure that the grasping action is accurately aligned with the target object and the placement action is accurately placed in the target area; For path points: Adjust their z-axis coordinates based on depth parameters, including: the preset safe height of the operation scenario and the object placement height; Adjust the z-axis coordinate of the path point r Furthermore, the three-dimensional coordinates of the grab point and the target point remain unchanged, and the target depth is calculated using the above formula.

6. The robot operation planning method based on adaptive visual cues according to claim 5, characterized in that, The specific formula for coordinate transformation in S4 is as follows: Where u,v are two-dimensional pixel coordinates, x c ,y c ,z c The coordinates are in the camera coordinate system (3D). Let x be the intrinsic parameter matrix of the camera. r ,y r ,z r The coordinates are in the robot's three-dimensional coordinate system. This is the transformation matrix from the camera coordinate system to the robot coordinate system.

7. The robot operation planning method based on adaptive visual cues according to claim 1, characterized in that, The inverse kinematics method is specifically used to solve the position of each joint of the robot in S6.

8. A robot operation planning system based on adaptive visual cues, characterized in that, The system operates according to any one of the methods in claims 1 to 8, and the system includes a visual language feature extraction module, a visual key point detection module, an optimal point prediction module, a coordinate transformation module, an end effector path planning module, and a joint trajectory generation module; The visual language feature extraction module is used to acquire robot first-view image data and task text instruction data. Based on the above data, a pre-trained visual language model is used to extract a list of task-related objects and their preliminary localization areas. The visual key point detection module performs visual key point detection based on the robot's first-view image data, the list of task-related objects and their preliminary positioning areas, generates and outputs a set of candidate key points for the operation of task-related objects, and pixel coordinate labels of the candidate key points in the first-view image. The optimal point prediction module predicts and selects the optimal gripping point, path point, and target point from the set of candidate key points based on the task text instruction data and the pixel coordinate annotation of candidate key points in the first-view image and the candidate key point annotation image, and outputs the pixel coordinates of the optimal gripping point, path point, and target point and the target direction of the robot end effector. The coordinate transformation module uses the coordinate transformation of the hand-eye calibration algorithm to transform the pixel coordinates of the optimal grasping point, path point and target point into three-dimensional coordinates in the robot coordinate system; The end effector path planning module is used to sequentially connect the optimal grasping point, path point and target point in the robot coordinate system, and perform smoothing to generate the robot end effector planning path. The joint trajectory generation module calculates the position of each joint of the robot based on the target direction and the planned path of the robot end effector, generates the continuous execution trajectory of each joint, and completes the robot operation planning.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images