Robot manipulation method, robot, storage medium, device and program product
By receiving and converting the two-dimensional visual information in the visual prompt image into three-dimensional task information, the robot end executor is controlled, which solves the problem of language command ambiguity and insufficient predefined action sequences, and improves the accuracy and efficiency of robot task execution.
Patent Information
- Application Number
- CN202410981895.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-12
AI Technical Summary
Existing robot manipulation methods rely on language instructions, are ambiguity, it is difficult to accurately perform complex tasks, and predefined action sequences are insufficient in complex environments.
By receiving the corresponding visual prompt images of multiple action steps of the end effector manipulating the object, the two-dimensional visual information is extracted and converted into three-dimensional task information is used to control the end effector to achieve accurate execution of complex tasks.
Overcoming the ambiguity of language instructions and the limitations of predefined action sequences, improving the accuracy and efficiency of robots in performing tasks, and adapting to complex and variable operating environments.
Smart Images

Figure CN118941619A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of robot manipulation algorithms, and in particular to robot manipulation methods, robots, storage media, devices and program products. Background Art
[0002] With the continuous development of robotics technology, robots are increasingly used in many fields such as industry, medicine, and home. As a close assistant to humans, robots need to have the ability to understand and perform complex tasks, which puts higher requirements on robot manipulation algorithms. Currently, many robot manipulation methods rely on language instructions. However, language instructions are somewhat ambiguous.
[0003] Based on this, the present application provides a robot manipulation method, a robot, a storage medium, a device and a program product to improve the relevant technology. Summary of the invention
[0004] The purpose of this application is to provide a robot manipulation method, a robot, a storage medium, a device and a program product to reduce the impact of the ambiguity of language instructions on the robot and improve the accuracy of the robot's manipulation of objects.
[0005] The purpose of this application is achieved by the following technical solutions:
[0006] In a first aspect, the present application provides a robot manipulation method, the method comprising:
[0007] Receiving visual prompt images corresponding to a plurality of action steps of manipulating an object by an end effector of the robot; the two-dimensional visual information in the visual prompt image is used to prompt two-dimensional task information, and the two-dimensional task information includes one or more of two-dimensional coordinate information of a contact point, two-dimensional posture information of the end effector, and two-dimensional moving direction information;
[0008] For at least one of the action steps, extracting two-dimensional visual information in a visual cue image corresponding to the action step to obtain three-dimensional task information corresponding to the action step; the three-dimensional task information includes one or more of three-dimensional coordinate information of the contact point, three-dimensional posture information of the end effector, and three-dimensional moving direction information;
[0009] The end effector is controlled to manipulate the object using the three-dimensional task information corresponding to at least one of the action steps.
[0010] In some embodiments, the three-dimensional task information includes the three-dimensional movement direction information;
[0011] The method further comprises: according to the sequence of at least one of the action steps, using the three-dimensional moving direction information corresponding to the action step, sequentially connecting the action step and its corresponding next action step to obtain the overall trajectory corresponding to at least one of the action steps;
[0012] The use of three-dimensional task information corresponding to at least one of the action steps to control the end effector to manipulate the object includes: using the three-dimensional task information corresponding to the action step to plan the motion trajectory of the end effector to execute the action step so that the end effector completes the overall trajectory.
[0013] In some embodiments, each action step corresponds to an atomic task, and the atomic task includes one or more of placement, movement, rotation, picking, pulling, and pushing.
[0014] In some embodiments, extracting the two-dimensional visual information in the visual prompt image corresponding to the action step to obtain the three-dimensional task information corresponding to the action step includes:
[0015] Extracting two-dimensional visual information from the visual cue image to obtain two-dimensional task information;
[0016] The visual cue image and the two-dimensional task information are input into an action prediction model to obtain the three-dimensional task information.
[0017] In some embodiments, the training process of the action prediction model includes:
[0018] Acquire a training set; each set of training data in the training set includes a training visual prompt image, training two-dimensional task information, and training three-dimensional task information;
[0019] For at least one of the training data, the following processing is performed:
[0020] Inputting the training visual cue image and the training two-dimensional task information in the training data into the action prediction model to obtain predicted three-dimensional task information;
[0021] Using the predicted three-dimensional task information and the training three-dimensional task information, a loss value corresponding to a target loss function is calculated;
[0022] Using the loss value, updating the model parameters of the action prediction model;
[0023] Check whether the training end condition is met; if yes, stop training the motion prediction model; if no, continue training the motion prediction model using the next set of training data.
[0024] In some embodiments, the training visual cue image is obtained by drawing visual elements on a two-dimensional image containing the training object, and the process of obtaining the training data includes:
[0025] Controlling the end effector to randomly interact with the training object in the simulator, and recording training 3D task information of successfully manipulating the training object, wherein the training 3D task information includes training 3D coordinate information of training contact points, training 3D posture information of the end effector, and training 3D moving direction information; and / or,
[0026] Projecting the training three-dimensional coordinate information of the training contact point onto the two-dimensional image to obtain the training two-dimensional coordinate information of the training contact point, and drawing the visual element corresponding to the training contact point on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point; and / or,
[0027] Determine the training three-dimensional coordinate information of the posture reference point based on the training three-dimensional coordinate information of the training contact point and the training three-dimensional posture information, project the training three-dimensional coordinate information of the posture reference point onto the two-dimensional image to obtain the training two-dimensional coordinate information of the posture reference point, and draw the visual element corresponding to the training two-dimensional posture information on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the posture reference point; and / or,
[0028] The training three-dimensional coordinate information of the moving direction reference point is determined based on the training three-dimensional coordinate information of the training contact point and the training three-dimensional moving direction information, the training three-dimensional coordinate information of the moving direction reference point is projected onto the two-dimensional image to obtain the training two-dimensional coordinate information of the moving direction reference point, and the visual elements corresponding to the training two-dimensional moving direction information are drawn on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the moving direction reference point.
[0029] In some embodiments, the three-dimensional posture information includes three-dimensional Y-axis direction information and three-dimensional Z-axis direction information, and the target loss function includes one or more of text supervision loss, orthogonal loss and projection loss.
[0030] In some embodiments, the two-dimensional visual information uses visual elements with different display parameters to prompt at least two of the two-dimensional coordinate information of the contact point, the two-dimensional posture information, and the two-dimensional moving direction information;
[0031] The visual elements include one or more of points, lines and shapes;
[0032] The display parameters include one or more of color, line width, line type, line arrow direction, brightness, contrast, saturation, exposure, hue, brightness, grayscale and color temperature.
[0033] In a second aspect, the present application provides a robot comprising a control module and an end effector, wherein the control module is used to execute any one of the above methods to control the end effector to manipulate an object.
[0034] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements any of the above methods when executed by a processor.
[0035] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.
[0036] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program implements any one of the above methods when executed by a processor.
[0037] The present application provides a robot manipulation method, a robot, a storage medium, a device and a program product, which receives visual prompt images corresponding to multiple action steps, extracts two-dimensional visual information from them, and further converts them into three-dimensional task information. Using this three-dimensional task information, the end effector is controlled to manipulate objects to achieve accurate execution of complex tasks. It not only overcomes the ambiguity of language instructions, but also improves the accuracy and efficiency of robot execution of tasks and adapts to various complex and changeable operating environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The present application is further described below in conjunction with the accompanying drawings and specific implementation methods.
[0039] Figure 1 It is a flow chart of a robot manipulation method provided in an embodiment of the present application.
[0040] Figure 2 It is a schematic diagram of a robot performing a task provided in an embodiment of the present application.
[0041] Figure 3 It is a flowchart of a training action prediction model provided in an embodiment of the present application.
[0042] Figure 4 It is a flowchart of using a motion prediction model for reasoning provided in an embodiment of the present application.
[0043] Figure 5It is a structural block diagram of a robot provided in an embodiment of the present application.
[0044] Figure 6 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the implementation mode of this application. Obviously, the described implementation mode is only a part of the implementation mode of this application, not all the implementation modes. Based on the implementation mode in this application, all other implementation modes obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0046] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0047] As human assistants, it is crucial for robots to understand the tasks they are supposed to perform. Currently, many robotic manipulation methods rely on language instructions or predefined action sequences.
[0048] Many approaches adopt language as a conditioning mechanism and design models to achieve command following and task execution. For example, some robots enhance semantic reasoning by leveraging large-scale data for model training, enabling the model to interact with a wider range of objects based on language instructions. In addition, others inject manipulation capabilities into existing multimodal large language models (MLLMs), leveraging their holistic understanding of the scene and command following capabilities to enhance performance in robotic tasks.
[0049] However, due to the ambiguity in language instructions, the training paradigm of using language as a task goal poses difficulties for models in interpreting the actions required to complete the task. For example, with text instructions like "fold the left sleeve in" or even detailed descriptions like "grasp at the cuff and place on the opposite shoulder", the model may face potential challenges in understanding the language and accurately capturing the precise position and direction of the sleeve to be grasped or placed, indicating that text is not sufficient to effectively describe the conditional signals of low-level actions. In addition, complex tasks involving multiple action steps require more detailed and longer instructions, which significantly increases the model's requirements for natural language understanding and instruction following capabilities, which in turn increases the complexity of training models to complete robotic tasks.
[0050] Secondly, predefined action sequences are insufficient when dealing with complex and changing tasks. This approach requires planning each action step in advance and hard-coding it into the robot control system. However, the actual operating environment is complex and changeable, and robots need to be able to flexibly respond to various emergencies and complex tasks. It is difficult to meet these requirements with only predefined action sequences.
[0051] Therefore, in the field of robotics, using visual cues as task control conditions and enabling models to predict action generation based on them has also attracted much attention. Some robots use hand-drawn sketches as target specifications to teach the robot to achieve the desired target state, but their application is largely limited to planar scenes, such as accessible reference frames such as tables. In addition, there are robots that use two-dimensional trajectories as conditional signals for rich expression of robot strategies. However, it treats the entire task as a single trajectory, and the model becomes more and more complex as the number of actions increases.
[0052] See also Figure 1 , Figure 1 It is a flow chart of a robot manipulation method provided in an embodiment of the present application.
[0053] In order to improve the relevant technology, the embodiment of the present application provides a robot manipulation method, the method comprising steps S101 to S103.
[0054] Step S101: Receive visual prompt images corresponding to multiple action steps of the robot's end effector manipulating an object; the two-dimensional visual information in the visual prompt image is used to prompt two-dimensional task information, and the two-dimensional task information includes one or more of the two-dimensional coordinate information of the contact point, the two-dimensional posture information of the end effector, and the two-dimensional moving direction information.
[0055] Step S102: For at least one of the action steps, extract the two-dimensional visual information in the visual prompt image corresponding to the action step to obtain the three-dimensional task information corresponding to the action step; the three-dimensional task information includes one or more of the three-dimensional coordinate information of the contact point, the three-dimensional posture information of the end effector and the three-dimensional moving direction information.
[0056] Step S103: using the three-dimensional task information corresponding to at least one of the action steps, controlling the end effector to manipulate the object.
[0057] The above embodiment receives visual prompt images corresponding to multiple action steps, extracts two-dimensional visual information from them, and further converts them into three-dimensional task information. Using this three-dimensional task information, the end effector is controlled to manipulate objects to achieve accurate execution of complex tasks. This method not only overcomes the ambiguity of language instructions and the limitations of predefined action sequences, but also improves the accuracy and efficiency of robot execution tasks and adapts to various complex and changing operating environments.
[0058] Among them, the robot can be a humanoid robot (also known as a bipedal robot or a humanoid robot), a quadruped robot, a wheeled robot, a robotic arm or other automated equipment. The robot can be used to manipulate objects, such as clothes, toys, tools, eggs, etc., and the above embodiment does not limit this. The visual prompt image is an image used to guide the robot to perform a task. For example, two-dimensional task information can be prompted by visual elements such as points and lines, such as two-dimensional coordinate information of the contact point, two-dimensional posture information of the end effector, and two-dimensional movement direction information after contact, etc., to help the robot understand and execute specific action steps. The action step is the basic manipulation unit when the robot performs a task, such as grasping, moving, rotating, etc., and each action step corresponds to a visual prompt image.
[0059] The two-dimensional visual information is the planar information presented in the visual prompt image, for example, in the form of visual elements, which may include the two-dimensional coordinate information of the contact point, the two-dimensional posture information of the end effector, and the visual elements corresponding to the two-dimensional moving direction information. The visual elements are represented in the form of colors, lines, graphics, etc., and may be drawn in the form of crayons, pens, pencils, etc., and the above embodiments do not limit this. In some embodiments, the two-dimensional visual information may use visual elements with different display parameters to prompt at least two of the two-dimensional coordinate information of the contact point, the two-dimensional posture information, and the two-dimensional moving direction information. In some embodiments, the visual elements include one or more of points, lines, and shapes. In some embodiments, the shapes include one or more of circles, ellipses, polygons, bows, arcs, rings, and runway shapes. In some embodiments, the display parameters include one or more of color, line width, line type, line arrow direction, brightness, contrast, saturation, exposure, hue, brightness, grayscale, and color temperature. As an example, a combination of four colors and shapes is used to represent two-dimensional visual information. A blue dot is used to indicate the location of the contact point, a red line and a green line are used to indicate the z-axis direction and the y-axis direction when the end effector contacts the object, and a yellow line is used to indicate the movement direction after contact.
[0060] The two-dimensional task information includes the task details in the two-dimensional visual information, such as the two-dimensional coordinate information of the contact point, the two-dimensional posture information of the end effector, and the two-dimensional moving direction information, which is used to guide the basic actions of the robot on the plane. The two-dimensional posture information includes, for example, two-dimensional y-axis direction information and two-dimensional z-axis direction information. The two-dimensional posture information and the two-dimensional moving direction information are represented, for example, in the form of a two-dimensional unit vector.
[0061] The three-dimensional task information may be three-dimensional space information obtained by extracting and converting two-dimensional visual information, such as three-dimensional coordinate information of the contact point, three-dimensional posture information of the end effector, and three-dimensional movement direction information, which is used to guide the operation of the robot in three-dimensional space. The three-dimensional posture information, for example, includes three-dimensional Y-axis direction information and three-dimensional Z-axis direction information, which together determine the rotation direction of the end effector in contact with the object. The three-dimensional movement direction information is used to indicate the movement direction of the end effector after contacting the object. The three-dimensional posture information and the three-dimensional movement direction information are expressed, for example, in the form of a three-dimensional unit vector. The end effector is the execution part of the robot, such as the gripper or tool of a robotic arm, which is used to complete specific actions such as grasping, moving, and rotating.
[0062] Related robot manipulation methods rely on language instructions, and there are problems with understanding language ambiguity and inaccurate actions. Especially in complex tasks, it is difficult for language instructions to accurately describe every detail of the action, resulting in low efficiency of the robot in performing the task, or even failure to complete the task. In the above embodiment, visual prompt images corresponding to multiple action steps of the robot end effector manipulating the object are first received. These visual prompt images contain two-dimensional visual information for prompting two-dimensional task information. Then, for each action step, the two-dimensional visual information is extracted from the visual prompt image and converted to obtain the corresponding three-dimensional task information. Afterwards, these three-dimensional task information are used to control the end effector to manipulate the object and complete the predetermined task steps.
[0063] The above embodiments can effectively improve the accuracy and efficiency of the robot when manipulating objects. The visual cue image provides clear task instructions. By converting two-dimensional visual information into three-dimensional task information, the robot can accurately locate and move the end effector in three-dimensional space to complete complex operation tasks. This method not only overcomes the ambiguity of understanding caused by language instructions, but also simplifies and standardizes the task planning process, improving the robot's adaptability and task execution success rate in different scenarios.
[0064] In some embodiments, the three-dimensional task information includes the three-dimensional moving direction information. The method may further include: according to the sequence of at least one of the action steps, using the three-dimensional moving direction information corresponding to the action step, sequentially connecting the action step and its corresponding next action step to obtain the overall trajectory corresponding to at least one of the action steps. Using the three-dimensional task information corresponding to at least one of the action steps to control the end effector to manipulate the object includes: using the three-dimensional task information corresponding to the action step to plan the motion trajectory of the end effector to execute the action step, so that the end effector completes the overall trajectory.
[0065] The overall trajectory is, for example, a continuous motion path formed by sequentially connecting a plurality of action steps, representing the complete manipulation route of the robot from the beginning to the end of the task.
[0066] In some embodiments, each action step may correspond to an atomic task, and the atomic task includes one or more of placement, movement, rotation, picking, pulling, and pushing.
[0067] The above-mentioned embodiment utilizes a series of visual cue images corresponding to the action steps as high-level planning, and each visual cue image corresponds to an atomic step, for example. The end effector is motion planned (for example, by utilizing a motion planner) to execute the atomic action strategy corresponding to each atomic task, and the end effector is able to complete the long-term trajectory (i.e., the overall trajectory). The advantage of doing so is that the complexity of the long-term task can be decomposed while optimizing the success rate of each atomic task to ensure the success of the overall task. In addition, by treating each atomic task as a basic unit, the atomic tasks can be combined into any combination, and complex manipulation tasks involving multiple action steps can be handled.
[0068] The above embodiment uses the three-dimensional moving direction information to sequentially connect each action step and its corresponding next action step in accordance with the sequence of the action steps to form an overall trajectory. The motion trajectory of the end effector is planned using the three-dimensional task information to ensure that the end effector completes the operation path of the entire task. The above embodiment forms a complete operation trajectory by sequentially connecting the three-dimensional moving direction information of each action step, so that the robot can smoothly perform complex multi-step tasks, thereby improving the robot's adaptability and task execution success rate in different scenarios.
[0069] See also Figure 2 , Figure 2 Schematic diagram of a robot performing a task provided by an embodiment of the present application. Figure 2In the figure, the model is an action prediction model, t is time, t1, t2, t3, and t4 represent different moments, and the input at each moment includes a visual cue image. The end effector posture includes two-dimensional z-axis direction information (i.e., z-axis, represented by red lines), two-dimensional y-axis direction information (i.e., y-axis, represented by green lines), and two-dimensional coordinate information of the contact point (i.e., contact point coordinates, represented by blue dots). The two-dimensional movement direction of the end effector is represented by a yellow line. It can be seen that under visual cues, the robot can complete a series of action steps, first picking up the left sleeve at the left cuff and placing it on the right shoulder, and then picking up the right sleeve at the right cuff and placing it on the left shoulder.
[0070] In some embodiments, extracting the two-dimensional visual information from the visual cue image corresponding to the action step to obtain the three-dimensional task information corresponding to the action step may include: extracting the two-dimensional visual information from the visual cue image to obtain the two-dimensional task information; inputting the visual cue image and the two-dimensional task information into an action prediction model to obtain the three-dimensional task information.
[0071] Among them, the action prediction model, for example, is a model that uses the input visual cue image and two-dimensional task information to predict three-dimensional task information, and can be a model based on deep learning.
[0072] In some embodiments, the two-dimensional task information and / or the three-dimensional task information is in text form. In this case, the action prediction model is a model that can process multimodal information. The above embodiment uses the action prediction model to formalize the task prediction problem into a language modeling task, and can output the three-dimensional task information in text form. The action prediction model is used to convert the visual prompt image and the two-dimensional task information into three-dimensional task information, and output it in text form, which reduces the risk of ambiguity in natural language instructions and ensures that the robot can accurately understand and execute each step of the operation, thereby improving the accuracy and reliability of task execution. Formalizing the task prediction problem as a language modeling task simplifies the planning and execution process of complex tasks, enabling the robot to efficiently complete multi-step tasks.
[0073] See also Figure 3 , Figure 3 is a flow chart of a training action prediction model provided in an embodiment of the present application. Figure 3 In the example, each input is an input pair, and each input pair includes a visual input and a text input. The visual input is, for example, a visual prompt image, and the text input is, for example, two-dimensional task information.
[0074] The action prediction model receives visual input I of objects drawn with visual cues. Figure 3In the figure, four colors are used for visual cues: blue dots represent contact points; red and green lines represent the two-dimensional z-axis direction and two-dimensional y-axis direction when the end effector contacts the object; and yellow lines represent the two-dimensional movement direction after contact. In order to reduce the ambiguity caused by overlapping visual cues, the corresponding values can be explicitly specified as language cues P in the text input, such as the two-dimensional coordinate information a0 of the contact point. p ; Two-dimensional z-axis direction information a0 of the end effector (e.g., gripper) z and two-dimensional y-axis direction information a0 y , and two-dimensional moving direction information a0 m , these three directional information can be represented by a two-dimensional unit vector. The goal of the action prediction model is to generate an action (i.e., three-dimensional task information) a0 = (a0 p ,a0 Z ,a0 Y ,a0 M ), where a0 p It can be obtained by mapping the depth information of the contact point and the camera parameters into the three-dimensional space, and a0 Z 、a0 Y and a0 M It can be represented by a three-dimensional unit vector, which corresponds to the three-dimensional Z-axis direction information, three-dimensional Y-axis direction information and three-dimensional moving direction information of the end effector. Z and a0 Y Together they determine the rotation direction of the end effector in contact with the object, while a0 M Controls the subsequent movement direction.
[0075] In some embodiments, the training process of the motion prediction model may include: obtaining a training set; each set of training data in the training set includes a training visual cue image, training two-dimensional task information, and training three-dimensional task information; for at least one of the training data, performing the following processing: inputting the training visual cue image and the training two-dimensional task information in the training data into the motion prediction model to obtain predicted three-dimensional task information; using the predicted three-dimensional task information and the training three-dimensional task information, calculating the loss value corresponding to the target loss function; using the loss value, updating the model parameters of the motion prediction model; detecting whether the training end condition is met; if yes, stopping the training of the motion prediction model; if not, continuing to train the motion prediction model using the next set of training data. Among them, the training three-dimensional task information can be obtained by user annotation, which can also be called annotated three-dimensional task information.
[0076] In some embodiments, the action prediction model may include a visual encoder, a projection module, a large language model (e.g., LLaMa), and a text adapter. The visual encoder is used to understand the visual elements in the training visual cue image. The projection module is used to project the three-dimensional coordinate information of the point to the two-dimensional image to obtain the corresponding two-dimensional coordinate information. The output of the projection module is processed by the large language model and the text adapter to obtain the training three-dimensional task information in text form, which can be used as the true value in the training data.
[0077] In some embodiments, the training visual cue image may be obtained by drawing visual elements on a two-dimensional image containing the training object, and the process of obtaining the training data may include: controlling the end effector to randomly interact with the training object in a simulator, recording training three-dimensional task information of successfully manipulating the training object, the training three-dimensional task information including training three-dimensional coordinate information of training contact points, training three-dimensional posture information of the end effector, and training three-dimensional moving direction information; and / or, projecting the training three-dimensional coordinate information of the training contact points onto the two-dimensional image to obtain the training two-dimensional coordinate information of the training contact points, and drawing the visual elements corresponding to the training contact points on the two-dimensional image based on the training two-dimensional coordinate information of the training contact points; and / or determining the posture information based on the training three-dimensional coordinate information of the training contact points and the training three-dimensional posture information. The training three-dimensional coordinate information of the posture reference point is determined based on the training three-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the posture reference point, and the training two-dimensional coordinate information of the posture reference point is projected onto the two-dimensional image to obtain the training two-dimensional coordinate information of the posture reference point, and the visual element corresponding to the training two-dimensional posture information is drawn on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the posture reference point; and / or the training three-dimensional coordinate information of the moving direction reference point is determined based on the training three-dimensional coordinate information of the training contact point and the training three-dimensional moving direction information, and the training three-dimensional coordinate information of the moving direction reference point is projected onto the two-dimensional image to obtain the training two-dimensional coordinate information of the moving direction reference point, and the visual element corresponding to the training two-dimensional moving direction information is drawn on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the moving direction reference point.
[0078] In some embodiments, for each direction in the training three-dimensional posture information, the number of posture reference points may be one or more. In some embodiments, the number of moving direction reference points may be one or more.
[0079] like Figure 3 As shown, for example, in the simulator, the end effector is used to randomly interact with the object. If a successful operation occurs after the random interaction, the successful contact posture of the end effector is recorded, such as the three-dimensional coordinate information a0 of the contact pointP’ , three-dimensional Z-axis direction information a0 Z’ , three-dimensional Y-axis direction information a0 Y’ And the three-dimensional moving direction information a0 after contact M’ , as a reference standard for training to guide training. By using the camera parameters, a0 P’ Projecting onto the two-dimensional image gives a0 p =(x,y), for example, represented by a blue dot. For directional visual cues, for example, a0 P’ Centered at a0 Z’ and a0 Y’ The attitude reference points are located in the direction. These attitude reference points are then projected onto the 2D image. Subsequently, these attitude reference points are connected with a0 p To calculate the two-dimensional Z-axis direction information a0 z and two-dimensional Y-axis direction information a0 y , and draw the corresponding red and green lines respectively. M’ Get two-dimensional moving direction information a0 m , and draw the corresponding yellow line.
[0080] In some embodiments, the training end condition may include at least one of the following: the loss value is less than the target loss value; the number of iterations is greater than the target number of iterations.
[0081] In some embodiments, the three-dimensional posture information may include three-dimensional Y-axis direction information and three-dimensional Z-axis direction information, and the target loss function may include one or more of text supervision loss, orthogonal loss, and projection loss. The above different types of losses are introduced to guide the training process of the model, so that the trained action prediction model can accurately predict the three-dimensional direction based on the two-dimensional direction prompt.
[0082] The above embodiment designs a fine-tuning task for multimodal large language models (MLLMs) by creating input pairs containing different levels of information and designing a target loss function to guide the model training process, so that the action prediction model can achieve the required low-level actions, that is, where to contact and how to interact. Therefore, when faced with new objects, as long as the action prediction model is provided with visual cues that can be interpreted by the action prediction model, the action prediction model can predict accurate actions, thereby improving the model generalization ability.
[0083] In order to make the action prediction model understand the meaning of various visual cues, multiple pairs of visual inputs and text inputs with different information levels can be designed for the action prediction model to learn, so that it can gradually understand the meaning of each visual cue. Figure 3The inputs (a)-(d) in Figure 1 show the following input pairs with different information levels: (a) only the two-dimensional coordinate information of the contact point a0 is provided p ; (b) Prompt the two-dimensional coordinate information of the contact point a0 p and the two-dimensional z-axis direction information a0 of the end effector z (c) Prompt the two-dimensional coordinate information of the contact point a0 p , the two-dimensional z-axis direction information a0 of the end effector z and two-dimensional y-axis direction information a0 y ; (d) Prompt the two-dimensional coordinate information of the contact point a0 p , the two-dimensional z-axis direction information a0 of the end effector z , two-dimensional y-axis direction information a0 y and two-dimensional moving direction information a0 m During the training process, for each set of training data, in each iteration, one of the above four input modes can be randomly selected, while the true value corresponding to each input (e.g., training 3D task information) remains unchanged.
[0084] It should be noted that in this article, "training" and "prediction" in training objects, training data, training visual cue images, training two-dimensional task information, training three-dimensional task information, and predicted three-dimensional task information are used as prefixes to distinguish them. Training visual cue images and visual cue images without prefixes are used to distinguish between visual cue images in the model training phase and visual cue images in the model use phase. In essence, they are both visual cue images. For related instructions, please refer to the description of visual cue images in the previous article, which will not be repeated here. The naming methods of training two-dimensional task information, training three-dimensional task information, and predicted three-dimensional task information are also similar, which will not be repeated in this article.
[0085] Among them, the text supervision loss (LT) is a basic loss used to ensure that the visual and language inputs of the action prediction model can be effectively aligned and that it can output the correct language pattern. In order to reduce the difficulty of direction regression prediction, for example, the normalized direction vector of continuous values can be discretized into 100 discrete values [-50, 50], each with a span of 0.02, and converted into a classification prediction. The cross entropy loss is used to supervise the matching of the model output with the true value.
[0086] For the orthogonality loss (LO), the orthogonality of the rotation matrix requires that its components maintain an orthogonal relationship between each pair of directions. However, forming separate language predictions for the 3D Z-axis direction and the 3D Y-axis direction does not explicitly consider their geometric relationship. To address this issue, a loss function can be introduced to extract the 3D Z-axis direction information and the 3D Y-axis direction information from the text output by the action prediction model and constrain them to ensure their orthogonal relationship.
[0087] For the projection loss (LP), the 3D direction prediction should be consistent with its corresponding 2D direction when projected back to a plane (e.g., a 2D image). In order to establish an explicit relationship between the input 2D direction information and the output 3D direction information, a projection loss can be introduced to guide the correlation between the 3D direction and its 2D direction. As an example, in the output of the motion prediction model, the training contact points can be extracted and projected to a 2D image. Subsequently, one or more posture reference points are located along the 3D Z-axis direction using the predicted 3D Z-axis direction information. These posture reference points are then projected to the 2D image and connected with the training contact points to obtain the predicted 2D Z-axis direction information. A similar method is used to obtain the predicted 2D Y-axis direction information and the 2D movement direction information, and then the input 2D direction information and its corresponding predicted 2D direction information are supervised using cosine similarity.
[0088] In some embodiments, the text supervision loss (LT) and the orthogonal loss (LO) remain consistent throughout all training processes, but the projection loss (LP) varies according to the presence of specific cues. For example, if the input visual cue image does not contain 2D motion direction information, the supervision of 2D motion direction information will be excluded in the projection loss (LP).
[0089] The above losses can be trained simultaneously under a total objective loss function: L = LT + LO + LP. The action prediction model is trained with dynamic input patterns to understand multiple visual cues and predict the desired action based on the provided cues as a task.
[0090] In order to eliminate ambiguity caused by overlapping visual cues, relevant numerical values can be explicitly specified as language cues in the input training two-dimensional task information (for example, expressed in text form), thereby improving the accuracy of task prediction.
[0091] In a specific application scenario, an embodiment of the present application provides a robot manipulation method, the method comprising:
[0092] Receive visual prompt images corresponding to multiple action steps of the robot's end effector manipulating an object; each action step corresponds to an atomic task, and the atomic task includes one or more of placement, movement, rotation, picking, pulling and pushing; the two-dimensional visual information in the visual prompt image is used to prompt the two-dimensional task information, and the two-dimensional task information includes the two-dimensional coordinate information of the contact point, the two-dimensional posture information of the end effector and the two-dimensional moving direction information; the two-dimensional visual information uses visual elements with different display parameters to prompt the two-dimensional coordinate information of the contact point, the two-dimensional posture information and the two-dimensional moving direction information; the visual elements include one or more of points, lines and shapes; the display parameters include one or more of color, line width, line type, line arrow direction, brightness, contrast, saturation, exposure, hue, brightness, grayscale and color temperature;
[0093] For at least one of the action steps, extracting two-dimensional visual information from the visual cue image to obtain two-dimensional task information; inputting the visual cue image and the two-dimensional task information into an action prediction model to obtain the three-dimensional task information; the three-dimensional task information includes three-dimensional coordinate information of the contact point, three-dimensional posture information of the end effector, and three-dimensional moving direction information;
[0094] According to the sequence of at least one of the action steps, using the three-dimensional moving direction information corresponding to the action step, sequentially connect the action step and its corresponding next action step to obtain the overall trajectory corresponding to at least one of the action steps;
[0095] The motion trajectory of the end effector executing the action step is planned by using the three-dimensional task information corresponding to the action step, so that the end effector completes the overall trajectory.
[0096] The training process of the action prediction model includes: obtaining a training set; each set of training data in the training set includes a training visual cue image, training two-dimensional task information and training three-dimensional task information; for at least one of the training data, performing the following processing: inputting the training visual cue image and the training two-dimensional task information in the training data into the action prediction model to obtain predicted three-dimensional task information; using the predicted three-dimensional task information and the training three-dimensional task information, calculating the loss value corresponding to the target loss function; using the loss value, updating the model parameters of the action prediction model; detecting whether the training end condition is met; if yes, stopping the training of the action prediction model; if not, continuing to train the action prediction model using the next set of training data. The target loss function includes one or more of text supervision loss, orthogonal loss and projection loss. The training visual cue image is obtained by drawing visual elements on a two-dimensional image containing the training object. The process of obtaining the training data includes: controlling the end effector to randomly interact with the training object in a simulator, recording training three-dimensional task information of successfully manipulating the training object, wherein the training three-dimensional task information includes training three-dimensional coordinate information of training contact points, training three-dimensional posture information of the end effector, and training three-dimensional moving direction information; projecting the training three-dimensional coordinate information of the training contact points onto the two-dimensional image to obtain the training two-dimensional coordinate information of the training contact points, and drawing the visual elements corresponding to the training contact points on the two-dimensional image based on the training two-dimensional coordinate information of the training contact points; determining the training three-dimensional coordinate information of the posture reference points based on the training three-dimensional coordinate information of the training contact points and the training three-dimensional posture information The training three-dimensional coordinate information of the posture reference point is projected onto the two-dimensional image to obtain the training two-dimensional coordinate information of the posture reference point, and the visual elements corresponding to the training two-dimensional posture information are drawn on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the posture reference point; the training three-dimensional coordinate information of the moving direction reference point is determined based on the training three-dimensional coordinate information of the training contact point and the training three-dimensional moving direction information, the training three-dimensional coordinate information of the moving direction reference point is projected onto the two-dimensional image to obtain the training two-dimensional coordinate information of the moving direction reference point, and the visual elements corresponding to the training two-dimensional moving direction information are drawn on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the moving direction reference point.
[0097] The above embodiment draws visual cues on a two-dimensional image to represent low-level actions and high-level planning. The visual cue images of a single action step in each task (e.g., an atomic task) are prompted with two-dimensional color visual elements. The prompt content may include the contact point, the z-axis direction of the end effector, the y-axis direction of the end effector, and the moving direction after contact. The constraint relationship between the directions, such as the relationship between the corresponding two-dimensional direction prompt and the predicted three-dimensional direction, is observed, and the corresponding model training strategy and supervision target (the design of the corresponding target loss function) are determined, aiming to gradually enhance the ability of MLLMs to understand each visual cue. At the same time, this enables the trained action prediction model to predict the corresponding contact point position (e.g., the three-dimensional coordinate information of the contact point), the three-dimensional contact posture (e.g., the three-dimensional posture information of the end effector) and the three-dimensional moving direction (e.g., the three-dimensional moving direction information after contact), thereby facilitating the completion of each atomic step. In addition, an image sequence containing multiple visual cue images is used as input to express the task planning process, each visual cue image uses a visual cue to represent the atomic action target (e.g., two-dimensional task information) of the corresponding action step, and an interaction strategy is designed to connect each action step in sequence using the predicted three-dimensional moving direction to form an overall trajectory.
[0098] See also Figure 4 , Figure 4 It is a flowchart of using a motion prediction model for reasoning provided in an embodiment of the present application.
[0099] In the inference stage, Figure 4 As shown, the user can draw visual cues on a two-dimensional image and then use it as visual input. As an example, the user can use blue dots, red lines, and green lines to indicate the position and posture that the end effector should reach, and use yellow lines to indicate the direction of movement after contact. The applicant has observed that many long-term tasks can be decomposed into several atomic tasks, which can be classified as placement, movement, rotation, picking, pulling, and pushing. Different atomic tasks exhibit different motion patterns: some require specific three-dimensional movement directions to determine subsequent actions, while others only rely on the posture of the next atomic task.
[0100] Specifically, placement and movement are simple actions that do not require additional movement after approaching the object. For example, once the object is placed according to the visual cue, the task is considered completed, e.g. Figure 4As shown. Therefore, these actions do not need to draw the corresponding visual elements of the 3D movement direction information for visual prompts. For rotation actions, such as rotating a button (here, a physical button), this action may not involve position translation after touching the button, but rotating the last joint of the multi-joint robot. Therefore, there is no need for 3D movement direction prompts, and it can be determined whether the rotation is clockwise or counterclockwise by comparing the prompts on the current image and the next image.
[0101] In addition, some actions require a moving direction to determine the moving action. For example, picking needs to move along a three-dimensional moving direction, such as moving upwards. Figure 4 As shown. For pushing and pulling, the robot needs to determine the direction of pushing or pulling. Therefore, for different types of action steps, users can use different prompt modes with or without moving direction prompts. It should be noted that although the moving direction prompt may overlap with other direction prompts, it does not necessarily need to start from the contact point, as long as its direction is correct and visible in the visual prompt image. Subsequently, by completing the drawn visual prompt image, the two-dimensional coordinate information and two-dimensional direction information (two-dimensional posture information, two-dimensional moving direction information) of the contact point can be automatically extracted and integrated into the language prompt based on the RGB values of the corresponding pixels of the visual elements. Given the visual input and language input, the action prediction model is able to output the predicted action as three-dimensional task information.
[0102] In order to make the robot interact with the object given the predicted action, each action step can correspond to an atomic action interaction strategy. p Use the depth information and camera parameters to project into three-dimensional space to obtain three-dimensional coordinate information a0 P . 3D Z-axis direction information a0 Z and three-dimensional Y-axis direction information a0 Y Together, they determine the rotation matrix, facilitating the establishment of initial contact with the object. The next goal is to determine the subsequent direction of movement. Place, move, and rotate are simple actions. The direction of movement after contacting the object can be determined by the prompt on the next visual cue image. Picking requires moving along a0. M In contrast, push and pull actions are more complex because they involve rotating and translating the object. These actions require adjusting the attitude and direction of the end effector during movement to avoid collision. To achieve this goal, the force applied to the robot arm can be used as a signal to determine how to adjust the direction. Specifically, after establishing initial contact with the object, the object is first moved an appropriate distance along the predicted three-dimensional movement direction. Subsequently, the actual position a reached by the end effector is analyzed. tP and through the previous position a t-1 P and the three-dimensional moving direction a t-1 M The calculated expected position a t P’ Due to the constraints of the joint object, a t P and a t P’ There may be a difference between them, reflecting the rotation of the moving parts. This difference can be used to approximate the joint axis A t And calculate the rotation adjustment θ required to align the z-axis direction of the end effector with the joint constraint. Therefore, the three-dimensional z-axis direction a t-1 Z Along axis A t Rotate θ to adjust a t Z .
[0103] In addition to the atomic action interaction strategy, a long-horizon task execution strategy can also be used to implement long-term tasks. For example, a series of visual cue images are used as high-level planning, and each visual cue image corresponds to an atomic task (or action step). By using a motion planner to execute the above atomic action interaction strategy, long-term trajectories can be completed. The benefit of this approach is that it can decompose the complexity of long-term tasks while optimizing the success rate of each atomic task to ensure the success of the overall task. At the same time, by treating each atomic task as a unit, they can be combined into any combination, enabling the action prediction model to handle a variety of manipulation tasks.
[0104] The above embodiments use visual cues to clearly communicate the task objectives, including both low-level actions (e.g., corresponding to a single action step) and high-level planning (e.g., corresponding to multiple action steps). The trained action prediction model can understand two-dimensional visual information and predict accurate contact postures and three-dimensional movement directions in three-dimensional space. A task execution strategy is designed for the robot to complete each atomic task based on the prediction, while connecting each action step one by one to form an overall trajectory.
[0105] The embodiment of the present application also provides a robot manipulation device, which includes an image receiving module, a visual extraction module and an object manipulation module. The image receiving module is used to receive visual prompt images corresponding to multiple action steps of the robot's end effector manipulating an object; the two-dimensional visual information in the visual prompt image is used to prompt two-dimensional task information, and the two-dimensional task information includes one or more of the two-dimensional coordinate information of the contact point, the two-dimensional posture information of the end effector, and the two-dimensional moving direction information. The visual extraction module is used to extract the two-dimensional visual information in the visual prompt image corresponding to at least one of the action steps to obtain the three-dimensional task information corresponding to the action step; the three-dimensional task information includes one or more of the three-dimensional coordinate information of the contact point, the three-dimensional posture information of the end effector, and the three-dimensional moving direction information. The object manipulation module is used to control the end effector to manipulate the object using the three-dimensional task information corresponding to at least one of the action steps.
[0106] In some embodiments, the three-dimensional task information may include the three-dimensional moving direction information; the device also includes a step connection module, which is used to connect the action steps and their corresponding next action steps in sequence according to the sequence of at least one of the action steps, using the three-dimensional moving direction information corresponding to the action step, so as to obtain the corresponding overall trajectory of at least one of the action steps; the object manipulation module is used to plan the motion trajectory of the end effector to execute the action step using the three-dimensional task information corresponding to the action step, so that the end effector completes the overall trajectory.
[0107] In some embodiments, the visual extraction module can extract the two-dimensional visual information in the visual cue image corresponding to the action step to obtain the three-dimensional task information corresponding to the action step in the following manner: extract the two-dimensional visual information in the visual cue image to obtain the two-dimensional task information; input the visual cue image and the two-dimensional task information into the action prediction model to obtain the three-dimensional task information.
[0108] See also Figure 5 , Figure 5 It is a structural block diagram of a robot provided in an embodiment of the present application.
[0109] An embodiment of the present application further provides a robot, comprising a control module and an end effector, wherein the control module is used to execute any one of the above methods to control the end effector to manipulate an object.
[0110] In some embodiments, the robot may be a multi-jointed robot.
[0111] In some embodiments, the robot may further include one or more of an angle encoder, a torque sensor, a camera, and a PIR sensor.
[0112] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.
[0113] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.
[0114] The computer program product may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer readable media.
[0115] An embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.
[0116] See also Figure 6 , Figure 6 It is a structural block diagram of a computer device provided in an embodiment of the present application.
[0117] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.
[0118] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.
[0119] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.
[0120] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solution of the embodiment of the present application is implemented by software or firmware, the computer program for implementing the solution of the embodiment of the present application can be stored in the processor 120 and executed by the processor 120.
[0121] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.
[0122] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor, etc.
[0123] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 120 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor 120 for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0124] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input output system (BIOS), application software, etc.
[0125] The operating system is used to manage the hardware and / or software resources of computer devices and is the core and cornerstone of computer devices. The operating system needs to handle basic tasks such as managing and configuring memory, determining the priority of system resource supply and demand, controlling input and output devices, operating networks, and managing file systems. In order to facilitate user operations, most operating systems will provide an operating interface for users to interact with the system.
[0126] BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and application programs. In some implementations, BIOS can also monitor and display processor temperature and execute functions such as adjusting temperature protection strategies.
[0127] Application software is also called application program, which can be understood as software written for a specific application purpose of the user. It is one of the main categories of computer software. For example, application software can be a program used to achieve power control, temperature management, etc.
[0128] It should be noted that although some embodiments of the present application take mobile robots as examples, the present application can be applied to other bionic robots, such as AGVs, drones, etc., and the present application is not limited to this.
[0129] It should be understood that the specific examples in this specification are only intended to help those skilled in the art to better understand the implementation methods of the present application, rather than to limit the scope of protection of the present application.
[0130] It is understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.
[0131] It can be understood that the various implementation modes described in this specification can be implemented individually or in combination, and this application is not limited to this.
[0132] Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0133] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.
[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each embodiment described above can refer to the corresponding process in other embodiments and will not be repeated here.
[0135] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the technical solution of the present application.
[0137] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0138] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0139] The above are only specific implementations of this specification, but the protection scope of this application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of this application should be based on the protection scope of the claims.
Claims
1. A robot manipulation method, characterized in that: The method comprises: Receiving visual prompt images corresponding to a plurality of action steps of manipulating an object by an end effector of the robot; the two-dimensional visual information in the visual prompt image is used to prompt two-dimensional task information, and the two-dimensional task information includes one or more of two-dimensional coordinate information of a contact point, two-dimensional posture information of the end effector, and two-dimensional moving direction information; For at least one of the action steps, extracting two-dimensional visual information in a visual cue image corresponding to the action step to obtain three-dimensional task information corresponding to the action step; the three-dimensional task information includes one or more of three-dimensional coordinate information of the contact point, three-dimensional posture information of the end effector, and three-dimensional moving direction information; The end effector is controlled to manipulate the object using the three-dimensional task information corresponding to at least one of the action steps.
2. The robot manipulation method according to claim 1, characterized in that: The three-dimensional task information includes the three-dimensional moving direction information; The method further comprises: according to the sequence of at least one of the action steps, using the three-dimensional moving direction information corresponding to the action step, sequentially connecting the action step and its corresponding next action step to obtain the overall trajectory corresponding to at least one of the action steps; The use of three-dimensional task information corresponding to at least one of the action steps to control the end effector to manipulate the object includes: using the three-dimensional task information corresponding to the action step to plan the motion trajectory of the end effector to execute the action step so that the end effector completes the overall trajectory.
3. The robot manipulation method according to claim 2, characterized in that: Each action step corresponds to an atomic task, and the atomic task includes one or more of placement, movement, rotation, picking, pulling and pushing.
4. The robot manipulation method according to claim 1, characterized in that: The extracting of the two-dimensional visual information in the visual prompt image corresponding to the action step to obtain the three-dimensional task information corresponding to the action step includes: Extracting two-dimensional visual information from the visual cue image to obtain two-dimensional task information; The visual cue image and the two-dimensional task information are input into an action prediction model to obtain the three-dimensional task information.
5. The robot manipulation method according to claim 4, characterized in that: The training process of the action prediction model includes: Acquire a training set; each set of training data in the training set includes a training visual prompt image, training two-dimensional task information, and training three-dimensional task information; For at least one set of the training data, the following processing is performed: Inputting the training visual cue image and the training two-dimensional task information in the training data into the action prediction model to obtain predicted three-dimensional task information; Using the predicted three-dimensional task information and the training three-dimensional task information, a loss value corresponding to a target loss function is calculated; Using the loss value, updating the model parameters of the action prediction model; Check whether the training end condition is met; if yes, stop training the motion prediction model; if no, continue training the motion prediction model using the next set of training data.
6. The robot manipulation method according to claim 5, characterized in that: The training visual prompt image is obtained by drawing visual elements on a two-dimensional image containing the training object. The process of obtaining the training data includes: Controlling the end effector to randomly interact with the training object in the simulator, and recording training 3D task information of successfully manipulating the training object, wherein the training 3D task information includes training 3D coordinate information of training contact points, training 3D posture information of the end effector, and training 3D moving direction information; and / or, Projecting the training three-dimensional coordinate information of the training contact point onto the two-dimensional image to obtain the training two-dimensional coordinate information of the training contact point, and drawing the visual element corresponding to the training contact point on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point; and / or, Determine the training three-dimensional coordinate information of the posture reference point based on the training three-dimensional coordinate information of the training contact point and the training three-dimensional posture information, project the training three-dimensional coordinate information of the posture reference point onto the two-dimensional image to obtain the training two-dimensional coordinate information of the posture reference point, and draw the visual element corresponding to the training two-dimensional posture information on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the posture reference point; and / or, The training three-dimensional coordinate information of the moving direction reference point is determined based on the training three-dimensional coordinate information of the training contact point and the training three-dimensional moving direction information, the training three-dimensional coordinate information of the moving direction reference point is projected onto the two-dimensional image to obtain the training two-dimensional coordinate information of the moving direction reference point, and the visual elements corresponding to the training two-dimensional moving direction information are drawn on the two-dimensional image based on the training two-dimensional coordinate information of the training contact point and the training two-dimensional coordinate information of the moving direction reference point.
7. The robot manipulation method according to claim 6, characterized in that: The three-dimensional posture information includes three-dimensional Y-axis direction information and three-dimensional Z-axis direction information, and the target loss function includes one or more of text supervision loss, orthogonal loss and projection loss.
8. The robot manipulation method according to claim 1, characterized in that: The two-dimensional visual information uses visual elements with different display parameters to prompt at least two of the two-dimensional coordinate information of the contact point, the two-dimensional posture information and the two-dimensional moving direction information; The visual elements include one or more of points, lines and shapes; The display parameters include one or more of color, line width, line type, line arrow direction, brightness, contrast, saturation, exposure, hue, brightness, grayscale and color temperature.
9. A robot, characterized in that: The robot comprises a control module and an end effector, wherein the control module is used to execute the method according to any one of claims 1 to 8 to control the end effector to manipulate an object.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
11. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.
12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Natural language control method for humanoid robot
CN119610090A
Robot, operation method thereof, operation device, storage medium, and program product
CN120773057A
Visual perception method of humanoid robot and related equipment
CN121962722A
Control method, device and system and storage medium
CN122172630A
