Robot control method and device based on visual language model and robot

Abstract

The application discloses a robot control method and device based on a visual language model and a robot, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring interaction data and image data; inputting the interaction data into an action-perception visual language model, determining a task based on the interaction data, identifying a control object corresponding to the task from the image data according to the task, and outputting kinematic parameters of the control object; and controlling the robot to operate the control object based on the task and the kinematic parameters. According to the application, the action-perception visual language model is centered on the control object, the operable part of the control object is located according to the interaction data and the image data, necessary joint information is provided, and operation is performed, the output of the model is related to the controlled object, and is not related to the robot hardware, so that the robot hardware configuration and capability need not be customized, different robot configurations can be adapted, and the universality and adaptability of robot control are improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a robot control method, device and robot based on a visual language model. Background Technology

[0002] In AI-driven robotics, the Visual Language Model (VLM) focuses on combining visual information with language instructions, enabling robots to understand and execute task descriptions based on natural language.

[0003] In related technologies, VLM model design mainly focuses on directly learning robot-centric actions. The instructions output by the model are strongly correlated with the robot hardware itself, which limits the model's generalization ability and makes it difficult to adapt to different hardware platforms or environments. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a robot control method, device, and robot based on a visual language model to adapt to different robot configurations and improve the versatility and adaptability of robot control.

[0005] In a first aspect, this application provides a robot control method based on a visual language model, comprising:

[0006] Acquire interactive data and image data;

[0007] Interactive data is input into the action-aware visual language model, which determines the task based on the interactive data, identifies the control object corresponding to the task from the image data according to the task, and outputs the kinematic parameters of the control object.

[0008] The robot is controlled to operate the controlled object based on the task and the kinematic parameters.

[0009] According to the robot control method based on a visual language model of this application, interactive data and image data are acquired; the interactive data is input into an action-aware visual language model, which determines a task based on the interactive data, identifies the control object corresponding to the task from the image data according to the task, and outputs the kinematic parameters of the control object; the robot is controlled to operate the control object based on the task and the kinematic parameters. This embodiment of the application enables the action-aware visual language model to center on the control object, identify the control object based on interactive data and image data, and output the kinematic parameters of the control object. The robot can then be controlled to operate the control object using these kinematic parameters, eliminating the need for customization based on the hardware configuration and capabilities of a specific robot. This allows for adaptation to different robot configurations, improving the versatility and adaptability of robot control.

[0010] According to one embodiment of this application, the kinematic parameters include the 3D position information of the controlled object, the joint structure, and the motion characteristics of the joint structure.

[0011] According to one embodiment of this application, the joint structure includes a motion axis, and the motion characteristics include the motion type and range of motion of the joint structure. The motion type includes linear motion and rotational motion, and the rotational motion includes rotation and rolling.

[0012] According to one embodiment of this application, controlling the robot to operate the controlled object based on the task and the kinematic parameters includes:

[0013] Path information is planned based on the 3D position information of the controlled object; the operable area of ​​the controlled object is determined based on the joint structure; and the motion type and range of motion of the controlled object are determined based on the motion characteristics of the joint structure.

[0014] Based on the task, the path information, the operable area, the action type, and the range of motion of the controlled object, a target instruction is matched from the robot's instruction library so that the robot moves to the position of the controlled object according to the target instruction, and operates the controlled object in the operable area according to the action type and the range of motion.

[0015] In this embodiment, by matching target instructions from the robot's instruction library based on path information, operable area, and action type, the robot can perform appropriate actions at the correct location based on instructions in its own instruction library, without relying on the robot's hardware configuration, thus further enhancing the versatility of the control strategy.

[0016] According to one embodiment of this application, controlling the robot to operate the controlled object based on the task and the kinematic parameters includes:

[0017] The operation point of the controlled object is determined according to the task; wherein, the operation point is the position of the controlled object away from the motion axis;

[0018] The robot is controlled to operate the controlled object at the operating point according to the kinematic parameters.

[0019] In this embodiment, by determining the operation point as a position in the controlled object that is far from the motion axis according to the task, the robot can use a longer lever arm to amplify its force output, and can generate a greater torque effect even when the force output remains unchanged, thereby reducing the robot's load and improving the efficiency of operating the controlled object.

[0020] According to one embodiment of this application, controlling the robot to operate the controlled object based on the task and the kinematic parameters includes:

[0021] The state of the robot's gripper is determined according to the task.

[0022] The robot is controlled to operate the controlled object in the gripper state according to the kinematic parameters.

[0023] In this embodiment, by determining the state of the robot gripper to adapt to a specific task type, the opening degree, shape, or force of the gripper can be adjusted according to task requirements, so that the robot can better fit the characteristics of the controlled object and achieve more precise operation of the controlled object.

[0024] According to one embodiment of this application, the action-aware visual language model is trained in the following manner:

[0025] Construct a sample set, wherein the samples in the sample set include images and corresponding question-and-answer texts for the images, and the labels of the samples include the kinematic parameters of the control objects in the images;

[0026] The pre-trained visual language model is fine-tuned based on the sample set to obtain the trained action-aware visual language model.

[0027] In this embodiment, by constructing a sample set, which includes images and corresponding question-and-answer text, and using the kinematic parameters of the controlled objects in the images as labels, the model can learn the correlation between the kinematic parameters of the controlled objects in the images and the natural language descriptions. This allows for a better understanding of complex task instructions and enables generalization to different robot platforms and operating scenarios, further improving the versatility and adaptability of robot control. Furthermore, the practice of fine-tuning the pre-trained model leverages the knowledge learned by the model on large-scale data, further enhancing learning efficiency and model performance.

[0028] According to one embodiment of this application, constructing the sample set includes:

[0029] ControlNet is used to perform data augmentation on the images, generating diverse images as the images in the samples;

[0030] Diverse question-and-answer texts were constructed using a large language model and prompting engineering to serve as the question-and-answer texts corresponding to the images in the sample.

[0031] In this embodiment, the images enhanced by ControlNet increase the diversity of the sample set, enabling the model to learn control objects from a wider range of perspectives and under more conditions, thus improving the model's adaptability to different environments. Meanwhile, the question-and-answer text generated by the large language model and prompting engineering can generate rich descriptions of reasoning tasks, covering a wider range of language scenarios and question types, thereby enhancing the model's ability to understand and respond to natural language instructions.

[0032] According to one embodiment of this application, fine-tuning the pre-trained visual language model based on the sample set includes:

[0033] The visual language model is fine-tuned in two stages; the first stage involves fine-tuning the visual projection layer of the visual language model, and the second stage involves simultaneously fine-tuning the visual projection layer and the large language model portion of the visual language model.

[0034] In this embodiment, a two-stage fine-tuning process is employed to train the visual language model. This allows for the gradual optimization of the model's ability to integrate visual and linguistic information. The first stage focuses on fine-tuning the visual projection layer, enabling the model to more accurately extract features related to the controlled object from images, thus enhancing its sensitivity to visual input and its feature representation capabilities. The second stage further fine-tunes the linguistic component of the visual language model, achieving deep fusion of visual features and linguistic information and improving the model's understanding of complex instructions. This phased and meticulous adjustment helps resolve potential inconsistencies that may arise during the fusion of different modalities, ultimately achieving superior multimodal learning results and enabling the robot to more accurately perform tasks based on natural language descriptions.

[0035] According to one embodiment of this application, the method further includes:

[0036] In the second stage of fine-tuning, the proportion of samples with motion characteristics is increased.

[0037] In this embodiment, by increasing the proportion of samples with motion features in the second-stage fine-tuning, the model is able to learn more complex motion patterns and dynamic changes, which can significantly improve the model's ability to recognize and understand motion-related expressions, thereby more accurately capturing and predicting the motion state of the controlled object in practical applications.

[0038] According to one embodiment of this application, the samples in the sample set are simulated data.

[0039] In this embodiment, by using simulated data as training samples, the cost of data collection can be significantly reduced because simulated data allows for the generation of a large number of image and question-and-answer text pairs in a controlled virtual environment.

[0040] Secondly, this application provides a robot control device based on a visual language model, comprising:

[0041] The acquisition module is used to acquire interactive data and image data;

[0042] An input module is used to input interactive data into an action-aware visual language model, wherein the action-aware visual language model determines a task based on the interactive data, identifies the control object corresponding to the task from the image data according to the task, and outputs the kinematic parameters of the control object.

[0043] A control module is used to control the robot to operate the controlled object based on the task and the kinematic parameters.

[0044] According to the robot control device based on a visual language model of this application, interactive data and image data are acquired; the interactive data is input into an action-aware visual language model, which determines a task based on the interactive data, identifies the control object corresponding to the task from the image data based on the task, and outputs the kinematic parameters of the control object; the robot is controlled to operate the control object based on the task and the kinematic parameters. This embodiment of the application, by centering the action-aware visual language model on the control object, identifies the control object based on interactive data and image data and outputs the kinematic parameters of the control object, and the robot can be controlled to operate the control object using the kinematic parameters of the control object. This eliminates the need for customization for specific robot hardware configurations and capabilities, enabling adaptation to different robot configurations and improving the versatility and adaptability of robot control.

[0045] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot control method based on the visual language model as described in the first aspect above.

[0046] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot control method based on a visual language model as described in the first aspect above.

[0047] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the robot control method based on a visual language model as described in the first aspect above.

[0048] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the robot control method based on a visual language model as described in the first aspect above.

[0049] In a seventh aspect, this application provides a robot controlled by a robot control method based on a visual language model as described in the first aspect above.

[0050] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0051] According to the robot control method based on visual language model of this application, by using action-aware visual language model to learn the joint structure and action feasibility of the controlled object from the collected interaction data and image data, the action-aware visual language model can locate the operable parts of the controlled object and provide the necessary joint information and execution operations through prompts and language task descriptions. The instructions output by the model are related to the robot's controlled object, not to the robot hardware itself. Therefore, it is not necessary to collect a large amount of robot interaction data to be customized for the hardware configuration and capabilities of a specific robot. It can adapt to different robot configurations and improve the versatility and adaptability of robot control.

[0052] Furthermore, in some embodiments, by matching target instructions from the robot's instruction library based on path information, operable area, and action type, the robot can perform appropriate actions at the correct location based on instructions in its own instruction library, without relying on the robot's hardware configuration, thereby further enhancing the versatility of the control strategy.

[0053] Furthermore, in some embodiments, by determining the operation point as a position in the controlled object far from the motion axis according to the task, the robot can use a longer lever arm to amplify its force output, generating a greater torque effect even when the force output remains unchanged, reducing the robot's load and improving the efficiency of operating the controlled object.

[0054] Furthermore, in some embodiments, by determining the state of the robot gripper to adapt to a specific task type, the opening degree, shape, or force of the gripper can be adjusted according to task requirements, so that the robot can better conform to the characteristics of the controlled object and achieve more precise operation of the controlled object.

[0055] Furthermore, in some embodiments, by constructing a sample set, which includes images and corresponding question-and-answer text, and using the kinematic parameters of the controlled objects in the images as labels, the model can learn the correlation between the kinematic parameters of the controlled objects in the images and the natural language descriptions. This allows for a better understanding of complex task instructions and enables generalization to different robot platforms and operating scenarios, further improving the versatility and adaptability of robot control. In addition, fine-tuning the pre-trained model leverages the knowledge learned by the model on large-scale data, further enhancing learning efficiency and model performance.

[0056] Furthermore, in some embodiments, images enhanced by ControlNet increase the diversity of the sample set, enabling the model to learn control objects from a wider range of perspectives and under more conditions, thus improving the model's adaptability to different environments. Meanwhile, question-and-answer text generated using large language models and prompt engineering can produce rich descriptions of reasoning tasks, covering a wider range of language scenarios and question types, thereby enhancing the model's ability to understand and respond to natural language instructions.

[0057] Furthermore, in some embodiments, a two-stage fine-tuning process is employed to train the visual language model. This allows for the gradual optimization of the model's ability to integrate visual and linguistic information. The first stage focuses on fine-tuning the visual projection layer, enabling the model to more accurately extract features related to the controlled object from images, thus enhancing its sensitivity to visual input and its feature representation capabilities. The second stage further fine-tunes the linguistic component of the visual language model, achieving deep fusion of visual features and linguistic information and improving the model's understanding of complex instructions. This phased and meticulous adjustment helps resolve potential inconsistencies that may arise during the fusion of different modalities, ultimately achieving superior multimodal learning results and enabling the robot to more accurately perform tasks based on natural language descriptions.

[0058] Furthermore, in some embodiments, by increasing the proportion of samples with motion features in the second-stage fine-tuning, the model can learn more complex motion patterns and dynamic changes, which can significantly improve the model's ability to recognize and understand motion-related expressions, thereby more accurately capturing and predicting the motion state of the controlled object in practical applications.

[0059] Furthermore, in some embodiments, using simulated data as training samples can significantly reduce the cost of data collection, as simulated data allows for the generation of a large number of image and question-and-answer text pairs in a controlled virtual environment.

[0060] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0061] The above and / or additional aspects and advantages of this application will become apparent and readily understood in conjunction with the following description of the embodiments in conjunction with the accompanying drawings, wherein:

[0062] Figure 1 This is a flowchart illustrating the robot control method based on a visual language model provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the three-dimensional bounding box of the control object provided in the embodiments of this application;

[0064] Figure 3 This is a schematic diagram of the motion characteristics of the joint structure provided in the embodiments of this application;

[0065] Figure 4 This is a schematic image of a sample provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the architecture of the action-aware visual language model provided in an embodiment of this application;

[0067] Figure 6 This is a schematic diagram of the simulation data provided in the embodiments of this application;

[0068] Figure 7 This is a schematic diagram of the structure of the robot control device based on a visual language model provided in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0071] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0072] Visual Machines (VLM) is an advanced artificial intelligence technology that combines the capabilities of computer vision and natural language processing to achieve complex control of robots. The core function of VLM is to understand and process visual information, such as images and video frames, as well as verbal instructions, such as natural language descriptions or commands. This model is designed to allow robots to receive task descriptions in natural language and translate them into sequences of actions that the robot can perform.

[0073] Visual Learning Models (VLMs) typically employ deep learning architectures, particularly Transformer-based models, as they are adept at handling sequential data and capturing long-range dependencies. In a VLM, a visual encoder, such as a CNN or ViT, first extracts features from the visual data, while a text encoder, such as BERT or GPT, processes the linguistic input to extract semantic information. These features are then combined in a fusion layer, using attention mechanisms to reinforce the correlation between visual and linguistic information. In this way, VLMs can translate tasks described in natural language into action commands for the robot. For example, given a task description "Get me the red cup on the table," a VLM will parse keywords in the linguistic command, such as "red cup" and "get me," and combine this with visual information to identify and locate the red cup on the table, then generate a series of action commands to guide the robot to perform the action of picking up the cup.

[0074] In related technologies, VLM model design mainly focuses on directly learning robot-centric actions. Because VLM model training is highly data-dependent, this method requires collecting vast amounts of robot interaction data, particularly specific robot-environment interaction data, which is difficult to obtain in the real world, and collecting large amounts of real-world robot interaction data is extremely costly. Furthermore, this method often involves a strong correlation between the model output and specific robot hardware, meaning the motion commands learned during training are tailored to the specific robot's hardware configuration and capabilities. For example, a VLM designed for a robot with a six-DOF robotic arm might generate specific motion commands for the end effector, arm, and legs, such as commands to control the speed, direction, and position of these parts. These commands are precise for that robot but may not be applicable to another robot with different DDOs or joint configurations. This strong correlation limits the model's generalization ability, as each different hardware platform or environment may require different motion commands. This necessitates retraining or recalibrating the VLM to adapt to the new hardware when migrating it from one platform to another. Furthermore, the model may require further adjustments when the environment changes, such as different workspace layouts or object positions.

[0075] For example, a VLM trained in a laboratory environment might encounter obstacles in an industrial environment because it may not have learned how to handle equipment or obstacles specific to industrial environments. Similarly, a VLM designed specifically for a home environment may not work effectively in a commercial environment because it may not have received training on commercial layouts or object types.

[0076] This application considers that if, in VLM applications, the model focuses on learning actions centered on the robot's controlled object rather than learning robot-centered actions, it is expected to solve the problem of limited generalization ability of models in related technologies, making it difficult to adapt to different hardware platforms or environments.

[0077] The robot control method, device, and robot based on visual language model provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0078] Among them, the robot control method based on visual language model can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0079] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0080] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0081] The robot control method based on visual language model provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the robot control method based on visual language model. The electronic devices mentioned in this application embodiment include, but are not limited to, servers, controllers, mobile phones, tablets, computers, cameras, and wearable devices. The robot control method based on visual language model provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0082] like Figure 1 As shown, the robot control method based on the visual language model includes steps 110, 120 and 130.

[0083] Step 110: Obtain interactive data and image data.

[0084] In this embodiment, the image data can be data acquired through a camera, depth camera, or other vision sensor. The camera, depth camera, or other vision sensor can be installed in the robot or in other areas, such as a room. Image data can be acquired through the vision sensor, and this image data can include various objects, such as scissors, a fan, a refrigerator, or any other arbitrary object.

[0085] Interactive data can be any data generated through interaction with the environment. For example, interactive data could be text data entered by a user via a keyboard or other text input device, voice data emitted through sound, or captured data such as the user's body movements and gestures. Of course, interactive data can also be data related to the robot itself; for example, interactive data could be data collected by various sensors mounted on the robot, such as data collected by distance sensors, force sensors, and temperature sensors.

[0086] Step 120: Input the interaction data into the action-aware visual language model. The action-aware visual language model determines the task based on the interaction data, identifies the control object corresponding to the task from the image data according to the task, and outputs the kinematic parameters of the control object.

[0087] In this embodiment, the action-aware visual language model is a trained model. For example, the action-aware visual language model can be a model trained from scratch using a large amount of data and computing resources. Of course, to save costs, a pre-trained visual language model can be used, and then fine-tuned to obtain the action-aware visual language model of this embodiment.

[0088] In this embodiment, the action-aware visual language model can include two parts: a language model and a visual model. The language model is responsible for processing natural language input (such as text descriptions and instructions), while the visual model is responsible for processing visual input (such as images and videos). These two models cooperate to understand the relationship between text and images through a shared representation space. The action-aware visual language model can determine the task corresponding to the interaction data by understanding the interaction data, and then identify the control object corresponding to the task from the image data based on the task. For example, for the interaction data being the user-inputted text instruction "I am thirsty," the language model in the action-aware visual language model can understand "I am thirsty" and determine the task as "get water." Then, the visual model can identify the image data. For example, if the image data includes objects such as a refrigerator, a bench, and toys, the visual model can identify the control object "refrigerator" in the image data that is related to the task "get water," because the model has been trained to associate it with water in the refrigerator.

[0089] In this embodiment of the application, after the action-aware visual language model identifies the controlled object from the image data, it can further determine the kinematic parameters of the controlled object so that in subsequent steps, the robot can be instructed on how to operate the controlled object based on the kinematic parameters.

[0090] In some embodiments, the kinematic parameters of the controlled object can represent its position and orientation in space, as well as the configuration and motion capabilities of its moving parts. Kinematic parameters describe information such as the controlled object's position, range of motion, and motion capabilities, thus facilitating instructing the robot on how to operate the controlled object. For example, for a controlled object like a refrigerator, its kinematic parameters could include the refrigerator's location, the fact that its operable part is the door, and how the door opens and closes. Knowing the refrigerator's location and how its door opens and closes allows the robot to operate the refrigerator.

[0091] In some embodiments, kinematic parameters include 3D position information of the controlled object, joint structures, and motion characteristics of the joint structures.

[0092] like Figure 2 As shown, 3D position information can be output in the form of a 3D bounding box, such as outputting the position coordinates of each vertex of the 3D bounding box, or outputting the 3D bounding box in a visual form, which is the position of the controlled object.

[0093] like Figure 3 As shown, a joint structure refers to a part of multiple components or objects that are interconnected through connection points or joints. Joint structures enable objects to perform complex movements and changes, much like the skeleton of a living organism is connected by joints. A joint structure can include an axis of motion, around which parts of the object can move. For example, in real life, a joint structure can be a door hinge, a door pivot, or a drawer slide; these can all be joint structures that allow them to open, close, or grip objects. Hinges and slides can be considered axes of motion.

[0094] The kinematic characteristics of a joint structure can be parameters such as the motion capability and range of motion exhibited by the various parts of an object connected by the joint structure. For example... Figure 3 As shown, the motion characteristics of a joint structure include the type of motion and range of motion of the controlled object. The type of motion includes linear motion and rotational motion. Linear motion can include sliding, and rotational motion includes turning and rolling. The range of motion can represent information such as the degrees of freedom of the controlled object's movement around the joint structure, the maximum angle or distance of movement, etc.

[0095] Step 130: Control the robot to operate the controlled object based on the task and kinematic parameters.

[0096] In this embodiment, the robot can be any type of robot. For example, the robot may include components such as joints and end effectors. The joints enable the movement of various parts of the robot, and these joints can be rotational or sliding. The end effector can be the end tool of a robot arm, such as a gripper, a gripper, a suction cup, or other specially designed mechanical devices, used to interact with the environment, such as grasping, carrying, welding, or drawing.

[0097] In this embodiment, kinematic parameters reveal how the controlled object is controlled, enabling the robot to manipulate the controlled object. Specifically, the robot can plan a path based on the kinematic parameters of the controlled object, allowing the robot body or end effector to reach the corresponding position of the controlled object. This also determines the graspable position and range of motion of the controlled object. Based on this information, the robot's end effector is controlled to reach the corresponding position of the controlled object and adapt to its shape and characteristics, performing grasping, moving, and other operations on the controlled object.

[0098] For example, for the task "open the refrigerator", the output kinematic parameters can include information such as the position of the refrigerator door, the rotation axis of the door, the direction and range of the door's movement. Based on this information, the robot's end effector, such as a gripper or a suction cup, can be controlled to move to the position of the refrigerator door. The gripping point is determined by referring to the direction and range of the refrigerator door's movement, and then the refrigerator door is moved from the gripping point, thereby controlling the robot to operate the refrigerator door.

[0099] According to the robot control method based on a visual language model of this application, interactive data and image data are acquired; the interactive data is input into an action-aware visual language model, which determines the task based on the interactive data, identifies the control object corresponding to the task from the image data, and outputs the kinematic parameters of the control object; the robot operates the control object based on the task and the kinematic parameters. This embodiment of the application utilizes an action-aware visual language model to learn the joint structure and action feasibility of the control object from the acquired interactive data and image data. Through prompts and language task descriptions, the action-aware visual language model can locate the operable parts of the control object and provide necessary joint information and execution operations. The instructions output by the model are related to the robot's controlled object, not to the robot's hardware itself. For example, the instructions output by the model do not control the robot's end effector, robotic arm, or other parts. Therefore, it is not necessary to collect a large amount of robot interactive data to tailor the control method to the specific robot's hardware configuration and capabilities. This approach can adapt to different robot configurations, improving the versatility and adaptability of robot control.

[0100] In some embodiments, controlling the robot's operation of the controlled object based on the task and kinematic parameters includes:

[0101] Path information is planned based on the 3D position information of the controlled object; the operable area of ​​the controlled object is determined based on the joint structure; and the motion type and range of motion of the controlled object are determined based on the motion characteristics of the joint structure.

[0102] Based on the task, path information, operable area, action type, and range of motion of the controlled object, the robot matches the target instruction from the robot's instruction library, so that the robot can move to the position of the controlled object according to the target instruction, and operate the controlled object in the operable area according to the action type and range of motion.

[0103] In this embodiment, the spatial relationship between the 3D position of the controlled object and the robot's current position can be analyzed to plan path information. Specifically, the robot can use its built-in sensor data and environmental maps, combined with path planning algorithms such as A*, Dijkstra's algorithm, or RRT algorithm, to calculate the optimal path to the 3D position of the controlled object. During path planning, the robot needs to consider obstacles, terrain features, and possible dynamic changes to ensure the feasibility and safety of the path. After path planning is completed, the robot will obtain a series of navigation points to guide its movement towards the controlled object. Of course, for robots with a fixed base, the path of the robot's end effector can be planned.

[0104] By analyzing the joint structure, the operable area of ​​the controlled object can be determined. For example, for a refrigerator door, the joint structure is the connecting shaft between the refrigerator door and the refrigerator, and the operable area can be determined to be any position on the refrigerator door, with the position away from the joint structure being the best. For a drawer, the joint structure is the slide rail, and the operable area can be determined to be any position on the drawer.

[0105] After determining the operable area, the type and range of motion of the controlled object need to be determined based on the motion characteristics of the joint structure. For example, if the target is a rotatable door handle, the robot needs to identify the rotational motion and the possible range of rotation angles.

[0106] In this embodiment, the instruction library is a resource within the robot control system, containing various predefined motion templates and instruction sequences for performing specific tasks, such as grasping instructions, movement instructions, and angle adjustment instructions. Different robots, due to their varying structures, will have different instructions in their instruction libraries. For example, regarding grasping instructions, different robots may be equipped with different types of end effectors, such as robotic arms, grippers, suction cups, or custom tools. Each end effector has a different grasping mechanism and control method, therefore their grasping instructions will also differ. Similarly, the movement instructions differ between quadruped and bipedal robots.

[0107] In this embodiment, based on the task, the planned path information, the determined operable area, and the action type and range of motion, the corresponding target instruction can be matched in the robot's instruction library. For example, the robot's movement instruction and grasping instruction can be matched, and the movement direction and movement angle of the robot grasping the controlled object can be calculated based on the action type and range of motion.

[0108] The robot can execute tasks based on matched target instructions, including moving to the location of the target object, adjusting the end effector to fit the operable area, and then performing operations according to the action type and range of motion. During execution, the task progress can be monitored in real time, and adjustments can be made based on sensor feedback to ensure the accuracy of the robot's movements and the successful completion of the task. For example, if the task is to rotate a door handle, the robot will move to the handle's location, adjust its gripper to correctly grasp the handle, and then perform the rotation action according to a predetermined range of motion.

[0109] Because the information output by the action-aware visual language model is the kinematic parameters of the controlled object, such as 3D position, action type of the operable area, etc., robots with different configurations or types can match the corresponding instructions from their own instruction library according to the information output by the model, thereby operating the controlled object. There is no need to collect a large amount of robot interaction data to customize the hardware configuration and capabilities of a specific robot.

[0110] In this embodiment, by matching target instructions from the robot's instruction library based on path information, operable area, and action type, the robot can perform appropriate actions at the correct location based on instructions in its own instruction library, without relying on the robot's hardware configuration, thus further enhancing the versatility of the control strategy.

[0111] In some embodiments, controlling the robot's operation of the controlled object based on the task and kinematic parameters includes:

[0112] The operation point of the controlled object is determined according to the task; where the operation point is the position of the controlled object that is farthest from the motion axis.

[0113] The robot is controlled to manipulate the controlled object at the operating point based on kinematic parameters.

[0114] In this embodiment, the operation point of the controlled object can be determined based on the different tasks. For example, if the task is "opening the door" or "closing the door," the operation point of the controlled object "door" can be a position away from the door hinge; if the task is "opening the refrigerator," the operation point of the controlled object "refrigerator door" can be a position away from the refrigerator door hinge. The operation point can be determined according to the different tasks. After the operation point is determined, the robot can be controlled to operate the controlled object at the operation point to complete the task execution.

[0115] In this embodiment, by determining the operation point as a position in the controlled object that is far from the motion axis according to the task, the robot can use a longer lever arm to amplify its force output, and can generate a greater torque effect even when the force output remains unchanged, thereby reducing the robot's load and improving the efficiency of operating the controlled object.

[0116] In some embodiments, controlling the robot's operation of the controlled object based on the task and kinematic parameters includes:

[0117] Determine the state of the robot's gripper based on the task;

[0118] The robot is controlled to manipulate the controlled object in a gripper state based on kinematic parameters.

[0119] In this embodiment, the gripper state of the robot can be the opening degree, force, gripper shape, etc., and different gripper states of the robot can be determined based on different task types.

[0120] For example, if the task is "opening a door," the robot's gripper can be in a state of opening or closing to a certain extent, grasping the controlled object "door" to perform the opening operation. When the task is "closing a door," the robot's gripper can be in a state of either opening or closing; as long as the robot's gripper contacts the controlled object "door," the closing operation can be completed. Of course, when the task is other, the robot's gripper state can be determined according to the specific content of the task, and this application embodiment does not limit this.

[0121] In this embodiment, by determining the state of the robot gripper to adapt to a specific task type, the opening degree, shape, or force of the gripper can be adjusted according to task requirements, so that the robot can better fit the characteristics of the controlled object and achieve more precise operation of the controlled object.

[0122] In some embodiments, the action-aware visual language model is trained as follows:

[0123] Construct a sample set, in which the samples include images and the corresponding question-and-answer texts, and the labels of the samples include the kinematic parameters of the control objects in the images;

[0124] The pre-trained visual language model is fine-tuned based on the sample set to obtain the action-aware visual language model.

[0125] In this embodiment, samples can be established in the form of Visual Question Answering (VQA), whereby the samples include images and corresponding question-and-answer text. Specifically, images of various objects can be collected, such as refrigerators, scissors, drawers, switches, etc. These control objects can include objects with joint structures or those without. In the question-and-answer text, the question can be a natural language question about the image content, related to objects in the image, object attributes, relationships between objects, or object actions; the answer is the corresponding response to the question, which can indicate the object's name, attributes, actions, or location, etc. For example, such as... Figure 4As shown in the image, there is a pair of scissors and a piece of paper; the question is: how do you cut the paper? The answer is: use scissors to cut it.

[0126] In this embodiment, sample labels can be established using structured data to represent the kinematic parameters of the controlled object in the image. For example, the kinematic parameters of the controlled object in the image can be represented by a triple (B, A, S). Here, B represents the bounding box locating the operable part of interest in the image, i.e., the 3D position of the controlled object; A represents the axis of motion of the joint structure; and S represents the semantic label, indicating the type of motion of the joint structure, such as linear or rotational motion, the name of the axis, etc. The 3D bounding box B can be composed of its eight vertices {(x...)} i ,y i ,z i )}, i = 1,...,8 represents. Here, (x i ,y i ) is the 2D projection position of vertex i in the given image plane, z i Normalize to the (0,1) range using the maximum and minimum depth values. Axis A can be represented by two edge points in 3D space {(α... i ,β i ,γ i )}, i = 1, 2 represents, and is normalized using the same method as bounding box B.

[0127] In this embodiment, a suitable pre-trained visual language model can be selected, such as... Figure 5 As shown, the architecture of a visual language model can include a large language model component and a visual projection layer. The large language model can be a model such as BERT (Bidirectional Encoder Representations from Transformers), a model combining Faster R-CNN and BERT, LLaMA (Large Language Model Meta AI), or GPT (Generative Pre-trained Transformer). The visual projection layer can include a visual model, such as a convolutional neural network, ViT (Vision Transformer), or self-attention networks. Of course, the visual language model can also be other models specifically pre-trained for visual language tasks.

[0128] In this embodiment, the fine-tuning phase requires setting some tasks so that the model knows which specific tasks it needs to learn and perform.

[0129] In some embodiments, four different subtasks can be constructed: detection, REC-Link, REG-Joint, and REC-Action. Each subtask includes an image, a text question, and a text answer. The detection task requires the VLM to locate all operable parts in an image by outputting a bounding box B for each part in the image. The REC-Link task involves locating joint structures based on a description. The REG-Joint task requires the VLM to provide a motion axis A and a semantic label S for the part specified by the bounding box B. The REC-Action task requires the VLM to locate a movable part based on an action task, such as "open the refrigerator," by providing a bounding box B and a corresponding semantic label S; for example, for "open the refrigerator," it can be located at any position on the refrigerator door. By defining the tasks, the model can be trained and fine-tuned according to the corresponding objective function. By defining explicit inputs (image, text question) and outputs (text answer, bounding box, joint axis, etc.) for each subtask, the model can better understand and learn how to handle different types of tasks.

[0130] Next, a suitable loss function and optimizer can be selected based on the nature of the task. Then, the visual language model can be fine-tuned using a sample set, and the model parameters can be optimized through backpropagation. During the fine-tuning process, different fine-tuning strategies can be tried based on the model's performance during training.

[0131] In this embodiment, by constructing a sample set, which includes images and corresponding question-and-answer text, and using the kinematic parameters of the controlled objects in the images as labels, the model can learn the correlation between the kinematic parameters of the controlled objects in the images and the natural language descriptions. This allows for a better understanding of complex task instructions and enables generalization to different robot platforms and operating scenarios, further improving the versatility and adaptability of robot control. Furthermore, the practice of fine-tuning the pre-trained model leverages the knowledge learned by the model on large-scale data, further enhancing learning efficiency and model performance.

[0132] In some embodiments, constructing a sample set includes:

[0133] ControlNet is used to perform data augmentation on images, generating diverse images as sample images;

[0134] We use a large language model and prompting engineering to construct diverse question-and-answer texts as the question-and-answer texts corresponding to the images in the samples.

[0135] ControlNet is a neural network for image data augmentation that can transform images based on their content to generate new image samples. Specifically, it can transform images through rotation, scaling, cropping, color adjustment, or adding noise to generate new images. For example, given an image of a robotic arm grasping an object, ControlNet can generate images of the arm grasping at different angles or against different backgrounds. By applying different transformations, the generated image samples exhibit diversity, covering a wide range of possible viewpoints and environmental conditions.

[0136] In this embodiment, the large language model can be a pre-trained large language model, such as GPT, BERT, or other models capable of generating text. Through prompting engineering, questions or instructions are designed based on the image content to serve as prompts for the model's generated text. For example, if the image is of a robot in a factory environment, the prompt could be "Describe how to cut paper." The language model then generates question-and-answer text based on the prompt. For example, the model might generate: "First, pick up the scissors and paper, then use its robotic arm to precisely aim the scissors at the paper and use the scissors to cut the paper." To ensure that the generated question-and-answer text is both diverse and relevant to the image content, the style and level of detail of the generated text can be controlled by adjusting the prompt information. For example, the prompt "Describe how to cut paper" could be adjusted to "I want to cut paper" or other similar expressions to generate the prompt information.

[0137] In this embodiment, the images enhanced by ControlNet increase the diversity of the sample set, enabling the model to learn control objects from a wider range of perspectives and under more conditions, thus improving the model's adaptability to different environments. Meanwhile, the question-and-answer text generated by the large language model and prompting engineering can generate rich descriptions of reasoning tasks, covering a wider range of language scenarios and question types, thereby enhancing the model's ability to understand and respond to natural language instructions.

[0138] In some embodiments, fine-tuning a pre-trained visual language model based on a sample set includes:

[0139] The visual language model is fine-tuned in two stages. The first stage involves fine-tuning the visual projection layer of the visual language model, while the second stage involves simultaneously fine-tuning the visual projection layer and the large language model portion of the visual language model.

[0140] In this embodiment, the Visual Projection Layer is the part of the visual language model responsible for processing visual information. It can use a Convolutional Neural Network (CNN) to extract features from the input image. The Visual Projection Layer transforms raw pixel data into high-level feature representations that capture information such as objects, shapes, and textures in the image. The large language model part is responsible for processing language information within the visual language model. It can use pre-trained language models such as BERT or GPT. The language model understands the semantic content of textual data and transforms it into embedded representations that can be combined with visual features.

[0141] The first stage of fine-tuning focuses primarily on adjusting the visual portion of the model to adapt it to a specific visual task. In this first stage, the parameters of the large language model can be frozen, and only the parameters of the visual projection layer are updated.

[0142] In the second stage of fine-tuning, the parameters of both the visual projection layer and the large language model part can be fine-tuned simultaneously. In this second stage, the parameters of the large language model part can be unfrozen, allowing them to be updated during training.

[0143] In this embodiment, a two-stage fine-tuning process is employed to train the visual language model. This allows for the gradual optimization of the model's ability to integrate visual and linguistic information. The first stage focuses on fine-tuning the visual projection layer, enabling the model to more accurately extract features related to the controlled object from images, thus enhancing its sensitivity to visual input and its feature representation capabilities. The second stage further fine-tunes the linguistic component of the visual language model, achieving deep fusion of visual features and linguistic information and improving the model's understanding of complex instructions. This phased and meticulous adjustment helps resolve potential inconsistencies that may arise during the fusion of different modalities, ultimately achieving superior multimodal learning results and enabling the robot to more accurately perform tasks based on natural language descriptions.

[0144] In some embodiments, the method further includes:

[0145] In the second stage of fine-tuning, the proportion of samples with motion characteristics is increased.

[0146] In this embodiment, a sample with motion characteristics is one in which the control object in the image is clearly related to motion, and the corresponding question-and-answer text is also related to motion. For example, if the control object in the image is scissors, and the question-and-answer text is "how to cut paper" and "use scissors to aim at the paper and open and close the scissors," then this sample is considered to have motion characteristics because both the image and the question-and-answer text are related to the motion of "cutting paper."

[0147] In the second stage of fine-tuning, the proportion of samples with motion features can be increased. For example, samples with motion features can be screened from the sample set, or new samples with motion features can be added through other means, thereby increasing the proportion of samples with motion features.

[0148] In this embodiment, by increasing the proportion of samples with motion features in the second-stage fine-tuning, the model is able to learn more complex motion patterns and dynamic changes, which can significantly improve the model's ability to recognize and understand motion-related expressions, thereby more accurately capturing and predicting the motion state of the controlled object in practical applications.

[0149] In some embodiments, the samples in the sample set are simulated data.

[0150] In this embodiment, the simulated data is taken from the PartNet-Mobility dataset, which contains thousands of 3D models with detailed motion annotations describing how the various parts of the model move. Using the simulated data, researchers can test the interaction of robots with these 3D models in a virtual environment without physical manipulation in the real world. This not only provides a safe and cost-effective way to develop and improve algorithms but also allows for the exploration of various operational scenarios without practical hardware limitations.

[0151] However, a limitation of the original PartNet-Mobility dataset is the lack of texture detail. For example... Figure 6 As shown, most objects are rendered in pure gray, which does not reflect real-world conditions. To bridge the gap between simulation and reality, ControlNet can be used to generate more realistic images. To generate richer descriptions of inference tasks, covering a wider range of language scenarios and question types, large language models and prompt engineering can be used to construct diverse question-and-answer texts corresponding to the images in the samples.

[0152] In this embodiment, by using simulated data as training samples, the cost of data collection can be significantly reduced because simulated data allows for the generation of a large number of image and question-and-answer text pairs in a controlled virtual environment.

[0153] The robot control method based on a visual language model provided in this application can be executed by a robot control device based on a visual language model. This application uses the execution of the robot control method based on a visual language model by a robot control device as an example to illustrate the robot control device based on a visual language model provided in this application.

[0154] This application also provides a robot control device based on a visual language model.

[0155] like Figure 7 As shown, the robot control device based on a visual language model includes:

[0156] The acquisition module 710 is used to acquire interactive data and image data;

[0157] The input module 720 is used to input interactive data into the action-aware visual language model. The action-aware visual language model determines the task based on the interactive data, identifies the control object corresponding to the task from the image data according to the task, and outputs the kinematic parameters of the control object.

[0158] The control module 730 is used for task and kinematic parameter-based control of the robot's operation of the controlled object.

[0159] According to the robot control device based on a visual language model of this application, interactive data and image data are acquired; the interactive data is input into a motion-aware visual language model, which determines the task based on the interactive data, identifies the control object corresponding to the task from the image data, and outputs the kinematic parameters of the control object; the robot is then controlled to operate the control object based on the task and the kinematic parameters. This embodiment of the application uses a visual language model centered on the control object to identify the control object based on interactive data and image data and output the kinematic parameters of the control object. The robot can then operate the control object using these kinematic parameters, eliminating the need for customization based on the hardware configuration and capabilities of a specific robot. This allows for adaptation to different robot configurations, improving the versatility and adaptability of robot control.

[0160] In some embodiments, kinematic parameters include 3D position information of the controlled object, joint structures, and motion characteristics of the joint structures.

[0161] In some embodiments, the joint structure includes a motion axis, and the motion characteristics include the type of motion and range of motion of the joint structure. The type of motion includes linear motion and rotational motion, and the rotational motion includes rotation and rolling.

[0162] In some embodiments, the control module 730 is further configured to:

[0163] Path information is planned based on the 3D position information of the controlled object; the operable area of ​​the controlled object is determined based on the joint structure; and the motion type and range of motion of the controlled object are determined based on the motion characteristics of the joint structure.

[0164] Based on path information, operable area, action type, and range of motion of the controlled object, the robot matches the target instruction from the robot's instruction library, so that the robot can move to the position of the controlled object according to the target instruction, and operate the controlled object in the operable area according to the action type and range of motion.

[0165] In some embodiments, the control module 730 is further configured to:

[0166] The operation point of the controlled object is determined according to the task; where the operation point is the position of the controlled object that is farthest from the motion axis.

[0167] The robot is controlled to manipulate the controlled object at the operating point based on kinematic parameters.

[0168] In some embodiments, the control module 730 is further configured to:

[0169] Determine the state of the robot's gripper based on the task;

[0170] The robot is controlled to manipulate the controlled object in a gripper state based on kinematic parameters.

[0171] In some embodiments, the action-aware visual language model is trained as follows:

[0172] Construct a sample set, in which the samples include images and the corresponding question-and-answer texts, and the labels of the samples include the kinematic parameters of the control objects in the images;

[0173] The pre-trained visual language model is fine-tuned based on the sample set to obtain the action-aware visual language model.

[0174] In some embodiments, constructing a sample set includes:

[0175] ControlNet is used to perform data augmentation on images, generating diverse images as sample images;

[0176] We use a large language model and prompting engineering to construct diverse question-and-answer texts as the question-and-answer texts corresponding to the images in the samples.

[0177] In some embodiments, fine-tuning a pre-trained visual language model based on a sample set includes:

[0178] The visual language model is fine-tuned in two stages. The first stage involves fine-tuning the visual projection layer of the visual language model, while the second stage involves simultaneously fine-tuning the visual projection layer and the large language model portion of the visual language model.

[0179] In some embodiments, during the second-stage fine-tuning, the proportion of samples with motion characteristics is increased.

[0180] In some embodiments, the samples in the sample set are simulated data.

[0181] The robot control device based on the visual language model in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific implementation.

[0182] The robot control device based on the visual language model in this application embodiment can be a device with an operating system. This operating system can be Microsoft (Windows), Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0183] In some embodiments, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-described robot control method embodiment based on a visual language model and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0184] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0185] This application also provides a robot, which is controlled by the above-described robot control method based on a visual language model.

[0186] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described robot control method embodiment based on a visual language model and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0187] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0188] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described robot control method based on a visual language model.

[0189] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0190] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described robot control method based on a visual language model, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0191] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0192] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0194] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0195] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0196] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A robot control method based on a visual language model, characterized in that, include: Acquire interactive data and image data; Interactive data is input into the action-aware visual language model, which determines the task based on the interactive data, identifies the control object corresponding to the task from the image data according to the task, and outputs the kinematic parameters of the control object; wherein, the kinematic parameters of the control object represent the position and posture of the control object in space, as well as the configuration and motion capability of its moving parts; The robot is controlled to operate the controlled object based on the task and the kinematic parameters.

2. The method according to claim 1, characterized in that, The kinematic parameters include the 3D position information of the controlled object, the joint structure, and the motion characteristics of the joint structure.

3. The method according to claim 2, characterized in that, The joint structure includes a motion axis, and the motion characteristics include the type of motion and range of motion of the joint structure. The type of motion includes linear motion and rotational motion, and the rotational motion includes rotation and rolling.

4. The method according to claim 2, characterized in that, The method of controlling the robot to operate the controlled object based on the task and the kinematic parameters includes: Path information is planned based on the 3D position information of the controlled object; the operable area of ​​the controlled object is determined based on the joint structure; and the motion type and range of motion of the controlled object are determined based on the motion characteristics of the joint structure. Based on the task, the path information, the operable area, the action type, and the range of motion of the controlled object, a target instruction is matched from the robot's instruction library so that the robot moves to the position of the controlled object according to the target instruction, and operates the controlled object in the operable area according to the action type and the range of motion.

5. The method according to claim 3, characterized in that, The method of controlling the robot to operate the controlled object based on the task and the kinematic parameters includes: The operation point of the controlled object is determined according to the task; wherein, the operation point is the position of the controlled object away from the motion axis; The robot is controlled to operate the controlled object at the operating point according to the kinematic parameters.

6. The method according to claim 1, wherein controlling the robot to operate the controlled object based on the task and the kinematic parameters comprises: The state of the robot's gripper is determined according to the task. The robot is controlled to operate the controlled object in the gripper state according to the kinematic parameters.

7. The method according to claim 1, characterized in that, The action-aware visual language model is trained in the following manner: Construct a sample set, wherein the samples in the sample set include images and corresponding question-and-answer texts for the images, and the labels of the samples include the kinematic parameters of the control objects in the images; The pre-trained visual language model is fine-tuned based on the sample set to obtain an action-aware visual language model.

8. The method according to claim 7, characterized in that, The construction of the sample set includes: ControlNet is used to perform data augmentation on the images, generating diverse images as the images in the samples; Diverse question-and-answer texts were constructed using a large language model and prompting engineering to serve as the question-and-answer texts corresponding to the images in the sample.

9. The method according to claim 7, characterized in that, The fine-tuning of the pre-trained visual language model based on the sample set includes: The visual language model is fine-tuned in two stages; the first stage involves fine-tuning the visual projection layer of the visual language model, and the second stage involves simultaneously fine-tuning the visual projection layer and the large language model portion of the visual language model.

10. The method according to claim 9, characterized in that, The method further includes: In the second stage of fine-tuning, the proportion of samples with motion characteristics is increased.

11. The method according to claim 7, characterized in that, The samples in the sample set are simulated data.

12. A robot control device based on a visual language model, characterized in that, include: The acquisition module is used to acquire interactive data and image data; An input module is used to input interactive data into an action-aware visual language model. The action-aware visual language model determines a task based on the interactive data, identifies the control object corresponding to the task from the image data according to the task, and outputs the kinematic parameters of the control object. The kinematic parameters of the control object represent the position and posture of the control object in space, as well as the configuration and motion capabilities of its moving parts. A control module is used to control the robot to operate the controlled object based on the task and the kinematic parameters.

13. A robot, characterized in that, The robot is controlled by the method as described in any one of claims 1-8.

14. An electronic device, characterized in that, The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the method as described in any one of claims 1-8.

Images