Robot control method and device based on base model knowledge

Abstract

The invention provides a robot control method and device based on base model knowledge, and relates to the technical field of robot control, and the method comprises the steps: inputting a text cue word and a visual cue word corresponding to a current task into a multi-modal base model, and obtaining a virtual view angle outputted by the multi-modal base model; projecting the three-dimensional point cloud corresponding to the multi-view-angle image to an image plane corresponding to the virtual view angle to generate a virtual view angle image; inputting the language instruction of the current task and the virtual view angle image into a key point prediction model to obtain spatial position information of a target key point output by the key point prediction model; and generating a motion track of the robot based on the spatial position information of the target key point, and controlling the robot to execute the current task based on the motion track. According to the method and the device provided by the invention, the sensing efficiency and the operation precision of the robot in the three-dimensional operation task are improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot control method and apparatus based on base model knowledge. Background Technology

[0002] With the rapid development of robotics technology in industrial manufacturing, warehousing and logistics, and daily services, the ability of robots to perform complex tasks in three-dimensional space has become a research hotspot. Compared to traditional two-dimensional grasping and placement tasks, three-dimensional operations require a more comprehensive and detailed understanding of the environment.

[0003] In related technologies, robots typically acquire multi-view images of their environment using multiple fixed cameras to achieve 3D reconstruction or visual information fusion, enabling high-precision perception and decision-making. However, the arrangement of fixed cameras in most current robot systems often suffers from overlapping views and information redundancy, resulting in the collection of a large amount of background information irrelevant to the task. This not only increases the computational burden on visual processing but may also introduce misleading noise, affecting the robot's accurate identification of the target and its task execution efficiency.

[0004] Therefore, how to improve the perception efficiency and operational accuracy of robots in three-dimensional manipulation tasks has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] This invention provides a robot control method and apparatus based on base model knowledge, which addresses the technical problem of how to improve the perception efficiency and operational accuracy of robots in three-dimensional operation tasks.

[0006] This invention provides a robot control method based on base model knowledge, comprising: The text and visual cues corresponding to the current task are input into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model; the text cues are determined based on the description information of the workspace of the current task; the visual cues are determined based on multi-view images captured by cameras with multiple fixed viewpoints. The three-dimensional point cloud corresponding to the multi-view image is projected onto the image plane corresponding to the virtual view to generate a virtual view image; The language instructions for the current task and the virtual viewpoint image are input into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model. Based on the spatial location information of the target key points, the robot's motion trajectory is generated, and the robot is controlled to perform the current task based on the motion trajectory.

[0007] In some embodiments, the step of inputting the text and visual cues corresponding to the current task into the multimodal base model to obtain the virtual perspective output by the multimodal base model includes: Based on the pose information of cameras from multiple fixed viewpoints, determine the environmental description information; Based on the pose definition information of the virtual camera corresponding to the virtual viewpoint, the task description information is determined; Based on the pose information of the virtual camera corresponding to the typical viewpoint of the workspace, determine the context example information; Based on the observation angle and observation sequence of the virtual camera over the workspace, constraint rule information is determined; Based on the environment description information, the task description information, the context example information, and the constraint rule information, generate text prompts corresponding to the current task; Based on the overall environmental image of the workspace and the multi-view image, a visual cue word corresponding to the current task is generated; the overall environmental image is used to annotate the coordinate axes of the workspace, as well as the pose information of the robot and the fixed-view camera in the workspace. The text prompts and visual prompts are input into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model.

[0008] In some embodiments, projecting the 3D point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual view to generate a virtual view image includes: The coordinates of the pixels in each of the multi-view images are transformed, and each image is projected onto the three-dimensional point cloud representation space to obtain the three-dimensional point cloud corresponding to each image. Based on the pose information of the virtual camera corresponding to the virtual viewpoint, the three-dimensional point cloud corresponding to each image is projected onto the image plane corresponding to the virtual viewpoint to obtain the virtual viewpoint image.

[0009] In some embodiments, inputting the language instructions of the current task and the virtual viewpoint image into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model includes: The language instructions for the current task, the virtual view image, the depth prediction token, and the robot's state information are input into the key point prediction model to obtain the spatial location information of the target key points output by the key point prediction model; the depth prediction token is used to capture the depth information of the target key points.

[0010] In some embodiments, the key point prediction model includes a multimodal feature fusion module and a key point prediction module; The step of inputting the language command of the current task, the virtual view image, the depth prediction token, and the robot's state information into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model includes: The language instructions for the current task, the virtual perspective image, the depth prediction token, and the robot's state information are input into the multimodal feature fusion module. The multimodal feature fusion module then fuses the text features of the language instructions, the image features of the virtual perspective image, and the text features of the state information based on a cross-modal attention mechanism to obtain multimodal fused features. The multimodal fusion features are input into the key point prediction module to obtain the spatial location information of the target key points output by the key point prediction module.

[0011] In some embodiments, inputting the multimodal fusion features into the keypoint prediction module to obtain the spatial location information of the target keypoints output by the keypoint prediction module includes: The multimodal fusion features are input into the key point prediction module to obtain the first key point location heatmap and the depth information of the target key point output by the key point prediction module; The local area where the key point is located in the heat map of the key point location is enlarged, and the enlarged heat map of the key point location is input into the key point prediction model again to obtain the second key point location heat map output by the key point prediction model. The planar position information of the target key point is determined based on the heat map of the second key point location; Based on the planar position information and depth information of the target key points, the spatial position information of the target key points is determined.

[0012] In some embodiments, the method further comprises: Based on the 3D point cloud corresponding to the multi-view images, determine the object boundary information and spatial obstacle information in the workspace of the current task; Based on the object boundary information and the spatial obstacle information, collision detection is performed on the robot's motion trajectory.

[0013] This invention provides a robot control device based on base model knowledge, comprising: The viewpoint determination unit is used to input the text prompts and visual prompts corresponding to the current task into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model; the text prompts are determined based on the description information of the workspace of the current task; the visual prompts are determined based on multi-view images captured by cameras with multiple fixed viewpoints. An image generation unit is used to project the three-dimensional point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual view to generate a virtual view image. The location prediction unit is used to input the language instructions of the current task and the virtual viewpoint image into the key point prediction model to obtain the spatial location information of the target key points output by the key point prediction model. The trajectory control unit is used to generate the robot's motion trajectory based on the spatial position information of the target key points, and to control the robot to perform the current task based on the motion trajectory.

[0014] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot control method based on pedestal model knowledge.

[0015] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the robot control method based on base model knowledge.

[0016] The robot control method and apparatus based on base model knowledge provided by this invention input text and visual cues corresponding to the current task into a multimodal base model to obtain a virtual perspective output by the multimodal base model. The text cues are determined based on the descriptive information of the workspace of the current task; the visual cues are determined based on multi-view images captured by cameras with multiple fixed perspectives. The 3D point cloud corresponding to the multi-view images is projected onto the image plane corresponding to the virtual perspective to generate a virtual perspective image. The language commands of the current task and the virtual perspective image are input into a keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model. Based on the spatial location information of the target keypoints, the robot's motion trajectory is generated, and the robot is controlled based on the motion trajectory. The system performs the current task effectively. Because the virtual viewpoint is predicted based on text and visual cues using a multimodal base model, and this virtual viewpoint is the optimal observation viewpoint, it effectively covers key parts of the current task's workspace, reducing redundant and interfering information. By utilizing both language commands and virtual viewpoint images, it fully leverages the task information provided by language commands and the visual information provided by virtual viewpoint images, improving the accuracy and robustness of keypoint prediction. This enhances the robot's perception efficiency and operational accuracy in 3D manipulation tasks. Furthermore, using a single virtual viewpoint image to guide policy learning significantly reduces the input size and computational burden of the keypoint prediction model during the training and inference phases. It is highly scalable, independent of specific camera configurations, and can be flexibly deployed in different robot systems. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the robot control method based on base model knowledge provided by this invention.

[0020] Figure 2 This is a schematic diagram of the workspace provided by the present invention.

[0021] Figure 3 This is the second flowchart of the robot control method based on base model knowledge provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the robot control device based on base model knowledge provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps, units, or modules is not necessarily limited to those explicitly listed, but may include other steps, units, or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0026] With the development of multimodal perception and multi-view learning methods, researchers have made a series of advances in improving the accuracy and robustness of 3D manipulation. Among these advancements, the choice of camera viewpoint is widely considered one of the core factors affecting robot manipulation performance. By selecting an appropriate observation viewpoint, robots can minimize occlusion, acquire key geometric features and semantic information, and thus make more accurate perception judgments and action planning. This viewpoint selection mechanism is essentially highly similar to the natural behavior of humans when dealing with 3D object interaction tasks; humans often actively adjust their observation position to obtain the most favorable viewpoint, thereby optimizing interaction strategies. However, the arrangement of fixed cameras in most current robot systems often suffers from overlapping views and information redundancy, resulting in a large amount of background information irrelevant to the task being collected. This not only increases the computational burden on the vision processing module but may also introduce misleading noise, affecting the system's accurate identification of the target and task execution efficiency.

[0027] To address the aforementioned issues, information fusion strategies such as voxelization and point cloud stitching can be employed to unify and integrate multi-view color (red, green, blue)-depth (RGB-D) images into a unified 3D representation, thereby enhancing environmental perception capabilities. While these methods offer certain advantages in 3D reconstruction and understanding, the resulting complete 3D representation inevitably contains a large number of redundant regions, leading to low efficiency in downstream policy learning and inference. Furthermore, the training cost of such methods is typically extremely high; for example, some models based on 3D point clouds require training on eight GPUs, which can take up to 16 days, severely limiting their deployability and scalability in real-world scenarios. To further improve the effectiveness of viewpoint selection, point cloud or RGB-D data can be projected onto several virtual camera planes specified by expert experience to obtain more information-density rendered images for specific tasks. These methods have achieved positive results in specific scenarios, but they are heavily reliant on expert knowledge, making them difficult to adapt to task changes or scene migrations, and lacking flexibility and generalization ability.

[0028] In order to address the shortcomings of related technologies, Figure 1 This is one of the flowcharts illustrating the robot control method based on base model knowledge provided by the present invention, such as... Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0029] Step 110: Input the text prompts and visual prompts corresponding to the current task into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model; the text prompts are determined based on the description information of the current task's workspace; the visual prompts are determined based on multi-view images captured by cameras with multiple fixed viewpoints.

[0030] Specifically, the robot control method provided in this embodiment of the invention is executed by a robot control device. This device can be implemented in software, such as a robot control program running on a computer; or it can be implemented in hardware, such as a computer or server that executes the robot control method.

[0031] The application scenario of the method provided in this embodiment of the invention is to control a robot to perform operational tasks in a set workspace. Figure 2 This is a schematic diagram of the workspace provided by the present invention, as shown below. Figure 2 As shown, the current task is an operation that the robot needs to perform in three-dimensional space, such as opening a drawer with a robotic arm. The workspace refers to the three-dimensional activity area where the robot performs the task. Multiple cameras with fixed viewing angles can be installed in the workspace to acquire images from multiple fixed perspectives, resulting in multi-view images. This embodiment of the invention does not specifically limit the location of the fixed viewing angles or the number of cameras. For example, four cameras can be set up, each acquiring images of a fixed area from different angles. The cameras can be RGB-D cameras, simultaneously acquiring color images and depth information.

[0032] Understandably, images captured from fixed viewpoints are prone to occlusion due to the fixed perspective, which affects the accuracy of the robot's task execution. A virtual viewpoint can be constructed to address this. A virtual viewpoint represents the optimal perspective for observing the robot's task under ideal conditions. From a virtual viewpoint, the details of the robot's current task can be observed to the maximum extent.

[0033] Multimodal foundation models are pre-trained large-scale multimodal models capable of processing and generating data across multiple modalities, such as text, images, and audio. These models, pre-trained on large-scale multimodal data, learn general features and patterns, possessing world knowledge and visual reasoning capabilities, assisting robots in automatically selecting the optimal viewpoint from candidate virtual camera poses. Multimodal foundation models can be selected from large-scale multimodal models in relevant technologies.

[0034] Prompts are text, images, or audio inputs to a model that guide it to generate specific outputs. The purpose of prompts is to provide the model with contextual information and task requirements, helping it understand the user's needs and generate appropriate text or perform corresponding actions.

[0035] In this embodiment of the invention, the prompts for the multimodal base model include text prompts and visual prompts.

[0036] Text prompts are determined based on descriptive information about the current task's workspace. This descriptive information typically includes the task's objective, the characteristics of the objects being manipulated, and the workspace layout. The purpose of text prompts is to provide the model with clear task context and objectives, helping the model better understand the task requirements.

[0037] Visual cues are determined based on multi-view images captured by cameras with multiple fixed viewpoints. These images provide multiple perspectives of the target object or workspace, helping the model understand the shape, position, and state of the target object from different angles. The role of visual cues is to provide the model with rich visual information, enabling the model to more comprehensively understand task-related objects and the environment.

[0038] Step 120: Project the 3D point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual view to generate the virtual view image.

[0039] Specifically, the coordinates of pixels in each visual image can be transformed to the same 3D point cloud representation space (e.g., world coordinate system), resulting in 3D point clouds corresponding to each viewpoint image. The 3D point clouds from different viewpoint images are then fused together to accurately represent the 3D workspace where the robot operates.

[0040] A virtual viewpoint can be understood as the observation perspective of a virtual camera. This is typically achieved by defining the intrinsic parameters (focal length, optical center, etc.) and extrinsic parameters (position, orientation) of the virtual camera. Based on these intrinsic and extrinsic parameters, a 3D point cloud can be projected onto the image plane corresponding to the virtual viewpoint to generate a virtual viewpoint image.

[0041] Step 130: Input the language instructions and virtual viewpoint images of the current task into the key point prediction model to obtain the spatial location information of the target key points output by the key point prediction model.

[0042] Specifically, the language instructions for the current task are used to describe the content of the current task. When the robot performs an operation task, its movement trajectory is displayed in the virtual view image.

[0043] Key target points refer to locations that are crucial for describing and understanding the robot's motion. These key target points provide important information about the robot's motion, including its start, end, turning points, and velocity changes. Spatial location information refers to the spatial coordinates of these key target points.

[0044] By constructing a keypoint prediction model, the spatial location information of target keypoints in a motion trajectory can be predicted. This allows for full utilization of the task context provided by language commands and the visual information provided by virtual perspective images, thereby improving the accuracy and robustness of keypoint prediction. The keypoint prediction model can be constructed based on neural network models in related technologies.

[0045] Step 140: Based on the spatial location information of the target key points, generate the robot's motion trajectory, and control the robot to perform the current task based on the motion trajectory.

[0046] Specifically, based on the spatial location information of key target points, the robot's motion trajectory can be generated, and the generated motion trajectory can control the robot to perform the current task.

[0047] The robot control method based on base model knowledge provided in this invention involves inputting text and visual cues corresponding to the current task into a multimodal base model to obtain a virtual perspective output by the multimodal base model. The text cues are determined based on the description information of the current task's workspace; the visual cues are determined based on multi-view images captured by cameras with multiple fixed perspectives. The 3D point cloud corresponding to the multi-view images is projected onto the image plane corresponding to the virtual perspective to generate a virtual perspective image. The language commands for the current task and the virtual perspective image are input into a keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model. Based on the spatial location information of the target keypoints, the robot's motion trajectory is generated, and the robot is controlled based on the motion trajectory. The system performs the current task effectively. Because the virtual viewpoint is predicted based on text and visual cues using a multimodal base model, and this virtual viewpoint is the optimal observation viewpoint, it effectively covers key parts of the current task's workspace, reducing redundant and interfering information. By utilizing both language commands and virtual viewpoint images, it fully leverages the task information provided by language commands and the visual information provided by virtual viewpoint images, improving the accuracy and robustness of keypoint prediction. This enhances the robot's perception efficiency and operational accuracy in 3D manipulation tasks. Furthermore, using a single virtual viewpoint image to guide policy learning significantly reduces the input size and computational burden of the keypoint prediction model during the training and inference phases. It is highly scalable, independent of specific camera configurations, and can be flexibly deployed in different robot systems.

[0048] It should be noted that each embodiment of the present invention can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0049] In some embodiments, text and visual cues corresponding to the current task are input into a multimodal base model to obtain a virtual perspective output by the multimodal base model, including: Based on the pose information of cameras from multiple fixed viewpoints, determine the environmental description information; Based on the pose definition information of the virtual camera corresponding to the virtual viewpoint, the task description information is determined; Context example information is determined based on the pose information of the virtual camera corresponding to the typical viewpoint of the workspace. Based on the observation perspective and order of the virtual camera in the workspace, the constraint rule information is determined; Based on environmental description information, task description information, contextual example information, and constraint rule information, generate text prompts for the current task. Based on the overall environment image of the workspace and multi-view images, visual cues corresponding to the current task are generated; the overall environment image is used to label the coordinate axes of the workspace, as well as the pose information of the robot and camera in the workspace. Input textual and visual cues into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model.

[0050] Specifically, text prompts can include four parts: environment description information, task description information, contextual example information, and constraint rule information.

[0051] Environmental description information is used to describe the pose information of cameras with multiple fixed viewpoints, which can clarify the position of the reference origin in the workspace and the direction of the coordinate axes in the image.

[0052] The task description information is used to describe the pose definition information of the virtual camera corresponding to the virtual viewpoint. For example, the pose of the virtual camera can be defined using functions of the PyTorch3D library, and modeled based on two key parameters: the angle between the virtual camera and the horizontal plane, and the angle between the projection of the direction vector onto the reference plane and the reference direction.

[0053] Contextual example information describes the pose information of the virtual camera corresponding to typical viewpoints in the workspace. Typical viewpoints include directly in front, directly above, and directly to the right. The viewpoint of a camera with a fixed viewpoint is not necessarily a typical viewpoint; therefore, it is necessary to construct virtual cameras with typical viewpoints to obtain images from these viewpoints. This can help the multimodal pedestal model more accurately understand the task objective and spatial structure.

[0054] The constraint rules describe the virtual camera's observation perspective and order within the workspace. Four constraint rules can be defined to guide the multimodal base model in outputting reasonable results. The first rule guides the model to prioritize the camera's viewpoint aligned with the coordinate axes as the observation perspective. The second rule emphasizes that judgments must be based on visual cues. The third and fourth rules restrict the virtual camera's observation order within the workspace to top-down and standardize the output format.

[0055] Visual cues can include two parts: an overall environment image and multi-view images. The overall environment image is an image acquired from the entire scope of the workspace. Visual labeling mechanisms from multimodal research can be used to explicitly label the origin and coordinate axes of the workspace, as well as the pose information of the robot and multiple fixed-view cameras within the workspace, in the overall environment image. This helps the base model more effectively combine existing knowledge to make reasonable inferences and viewpoint judgments.

[0056] The robot control method based on base model knowledge provided in this invention can help the base model to more effectively combine existing knowledge to make reasonable reasoning and perspective judgment by reasonably setting text prompts and visual prompts, and output accurate virtual perspective.

[0057] In some embodiments, projecting the 3D point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual viewpoint to generate a virtual viewpoint image includes: The coordinates of the pixels in each image in the multi-view image are transformed, and each image is projected onto the three-dimensional point cloud representation space to obtain the three-dimensional point cloud corresponding to each image. Based on the pose information of the virtual camera corresponding to the virtual viewpoint, the 3D point cloud corresponding to each image is projected onto the image plane corresponding to the virtual viewpoint to obtain the virtual viewpoint image.

[0058] Specifically, multi-view images are color images with depth information captured by an RGB-D camera. The coordinates of pixels in each image can be transformed, and each image can be projected onto a 3D point cloud representation space to obtain the corresponding 3D point cloud, which can be expressed by the formula: in, These are the planar coordinates of pixels in the image; This refers to the depth information of pixels in the image; The spatial coordinates of a pixel after it has been converted to a 3D point cloud representation space; This is the intrinsic parameter matrix of the camera; This is the extrinsic parameter matrix of the camera. The intrinsic parameter matrix contains... and The focal length of the camera; and The coordinates of the optical center; Represents the rotation matrix; This represents the translation vector.

[0059] Based on the pose information of the virtual camera corresponding to the virtual viewpoint, the intrinsic and extrinsic parameter matrices of the virtual camera can be calculated. Using the same projection transformation formula as above, the 3D point cloud corresponding to each image is projected onto the image plane corresponding to the virtual viewpoint, thereby generating a virtual viewpoint image under that viewpoint, which is used for subsequent feature extraction and task decision-making.

[0060] The robot control method based on base model knowledge provided in this invention generates accurate virtual view images by projecting the three-dimensional point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual view. This effectively covers the key parts of the current task's workspace and reduces redundant interference information.

[0061] In some embodiments, the language instructions for the current task and the virtual viewpoint image are input into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model, including: The language instructions for the current task, virtual view images, depth prediction tokens, and robot state information are input into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model; the depth prediction token is used to capture the depth information of the target keypoints.

[0062] Specifically, a keypoint prediction model can be built using a neural network as the initial model. The neural network can be a convolutional neural network, a recurrent neural network, a generative adversarial network, or an attention mechanism network (such as the Transformer model), etc.

[0063] The language command for the current task is a voice-based instruction given to control the robot to perform the current task. Depth prediction tokens are a set of learnable prediction tokens used to capture depth information of key points on the target. The robot's state information refers to its position, posture, velocity, acceleration, joint angles, joint torques, etc. State information is crucial for the robot's control, planning, and decision-making.

[0064] It can collect language instructions from historical tasks, virtual perspective images, depth prediction tokens, and robot state information as samples, and use the actual spatial positions of target key points in the robot's motion trajectory in historical tasks as labels to train the initial model, thereby improving the feature learning ability of the initial model and its ability to predict the spatial positions of target key points, and finally obtaining a key point prediction model.

[0065] The robot control method based on pedestal model knowledge provided in this invention processes the language instructions, virtual view images, depth prediction tokens and robot state information of the current task through a key point prediction model, predicts the spatial position information of target key points, realizes deep coupling between language instructions and image information, and improves the accuracy of key point spatial position prediction.

[0066] In some embodiments, the keypoint prediction model includes a multimodal feature fusion module and a keypoint prediction module; Input the current task's language commands, virtual viewpoint images, depth prediction tokens, and robot state information into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model, including: The language instructions for the current task, virtual viewpoint images, depth prediction tokens, and robot state information are input into the multimodal feature fusion module. The multimodal feature fusion module fuses the text features of the language instructions, the image features of the virtual viewpoint images, and the text features of the state information based on a cross-modal attention mechanism to obtain multimodal fused features. The multimodal fusion features are input into the keypoint prediction module to obtain the spatial location information of the target keypoints output by the keypoint prediction module.

[0067] Specifically, structurally, the key point prediction model can include a multimodal feature fusion module and a key point prediction module.

[0068] Textual features of language instructions can be extracted. For example, for user-input task-oriented language instructions, the ResNet-50 text encoder in the Contrastive Language-Image Pre-training (CLIP) model can be used to transform the instruction text into text feature vectors with semantic information. Image features of virtual perspective images can be extracted. For example, a randomly initialized convolutional neural network can be used to encode the image to obtain an intermediate visual representation that characterizes the spatial structure and object information in the image. In all of the above processes, it is necessary to ensure that textual features and image features have good alignment within the same representation space.

[0069] The text features of the current task's language instructions, the image features of the virtual perspective image, the depth prediction token, and the robot's state information are input into the multimodal feature fusion module. The multimodal feature fusion module first performs self-attention processing on the image features to enhance the feature's ability to model the relationship between local and global spaces. Second, a cross-modal attention mechanism is introduced, enabling the model to focus on regions in the image that are highly relevant to the task objective based on the semantic information in the text features. The text features of the language instructions, the image features of the virtual perspective image, and the text features of the state information are fused to obtain multimodal fused features, providing an accurate and semantically consistent representation for subsequent trajectory key point prediction.

[0070] In the above process, the depth prediction token captures the depth information of the current task target in space through cross-modal attention interaction with image and text features. Specifically, the model decouples and extracts the positional information of target key points in 3D space from images and language through multi-layered interactive operations, ultimately achieving accurate prediction of the depth information of target key points.

[0071] Finally, the multimodal fusion features are input into the keypoint prediction module to obtain the spatial location information of the target keypoints output by the keypoint prediction module.

[0072] The robot control method based on base model knowledge provided in this invention fuses the text features of language instructions, the image features of virtual perspective images, and the text features of state information based on a cross-modal attention mechanism to obtain multimodal fusion features. This enables a joint understanding of the semantics of the task and its position in the current environment, improving the perception efficiency and decision-making accuracy of the robot when performing operational tasks in three-dimensional space.

[0073] In some embodiments, multimodal fusion features are input into the keypoint prediction module to obtain the spatial location information of the target keypoints output by the keypoint prediction module, including: The multimodal fusion features are input into the key point prediction module to obtain the first key point location heatmap and the depth information of the target key point output by the key point prediction module; The local area where the key point is located in the key point location heatmap is enlarged, and the enlarged key point location heatmap is input again into the key point prediction model to obtain the second key point location heatmap output by the key point prediction model. Determine the planar location information of the target key points based on the heat map of the second key point location; Based on the planar position information and depth information of the target key points, the spatial position information of the target key points is determined.

[0074] Specifically, multimodal fusion features can be input into the keypoint prediction module to obtain a keypoint location heatmap and depth information of the target keypoints (obtained through a depth prediction token). The keypoint location heatmap shows the distribution of target keypoints in a specific area and can be used to represent the planar location information of the target keypoints. To improve the accuracy of target keypoint localization, this embodiment of the invention employs a two-stage keypoint heatmap prediction method.

[0075] First, the multimodal fusion features are input into the keypoint prediction module to obtain the first keypoint location heatmap and the depth information of the target keypoints. The first keypoint location heatmap can coarsely represent the approximate location area of ​​the task target, which is used to guide the focus range of subsequent fine prediction.

[0076] Secondly, the local area where the key point is located in the key point location heatmap is magnified, and the magnified key point location heatmap is input again into the key point prediction model to obtain the second key point location heatmap output by the key point prediction model. The second key point location heatmap can represent the location area of ​​the task target with fine granularity, thereby achieving high-precision key point localization.

[0077] Finally, the planar position information of the target key point is determined based on the heat map of the second key point location. By fusing the planar position information and depth information of the target key point, the spatial position information of the target key point can be obtained.

[0078] In some cases, the planar position information of the target key points in the first key point location heatmap is accurate enough, and there is no need for further prediction. To adapt to the varying complexity of different tasks and their stages, this embodiment of the invention introduces a dynamic two-stage prediction mechanism. By setting a dynamic predictor, based on the spatial distance between the key points in the coarse-grained prediction result (first key point location heatmap) and the fine-grained prediction result (second key point location heatmap) during training, it adaptively determines whether to adopt a two-stage structure: when the predicted positions of the two are close, it indicates that the current task difficulty is low, and the fine-grained stage can be skipped, directly outputting the coarse-grained prediction result (determining the spatial position information of the target key points based on the first key point location heatmap and depth information), thereby effectively reducing inference time and improving the overall system execution efficiency; when the task is more complex, the two-stage process is retained to ensure positioning accuracy and achieve a dynamic balance between speed and performance.

[0079] The robot control method based on base model knowledge provided in this invention improves the accuracy of positioning by using a two-stage key point heatmap prediction method to input multimodal fusion features into the key point prediction module and obtain the spatial location information of the target key points output by the key point prediction module.

[0080] In some embodiments, the robot's motion trajectory is generated based on the spatial location information of key target points, and the robot is controlled to perform the current task based on the motion trajectory. This can be achieved through forward kinematics modeling and inverse kinematics solving.

[0081] By using forward kinematics modeling, the joint angles of the robot's robotic arm can be represented. Mapped to the pose of the end effector ,Right now . The number of joints. For the first The joint angles of each joint. Let T be the space of a Lie group. T is the transpose operator.

[0082] in, This represents the kinematic chain transformation established based on the Denavit-Hartenberg (DH) parameters of the robotic arm, with the output being a homogeneous transformation matrix consisting of displacement and translation matrices. .

[0083] When the spatial position of the target key point in the motion trajectory and orientation Given a given value, the joint angle solution can be obtained by solving inverse kinematics. : .in, Let T be the space of Lie algebras. Let T be the transpose operator.

[0084] To improve solution efficiency and accuracy, this invention employs numerical optimization methods (such as gradient descent and Newton's method) combined with redundancy analysis to address the problems of multiple solutions and singular poses. When multiple feasible solutions exist, the solution that satisfies both motion continuity and energy minimization is preferentially selected.

[0085] In some embodiments, the method further includes: Based on the 3D point cloud corresponding to multi-view images, determine the object boundary information and spatial obstacle information in the workspace of the current task; Collision detection is performed on the robot's motion trajectory based on object boundary information and spatial obstacle information.

[0086] Specifically, after transforming multi-view images to obtain corresponding 3D point clouds, preprocessing (such as filtering, boundary culling, and resampling) can be performed to acquire accurate object boundary information and spatial obstacle information. Object boundary information refers to the boundary position and shape of an object in 3D space. Spatial obstacle information refers to the position and shape of obstacles existing in the environment.

[0087] Based on object boundary information and spatial obstacle information, collision detection is performed on the robot's motion trajectory to optimize the trajectory.

[0088] The robot control method based on base model knowledge provided in this invention obtains object boundary information and spatial obstacle information from the 3D point cloud corresponding to multi-view images, performs collision detection on the motion trajectory, realizes effective perception of scene information and dynamic adjustment of motion trajectory, provides high-precision and high-reliability path control guarantee for 3D operation tasks, and further improves the intelligence level and practicality of the overall system.

[0089] In some embodiments, a continuous trajectory curve can be constructed between the target key points using a trajectory interpolation method based on the obtained spatial locations of the target key points. Cubic spline interpolation can be used to ensure that the path has smoothness and continuity in the dimensions of position, velocity, and acceleration. The generated trajectory takes the following form: in, The joint angle vector corresponding to each joint point Total execution time For execution time.

[0090] In some embodiments, the generated motion trajectory can be encapsulated as control commands and sent to the control interface of the robotic arm via the Robot Operating System (ROS) control package. The ROS control system is responsible for handling real-time tracking and execution of the trajectory, including time synchronization, speed control, and dynamic constraint monitoring, ensuring that the robot end effector moves along the predetermined trajectory during task execution and has a certain degree of robustness and compensation capability in the presence of slight disturbances or external changes.

[0091] The robot control method based on base model knowledge provided in this invention effectively improves the robot's perception efficiency, operation accuracy, and system response speed in three-dimensional operation tasks, and exhibits good generalization ability and execution robustness in complex operation environments.

[0092] Based on the above embodiments, Figure 3 This is the second flowchart illustrating the robot control method based on base model knowledge provided by this invention, as shown below. Figure 3 As shown, the method mainly includes: Step 310: Viewpoint selection driven by the base model.

[0093] By using pre-designed text prompts and multi-view images, which are input into a multimodal base model, the model infers and selects the optimal virtual viewpoint for the current 3D operation task to enhance the perception of task-related information.

[0094] Step 320: Virtual perspective image generation.

[0095] Images acquired by multiple fixed RGB-D cameras are projected into a unified 3D point cloud representation space, and their extrinsic parameters are determined according to the selected virtual camera viewpoint. Then, the 3D point cloud is projected onto that viewpoint to generate the corresponding virtual viewpoint image.

[0096] Step 330: Fusion of visual features and language instructions.

[0097] Visual features of the virtual perspective image are extracted and fused with the user-input language commands via a keypoint prediction model (Transformer structure) to achieve a joint understanding of the task semantics and its position in the current visual environment.

[0098] Step 340: Prediction of key points of the two-stage robot trajectory.

[0099] Based on a coarse-to-fine granular neural network structure, coarse-grained prediction of the location of key points in the trajectory is first performed, and then the image is magnified locally in this area and fine-grained prediction is performed to improve the positioning accuracy in fine manipulation tasks.

[0100] Step 350: Robot control execution.

[0101] Based on the predicted trajectory key point positions, and combined with the forward and inverse kinematics models of the seven-degree-of-freedom robotic arm, the robotic arm is calculated and controlled to move to the target position to complete the operation task.

[0102] Step 360, Environmental Perception and Trajectory Planning System An RGB-D camera is used to acquire real-time color and depth images of the workspace, which are then converted into a dense 3D point cloud representation. By preprocessing the depth map (such as filtering, boundary culling, and resampling), accurate object boundary and spatial obstacle information is obtained, further assisting the trajectory planning module in collision detection and path optimization.

[0103] The apparatus provided in the embodiments of the present invention will be described below. The apparatus described below can be referred to in correspondence with the method described above.

[0104] Figure 4 This is a schematic diagram of the robot control device based on base model knowledge provided by the present invention, as shown below. Figure 4 As shown, the device includes: The viewpoint determination unit 410 is used to input the text prompts and visual prompts corresponding to the current task into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model; the text prompts are determined based on the description information of the workspace of the current task; the visual prompts are determined based on multi-view images captured by cameras with multiple fixed viewpoints. The image generation unit 420 is used to project the three-dimensional point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual view to generate the virtual view image. The location prediction unit 430 is used to input the language instructions and virtual viewpoint images of the current task into the key point prediction model to obtain the spatial location information of the target key points output by the key point prediction model. The trajectory control unit 440 is used to generate the robot's motion trajectory based on the spatial position information of the target key points, and to control the robot to perform the current task based on the motion trajectory.

[0105] The robot control device based on a base model knowledge provided in this invention inputs text and visual cues corresponding to the current task into a multimodal base model to obtain a virtual perspective output by the multimodal base model. The text cues are determined based on the description information of the workspace of the current task; the visual cues are determined based on multi-view images captured by cameras with multiple fixed perspectives. The 3D point cloud corresponding to the multi-view images is projected onto the image plane corresponding to the virtual perspective to generate a virtual perspective image. The language commands for the current task and the virtual perspective image are input into a keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model. Based on the spatial location information of the target keypoints, the robot's motion trajectory is generated, and the robot is controlled based on the motion trajectory. The system performs the current task effectively. Because the virtual viewpoint is predicted based on text and visual cues using a multimodal base model, and this virtual viewpoint is the optimal observation viewpoint, it effectively covers key parts of the current task's workspace, reducing redundant and interfering information. By utilizing both language commands and virtual viewpoint images, it fully leverages the task information provided by language commands and the visual information provided by virtual viewpoint images, improving the accuracy and robustness of keypoint prediction. This enhances the robot's perception efficiency and operational accuracy in 3D manipulation tasks. Furthermore, using a single virtual viewpoint image to guide policy learning significantly reduces the input size and computational burden of the keypoint prediction model during the training and inference phases. It is highly scalable, independent of specific camera configurations, and can be flexibly deployed in different robot systems.

[0106] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communications bus 540. The processor 510 can call logical commands stored in the memory 530 to execute the methods described in the above embodiments, for example: The text and visual cues corresponding to the current task are input into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model. The text cues are determined based on the description information of the current task's workspace. The visual cues are determined based on multi-view images captured by cameras with multiple fixed viewpoints. The 3D point cloud corresponding to the multi-view images is projected onto the image plane corresponding to the virtual viewpoint to generate the virtual viewpoint image. The language command of the current task and the virtual viewpoint image are input into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model. Based on the spatial location information of the target keypoints, the robot's motion trajectory is generated, and the robot is controlled to execute the current task based on the motion trajectory.

[0107] Furthermore, when the logical commands in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The processor in the electronic device provided in this embodiment of the invention can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0109] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0110] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.

[0111] This invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot control method based on base model knowledge, characterized in that, include: Input the text and visual cues corresponding to the current task into the multimodal base model to obtain the virtual perspective output by the multimodal base model; The text prompts are determined based on the description information of the workspace of the current task; the visual prompts are determined based on multi-view images captured by cameras with multiple fixed viewpoints. The three-dimensional point cloud corresponding to the multi-view image is projected onto the image plane corresponding to the virtual view to generate a virtual view image; The language instructions for the current task and the virtual viewpoint image are input into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model. Based on the spatial location information of the target key points, the robot's motion trajectory is generated, and the robot is controlled to perform the current task based on the motion trajectory.

2. The robot control method based on base model knowledge according to claim 1, characterized in that, The step of inputting the text and visual cues corresponding to the current task into the multimodal base model to obtain the virtual perspective output by the multimodal base model includes: Based on the pose information of cameras from multiple fixed viewpoints, determine the environmental description information; Based on the pose definition information of the virtual camera corresponding to the virtual viewpoint, the task description information is determined; Based on the pose information of the virtual camera corresponding to the typical viewpoint of the workspace, determine the context example information; Based on the observation angle and observation sequence of the virtual camera over the workspace, constraint rule information is determined; Based on the environment description information, the task description information, the context example information, and the constraint rule information, generate text prompts corresponding to the current task; Based on the overall environmental image of the workspace and the multi-view image, a visual cue word corresponding to the current task is generated; the overall environmental image is used to annotate the coordinate axes of the workspace, as well as the pose information of the robot and the fixed-view camera in the workspace. The text prompts and visual prompts are input into the multimodal base model to obtain the virtual viewpoint output by the multimodal base model.

3. The robot control method based on base model knowledge according to claim 1, characterized in that, The step of projecting the 3D point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual view to generate a virtual view image includes: The coordinates of the pixels in each of the multi-view images are transformed, and each image is projected onto the three-dimensional point cloud representation space to obtain the three-dimensional point cloud corresponding to each image. Based on the pose information of the virtual camera corresponding to the virtual viewpoint, the three-dimensional point cloud corresponding to each image is projected onto the image plane corresponding to the virtual viewpoint to obtain the virtual viewpoint image.

4. The robot control method based on base model knowledge according to claim 1, characterized in that, The step of inputting the language command of the current task and the virtual viewpoint image into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model includes: The language instructions for the current task, the virtual view image, the depth prediction token, and the robot's state information are input into the key point prediction model to obtain the spatial location information of the target key points output by the key point prediction model; the depth prediction token is used to capture the depth information of the target key points.

5. The robot control method based on base model knowledge according to claim 4, characterized in that, The key point prediction model includes a multimodal feature fusion module and a key point prediction module; The step of inputting the language command of the current task, the virtual view image, the depth prediction token, and the robot's state information into the keypoint prediction model to obtain the spatial location information of the target keypoints output by the keypoint prediction model includes: The language instructions for the current task, the virtual perspective image, the depth prediction token, and the robot's state information are input into the multimodal feature fusion module. The multimodal feature fusion module then fuses the text features of the language instructions, the image features of the virtual perspective image, and the text features of the state information based on a cross-modal attention mechanism to obtain multimodal fused features. The multimodal fusion features are input into the key point prediction module to obtain the spatial location information of the target key points output by the key point prediction module.

6. The robot control method based on base model knowledge according to claim 5, characterized in that, The step of inputting the multimodal fusion features into the keypoint prediction module to obtain the spatial location information of the target keypoints output by the keypoint prediction module includes: The multimodal fusion features are input into the key point prediction module to obtain the first key point location heatmap and the depth information of the target key point output by the key point prediction module; The local area where the key point is located in the heat map of the key point location is enlarged, and the enlarged heat map of the key point location is input into the key point prediction model again to obtain the second key point location heat map output by the key point prediction model. The planar position information of the target key point is determined based on the heat map of the second key point location; Based on the planar position information and depth information of the target key points, the spatial position information of the target key points is determined.

7. The robot control method based on base model knowledge according to any one of claims 1 to 6, characterized in that, The method further includes: Based on the 3D point cloud corresponding to the multi-view images, determine the object boundary information and spatial obstacle information in the workspace of the current task; Based on the object boundary information and the spatial obstacle information, collision detection is performed on the robot's motion trajectory.

8. A robot control device based on base model knowledge, characterized in that, include: The perspective determination unit is used to input the text prompts and visual prompts corresponding to the current task into the multimodal base model to obtain the virtual perspective output by the multimodal base model. The text prompts are determined based on the description information of the workspace of the current task; the visual prompts are determined based on multi-view images captured by cameras with multiple fixed viewpoints. An image generation unit is used to project the three-dimensional point cloud corresponding to the multi-view image onto the image plane corresponding to the virtual view to generate a virtual view image. The location prediction unit is used to input the language instructions of the current task and the virtual viewpoint image into the key point prediction model to obtain the spatial location information of the target key points output by the key point prediction model. The trajectory control unit is used to generate the robot's motion trajectory based on the spatial position information of the target key points, and to control the robot to perform the current task based on the motion trajectory.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot control method based on base model knowledge as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot control method based on the knowledge of the base model as described in any one of claims 1 to 7.

Images