A multimodal embodied intelligent control method and device and electronic equipment

Abstract

The present disclosure provides a multi-modal embodied intelligent control method and device, and electronic equipment. A plurality of multi-view images of a robot are obtained, as well as task text. Two-dimensional trajectory supervision information and key point supervision information are generated based on the two, and are converted into a visual supervision graph. Key point features are generated from the supervision graph, and trajectory features are generated in combination with the multi-view images. Depth images, optical flow information, matched segmentation masks, and actuator postures and motion frequencies are extracted to generate corresponding features. Each feature is input into a multi-modal fusion inference model to generate relative action prediction features and convert them into action control instructions. The actuator completes the task. The method introduces trajectory supervision, key point supervision, and multi-modal prior under the visual-language condition, significantly improving the perception integrity, robustness, and execution accuracy.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and robot control technology, and more specifically, to a multimodal embodied intelligent control method, device, and electronic device. Background Technology

[0002] Embodied intelligence aims to enable machines to autonomously interact with their real-world environments through a closed loop of perception, understanding, and action. In the field of robot control, the fusion of multimodal inputs (such as vision, language, depth, and state sensing) is one of the core directions for improving execution accuracy and task generalization capabilities.

[0003] Existing multimodal robot control methods typically input images and task text directly into the control policy model, relying solely on the final executed action as a supervision signal during training. This makes it difficult for the model to learn an effective mapping from high-dimensional perceptual information to the action space, resulting in a lack of controllability and transparency in action planning. In tasks requiring millimeter-level precision, such as grasping, insertion / removal, and assembly, the model is susceptible to accumulated errors, leading to a high failure rate. Furthermore, its generalization and transfer capabilities are limited; model performance significantly degrades when training data is insufficient or the execution environment changes (lighting, occlusion, multi-view differences, etc.). Summary of the Invention

[0004] This disclosure provides at least one multimodal embodied intelligence control method, device, and electronic device. The embodied intelligence method, which introduces trajectory supervision, key point supervision, and multimodal prior features under vision-language conditions, can more stably learn the mapping relationship from multimodal perception to fine movements, significantly improve the integrity and robustness of task perception, and has higher execution accuracy.

[0005] This disclosure provides a multimodal embodied intelligent control method, including:

[0006] Acquire multi-view images taken by cameras at multiple locations on the robot, along with task text;

[0007] Based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated and converted into a visual supervision map;

[0008] Based on the visualization supervision map, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated by combining the multi-view images.

[0009] Determine the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view image, as well as the pose information and motion frequency information corresponding to the actuator, and generate the corresponding depth features, optical flow features, segmentation features, pose features, and frequency features respectively.

[0010] The trajectory features, key point features, depth features, optical flow features, segmentation features, pose features, and frequency features are input into a multimodal fusion inference model to generate relative action prediction features and convert them into action control commands.

[0011] The robot's actuators are controlled to complete preset tasks according to the motion control instructions.

[0012] In one optional implementation, based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated, specifically including:

[0013] The image information from the robot's left arm, right arm, and head camera in the multi-view image is input into the image encoder and linearly projected to obtain the corresponding image visual features.

[0014] The task text is input into a text encoder to obtain task text features, and multimodal fusion is performed on the image visual features and the task text features to output the two-dimensional trajectory supervision information and the key point supervision information of the text modality.

[0015] Specifically, the two-dimensional coordinates and execution state of the robot's end effector in a preset viewpoint image coordinate system are extracted, and the trajectory sequence information of the text modality is generated as the two-dimensional trajectory supervision information;

[0016] The entity words of the manipulated object are parsed from the task text, and the key point positions of the manipulated object and the robot end effector are predicted. The key point coordinates and key point names of the text modality are generated as the key point supervision information.

[0017] In one optional implementation, the two-dimensional trajectory supervision information and the key point supervision information are converted into a visual supervision map, specifically including:

[0018] Based on the two-dimensional trajectory supervision information, a trajectory supervision image is generated, which is based on the robot head camera view and includes the two-dimensional trajectory path and the corresponding actuator state of the robot end effector.

[0019] Based on the key point supervision information, a key point supervision image is generated, which is based on the view of the robot's head camera and includes the key point positions of the manipulated object and the robot's end effector.

[0020] In one optional implementation, for the trajectory supervision image, the starting point, path, and ending point in the two-dimensional trajectory path are encoded with different colors, and the action nodes corresponding to the robot end effector are marked in the two-dimensional trajectory path using a preset pattern, and different states of the actuator are distinguished by color.

[0021] For the key point supervision image, based on the key point position, the key points corresponding to the manipulated object and the robot end effector are marked with the preset pattern and color-coded.

[0022] In one optional implementation, based on the visualized supervision map, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated in combination with the multi-view images, specifically including:

[0023] The key point supervision image is convolved and linearly projected to extract the key point features;

[0024] The trajectory supervision image is multiplied by a preset first coefficient, and the image information captured by the robot's head camera is multiplied by a preset second coefficient. Then, different levels of transparency are used to fuse the images to generate a global trajectory running image from the robot's head perspective.

[0025] The trajectory features are generated by performing convolution and linear projection on the global trajectory image.

[0026] In one optional implementation, the optical flow feature is generated based on the following steps:

[0027] For each viewpoint camera image, extract the starting camera image and the ending camera image of the trajectory sequence under that viewpoint, and determine the optical flow information between the starting camera image and the ending camera image under that viewpoint;

[0028] For the optical flow information under each viewpoint, reprojection is performed to extract projection features. After performing cross-attention operation on the projection features under the robot's left and right arm viewpoints and the projection features under the robot's head viewpoint, the resulting features of the cross-attention operation are concatenated to determine the optical flow features.

[0029] In one alternative implementation, model training is performed based on the following steps:

[0030] Acquire sample operation video data of the robot actuator, and configure corresponding text for the execution actions corresponding to the sample operation video data;

[0031] The motion posture of the robot actuator is labeled as a preset requirement label and a key point label, and an image feature extraction model corresponding to the multi-view image is trained based on the preset requirement label and the key point label;

[0032] After the image feature extraction model is trained, the image feature extraction model is frozen, and the multimodal fusion inference model is trained so that the multimodal fusion inference model has the ability to output relative action prediction features.

[0033] This disclosure also provides a multimodal embodied intelligent control device, including:

[0034] The data acquisition module is used to acquire multi-view images taken by cameras at multiple locations on the robot, as well as task text;

[0035] The trajectory and key point supervision module is used to generate two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view image, and convert them into a visual supervision map;

[0036] The first feature extraction module is used to generate key point features corresponding to the key point supervision information based on the visualization supervision map, and to generate corresponding trajectory features and key point features in combination with the multi-view image.

[0037] The second feature extraction module is used to determine the depth image, optical flow information, segmentation mask matching the task text, and pose information and motion frequency information corresponding to the multi-view image, and generate corresponding depth features, optical flow features, segmentation features, pose features and frequency features respectively.

[0038] The motion prediction module is used to input the trajectory features, key point features, depth features, optical flow features, segmentation features, pose features and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands.

[0039] The control execution module is used to control the robot's actuator to complete the preset task according to the action control instructions.

[0040] This disclosure also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the above-described multimodal embodied intelligent control method, or any possible implementation of the above-described multimodal embodied intelligent control method.

[0041] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described multimodal embodied intelligent control method or any possible implementation thereof.

[0042] This disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described multimodal embodied intelligent control method, or any possible implementation of the above-described multimodal embodied intelligent control method.

[0043] This disclosure provides a multimodal embodied intelligent control method, device, and electronic device. The method acquires multi-view images captured by cameras at multiple locations on a robot, along with task text. Based on the task text and the multi-view images, it generates two-dimensional trajectory supervision information and keypoint supervision information, converting them into a visual supervision map. According to the visual supervision map, it generates keypoint features corresponding to the keypoint supervision information, and combines this with the multi-view images to generate corresponding trajectory features and keypoint features. It determines the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view images, as well as the posture information and motion frequency information corresponding to the actuator, and generates corresponding depth features, optical flow features, segmentation features, posture features, and frequency features respectively. The trajectory features, keypoint features, depth features, optical flow features, segmentation features, posture features, and frequency features are input into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. The method controls the robot actuator to complete a preset task according to the motion control commands. Embodied intelligence methods that incorporate trajectory supervision, keypoint supervision, and multimodal prior features under vision-language conditions can more stably learn the mapping relationship from multimodal perception to fine movements, significantly improving the integrity and robustness of task perception and achieving higher execution accuracy.

[0044] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0046] Figure 1 A flowchart of a multimodal embodied intelligent control method provided in an embodiment of this disclosure is shown;

[0047] Figure 2A flowchart of a model training method provided by an embodiment of this disclosure is shown;

[0048] Figure 3 A schematic diagram of a multimodal embodied intelligent control device provided in an embodiment of this disclosure is shown;

[0049] Figure 4 A schematic diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0051] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0052] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0053] Research has revealed that existing multimodal robot control methods typically input images and task text directly into the control policy model, relying solely on the final executed action as a supervisory signal during training. This makes it difficult for the model to learn an effective mapping from high-dimensional perceptual information to the action space, resulting in a lack of controllability and transparency in action planning. In tasks requiring millimeter-level precision, such as grasping, insertion / removal, and assembly, the model is susceptible to accumulated errors, leading to a high failure rate. Furthermore, its generalization and transfer capabilities are limited; performance significantly degrades when training data is insufficient or the execution environment changes (lighting, occlusion, multi-view differences, etc.).

[0054] Based on the above research, this disclosure provides a multimodal embodied intelligent control method, device, and electronic device. The method acquires multi-view images captured by cameras at multiple locations on a robot, along with task text. Based on the task text and the multi-view images, it generates two-dimensional trajectory supervision information and keypoint supervision information, and converts them into a visual supervision map. According to the visual supervision map, it generates keypoint features corresponding to the keypoint supervision information, and combines this with the multi-view images to generate corresponding trajectory features and keypoint features. It determines the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view images, as well as the posture information and motion frequency information corresponding to the actuator, and generates corresponding depth features, optical flow features, segmentation features, posture features, and frequency features respectively. The trajectory features, keypoint features, depth features, optical flow features, segmentation features, posture features, and frequency features are input into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. The method controls the robot actuator to complete a preset task according to the motion control commands. Embodied intelligence methods that incorporate trajectory supervision, keypoint supervision, and multimodal prior features under vision-language conditions can more stably learn the mapping relationship from multimodal perception to fine movements, significantly improving the integrity and robustness of task perception and achieving higher execution accuracy.

[0055] To facilitate understanding of this embodiment, a detailed description of the multimodal embodied intelligent control method disclosed in this disclosure will be provided first. The executing entity of the multimodal embodied intelligent control method provided in this disclosure is generally a computer device with a certain computing capability. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, this multimodal embodied intelligent control method can be implemented by a processor calling computer-readable instructions stored in memory.

[0056] See Figure 1 The diagram shows a flowchart of a multimodal embodied intelligent control method provided in this disclosure embodiment. The method includes steps S101 to S105, wherein:

[0057] S101. Acquire multi-view images taken by cameras at multiple locations on the robot, as well as task text.

[0058] In one specific embodiment of the present invention, multi-view images captured by cameras at multiple locations on the robot, along with task text, are first acquired to achieve comprehensive perception and semantic understanding of the task scene. The robot includes multiple sets of cameras positioned on the left arm, right arm, and head. Preferably, the left and right arm cameras are mounted at the end of the robotic arm or near the joints, respectively, to acquire detailed images of the close-range operation area, accurately capturing the state of objects, posture changes, and local occlusion around the end effector. The head camera is mounted at a high position on the robot body to acquire global scene images, providing overall spatial layout information and environmental background information during task execution.

[0059] Here, to ensure the temporal consistency of multi-view images, this embodiment adopts a unified time synchronization mechanism during the data acquisition process. By using hardware trigger signals or software timestamps, image frames from various cameras are acquired synchronously at the same time point, thereby ensuring the spatiotemporal consistency of multi-view images during fusion and comparative analysis.

[0060] The task text is generated by an external interactive system and can originate from user natural language commands, preset task scripts, or scene tag information. Specifically, it includes descriptions of the target object, operation method, target location, and constraints of the task. For example, in the task of "placing the red cup on the right side of the table," the task text would contain entity and action descriptions such as "red cup," "placing," and "right side of the table."

[0061] In this way, by simultaneously inputting the task text and multi-view images into the subsequent multimodal processing module, the robot can establish the correspondence between the semantic context of the task objective and the visual information of the environment during the perception stage.

[0062] It should be noted that during the acquisition process, multi-view images are not limited to color images; they can also include depth maps acquired by head cameras or other depth sensors to provide spatial geometric information of the scene, assisting in subsequent 3D position reasoning and path planning. To improve image data quality, the system can perform preprocessing operations such as automatic exposure, white balance, and noise suppression before acquisition to reduce the impact of illumination variations, color distortion, and sensor noise on subsequent feature extraction. In some application scenarios, image distortion correction algorithms can also be introduced to correct geometric distortions caused by wide-angle or fisheye lenses, thereby ensuring the coordinate accuracy of trajectory monitoring and keypoint prediction.

[0063] S102. Based on the task text and the multi-view image, generate two-dimensional trajectory supervision information and key point supervision information, and convert them into a visual supervision map.

[0064] In practice, the process of generating two-dimensional trajectory supervision information and key point supervision information based on task text and multi-view images introduces highly interpretable intermediate supervision signals for multimodal embodied intelligent control, and transforms abstract path and target information into image representations that the model can directly perceive through visualization.

[0065] Specifically, firstly, multi-view images from the robot's left arm, right arm, and head cameras are input into the image encoding module. This module preferably employs a visual encoder with cross-scene generalization capabilities and maps its output high-dimensional feature vectors to a unified feature space through linear projection, facilitating subsequent multimodal fusion with task text features. For text processing, the task text is input into the text encoder to extract semantic information, including the operation object, target location, action type, and related constraints. Then, a multimodal fusion network interacts the task text features with the image features from each viewpoint to generate two types of textual prediction results: two-dimensional trajectory supervision information and keypoint supervision information.

[0066] Here, 2D trajectory supervision information is used to describe the 2D path trajectory and corresponding execution state of the robot's end effector under a specific camera view during task execution. In this embodiment, the head camera view is preferably used as the reference coordinate system. The 2D coordinate points of the end effector in continuous time frames are extracted and combined with execution state parameters such as the gripper's opening and closing state and velocity changes, expressing this information as a text sequence. For example, the trajectory sequence information may include the (x, y) coordinates of each path point and the corresponding execution state label, forming a set of path-state pairs. Keypoint supervision information is used to characterize the spatial relationship between the task target object and the end effector. By parsing entity words in the task text, the category of the manipulated object is identified, and the positions of the object and key parts related to the end effector, such as gripping points, support points, or placement points, are predicted. Finally, the results are output in the form of keypoint coordinates and names.

[0067] After obtaining the two types of supervision information, they are converted into visual supervision maps to enhance the model's spatial intuitive perception capabilities. Specifically, for trajectory supervision information, a trajectory supervision image containing the complete motion trajectory is generated based on the two-dimensional path of the end effector from the head's perspective. The starting point, middle segment, and ending point of the trajectory are distinguished by different color codes, and the gripper's execution state is reflected by preset patterns or color markings at the trajectory nodes, thus visually clarifying the action sequence and state changes. For keypoint supervision information, the predicted keypoint positions are marked on the image with the head camera's perspective as the background. Different colors and graphic symbols are used to distinguish between object keypoints and end effector keypoints, and text labels or color codes are added near the keypoints to indicate their semantic categories. Both trajectory supervision images and keypoint supervision images use a black background or task scene image as a base to ensure that subsequent feature extraction can accurately focus on the trajectory and keypoint regions and reduce background interference.

[0068] As one possible implementation, image information from the robot's left arm, right arm, and head cameras in multi-view images is input into an image encoder and linearly projected to obtain corresponding image visual features. The task text is input into a text encoder to obtain task text features, and multimodal fusion is performed on the image visual features and task text features to output two-dimensional trajectory supervision information and key point supervision information of the text modality. Specifically, the two-dimensional coordinates and execution state of the robot's end effector in the preset view image coordinate system are extracted to generate trajectory sequence information of the text modality as two-dimensional trajectory supervision information. The entity words of the manipulated object are parsed from the task text, and the key point positions of the manipulated object and the robot's end effector are predicted to generate key point coordinates and key point names of the text modality as key point supervision information.

[0069] In practice, image information from the robot's left arm, right arm, and head cameras in multi-view images is input into an image encoder for feature extraction. After feature extraction, a linear projection layer maps the high-dimensional features from different camera perspectives to a unified feature space, ensuring consistency in the dimensionality and distribution of features from each perspective, which facilitates subsequent multimodal fusion processing.

[0070] Here, the image features of the left and right arm cameras can focus on capturing fine-grained texture information and local geometric changes in the operating area, while the image features of the head camera provide global layout and environmental context information of the task scene. Through this complementary perspective, accurate spatial perception can be achieved in subsequent fusion.

[0071] Simultaneously, the task text is input into a text encoder to extract task semantic information. The text encoder can be a pre-trained language model such as T5, BERT, or Gemma. Through word segmentation, embedding, and context modeling, it obtains task text features that can represent information such as task intent, target object, action type, and positional constraints. After obtaining the image visual features and task text features, they are input into a multimodal fusion module, allowing the text features to selectively focus on task-related regions and details from different visual perspectives, thereby generating a more task-oriented fusion representation.

[0072] Furthermore, based on the fusion results, two types of text modal prediction information are output: two-dimensional trajectory supervision information and keypoint supervision information. The generation process of the two-dimensional trajectory supervision information includes: extracting the two-dimensional coordinate positions of the robot's end effector in continuous time frames under a preset viewpoint image coordinate system, while simultaneously recording execution state information, such as gripper opening and closing states, speed changes, and grasping or placing action markers. These temporal coordinates and states are combined to form trajectory sequence information, which is then encoded and output in a textual manner. The generation process of the keypoint supervision information includes: first, parsing the entity words of the manipulated object from the task text, such as "cup," "screw," and "button," then predicting the keypoint positions related to the object and the keypoint relationships between the object and the end effector, such as grasping points, contact surfaces, and insertion ports, from the fused features. Finally, the two-dimensional coordinates of the keypoints and their corresponding semantic names are output as text modal information.

[0073] As one possible implementation, based on the 2D trajectory supervision information, a trajectory supervision image is generated, based on the robot's head camera perspective, containing the 2D trajectory path and the corresponding actuator state of the robot's end effector. Based on the keypoint supervision information, a keypoint supervision image is generated, based on the robot's head camera perspective, containing the positions of key points corresponding to the manipulated object and the robot's end effector. For the trajectory supervision image, the start point, path, and end point in the 2D trajectory path are color-coded, and the motion nodes corresponding to the robot's end effector are marked in the 2D trajectory path using a preset pattern, distinguishing different actuator states by color. For the keypoint supervision image, based on the keypoint positions, the key points corresponding to the manipulated object and the robot's end effector are marked using a preset pattern and color-coded.

[0074] In practice, after obtaining two-dimensional trajectory supervision information and key point supervision information, the viewpoint of the robot's head camera is used as a spatial reference benchmark to convert this information into a visual supervision image that can be directly used for visual feature extraction.

[0075] Specifically, regarding trajectory monitoring information, the system draws the trajectory path on a blank background or scene image based on the two-dimensional trajectory sequence of the end effector during task execution in the coordinate system of the head camera's viewpoint, so as to intuitively reflect the motion trajectory of the end effector throughout the task.

[0076] Here, in order to make the trajectory supervision graph clearly reflect the execution steps and state changes in the subsequent feature learning process, the starting point, intermediate path segments and the ending point of the trajectory path are respectively encoded with different colors. For example, green is used to mark the starting point, gradient colors are used to mark the path and red is used to mark the ending point, so as to visually distinguish different trajectory stages.

[0077] Meanwhile, preset pattern markers, such as dots, triangles, or specific shapes, are superimposed at key action nodes of the trajectory path to indicate the execution state of the end effector at that node (such as gripper opening and closing, pause, acceleration, etc.). Different state categories are further distinguished by color or fill method, so that the trajectory supervision map can carry the dual semantics of path information and action state.

[0078] Furthermore, regarding the keypoint supervision information, the viewpoint of the robot's head camera is also used as a reference. The keypoint positions of the manipulated object and the keypoint positions related to the end effector are marked on the keypoint supervision image. To ensure distinguishability in multi-keypoint scenarios, this embodiment uses different preset pattern shapes (such as squares, rhombuses, stars, etc.) to distinguish between object keypoints and actuator keypoints when marking keypoints, and assigns specific color codes to different categories of keypoints. For example, yellow is used to mark keypoints of the target object, and purple is used to mark keypoints of the end effector, thereby establishing a clear category classification visually.

[0079] Thus, through the aforementioned visualization transformation process, this embodiment transforms the originally abstract trajectory data and key point data into information images with intuitive spatial meaning and semantic hierarchy. This preserves accurate geometric and state information while providing significant intermediate supervision signals for model training. When these supervision images are subsequently input into the feature extraction network, they enable the network to learn explicit representations of trajectories and key points, thereby significantly improving the accuracy and robustness of control strategies in path planning, target localization, and action execution.

[0080] S103. Based on the visualization supervision map, generate key point features corresponding to the key point supervision information, and combine the multi-view image to generate corresponding trajectory features and key point features.

[0081] In one specific embodiment of the present invention, after generating the trajectory supervision image and the keypoint supervision image, feature extraction is first performed on the keypoint supervision image to obtain the corresponding keypoint features. Specifically, the keypoint supervision image is input to a feature extraction module, which preferably includes convolutional layers, batch normalization layers, and nonlinear activation units. Through multi-layer convolutional operations, the spatial distribution features and shape semantic information of the keypoint region are extracted. After convolution processing, the features are mapped to a unified multimodal fusion space via a linear projection layer, thereby forming a keypoint feature vector that can characterize the positional relationship between the manipulated object and the key parts of the end effector. This feature vector not only contains the spatial coordinate information of the keypoints but also the category and functional semantics implied by color, shape, and other markings, providing a direct basis for subsequent motion planning.

[0082] Here, the key point supervision image is convolved and linearly projected to extract key point features; the trajectory supervision image is multiplied with a preset first coefficient, and the image information acquired by the robot head camera is multiplied with a preset second coefficient, and then fused with different transparency to generate a global trajectory running image from the robot head perspective; the global trajectory running image is then convolved and linearly projected to generate trajectory features.

[0083] In the trajectory feature generation process, this embodiment fuses the trajectory supervision image with multi-view images to simultaneously preserve trajectory path and environmental background information. Specifically, the trajectory supervision image is multiplied by a preset first coefficient, and the corresponding view image captured by the robot's head camera is multiplied by a preset second coefficient. The two are then superimposed with different levels of transparency to generate a global trajectory running image that includes the global scene background and clearly defined trajectory markers. This preserves trajectory visibility and status markers while retaining the layout of objects in the background, obstacle positions, and environmental texture information, thus providing conditions for joint modeling of the trajectory and the scene.

[0084] Subsequently, the global trajectory image is input into a convolutional feature extraction network. After multiple convolutions and feature downsampling, the geometric shape, color coding distribution, and relative positional relationship with the environment of the path are extracted. At the output end, the image is uniformly mapped to the same feature space as the key point features through a linear projection layer to obtain the trajectory feature vector.

[0085] Furthermore, in the process of generating trajectory features and keypoint features by combining multi-view images, this embodiment can complementaryly fuse features from the left arm, right arm, and head cameras to fully utilize close-up operational details and panoramic layout information. For example, a cross-view attention mechanism can be used to fuse local object texture and motion detail features captured by the left and right arm cameras with global spatial features provided by the head camera, thereby enabling the generated trajectory features and keypoint features to possess both fine-grained local accuracy and global spatial consistency.

[0086] This fusion approach can effectively improve the model's adaptability to accurate trajectory planning and key point localization in complex task scenarios, and provide high-quality input for action prediction in subsequent multimodal fusion inference models.

[0087] S104. Determine the depth image, optical flow information, segmentation mask matching the task text, and pose information and motion frequency information corresponding to the multi-view image, and generate corresponding depth features, optical flow features, segmentation features, pose features and frequency features respectively.

[0088] In one specific embodiment of the present invention, after obtaining trajectory features and key point features, it is necessary to further extract various auxiliary modal features from multi-view images and task-related information to improve the perception accuracy and action stability of the control strategy in dynamic environments.

[0089] Here, the depth images corresponding to the multi-view images are first determined. These depth images can be directly acquired by a head-mounted camera with depth perception capabilities, or calculated from multi-view color images using binocular matching, structured light, or depth estimation algorithms. Depth images provide spatial distance information for each pixel in the scene, which helps in establishing target positions and path planning constraints in 3D space. In this embodiment, the depth images are input into a convolutional coding network. After multiple layers of convolution, pooling, and feature compression, multi-scale spatial geometric features are extracted and mapped to a unified-dimensional depth feature vector through linear projection for fusion with other modal features.

[0090] Here, in terms of acquiring optical flow information, this embodiment preferably selects the starting frame and the ending frame of the trajectory sequence for each viewpoint, and uses an optical flow estimation algorithm (such as RAFT, PWC-Net or other high-precision optical flow networks) to calculate the motion vector field of the pixels between frames, so as to obtain an optical flow map that reflects the dynamic changes of the scene.

[0091] Among them, optical flow maps can explicitly capture changes in object position and occlusion caused by robot actions during task execution, thus providing a basis for dynamic environment modeling. After feature reprojection, the optical flow maps are fused with cross-view cross-attention features from the perspectives of the left and right arms and the head camera to obtain fused optical flow features that combine local fine motion and global dynamic relationships. Finally, optical flow feature vectors are generated through convolution and linear projection.

[0092] Specifically, for each viewpoint camera image, the starting and ending camera images of the trajectory sequence under that viewpoint are extracted to determine the optical flow information between the starting and ending camera images under that viewpoint; for the optical flow information under each viewpoint, reprojection is performed to extract projection features; after performing cross-attention operation on the projection features under the robot's left and right arm viewpoints and the projection features under the robot's head viewpoint, the resulting features of the cross-attention operation are spliced ​​together to determine the optical flow features.

[0093] Furthermore, in terms of generating the segmentation mask, this embodiment inputs the task text and multi-view images into a multimodal segmentation model. The task semantic information guides the model to locate and segment object regions related to the task target in the image, outputting corresponding binary or multi-value segmentation mask images. The segmentation mask can explicitly identify the task-related regions of interest in visual features, reducing background interference. This mask image is then input into a convolutional coding network to extract the shape, boundary, and spatial distribution features of the target region, and a segmentation feature vector is obtained through linear mapping.

[0094] Furthermore, regarding the attitude information of the actuator, this embodiment constructs a real-time motion state vector of the actuator by reading the angle data from the robotic arm joint encoder, the pose information of the end effector, and the velocity data, and inputs it into the attitude feature encoding module. This module preferably uses a KAN structure or a multilayer perceptron with nonlinear fitting capabilities to map the attitude information into a low-dimensional attitude feature vector, preserving the inter-joint coupling relationship and the global constraint information of the end effector attitude.

[0095] Furthermore, regarding motion frequency information, a parameter vector reflecting the rhythm and frequency changes of motion is obtained by performing Fourier transform or zero-crossing rate analysis on the motion data of the actuator over a period of time. This vector is then encoded by a KAN module or a fully connected network to generate a frequency feature vector, which is used to provide dynamic temporal constraints during the motion prediction stage.

[0096] In this way, this embodiment generates depth features, optical flow features, segmentation features, pose features, and frequency features. These features, together with the aforementioned trajectory features and key point features, constitute the multimodal input required for task execution, providing comprehensive environmental perception and action state description for the subsequent multimodal fusion inference model, thereby significantly improving the robustness and generalization ability of the control strategy in multiple tasks and multiple environments.

[0097] S105. The trajectory features, key point features, depth features, optical flow features, segmentation features, pose features, and frequency features are input into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. The robot actuator is controlled to complete the preset task according to the motion control commands.

[0098] In one specific embodiment of the present invention, the trajectory features, keypoint features, depth features, optical flow features, segmentation features, pose features, and frequency features are input into a multimodal fusion inference model. The model aligns and correlates the different modal features to generate relative motion prediction features reflecting the current task state and environmental constraints. These relative motion prediction features describe the incremental control quantities relative to the previous moment, including joint angle changes, end-effector pose changes, and actuator opening / closing changes. Subsequently, these relative motion prediction features are converted into robot-executable motion control commands and sent to the actuator control module to drive the robot to complete the task execution of the current step according to the control commands.

[0099] As one possible implementation method, see Figure 2 The diagram shows a flowchart of a multimodal embodied intelligent control method provided in an embodiment of this disclosure. The method includes steps S201 to S203, wherein:

[0100] S201. Obtain sample operation video data of the robot actuator, and configure corresponding text for the execution actions corresponding to the sample operation video data.

[0101] S202. Label the motion posture of the robot actuator as preset requirement labels and key point labels, and train the image feature extraction model corresponding to the multi-view image based on the preset requirement labels and key point labels.

[0102] S203. After the image feature extraction model is trained, freeze the image feature extraction model and train the multimodal fusion inference model so that the multimodal fusion inference model has the ability to output relative action prediction features.

[0103] In practice, the training process is divided into two stages. The first stage mainly trains the image feature extraction model, which is designed to better predict motion trajectories and key point information. However, this stage does not rely on any motion information or robot structure and can be trained based on a large amount of video data of network-related robotic arm operations.

[0104] First, the actions performed in the video data are paired with corresponding text, and the overall motion posture of the robotic arm is labeled as the prediction label and key point label for the model. Then, the image feature extraction model is continuously trained based on these labels, so that the model can better predict the motion trajectory corresponding to the task actions.

[0105] The second stage freezes the pre-trained image feature extraction model. Then, a multimodal fusion inference model is used to predict the action features to be performed. This feature training employs the MSE loss function. During model inference, these features are converted into angle information for each joint of the robotic arm, enabling the robot to perform tasks based on changes in joint angles.

[0106] To enhance the model's environmental adaptability and multimodal information fusion capabilities, a multi-level data augmentation strategy is employed during training. This includes: random image masking: randomly occluding input camera images, with the masked area accounting for 10%-30% (randomly generated rectangular or irregular polygonal masks), ensuring that the masked area in a single image does not exceed 50% of the key semantic region (such as the robotic arm's end effector or the target object), forcing the model to reduce its reliance on local image information; and image augmentation: including color dithering (brightness adjustment range ±20%, contrast adjustment range 0.8-1.2 times, saturation adjustment range 0.7-1.3 times) and arbitrary cropping (cropped images retain 60%-100% of their original size, and the offset between the cropped area center and the original image center does not exceed 20% of the original size).

[0107] These operations not only enable the model to adapt to different lighting intensities and color temperatures, but also, by disrupting the integrity of a single camera image, prompt the model to fuse complementary information from three cameras (such as target contours from different perspectives and robotic arm postures), thereby improving the global consistency of behavior prediction.

[0108] During the predictive execution phase, the robotic arm employs a closed-loop execution mechanism of sequence traversal and dynamic inference: First, it traverses and executes the motion sequence output by the model (including parameters such as joint angular velocity and end effector movement distance at each time step) frame by frame. After completing the action at each time step, it collects the current environmental image and robotic arm state data in real time, and re-inputs them into the model for inference to generate the next motion sequence. This process iterates continuously until the robotic arm completes the preset action target (such as grasping a target object and placing it in a designated position), thereby correcting accumulated errors through dynamic feedback and ensuring the execution accuracy of complex actions.

[0109] This disclosure provides a multimodal embodied intelligent control method that acquires multi-view images captured by cameras at multiple locations on a robot, as well as task text. Based on the task text and the multi-view images, it generates two-dimensional trajectory supervision information and key point supervision information, and converts them into a visual supervision map. According to the visual supervision map, it generates key point features corresponding to the key point supervision information, and combines the multi-view images to generate corresponding trajectory features and key point features. It determines the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view images, as well as the posture information and motion frequency information corresponding to the actuator, and generates corresponding depth features, optical flow features, segmentation features, posture features, and frequency features respectively. It inputs the trajectory features, key point features, depth features, optical flow features, segmentation features, posture features, and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. It controls the robot actuator to complete a preset task according to the motion control commands. Embodied intelligence methods that incorporate trajectory supervision, keypoint supervision, and multimodal prior features under vision-language conditions can more stably learn the mapping relationship from multimodal perception to fine movements, significantly improving the integrity and robustness of task perception and achieving higher execution accuracy.

[0110] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0111] Based on the same inventive concept, this disclosure also provides a multimodal embodied intelligent control device corresponding to the multimodal embodied intelligent control method. Since the principle of the device in this disclosure for solving the problem is similar to the multimodal embodied intelligent control method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0112] Please see Figure 3 , Figure 3 This is a schematic diagram of a multimodal embodied intelligent control device provided in an embodiment of this disclosure. Figure 3 As shown in the embodiments of this disclosure, the multimodal embodied intelligent control device 300 includes:

[0113] The data acquisition module 310 is used to acquire multi-view images taken by cameras at multiple locations on the robot, as well as task text.

[0114] The trajectory and key point supervision module 320 is used to generate two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view image, and convert them into a visual supervision map.

[0115] The first feature extraction module 330 is used to generate key point features corresponding to the key point supervision information based on the visualization supervision map, and to generate corresponding trajectory features and key point features in combination with the multi-view image.

[0116] The second feature extraction module 340 is used to determine the depth image, optical flow information, segmentation mask matching the task text, and pose information and motion frequency information corresponding to the multi-view image, and generate corresponding depth features, optical flow features, segmentation features, pose features and frequency features respectively.

[0117] The motion prediction module 350 is used to input the trajectory features, key point features, depth features, optical flow features, segmentation features, pose features and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands.

[0118] The control execution module 360 ​​is used to control the robot's actuator to complete the preset task according to the action control command.

[0119] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0120] This disclosure provides a multimodal embodied intelligent control device that acquires multi-view images taken by cameras at multiple locations on a robot, as well as task text. Based on the task text and the multi-view images, it generates two-dimensional trajectory supervision information and key point supervision information, and converts them into a visual supervision map. According to the visual supervision map, it generates key point features corresponding to the key point supervision information, and combines the multi-view images to generate corresponding trajectory features and key point features. It determines the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view images, as well as the posture information and motion frequency information corresponding to the actuator, and generates corresponding depth features, optical flow features, segmentation features, posture features, and frequency features respectively. It inputs the trajectory features, key point features, depth features, optical flow features, segmentation features, posture features, and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. It controls the robot actuator to complete a preset task according to the motion control commands. Embodied intelligence methods that incorporate trajectory supervision, keypoint supervision, and multimodal prior features under vision-language conditions can more stably learn the mapping relationship from multimodal perception to fine movements, significantly improving the integrity and robustness of task perception and achieving higher execution accuracy.

[0121] Corresponding to Figure 1The multimodal embodied intelligent control method disclosed herein also provides an electronic device 400, such as... Figure 4 The diagram shown is a structural schematic of an electronic device 400 provided in an embodiment of this disclosure, including:

[0122] Processor 41, memory 42, and bus 43; memory 42 is used to store execution instructions, including main memory 421 and external memory 422; the main memory 421, also called internal memory, is used to temporarily store the computational data in processor 41, as well as the data exchanged with external memory 422 such as hard disk. Processor 41 exchanges data with external memory 422 through main memory 421. When the electronic device 400 is running, processor 41 and memory 42 communicate through bus 43, enabling processor 41 to execute... Figure 1 The steps of the multimodal embodied intelligent control method.

[0123] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the multimodal embodied intelligent control method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0124] This disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, they can perform the steps of the multimodal embodied intelligent control method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0125] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0129] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A multimodal embodied intelligent control method, characterized in that, include: Acquire multi-view images taken by cameras at multiple locations on the robot, along with task text; Based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated and converted into a visual supervision map; Based on the visualization supervision map, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated by combining the multi-view images. Determine the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view image, as well as the pose information and motion frequency information corresponding to the actuator, and generate the corresponding depth features, optical flow features, segmentation features, pose features, and frequency features respectively. The trajectory features, key point features, depth features, optical flow features, segmentation features, pose features, and frequency features are input into a multimodal fusion inference model to generate relative action prediction features and convert them into action control commands. The robot's actuators are controlled to complete preset tasks according to the motion control instructions.

2. The method according to claim 1, characterized in that, Based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated, specifically including: The image information from the robot's left arm, right arm, and head camera in the multi-view image is input into the image encoder and linearly projected to obtain the corresponding image visual features. The task text is input into a text encoder to obtain task text features, and multimodal fusion is performed on the image visual features and the task text features to output the two-dimensional trajectory supervision information and the key point supervision information of the text modality. Specifically, the two-dimensional coordinates and execution state of the robot's end effector in a preset viewpoint image coordinate system are extracted, and the trajectory sequence information of the text modality is generated as the two-dimensional trajectory supervision information; The entity words of the manipulated object are parsed from the task text, and the key point positions of the manipulated object and the robot end effector are predicted. The key point coordinates and key point names of the text modality are generated as the key point supervision information.

3. The method according to claim 1, characterized in that, Converting the two-dimensional trajectory supervision information and the key point supervision information into a visual supervision map specifically includes: Based on the two-dimensional trajectory supervision information, a trajectory supervision image is generated, which is based on the robot head camera view and includes the two-dimensional trajectory path and the corresponding actuator state of the robot end effector. Based on the key point supervision information, a key point supervision image is generated, which is based on the view of the robot's head camera and includes the key point positions of the manipulated object and the robot's end effector.

4. The method according to claim 3, characterized in that: For the trajectory supervision image, the starting point, path, and ending point in the two-dimensional trajectory path are encoded with different colors, and the action nodes corresponding to the robot end effector are marked in the two-dimensional trajectory path using a preset pattern, and different states of the actuator are distinguished by color; For the key point supervision image, based on the key point position, the key points corresponding to the manipulated object and the robot end effector are marked with the preset pattern and color-coded.

5. The method according to claim 3, characterized in that, Based on the visualized supervision map, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated in combination with the multi-view images, specifically including: The key point supervision image is subjected to convolution and linear projection to extract the key point features; The trajectory supervision image is multiplied by a preset first coefficient, and the image information captured by the robot's head camera is multiplied by a preset second coefficient. Then, different levels of transparency are used to fuse the images to generate a global trajectory running image from the robot's head perspective. The trajectory features are generated by performing convolution and linear projection on the global trajectory image.

6. The method according to claim 1, characterized in that, The optical flow feature is generated based on the following steps: For each viewpoint camera image, extract the starting camera image and the ending camera image of the trajectory sequence under that viewpoint, and determine the optical flow information between the starting camera image and the ending camera image under that viewpoint; For the optical flow information under each viewpoint, reprojection is performed to extract projection features. After performing cross-attention operation on the projection features under the robot's left arm and right arm viewpoints and the projection features under the robot's head viewpoint, the resulting features of the cross-attention operation are concatenated to determine the optical flow features.

7. The method according to claim 1, characterized in that, The model is trained based on the following steps: Acquire sample operation video data of the robot actuator, and configure corresponding text for the execution actions corresponding to the sample operation video data; The motion posture of the robot actuator is labeled as a preset requirement label and a key point label, and an image feature extraction model corresponding to the multi-view image is trained based on the preset requirement label and the key point label; After the image feature extraction model is trained, the image feature extraction model is frozen, and the multimodal fusion inference model is trained so that the multimodal fusion inference model has the ability to output relative action prediction features.

8. A multimodal embodied intelligent control device, characterized in that, include: The data acquisition module is used to acquire multi-view images taken by cameras at multiple locations on the robot, as well as task text; The trajectory and key point supervision module is used to generate two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view image, and convert them into a visual supervision map; The first feature extraction module is used to generate key point features corresponding to the key point supervision information based on the visualization supervision map, and to generate corresponding trajectory features and key point features in combination with the multi-view image. The second feature extraction module is used to determine the depth image, optical flow information, segmentation mask matching the task text, and pose information and motion frequency information corresponding to the multi-view image, and generate corresponding depth features, optical flow features, segmentation features, pose features and frequency features respectively. The motion prediction module is used to input the trajectory features, key point features, depth features, optical flow features, segmentation features, pose features and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. The control execution module is used to control the robot's actuator to complete the preset task according to the action control instructions.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the multimodal embodied intelligent control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the multimodal embodied intelligent control method as described in any one of claims 1 to 7.

Images