Multimodal-based embodied robot control method and device, electronic equipment, readable storage medium and program product

Abstract

The application relates to a multi-modal-based embodied robot control method and device, electronic equipment, readable storage medium and program product. By acquiring a task instruction, multi-modal environment data is acquired based on the task instruction, the multi-modal environment data is fused and processed, multi-modal perception information after fusion is obtained, and based on the multi-modal perception information and the state of the embodied robot, a task path and a task action sequence for a target task are determined, the task path and the task action sequence are converted into control instructions for the embodied robot, so that the embodied robot is controlled to execute the target task according to the control instructions. Through the fusion processing of the multi-modal environment data, not only the perception robustness of the embodied robot in a dynamic and uncertain environment can be ensured, but also the real-time response capability and the accuracy of intelligent decision-making can be optimized, so that the adaptability and execution capability of the embodied robot to complex working conditions are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal based embodied robot control method and device, electronic equipment, readable storage medium and program product. BACKGROUND

[0002] With the challenges of rising labor costs and labor shortage in the global manufacturing, construction and civil engineering fields, automated, intelligent and efficient robot equipment has become an inevitable trend of industrial transformation and upgrading. Embodied intelligence is an important branch of emerging artificial intelligence, which refers to the learning and execution of tasks by an intelligent agent through physical interaction with the environment, emphasizing the perception, action and adaptation of intelligent agents in real or simulated environments. In industrial production, embodied robots can flexibly handle different production tasks, improving production efficiency and product quality.

[0003] However, environmental perception is a key link to realize autonomous perception, decision-making and execution of embodied robots. Traditional laser radar point cloud single modal detection lacks texture, color and context information, and is easily affected by occlusion, resulting in poor detection accuracy in many scenes, which further affects the perception and execution ability of embodied robots. SUMMARY

[0004] Therefore, it is necessary to provide a multi-modal based embodied robot control method, device, electronic equipment, computer readable storage medium and computer program product capable of improving the perception and execution ability of embodied robots.

[0005] In a first aspect, the present application provides a multi-modal based embodied robot control method, which comprises:

[0006] obtaining a task instruction and multi-modal environment data based on the task instruction; the task instruction is used to instruct the embodied robot to execute a target task;

[0007] fusing the multi-modal environment data to obtain fused multi-modal perception information;

[0008] determining a task path and a task action sequence for the target task based on the multi-modal perception information and the state of the embodied robot;

[0009] converting the task path and the task action sequence into a control instruction for the embodied robot, and controlling the embodied robot to execute the target task according to the control instruction.

[0010] In one of the embodiments, the multi-modal environment data at least includes image data and point cloud data; the fusion processing of the multi-modal environment data to obtain the fused multi-modal perception information comprises: performing feature extraction on the point cloud data to obtain voxelized point cloud features; performing feature extraction on the image data to obtain an image feature map; projecting the voxelized point cloud features into a feature domain of the image data to obtain a point cloud projection feature map aligned with the image feature map; and performing enhanced fusion processing on the image feature map and the point cloud projection feature map to obtain the fused multi-modal perception information.

[0011] In one of the embodiments, the enhanced fusion processing of the image feature map and the point cloud projection feature map to obtain the fused multi-modal perception information comprises: performing point multiplication on the image feature map and the point cloud projection feature map to obtain a point multiplication result; processing the point multiplication result through two independent two-dimensional convolution layers to obtain a point cloud confidence weight and an image confidence weight; and performing weighted splicing on the image feature map and the point cloud projection feature map based on the point cloud confidence weight and the image confidence weight to obtain the multi-modal perception information.

[0012] In one of the embodiments, the multi-modal environment data at least includes image data and point cloud data; the fusion processing of the multi-modal environment data to obtain the fused multi-modal perception information comprises: performing feature extraction on the point cloud data to obtain voxel features; performing feature extraction on the image data to obtain an image feature map; determining a centroid coordinate of each non-empty voxel in the voxel features based on centering; projecting the centroid coordinate into a feature domain of the image data to obtain a reference point of the centroid of the voxel on the image feature map; sampling image features from the image feature map based on a sampling offset and the reference point to obtain a plurality of sampled image features; determining an attention weight of each image feature, and performing weighted aggregation on the plurality of image features according to the attention weight of each image feature to obtain a fusion feature, which is taken as the multi-modal perception information.

[0013] In one of the embodiments, the determination of the attention weight of each image feature comprises: for each image feature, generating a corresponding attention weight by using a multi-layer perception machine.

[0014] In one of the embodiments, the determining the task path and the task action sequence for the target task based on the multi-modal perception information and the self state of the embodied robot comprises: identifying a key target in an environment according to the multi-modal perception information; determining a sub task for the key target based on the target task and the identified key target; decomposing the sub task into a series of executable task action sequences according to the self state of the embodied robot, the task action sequences comprising a plurality of actions; and generating a corresponding task path based on the plurality of actions in the task action sequences.

[0015] In a second aspect, the present application further provides a multi-modal based embodied robot control device, the device comprising:

[0016] a data acquisition module configured to execute an acquisition task instruction and acquire multi-modal environment data based on the task instruction, the task instruction being used to instruct the embodied robot to execute a target task;

[0017] a multi-modal fusion module configured to execute a fusion processing on the multi-modal environment data to obtain fused multi-modal perception information;

[0018] a task determination module configured to execute a determination of a task path and a task action sequence for the target task based on the multi-modal perception information and the self state of the embodied robot;

[0019] a control module configured to execute a conversion of the task path and the task action sequence into a control instruction for the embodied robot, and control the embodied robot to execute the target task according to the control instruction.

[0020] In a third aspect, the present application further provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps of the method of the first aspect.

[0021] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to realize the steps of the method of the first aspect.

[0022] In a fifth aspect, the present application further provides a computer program product comprising a computer program, the computer program being executed by a processor to realize the steps of the method of the first aspect.

[0023] The multi-modal based embodied robot control method, device, electronic device, computer readable storage medium and computer program product can obtain a task instruction, obtain multi-modal environment data based on the task instruction, perform fusion processing on the multi-modal environment data, obtain fused multi-modal perception information, determine a task path and a task action sequence for a target task based on the multi-modal perception information and a state of the embodied robot, convert the task path and the task action sequence into a control instruction for the embodied robot, and control the embodied robot to perform the target task according to the control instruction. The multi-modal environment data is fused and processed, so that the perception robustness of the embodied robot in a dynamic and uncertain environment can be ensured, the real-time response capability and the accuracy of intelligent decision-making can be optimized, accurate environment representation is provided for task execution, and the adaptability and execution capability of the embodied robot to complex working conditions are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A flowchart of an embodiment of a multi-modal based embodied robot control method is shown.

[0026] Figure 2 A flowchart of an embodiment of a fusion processing step is shown.

[0027] Figure 3 An adaptive gating information fusion diagram of an embodiment is shown.

[0028] Figure 4 A flowchart of an embodiment of a fusion processing step is shown.

[0029] Figure 5 A fusion diagram of an embodiment is shown.

[0030] Figure 6 A flowchart of an embodiment of a task determination step is shown.

[0031] Figure 7 An application diagram of an embodiment is shown.

[0032] Figure 8 A model optimization diagram of an embodiment is shown.

[0033] Figure 9A structural block diagram of a multi-modal based embodied robot control device in an embodiment;

[0034] Figure 10 An internal structural diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0035] For the purpose of the present application, technical solutions and advantages, the following will be further described in detail in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0036] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" used in the present application and any variants thereof are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application means two or more. The term "and / or" used in the present application means one of the schemes or any combination of a plurality of schemes.

[0037] In an exemplary embodiment, as shown in Figure 1 A multi-modal based embodied robot control method is provided, which is applied to an embodied robot, and can specifically include the following steps:

[0038] Step 102, obtaining a task instruction, and obtaining multi-modal environment data based on the task instruction.

[0039] The task instruction is used to instruct the embodied robot to perform a target task, and the target task is a task set up for achieving a certain specific target. The task instruction can be a voice instruction, a text instruction or an instruction based on a control operation, etc. For example, the task instruction can be a specific task instruction such as "cleaning the floor" or "arranging the desktop clutter". The multi-modal environment data at least includes image data and point cloud data, which are collected by the embodied robot based on the obtained task instruction.

[0040] Specifically, after obtaining the task instruction, the embodied robot can collect multi-modal environment data based on the task instruction, for example, collect image data through a camera and collect point cloud data through a laser radar. And through subsequent steps for action planning to perform the target task.

[0041] Step 104, performing fusion processing on the multi-modal environment data to obtain fused multi-modal perception information.

[0042] The fusion processing refers to combining data from different modalities to achieve semantic association and spatial consistency alignment between different modalities, thereby improving information processing and understanding ability. The multi-modal perception information is structured data obtained by fusing multi-modal environment data (such as image data and point cloud data).

[0043] The fusion processing can be implemented by using a VLM (Visual Language Model) large model. The VLM large model can extract features, align features and fuse features of input data of various modalities (such as image data and point cloud data), thereby converting multi-modal information of the physical world into structured multi-modal perception information.

[0044] In this embodiment, by fusing multi-modal environment data, the comprehensiveness of environment state acquisition can be improved, the accuracy and timeliness of multi-source information fusion can be enhanced, and the perception robustness of the embodied robot in a dynamic and uncertain environment can be ensured. The real-time response capability can be optimized to support the accuracy of subsequent intelligent decision-making, and a highly reliable data support system can be constructed to provide accurate environment representation for task execution, thereby significantly improving the adaptability of the embodied robot to complex working conditions and the depth of environment understanding.

[0045] In step 106, a task path and a task action sequence for the target task are determined based on the multi-modal perception information and the state of the embodied robot.

[0046] The state of the embodied robot refers to the state information of the embodied robot, for example, including but not limited to the pose of the embodied robot, such as the position of the robot, the pose of the mechanical arm, and the state of the gripper. The task path refers to the movement path of the embodied robot for the target task. The task action sequence includes a series of executable actions, and the corresponding target task can be completed based on the task action sequence.

[0047] In this embodiment, the embodied robot determines the task path and the task action sequence for the target task based on the multi-modal perception information obtained by the fusion processing and the state of the embodied robot.

[0048] In step 108, the task path and the task action sequence are converted into control instructions for the embodied robot, and the embodied robot is controlled to execute the target task according to the control instructions.

[0049] The control instructions are specific signals for directly controlling the embodied robot to execute actions, such as signals for controlling the movement speed or angle of the mechanical arm, signals for controlling the clamping force of the gripper, or control signals for the rotation speed or angle of the motor of each joint.

[0050] In the embodiment, the embodied robot converts the task path and the task action sequence into bottom-layer control instructions that can be understood by actuators such as motors and steering gears, and controls the actuators according to the control instructions to complete the target task.

[0051] In the above multi-modal-based embodied robot control method, the embodied robot acquires a task instruction, acquires multi-modal environment data based on the task instruction, performs fusion processing on the multi-modal environment data to obtain fused multi-modal perception information, and determines a task path and a task action sequence for a target task based on the multi-modal perception information and a state of the embodied robot, converts the task path and the task action sequence into control instructions for the embodied robot, and controls the embodied robot to execute the target task according to the control instructions. Through the fusion processing on the multi-modal environment data, the embodied robot can not only ensure the perception robustness in a dynamic and uncertain environment, but also optimize the real-time response capability and the accuracy of intelligent decision-making, thereby providing accurate environment representation basis for task execution, and significantly improving the adaptability and execution capability of the embodied robot to complex working conditions.

[0052] In an exemplary embodiment, the multi-modal environment data at least includes image data and point cloud data, and as shown in FIG. 1, in step 104, the multi-modal environment data is fused to obtain fused multi-modal perception information, which can specifically include: Figure 2

[0053] In step 202, the point cloud data is feature-extracted to obtain voxelized point cloud features.

[0054] The voxelized point cloud features are structured voxel features obtained by voxelizing and feature-extracting unordered point cloud data. Specifically, voxelization and feature extraction can be realized by a 3D backbone network (such as VoxelNet, PointPillars, etc.).

[0055] In the embodiment, the original 3D laser radar point cloud (i.e., point cloud data) is first converted into structured voxel features by a 3D backbone network, so as to effectively extract the geometric features of the 3D space.

[0056] In step 204, the image data is feature-extracted to obtain an image feature map.

[0057] The image feature map is a feature map obtained by extracting rich texture and semantic features from the image data. Specifically, a 2D backbone network (such as ResNet, EfficientNet, etc.) can be used to feature-extract the image data to extract rich texture and semantic features, thereby obtaining the image feature map.

[0058] ​Step 206: Project the voxelized point cloud features into the feature domain of the image data to obtain a point cloud projection feature map aligned with the image feature map.

[0059] Since point cloud data and image data come from different coordinate systems (point clouds correspond to the 3D world, while images correspond to 2D), direct fusion is meaningless. Therefore, by using camera intrinsic and extrinsic parameters, the voxelized point cloud features are projected onto the 2D camera plane, i.e., the feature domain of the image data (such as the camera domain), thus obtaining a point cloud projection feature map aligned with the spatial dimensions of the image feature map, thereby laying the foundation for subsequent multimodal fusion.

[0060] Step 208: Enhance and fuse the image feature map and the point cloud projection feature map to obtain the fused multimodal perception information.

[0061] Among them, multimodal perception information is fused data obtained by enhancing and fusing image feature maps and point cloud projection feature maps.

[0062] For example, the above fusion process is further illustrated using the adaptive gating information fusion process. Specifically, as follows... Figure 3 As shown, firstly, the voxelized point cloud features extracted from the 3D backbone network are projected into the feature domain to obtain the point cloud projection feature map. Then, the point cloud is projected with feature maps. Image feature maps extracted with 2D backbone networks Perform a dot product to obtain the result. Then, process the dot product result using two independent 2D convolutional layers to obtain two weights. and Among them, weight This represents the point cloud confidence weight. A value close to 1 indicates that "the point cloud geometric information at this location is very reliable and should be given priority consideration." The image credibility weight is represented by a value close to 1, indicating that "the image texture / semantic information at that location is very reliable and should be given priority." The weight calculation can be performed using the following equations (1) and (2):

[0063] (1)

[0064] (2)

[0065] in, This represents the sigmoid activation function. This represents the image feature map extracted from the 2D backbone network. This means that the voxelized point cloud features extracted from the 3D backbone network are projected into a feature domain such as the camera domain to obtain a point cloud projection feature map. and This represents two independent two-dimensional convolutional layers.

[0066] Since dot product is an efficient interaction method, it can initially filter out regions where both modal features respond strongly (e.g., regions in an image identified as vehicles that also exhibit 3D geometric structures in the point cloud). The dot product results are then fed into two independent two-dimensional convolutional layers (e.g., ...). and In this context, the convolutional layer learns how to generate a spatially adaptive weight map based on the initial interaction results. Then, a Sigmoid (σ) activation function is applied to the convolutional output to compress the weight values ​​to the range [0, 1], generating two weight maps. and .

[0067] Then, based on the point cloud confidence weight and the image confidence weight, the image feature map and the point cloud projection feature map are weighted to obtain the weighted feature maps. Specifically, this is shown in equations (3) and (4):

[0068] (3)

[0069] (4)

[0070] in, Represents the projection feature map of the point cloud The weighted point cloud feature map obtained after weighting enhances the point cloud features in its high confidence region and suppresses them in its low confidence region. Represents the image feature map The weighted image feature map obtained after weighting enhances the image features in high confidence regions and suppresses them in low confidence regions.

[0071] Finally, the two weighted feature maps are concatenated along the channel dimension, and the concatenated fused feature is output through a final two-dimensional convolutional layer. This not only learns how to optimally combine the concatenated channel information, but also adjusts the number of channels to the desired output dimension, thereby ultimately outputting the enhanced fused feature, that is, outputting the fused multimodal perception information. Specifically, it is shown in the following equation (5):

[0072] (5)

[0073] in, This represents the fused features after splicing and enhancement, which is also the fused multimodal perceptual information. This represents a two-dimensional convolutional layer that performs additional processing on the fused features.

[0074] The above process has an adaptive feature, so that the weights are not fixed or manually set, but dynamically calculated by the network according to the input data. This enables the model to flexibly cope with various complex scenarios (for example, relying more on point clouds in dim light; relying more on images when the point cloud is sparse). And the Sigmoid function acts as a "soft switch" or "gate" to control the degree of information flow, which is an efficient strategy inspired by the gating mechanism in recurrent neural networks (such as LSTM). This process enables the model to learn to "play to its strengths" - to make more use of images where image information is reliable, and to make more use of point clouds where point cloud information is reliable, thereby generating a powerful multi-modal feature representation far superior to any single modality. By adaptively fusing point cloud geometric information and image features, it achieves enhanced feature image information, which helps to improve the performance of subsequent tasks.

[0075] In an exemplary embodiment, the multi-modal environment data at least includes image data and point cloud data, and as shown in FIG. 1, in step 104, the multi-modal environment data is fused to obtain fused multi-modal perception information, which can specifically include: Figure 4

[0076] Step 402, feature extraction is performed on the point cloud data to obtain voxel features.

[0077] Specifically, a 3D backbone network (such as VoxelNet, PointPillars, etc.) can be used to extract features from the point cloud data, thereby converting unordered point clouds into structured voxel features.

[0078] Step 404, feature extraction is performed on the image data to obtain an image feature map.

[0079] The image feature map is obtained by extracting rich texture and semantic features from the image data. Specifically, a 2D backbone network (such as ResNet, EfficientNet, etc.) can be used to extract features from the image data to extract rich texture and semantic features, thereby obtaining an image feature map.

[0080] Step 406, for each non-empty voxel in the voxel features, the centroid coordinates of the voxel are determined based on the centering.

[0081] The non-empty voxel refers to a voxel containing points. Specifically, as shown in FIG. 4, for each non-empty voxel in the voxel features extracted above, the centroid coordinates of the voxel can be determined based on the centering. Specifically, as shown in the following formula (6): Figure 5

[0082] (6)

[0083] where n is the number of points contained in the voxel,​​ x, y and z are the corresponding coordinate points, respectively.

[0084] Step 408, projecting the centroid coordinates into the feature domain of the image data to obtain the reference point of the centroid of the voxel on the image feature map.

[0085] Specifically, the centroid coordinates can be projected into the feature domain of the image data, such as the camera domain, by using the camera projection matrix, so as to obtain the reference point of the centroid of the voxel on the image feature map. Specifically, as shown in the following formula (7):

[0086] (7)

[0087] wherein RC is the combination of the rotation matrix and the camera calibration matrix, denotes the camera-lidar calibration matrix, denotes the centroid coordinates of a certain non-empty voxel, is the reference point of the centroid coordinates projected into the camera domain of the image data.

[0088] Step 410, sampling the image features from the image feature map based on the sampling offsets and the reference points to obtain the sampled multiple image features.

[0089] wherein the sampling offsets are used for sampling on the image feature map. Specifically, each voxel feature can predict a set of offsets by using a learnable convolutional layer (such as Iinear), which is used for sampling the image features around the reference point . m denotes the attention number, q denotes the query index, and i denotes the sampling point index.

[0090] Then, the image features are sampled from the image feature map by using the sampling offsets. Specifically, for each sampling point i, the image features are obtained by interpolation on the image feature map according to the offsets . Specifically, as shown in the following formula (8):

[0091] (8)

[0092] wherein denotes the image feature map, denotes the image features obtained by interpolation.

[0093] Step 412, determining the attention weights of each image feature, and weighting and aggregating the multiple image features according to the attention weights of each image feature to obtain the fusion features.

[0094] Specifically, for each image feature, a multi-layer perceptron can be used to generate the corresponding attention weight ; the attention weights can also be calculated through a linear layer and a softmax layer .

[0095] The sampled image features are then multiplied by the attention weights, the features are fused through an aggregation operation, and finally a linear layer is used to output the fusion features, which are the multi-modal perception information.

[0096] For example, for each attention head m, the sampled image features are weighted and summed with the attention weights, and multiplied by a learnable weight , to obtain the aggregated features, as shown in equation (9):

[0097] (9)

[0098] where m represents the attention number, q represents the query index, i represents the sampling point index, and is a learnable weight matrix, represents the attention weight.

[0099] Finally, the aggregated features are fused with the original point cloud features (usually by adding or concatenating and linear transformation) to obtain the enhanced fusion features (i.e., multi-modal perception information).

[0100] The above deformable sampling mechanism enhances the point cloud's ability to perceive the context of the image, and is suitable for scenarios of feature fusion in multi-modal 3D perception tasks. In this way, the model can adaptively learn how to combine point cloud and image features, thereby improving the perception ability of the image features around the reference point and obtaining more context information. This deformable cross-modal attention mechanism helps to achieve more accurate feature fusion and information extraction in complex visual tasks.

[0101] In an exemplary embodiment, as shown in Figure 6 , in step 106, based on the multi-modal perception information and the embodiment robot's own state, the task path and task action sequence for the target task are determined, which can specifically include:

[0102] Step 602, identifying key targets in the environment according to multi-modal perception information.

[0103] Among them, the key target can be a target that needs to be processed by analyzing the task instruction. In this embodiment, taking the task instruction "arrange the desktop clutter" as an example, the key target can be the specific clutter target on the desktop that needs to be arranged.

[0104] Exemplarily, the embodied robot understands the multi-modal perception information by combining the task instruction, so as to identify the key objects (such as books, snack bags, coffee cups) and their states (such as whether the coffee cup is full or empty? Is the snack bag puffy or flat?). In addition, the precise position and pose of these objects and environmental features (such as the opening of the trash can) can also be determined.

[0105] Step 604, based on the target task and the identified key objects, determine the sub-tasks for the key objects.

[0106] That is, by identifying the key objects, the target task is decomposed into smaller, manageable sub-tasks. For example, taking the task instruction, i.e. the target task, as "clean up the desktop clutter", based on the above-identified key objects such as books, snack bags, coffee cups, it can be decomposed into the sub-task of arranging books, the sub-task of cleaning snack bags, and the sub-task of cleaning coffee cups.

[0107] The priority of each sub-task can also be evaluated based on common sense (such as "liquids are more likely to spill than solids, so the risk is higher"). Thus, strategic decisions on "what to do" and "what to do first" are made.

[0108] Step 606, according to the state of the embodied robot, the sub-tasks are decomposed into a series of executable task action sequences.

[0109] Among them, the task action sequence includes multiple actions. Specifically, the sub-tasks can be decomposed into a series of executable task action sequences according to the state of the embodied robot. For example, for the sub-task of "cleaning the coffee cup", it can be decomposed into a series of sub-actions, i.e. task action sequences, such as "approach the cup", "close the gripper", "lift smoothly", "move above the trash can", "tilt", "reset", etc. based on the self-position.

[0110] Step 608, based on the multiple actions in the task action sequence, generate a corresponding task path.

[0111] Specifically, a corresponding task path can also be generated based on the multiple actions in the task action sequence. For example, for the task action sequence of the above-mentioned "cleaning the coffee cup" sub-task, an optimal path can be calculated for the end of the robotic arm to grasp the coffee cup and move to the trash can, such as collision-free, efficient and stable during movement.

[0112] Further, the planned action sequence can be converted into real physical movement and cope with uncertainties during execution. For example, the planned path can be converted into low-level control instructions (such as the rotation speed and rotation angle of the motor of each joint) that the actuators such as motors and steering engines can understand. During execution, control parameters (such as increasing the clamping force and adjusting the speed of the robot arm) can also be dynamically adjusted according to real-time feedback (such as the tactile sensor finding that the gripping force is not enough, and the visual sensor finding that the cup is slipping) to ensure the successful completion of the task. Thus, compliant and safe operation is achieved.

[0113] In an exemplary embodiment, the above-mentioned multi-modal-based embodied robot control method is further illustrated by taking the task instruction "organize desktop clutter" as an example. As shown in Figure 7 The above-mentioned method can be implemented based on an embodied intelligent large model. Specifically, it can include a perception layer, a decision layer, a rule layer, and an execution layer.

[0114] The perception layer uses a pre-trained VLM large model to extract, align, and fuse features of input data of multiple modalities, converting multi-modal information of the physical world into structured data to provide a basis for subsequent planning and decision-making and execution control; the decision layer is the core hub connecting environmental perception and action planning, and its core function can be compared to the "prefrontal cortex" of the human brain, which is responsible for value judgment, risk assessment, and action strategy selection; the planning layer converts abstract task goals into action sequences that can be physically executed, planning the motion trajectory of the robot end effector, while solving problems such as time and space constraints, resource allocation, and dynamic adjustment. The execution layer is the last key link that converts abstract planning into physical action, and is directly responsible for the interaction between the robot and the real world. Its core function can be compared to the "spinal cord + motor neuron" of the human body, which converts the path / skill generated by the planning layer into motor control signals to achieve high-precision and high-response body control, and dynamically adjusts parameters such as force and speed according to sensor feedback.

[0115] Exemplarily, to meet the requirements for action accuracy and stability in industrial tasks, the following optimization directions are proposed for the model structure:

[0116] (1) Multi-modal feature fusion

[0117] This module integrates multi-modal inputs such as vision, language, and force sensation for feature fusion. The vision encoder extracts features from RGB images, which are mapped into the large language model and aligned with features of other modalities, and the attention mechanism is used to enhance task-related features.

[0118] As shown in Figure 8As shown, the visual encoder can select pre-trained DINOv2 and SigLIP, DINOv2 is used to extract low-level spatial information, and SigLIP is used to extract high-level semantic information. The input image is encoded by two sub-modules, the output features are spliced and embedded in the input space of the language model in a channel-wise manner, and visual features containing low-level spatial information and high-level semantic information can be obtained, which is beneficial to the model to better understand the physical environment.

[0119] In addition, considering the problems of poor acquisition quality or low acquisition efficiency of sensors in actual application, another optimization direction is to use a monocular depth estimation base model as a visual encoder to directly predict the depth information of the object, instead of the depth map or laser point cloud collected by the sensor, to help the embodied intelligent large model better understand the 3D spatial relationship, avoid perspective misjudgment caused by only using RGB images, and reduce the deviation of action prediction.

[0120] (2) Action generation

[0121] This module parses the continuous action sequence from the fusion features output by the model to control the end effector to perform actions. A diffusion model-based strategy can be used to generate natural, smooth and effective action sequences, and high-robustness actions can be obtained through iterative optimization. Compared with traditional methods, the diffusion model-based strategy can model various styles of action distribution and perform outstandingly in terms of temporal action consistency, which is suitable for high-dimensional action generation in robot tasks and provides a guarantee for generating natural, smooth and accurate actions in unstructured environments.

[0122] For example, the basic perception and reasoning ability of the embodied intelligent large model can be pre-trained on general scene data, and a self-collected dataset can be constructed to fine-tune the embodied intelligent large model to adapt to specific industrial scene tasks.

[0123] The specific fine-tuning strategy can include full-parameter fine-tuning, joint main body training only part of the adaptive layer, introduction of lightweight modules, and control layer or effector part can insert expert rules or safety boundaries according to process requirements, etc.

[0124] For fine-tuning of the embodied intelligent large model for industrial scenes, a composite fine-tuning method can be designed to cope with different parameter updating methods for different modules. For example, LoRA and other parameter efficient fine-tuning methods are used for text encoder, visual encoder, etc., and only a small number of parameters are updated; and the action decoder is retrained to update all parameters to adapt to downstream tasks. In addition, a simulation-measurement comparison verification mechanism can be introduced to improve task migration and action robustness in complex real working conditions. Thus, the perception and execution capabilities of the model are further improved, and the adaptability of the robot to complex working conditions is significantly improved.

[0125] It should be understood that although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0126] Based on the same inventive concept, the embodiments of the present application also provide a multi-modal-based embodied robot control device for implementing the multi-modal-based embodied robot control method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-modal-based embodied robot control device embodiments provided below can refer to the limitations of the multi-modal-based embodied robot control method described above, which will not be repeated here.

[0127] In one exemplary embodiment, as shown in Figure 9 A multi-modal-based embodied robot control device is provided, comprising: a data acquisition module 902, a multi-modal fusion module 904, a task determination module 906, and a control module 908, wherein:

[0128] The data acquisition module 902 is configured to execute the acquisition of a task instruction, and acquire multi-modal environment data based on the task instruction; the task instruction is used to instruct the embodied robot to execute a target task;

[0129] The multi-modal fusion module 904 is configured to execute fusion processing on the multi-modal environment data to obtain fused multi-modal perception information;

[0130] The task determination module 906 is configured to execute determination of a task path and a task action sequence for the target task based on the multi-modal perception information and the self-state of the embodied robot;

[0131] The control module 908 is configured to execute conversion of the task path and the task action sequence into a control instruction for the embodied robot, and control the embodied robot to execute the target task according to the control instruction.

[0132] In an example embodiment, the multi-modal environment data comprises at least image data and point cloud data; the multi-modal fusion module is further configured to perform: feature extraction on the point cloud data to obtain voxelized point cloud features; feature extraction on the image data to obtain image feature maps; projecting the voxelized point cloud features into a feature domain of the image data to obtain point cloud projected feature maps aligned with the image feature maps; performing enhanced fusion processing on the image feature maps and the point cloud projected feature maps to obtain the multi-modal perception information after fusion.

[0133] In an example embodiment, the multi-modal fusion module is further configured to perform: point-wise multiplication on the image feature maps and the point cloud projected feature maps to obtain a point-wise multiplication result; processing the point-wise multiplication result through two independent two-dimensional convolution layers to obtain a point cloud confidence weight and an image confidence weight; performing weighted concatenation on the image feature maps and the point cloud projected feature maps based on the point cloud confidence weight and the image confidence weight to obtain the multi-modal perception information.

[0134] In an example embodiment, the multi-modal environment data comprises at least image data and point cloud data; the multi-modal fusion module is further configured to perform: feature extraction on the point cloud data to obtain voxel features; feature extraction on the image data to obtain image feature maps; determining, for each non-empty voxel in the voxel features, a centroid coordinate of the voxel based on centering; projecting the centroid coordinate into a feature domain of the image data to obtain a reference point of the centroid of the voxel on the image feature maps; sampling image features from the image feature maps based on a sampling offset and the reference point to obtain a plurality of sampled image features; determining an attention weight of each of the image features, and performing weighted aggregation on the plurality of image features according to the attention weight of each of the image features to obtain a fusion feature, wherein the fusion feature is taken as the multi-modal perception information.

[0135] In an example embodiment, the multi-modal fusion module is further configured to perform: for each of the image features, generating a corresponding attention weight using a multi-layer perception machine.

[0136] In an example embodiment, the task determination module is further configured to perform: identifying a key target in an environment according to the multi-modal perception information; determining a sub-task for the key target based on the target task and the identified key target; decomposing the sub-task into a series of executable task action sequences according to a self-state of the embodied robot, wherein the task action sequences comprise a plurality of actions; and generating a corresponding task path based on the plurality of actions in the task action sequences.

[0137] The various modules in the above multi-modal embodiment-based embodied robot control device can be implemented in whole or in part by software, hardware, and combinations thereof. The various modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be invoked by the processor to perform the operations corresponding to the various modules.

[0138] In an example embodiment, an electronic device is provided, an internal structure diagram of which can be as shown in Figure 10 The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is configured to exchange information between the processor and external devices. The communication interface of the electronic device is configured to perform wired or wireless communication with external terminals, and the wireless communication can be implemented through WIFI, mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement a multi-modal embodiment-based embodied robot control method. The display unit of the electronic device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0139] Those skilled in the art can understand that Figure 10 The structure shown in the above

[0140] In an example embodiment, an electronic device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0141] In an embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of any of the above method embodiments.

[0142] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0143] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above-mentioned embodiments. Any reference to a memory, a database or other medium used in the embodiments provided in the present application can include at least one of a non-volatile memory and a volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., and is not limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., and is not limited thereto.

[0144] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0145] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A multi-modal based embodied robot control method, characterized by, The method comprises: acquiring a task instruction and acquiring multi-modal environment data based on the task instruction; the task instruction is used to instruct the embodied robot to perform a target task; the multi-modal environment data at least comprises image data and point cloud data; performing fusion processing on the multi-modal environment data to obtain fused multi-modal perception information, which comprises: performing feature extraction on the point cloud data to obtain voxelized point cloud features; and performing feature extraction on the image data to obtain an image feature map; projecting the voxelized point cloud features into a feature domain of the image data to obtain a point cloud projection feature map aligned with the image feature map; performing point multiplication on the image feature map and the point cloud projection feature map to obtain a point multiplication result; processing the point multiplication result through two independent two-dimensional convolution layers respectively to obtain a point cloud credibility weight and an image credibility weight; performing weighted splicing on the image feature map and the point cloud projection feature map based on the point cloud credibility weight and the image credibility weight to obtain the multi-modal perception information; identifying a key target in an environment according to the multi-modal perception information; determining a subtask for the key target based on the target task and the identified key target; decomposing the subtask into a series of executable task action sequences according to a self state of the embodied robot, wherein the task action sequences comprise a plurality of actions; generating a corresponding task path based on the plurality of actions in the task action sequences; the self state comprises but is not limited to a pose of the embodied robot, a robot position, a mechanical arm posture, and a gripper state; converting the task path and the task action sequences into a control instruction for the embodied robot, and controlling the embodied robot to perform the target task according to the control instruction.

2. A multi-modal based embodied robot control device, comprising: The device comprises: a data acquisition module configured to perform acquiring a task instruction and acquiring multi-modal environment data based on the task instruction; the task instruction is used to instruct the embodied robot to perform a target task; the multi-modal environment data at least comprises image data and point cloud data; performing fusion processing on the multi-modal environment data to obtain fused multi-modal perception information, which comprises: a multi-modal fusion module configured to perform feature extraction on the point cloud data to obtain voxelized point cloud features; and performing feature extraction on the image data to obtain an image feature map; projecting the voxelized point cloud features into a feature domain of the image data to obtain a point cloud projection feature map aligned with the image feature map; performing point multiplication on the image feature map and the point cloud projection feature map to obtain a point multiplication result; processing the point multiplication result through two independent two-dimensional convolution layers respectively to obtain a point cloud credibility weight and an image credibility weight; performing weighted splicing on the image feature map and the point cloud projection feature map based on the point cloud credibility weight and the image credibility weight to obtain the multi-modal perception information; a task determination module configured to perform identifying a key target in an environment according to the multi-modal perception information; determine a sub-task for the key target based on the target task and the identified key target; decompose the sub-task into a series of executable task action sequences according to a self-state of the embodied robot, the task action sequences including a plurality of actions; generate a corresponding task path based on the plurality of actions in the task action sequences; the self-state includes but is not limited to a pose of the embodied robot, a robot position, a mechanical arm posture, a gripper state; a control module configured to execute conversion of the task path and the task action sequences into control instructions for the embodied robot, and control the embodied robot to execute the target task according to the control instructions. 3.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to realize the steps of the method of claim 1.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of claim 1.

5. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of claim 1. The computer program is executed by the processor to realize the steps of the method of claim 1.

Images