A robot control method, system, electronic device and storage medium

Abstract

The present application discloses a robot control method, system, electronic device and storage medium, and the technical field to which it belongs is artificial intelligence technology. The robot control method comprises: receiving a robot control instruction, and extracting semantic features in the robot control instruction; obtaining a color image of the robot's environment, performing image segmentation on the color image to obtain a mask corresponding to the image content, and determining the semantic category of the mask; fusing the mask, the semantic category of the mask and the semantic feature to obtain a fusion feature; inputting the fusion feature and the preset feature into a multimodal large model to obtain control information; controlling the robot according to the control information to complete the task corresponding to the robot control instruction. The present application can accurately identify user intentions and improve the flexibility and accuracy of robot control.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a robot control method, system, electronic device and storage medium. Background Art

[0002] Robotics is an interdisciplinary science that involves multiple fields such as mechanical engineering, electronic engineering, computer science, and artificial intelligence.

[0003] In related technologies, robots can perform corresponding operations according to control instructions input by users. However, when performing complex operations, the robot's understanding of human language instructions is often biased; and because the external environment and the robot's internal state change in real time, the robot lacks the ability to adjust flexibly.

[0004] Therefore, how to accurately identify user intentions and improve the flexibility and accuracy of robot control is a technical problem that technicians in this field currently need to solve. Summary of the invention

[0005] The purpose of this application is to provide a robot control method, system, electronic device and storage medium that can accurately identify user intentions and improve the flexibility and accuracy of robot control.

[0006] In order to solve the above technical problems, the present application provides a robot control method, comprising:

[0007] Receiving robot control instructions and extracting semantic features in the robot control instructions;

[0008] Acquire a color image of the environment in which the robot is located, perform image segmentation on the color image to obtain a mask corresponding to the image content, and determine the semantic category of the mask;

[0009] fusing the mask, the semantic category of the mask, and the semantic feature to obtain a fused feature;

[0010] Inputting the fused features and the preset features into the multimodal large model to obtain control information; wherein the preset features include at least one of the semantic features, the color image features of the color image of the robot's environment, the depth image features of the depth image of the robot's environment, and the robot state features;

[0011] The robot is controlled according to the control information so as to complete the task corresponding to the robot control instruction.

[0012] Optionally, the mask, the semantic category of the mask and the semantic feature are fused to obtain a fused feature, including:

[0013] Inputting the mask into a mask feature extractor to obtain mask features;

[0014] Inputting the semantic category of the mask into the feature extractor of the large language model to obtain a label feature;

[0015] Constructing a joint feature including the mask feature and the label feature, and adding a corresponding position code to the joint feature;

[0016] The joint feature including the position code is fused with the semantic feature to obtain the fused feature.

[0017] Optionally, constructing a joint feature including the mask feature and the label feature, and adding a corresponding position code to the joint feature, including:

[0018] The mask feature and the label feature corresponding to each of the masks are concatenated to obtain the joint feature;

[0019] A corresponding position code is added to the joint feature according to the position of the mask in the color image.

[0020] Optionally, the joint feature including the position code is fused with the semantic feature to obtain the fused feature, including:

[0021] Processing the joint feature including the position encoding and the semantic feature based on a multi-head cross attention mechanism to obtain an intermediate processing result;

[0022] The intermediate processing result is processed using the fully connected layer, the pooling layer and the normalization layer of the semantic fusion model to obtain the fusion feature.

[0023] Optionally, the color image feature is obtained in the following manner:

[0024] The color image coding model is used to encode and extract the color image of the robot's environment to obtain the color image features;

[0025] The depth image features are obtained in the following manner:

[0026] The depth image encoding model is used to encode and extract the depth image of the robot's environment to obtain the depth image features;

[0027] The robot status characteristics are obtained in the following manner:

[0028] The robot state encoding model is used to encode and extract the state information of the robot to obtain the robot state characteristics; wherein the state information includes at least one of the joint angle, the end effector state and the positioning information.

[0029] Optionally, the color image of the environment in which the robot is located includes a first color image taken by a first camera and a second color image taken by a second camera; wherein the first camera is arranged on the robot, and the second camera is arranged in the field in which the robot is located;

[0030] The color image coding model is used to encode and extract the color image of the robot's environment to obtain color image features, including:

[0031] The first color image and the second color image are coded and extracted using a color image coding model to obtain color image features.

[0032] Optionally, the depth image of the environment in which the robot is located includes a first depth image taken by the first camera and a second depth image taken by the second camera;

[0033] The depth image encoding model is used to encode and extract the depth image of the robot's environment to obtain the depth image features, including:

[0034] The first depth image and the second depth image are encoded and extracted using a depth image encoding model to obtain depth image features.

[0035] The present application also provides a robot control system, the system comprising:

[0036] An instruction processing module, used for receiving robot control instructions and extracting semantic features in the robot control instructions;

[0037] An image processing module, used to obtain a color image of the environment in which the robot is located, perform image segmentation on the color image to obtain a mask corresponding to the image content, and determine the semantic category of the mask;

[0038] A feature fusion module, used for fusing the mask, the semantic category of the mask and the semantic feature to obtain a fused feature;

[0039] A feature processing module, used for inputting the fusion feature and the preset feature into the multimodal large model to obtain control information; wherein the preset feature includes at least one of the semantic feature, the color image feature of the color image of the robot's environment, the depth image feature of the depth image of the robot's environment, and the robot state feature;

[0040] The execution module is used to control the robot according to the control information so as to complete the task corresponding to the robot control instruction.

[0041] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned robot control method when calling the computer program in the memory.

[0042] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps of the robot control method are implemented.

[0043] The present application provides a robot control method, which extracts semantic features from robot control instructions, segments the color image of the environment to generate a corresponding mask, and determines the semantic category of the mask; fuses the mask, the semantic category of the mask and the semantic feature to generate a fusion feature, so as to associate the robot control instruction with the specific object in the environment; inputs the fusion feature and the preset feature into a multimodal large model to obtain control information to complete the task; the above-mentioned preset feature includes at least one of a semantic feature, a color image feature, a depth image feature and a robot state feature. The present application inputs the fusion feature and the preset feature into the multimodal large model, so that the multimodal large model captures the correlation between each feature, and generates control information by integrating these features. Therefore, the present application can accurately identify the user's intention and improve the flexibility and accuracy of the robot control. The present application also provides a robot control system, an electronic device and a storage medium, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a robot control method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the workflow of a large model framework of embodied intelligent operation based on multimodal semantic fusion provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the principle of a large model framework of embodied intelligent operation based on multimodal semantic fusion provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a mask feature generation principle provided in an embodiment of the present application;

[0049] Figure 5A label feature generation principle diagram provided in an embodiment of the present application;

[0050] Figure 6 A structural diagram of a semantic fusion model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] See below Figure 1 , Figure 1 A flowchart of a robot control method provided in an embodiment of the present application.

[0053] Specific steps may include:

[0054] S101: receiving a robot control instruction, and extracting semantic features from the robot control instruction.

[0055] Among them, this embodiment can be applied to robots such as embodied robots, handling robots, and sweeping robots. The above-mentioned robot control instructions are information containing natural language, that is, language instructions. The user can issue robot control instructions by voice or text, for example: "Please take the book on the table to the sofa", and the robot can capture the user's robot control instructions through a microphone or input device, and convert the robot control instructions into digital signals or text form. The above-mentioned robot control instructions can also be instructions transmitted by other electronic devices (such as smart speakers, smart phones, etc.). In this embodiment, the above-mentioned robot control instructions can be processed by natural language to obtain semantic features. The semantic features can be vectors, which contain the main content and intention of the robot control instructions, such as action type, object category, location information, etc. This step can use a large language model to extract semantic features in the robot control instructions.

[0056] S102: Acquire a color image of the environment in which the robot is located, perform image segmentation on the color image to obtain a mask corresponding to the image content, and determine a semantic category of the mask.

[0057] Among them, in this embodiment, a camera can be set on the robot, or a camera can be set in the environment where the robot is located, so as to obtain a color image of the environment where the robot is located. The above-mentioned camera can be a depth camera. In this step, the color image can be segmented to obtain a mask corresponding to the image content, and each mask has its corresponding semantic category. The semantic category is used to describe the classification of the content corresponding to each mask in the color image, such as cars, pedestrians, trees, houses, etc. In this embodiment, a segmentation model (i.e., an image segmentation model) can be used to obtain a mask corresponding to the image content.

[0058] S103: Fusing the mask, the semantic category of the mask, and the semantic feature to obtain a fused feature.

[0059] As a feasible implementation method, this embodiment can use feature splicing to fuse masks, semantic categories and semantic features; as another feasible implementation method, this embodiment can use a feature fusion network to fuse masks, semantic categories and semantic features.

[0060] The fused features obtained in this step can associate user instructions with specific objects in the environment so that the robot can perform tasks more accurately.

[0061] S104: Input the fused features and preset features into a multimodal large model to obtain control information.

[0062] The preset features include at least one of semantic features in robot control instructions, color image features of a color image of the robot's environment, depth image features of a depth image of the robot's environment, and robot state features.

[0063] In this step, semantic features and preset features can be input into a multimodal large model. The multimodal large model can comprehensively process the features of different modalities to capture the complex relationship between these features. The multimodal large model can fuse the features of different modalities through a multi-layer neural network, and then use the control information generated by the output layer. The above control information can be a specific action sequence, such as moving, grasping, navigating, etc., or a higher-level strategy, such as path planning and task scheduling.

[0064] Through the above operations, the multimodal large model can generate accurate control information based on features from multiple sources, ensuring that the robot can complete the tasks corresponding to the robot control instructions efficiently and accurately.

[0065] S105: Control the robot according to the control information so as to complete the task corresponding to the robot control instruction.

[0066] This embodiment extracts semantic features from the robot control instructions, segments the color image of the environment to generate a corresponding mask, and determines the semantic category of the mask; fuses the mask, the semantic category of the mask, and the semantic features to generate fused features so as to associate the robot control instructions with specific objects in the environment; inputs the fused features and the preset features into the multimodal large model to obtain control information so as to complete the task; the preset features include at least one of the semantic features, color image features, depth image features, and robot state features. This embodiment inputs the fused features and the preset features into the multimodal large model so that the multimodal large model captures the correlation between the features and generates control information by integrating these features. Therefore, this embodiment can accurately identify the user's intentions and improve the flexibility and accuracy of robot control.

[0067] As for Figure 1 For further description of the corresponding embodiment, the fusion feature can be generated in the following manner:

[0068] Step A1: input the mask into a mask feature extractor to obtain mask features.

[0069] Step A2: Input the semantic category of the mask into the feature extractor of the large language model to obtain label features.

[0070] Step A3: construct a joint feature including the mask feature and the label feature, and add a corresponding position code to the joint feature.

[0071] In this embodiment, the mask feature and the label feature corresponding to each mask may be concatenated to obtain the joint feature, and a corresponding position code may be added to the joint feature according to the position of the mask in the color image.

[0072] Step A4: Fusing the joint feature including the position code with the semantic feature to obtain the fused feature.

[0073] Specifically, this embodiment can process the joint features containing the position encoding and the semantic features based on the multi-head cross-attention mechanism to obtain an intermediate processing result; use the fully connected layer, pooling layer and normalization layer of the semantic fusion model to process the intermediate processing result to obtain the fusion feature.

[0074] As for Figure 1 Further introduction to the corresponding embodiment: if the preset features include color image features, then before the preset features are input into the multimodal large model, the color image features can be obtained in the following manner: a color image coding model is used to encode and extract the color image of the robot's environment to obtain the color image features.

[0075] If the preset features include depth image features, then before the preset features are input into the multimodal large model, the depth image features can be obtained in the following manner: a depth image encoding model is used to encode and extract the depth image of the robot's environment to obtain the depth image features.

[0076] If the preset features include robot state features, before the preset features are input into the multimodal large model, the robot state features can be obtained in the following manner: the robot state information is encoded and extracted using a robot state encoding model to obtain robot state features; wherein the state information includes at least one of joint angles, end effector states, positioning information, acceleration, battery power, and sensor readings (such as temperature, humidity, light intensity, etc.). This embodiment can preprocess the acquired state information, such as standardizing or normalizing the numerical range to ensure data consistency and comparability. This embodiment can encode the preprocessed state information to obtain robot state features. This embodiment can use a robot state encoding model to extract the above robot state features.

[0077] As for Figure 1 In the further description of the corresponding embodiment, the robot is provided with a first camera, and a second camera is provided in the field where the robot is located. The second camera is also called a third-view camera, which is a camera independent of the robot and does not move with the movement of the robot. If the above robot is a robot with a mechanical arm, the first camera can be provided at the end of the mechanical arm of the robot to effectively obtain the environmental information of the robot.

[0078] The color image of the environment in which the robot is located includes: a first color image taken by a first camera and a second color image taken by a second camera. Correspondingly, the process of extracting color image features using a color image coding model includes: encoding and extracting the first color image and the second color image using the color image coding model to obtain color image features.

[0079] The depth image of the environment in which the robot is located includes: a first depth image taken by a first camera and a second depth image taken by a second camera. Correspondingly, the process of extracting depth image features using a depth image coding model includes: encoding and extracting the first depth image and the second depth image using the depth image coding model to obtain depth image features.

[0080] The process described in the above embodiment is explained below through an embodiment in actual application.

[0081] Although with the rapid development of large models in recent years, some methods have attempted to improve the robot's ability to understand and execute tasks by directly fusing language instructions with visual signals in large models. However, these methods do not explicitly provide an effective mechanism to combine language instructions with visual environment, resulting in insufficient information transfer in the fusion process. When performing complex operations, robots often have deviations in their understanding of human language instructions, which affects the improvement of the overall operation success rate. This phenomenon is particularly prominent in practical applications, because existing robot systems often cannot accurately understand or parse complex task descriptions from humans, and lack the ability to flexibly adjust when facing dynamic environments. In contrast, humans usually have stronger intention understanding and environmental perception capabilities when performing tasks. When humans receive instructions such as "grab an apple", they first analyze the key information in the instruction, that is, identify the object "apple" and quickly determine whether it exists in the current visual environment. Next, humans will adjust their actions based on the position and posture information of the object to complete the grasping task. This process not only involves the close combination of language information and visual information, but also includes real-time perception of the environment and adjustment of operation strategies. Obviously, humans demonstrate a highly intelligent perception and response mechanism when performing tasks, which is also a key point that existing robotics technology urgently needs to improve.

[0082] Related technologies focus on directly fusing multimodal information, but lack in-depth analysis and optimization strategies, and fail to effectively solve the problem of insufficient integration of language instructions and visual environment. This deficiency limits the robot's ability to understand complex instructions, especially when facing a diverse and dynamically changing environment, the accuracy and success rate of operations still need to be improved. To meet these challenges, more attention needs to be paid to how to more closely integrate language instructions with visual perception information, so that robots can understand the key elements of the task like humans, and make intelligent decisions and actions based on the perception of the environment. This requires not only more efficient multimodal fusion algorithms, but also further optimization of the robot's learning and reasoning mechanisms to ensure that it can accurately respond to and execute human instructions in the complex real world, thereby improving the efficiency and accuracy of the overall operation. Therefore, how to effectively combine language and visual information so that robots can understand human intentions more deeply and achieve intelligent operations through comprehensive perception of the environment is still a problem that needs to be solved in the future development of robotics technology.

[0083] Based on the above technical problems, this application provides a large model framework of embodied intelligent operation based on multimodal semantic fusion. This embodiment further enhances the robot's ability to understand and execute complex tasks by deeply integrating visual and language command information, thereby significantly improving the accuracy of operations. The system uses data from multiple modalities, including but not limited to visual perception information, language instructions, and environmental perception information, to help the model fully and meticulously understand the external environment and human intentions, and provide precise guidance for robot operations.

[0084] First, this embodiment uses a large language model (LLM) to perform in-depth intent analysis on language instructions input by humans, and converts the analysis results into a feature form acceptable to the multimodal large model, so that the robot can more accurately understand and execute language instructions from humans. This conversion process not only complements language information and visual information, but also enhances the robot's ability to respond to instructions in complex task scenarios. At the same time, this embodiment also uses a depth camera to obtain RGB images and depth information in the environment, and introduces a segmentation model to perform detailed positioning and segmentation of all objects in the image, thereby generating object categories and corresponding mask outputs.

[0085] The depth camera in this embodiment includes a hand-eye camera and a third-view camera. The hand-eye camera is installed 3 cm from the end of the robotic arm, which can observe the dynamics of the gripper at the end of the robotic arm and its surrounding environment in real time to ensure accurate operation execution; while the third-view camera is placed in front of the robotic arm, which can overlook the entire working environment from a more macro perspective, covering the robotic arm and its operating space. Through the combination of these two perspectives, this embodiment can obtain more comprehensive environmental perception information, thereby supporting the robot to make more accurate judgments and operations.

[0086] This embodiment uses a semantic fusion model to deeply fuse the object mask results obtained by instance segmentation with the language instruction features generated by the large language model, thereby generating multimodal information after semantic integration. The semantic fusion model combines the cross-attention mechanism to perform detailed fusion and attention extraction of language and visual information, ensuring that language instructions and visual perception information complement and reinforce each other at multiple levels, and obtain operation instructions that the robot can execute.

[0087] In terms of image information, the RGB (red, green, and blue) images and depth images taken by the hand-eye camera and the third-person camera will be respectively subjected to feature extraction through the image coding model. Different types of images (RGB images and depth images) are processed by different encoders to meet the requirements of different input dimensions.

[0088] After all feature extraction is completed, this embodiment can input these multimodal data from vision, language and depth sensing into the multimodal large model, which is responsible for the final feature fusion and decision output. The multimodal large model can output various types of robot control information. These instructions include the robot's joint state at a future moment, the terminal posture, and the status signal of whether the robot has completed the task. Among them, the future joint state of the robot covers the angle value of each joint and the state of the gripper, and the terminal posture is represented by a six-dimensional vector, which specifically describes the translation and rotation information of the robot's terminal in the three directions of x, y, and z. By combining this information, the robot can perform more accurate and efficient operations in complex and changing environments.

[0089] See also Figure 2 , Figure 2 A schematic diagram of the workflow of an embodied intelligent operation big model framework based on multimodal semantic fusion provided in the embodiment of the present application is provided to realize the robot's comprehensive perception and timely response to the environment, language instructions and its own status. The above-mentioned embodied intelligent operation big model framework includes a multi-sensor system, a multi-modal encoding module, a multi-modal fusion module and a robot control execution module. The multi-sensor system can realize 2D (dimensionality) environmental perception, 3D environmental perception, language perception and current state perception.

[0090] First, the external environment and language instructions are comprehensively perceived through a multi-sensor system. Among them, the 2D and 3D information of the environment is obtained through a depth camera, and the size of the color image is the same as that of the depth image. Through this 2D and 3D environmental perception method, the visual information of the surrounding environment can be fully captured, ensuring that the robot can obtain rich environmental data in real time. The language perception module is responsible for processing the input text instructions and converting the task description provided by humans into language features that the system can understand. These text instructions usually contain a clear description of the operation task, such as object recognition, position indication or operation requirements. In addition, the current state information of the robot is also perceived by the system at this stage, mainly collected through the current state perception module, which is directly connected to the control interface of the robot and can obtain the key state parameters of the robot when performing the task in real time. These state information includes but is not limited to the robot's joint angle, end effector position, open or closed state of the gripper, etc. These data effectively assist the robot in performing precise operations, and they provide real-time feedback on the specific posture and operation ability of the robot in the environment. In this embodiment, all perceived information can be uniformly input into the multimodal encoding module for processing. The multimodal encoding module can extract features from data from different sources and convert visual, language, and robot state information into high-dimensional feature vectors. Through this feature extraction, the system can uniformly represent heterogeneous data (such as images, text, state parameters, etc.) into a feature space that can be used for subsequent processing, providing a basis for multimodal fusion. Once all features are extracted, they will be input into the multimodal fusion module. In the multimodal fusion module, the system adopts a multi-level feature fusion strategy to combine different modal features from vision, language, and state perception to generate a comprehensive fusion feature. This fusion process not only aligns and jointly processes information from different modalities, but also ensures that the system can capture the most critical details for task execution in each modality through hierarchical feature extraction and attention mechanisms. The multimodal fusion module will generate a set of specific robot execution instructions that can accurately guide the robot to complete the predetermined task. After the robot executes the instructions, the system will re-acquire the information of the current environment, update the 2D and 3D visual perception data, and obtain the new state of the robot at the same time. This closed-loop process enables the system to dynamically adjust according to changes in the environment, ensuring that the robot can always make the best decision based on the latest perception information when performing continuous tasks. Through this iterative cycle of continuous perception-execution-reperception, the system can effectively improve the robot's operating efficiency and accuracy, enabling it to flexibly respond to and complete tasks in complex and changing environments.

[0091] See also Figure 3 , Figure 3A schematic diagram of the principle of a large model framework of embodied intelligent operation based on multimodal semantic fusion provided by an embodiment of the present application, showing the process of acquiring, processing and fusing multimodal information. The input of the framework includes human language instructions, RGB images taken by a hand-eye camera, RGB images taken by a third-view camera, depth images taken by a hand-eye camera, depth images taken by a third-view camera, and robot status. The models used in the above framework include: a large language model, an instance segmentation model, a semantic fusion model, an RGB image encoding model, a depth image encoding model, a robot state encoding model, and a multimodal large model. The output of the above framework includes the robot's future state, the robot's terminal posture, and the detection result of whether the robot has completed the task.

[0092] In this embodiment, human language instructions can be obtained through the text input module, and the instructions are input into the large language model for extracting language features. In this process, the large language model will perform deep semantic analysis on the input text and generate corresponding feature vectors, and the feature dimension is the preset dimension value n. These language features lay the foundation for subsequent multimodal fusion, ensuring that the system can understand and process complex instruction information from humans.

[0093] In terms of visual perception, this embodiment uses two depth cameras, which are located at different positions, to capture RGB images and depth data in the environment. First, the hand-eye camera is installed 3 cm above the end of the robot arm. Its unique position ensures that it can monitor the operation of the end effector of the robot arm and the detailed changes of its surrounding environment in real time; secondly, the third-view camera is set up in the upper position directly in front of the robot, which can overlook the entire working environment, including the robot arm and its operating space. The perspectives of the two cameras complement each other, ensuring that the system can fully perceive the detailed changes in the robot's operating environment from multiple dimensions. Next, the RGB images captured by these cameras will be input into the segmentation model for image segmentation prediction. Through the segmentation model, this embodiment can generate corresponding masks according to the image content, and each mask corresponds to a specific semantic category. These masks and their category information are further passed to the semantic fusion model for deep fusion with the language features extracted from the large language model. The dimension of the fused feature vector is the preset dimension value n. This step of multimodal feature fusion ensures the close integration of language instructions and visual perception information, providing more comprehensive input for subsequent robot control decisions.

[0094] This embodiment can also perform separate feature extraction on RGB images. Specifically, the RGB images of the hand-eye camera and the third-view camera will be input into the image coding model respectively, and the image will be encoded and extracted through the model to generate image feature vectors. This embodiment can input these image feature vectors into the fully connected layer for further processing and convert them into n-dimensional feature vectors to align with other feature dimensions. Similarly, the depth images obtained by the two depth cameras will also be processed by the image coding model and the fully connected layer to finally generate an n-dimensional depth feature vector. This process ensures that the system can make full use of visual and depth information to support robot operation tasks.

[0095] This embodiment can also extract the robot's own state information. The current robot state can be obtained through a special robot state encoding model, which includes two fully connected layers, and can encode key parameters such as robot joint angles and end effector states to generate a feature vector of length n. Through this state encoding process, the system can understand the robot's posture and state in real time, providing accurate feedback basis for subsequent action decisions.

[0096] All the above-mentioned extracted feature vectors, including language features, visual features, depth features and robot state features, will be input into the multimodal large model in a predetermined order. The predetermined order is semantic features, fusion features, color image features, depth image features and robot state features. As the core processing module, the multimodal large model generates the final feature instruction vector through multi-feature fusion and deep learning. The feature instruction vector is then processed by a fully connected layer to generate the output result of the system. These results include three aspects: first, the state information of the robot at the future moment, specifically the angle of all joints of the robot in the next operation and the state of the gripper, to ensure the continuity and coordination of the robot when performing complex tasks; second, the posture information of the robot end, including its precise position and posture in the operation space, which is crucial to improving the operation accuracy; finally, the system will output a signal about whether the task is completed or not, indicating whether the robot has completed the current task. If completed, the system will stop subsequent actions, thereby improving the efficiency of task execution.

[0097] See also Figure 4 , Figure 4A mask feature generation principle diagram provided in an embodiment of the present application, after inputting mask 1~k into the mask feature extractor, mask features 1~k can be obtained. In this embodiment, the mask output by the segmentation model can be input into the mask feature extractor, and the mask feature extractor can perform deep feature extraction on the mask corresponding to each semantic category. This feature extraction process can effectively capture the spatial structure and detail information of each segmented object in the image, ensure that the mask can accurately reflect the shape and position of the object in the scene, and provide a basis for subsequent feature fusion.

[0098] See also Figure 5 , Figure 5 A label feature generation principle diagram provided in an embodiment of the present application, after inputting mask labels 1~k into the large language model feature extractor, label features 1~k can be obtained. In this embodiment, the feature extractor of the large language model can further process the category label corresponding to each mask. The feature extractor of the large language model can perform semantic analysis on each category label and extract its semantic features. These semantic features reflect the high-level information contained in the category label, such as the function, attributes and association of an object with other categories. The extracted label features are processed by dimensionality reduction to finally obtain a feature vector of the corresponding dimension.

[0099] See also Figure 6 , Figure 6A structural diagram of a semantic fusion model provided in an embodiment of the present application, the semantic fusion model includes a multi-head cross attention structure, a multi-layer perceptron (MLP, Multilayer perceptron) layer, a pooling layer, a normalization layer, and the input parameters include mask features 1~k, label features 1~k, position codes 1~k, human instruction features (i.e., semantic features in human language instructions), and the output parameter is a fusion feature. After the feature extraction is completed, the mask feature can be concatenated with the label feature in this embodiment. This connection operation combines the information of two different modalities to form a joint feature vector, which contains both the visual information of the object and its corresponding semantic information. In this embodiment, the corresponding position code can be added to each joint feature vector. These joint features with position codes are input into the multi-head cross attention structure. In the multi-head cross attention mechanism, the system fuses the mask features, label features and human instruction features previously extracted from the language input through multi-layer cross attention operations. The cross attention mechanism can capture long-range dependencies between features to ensure that the system can understand the relationship between language instructions and different objects in the scene. For example, when a human instruction asks the robot to "grab an apple", the cross-attention mechanism can effectively match and associate the language instruction "apple" with the corresponding object features in the visual mask, thereby generating a feature vector that can guide the robot's operation. The above method ensures a high degree of integration of semantic information and visual information through multi-level feature extraction and fusion to support the robot's understanding and execution of complex tasks. After completing the cross-attention fusion, this embodiment can input the fused feature vector into a series of post-processing modules, including a fully connected layer, a pooling layer, and a normalization layer. The fully connected layer further adjusts and compresses the dimension of the feature to ensure that the fused information can be efficiently transmitted and processed; the pooling layer extracts the most important features through downsampling operations to reduce redundant information; the normalization layer standardizes the features to make the feature distribution more balanced and avoid deviations in subsequent operations. Finally, this embodiment generates a semantic fusion feature with a dimension of n. The feature vector not only contains the spatial details and category information in the visual information, but also integrates the instruction intention in the language input, which can provide a comprehensive task execution reference for the robot.

[0100] The multimodal fusion framework proposed in this embodiment improves the intelligence of the robot in the process of task execution and improves the interaction between the robot and humans to a certain extent. Through this deep fusion method, the robot's ability to understand and respond to human instructions is enhanced, and the accuracy of performing operational tasks is improved. This embodiment improves the reliability and accuracy of the robot when performing tasks, strengthens the ability to understand the needs in complex application scenarios, makes the collaboration between humans and robots more efficient, and optimizes the user experience.

[0101] The structure of a robot control system provided in an embodiment of the present application includes:

[0102] An instruction processing module, used for receiving robot control instructions and extracting semantic features in the robot control instructions;

[0103] An image processing module, used to obtain a color image of the environment in which the robot is located, perform image segmentation on the color image to obtain a mask corresponding to the image content, and determine the semantic category of the mask;

[0104] A feature fusion module, used for fusing the mask, the semantic category of the mask and the semantic feature to obtain a fused feature;

[0105] A feature processing module, used for inputting the fusion feature and the preset feature into the multimodal large model to obtain control information; wherein the preset feature includes at least one of the semantic feature, the color image feature of the color image of the robot's environment, the depth image feature of the depth image of the robot's environment, and the robot state feature;

[0106] The execution module is used to control the robot according to the control information so as to complete the task corresponding to the robot control instruction.

[0107] Furthermore, the process in which the feature fusion module fuses the mask, the semantic category of the mask and the semantic feature to obtain a fused feature includes: inputting the mask into a mask feature extractor to obtain a mask feature; inputting the semantic category of the mask into a feature extractor of a large language model to obtain a label feature; constructing a joint feature including the mask feature and the label feature, and adding a corresponding position code to the joint feature; and fusing the joint feature including the position code with the semantic feature to obtain the fused feature.

[0108] Furthermore, the feature fusion module constructs a joint feature including the mask feature and the label feature, and the process of adding a corresponding position code to the joint feature includes: concatenating the mask feature and the label feature corresponding to each mask to obtain the joint feature; and adding a corresponding position code to the joint feature according to the position of the mask in the color image.

[0109] Furthermore, the feature fusion module fuses the joint features containing the position code with the semantic features, and the process of obtaining the fused features includes: processing the joint features containing the position code and the semantic features based on a multi-head cross-attention mechanism to obtain an intermediate processing result; and processing the intermediate processing result using the fully connected layer, pooling layer and normalization layer of the semantic fusion model to obtain the fused features.

[0110] Furthermore, the robot control system also includes:

[0111] A color image feature extraction module is used to encode and extract the color image of the robot's environment using a color image encoding model to obtain color image features;

[0112] A depth image feature extraction module is used to encode and extract the depth image of the robot's environment using a depth image encoding model to obtain depth image features;

[0113] The robot state feature extraction module is used to encode and extract the robot state information using the robot state encoding model to obtain the robot state feature; wherein the state information includes at least one of the joint angle, the end effector state and the positioning information.

[0114] Furthermore, the color image of the environment in which the robot is located includes a first color image taken by a first camera and a second color image taken by a second camera; wherein the first camera is arranged on the robot, and the second camera is arranged in the field in which the robot is located;

[0115] Correspondingly, the color image feature extraction module is used to perform encoding extraction on the first color image and the second color image using a color image encoding model to obtain color image features.

[0116] Furthermore, the depth image of the environment in which the robot is located includes a first depth image taken by the first camera and a second depth image taken by the second camera;

[0117] Correspondingly, the depth image feature extraction module is used to encode and extract the first depth image and the second depth image using a depth image encoding model to obtain depth image features.

[0118] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, which will not be repeated here.

[0119] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0120] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.

[0121] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of this application.

[0122] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A robot control method, characterized in that: include: Receive robot control instructions and obtain color images of the robot's environment; Extracting semantic features in the robot control instructions using a large language model; Performing image segmentation on the color image using an image segmentation model to obtain a mask corresponding to the image content, and determining a semantic category of the mask; Inputting the mask into a mask feature extractor to obtain mask features; Inputting the semantic category of the mask into the feature extractor of the large language model to obtain a label feature; The mask feature and the label feature corresponding to each of the masks are concatenated to obtain a joint feature; adding a corresponding position code to the joint feature according to the position of the mask in the color image; Processing the joint feature including the position encoding and the semantic feature based on a multi-head cross attention mechanism to obtain an intermediate processing result; The intermediate processing results are processed using a fully connected layer, a pooling layer, and a normalization layer of a semantic fusion model to obtain fusion features; Inputting the fused features and the preset features into the multimodal large model to obtain control information; wherein the preset features include the semantic features, the color image features of the color image of the robot's environment, the depth image features of the depth image of the robot's environment, and the robot state features; The robot is controlled according to the control information so as to complete the task corresponding to the robot control instruction.

2. The robot control method according to claim 1, characterized in that: The color image features are obtained in the following manner: The color image coding model is used to encode and extract the color image of the robot's environment to obtain the color image features; The depth image features are obtained in the following manner: The depth image encoding model is used to encode and extract the depth image of the robot's environment to obtain the depth image features; The robot status characteristics are obtained in the following manner: The robot state encoding model is used to encode and extract the state information of the robot to obtain the robot state characteristics; wherein the state information includes at least one of the joint angle, the end effector state and the positioning information.

3. The robot control method according to claim 2, characterized in that: A color image of the environment in which the robot is located, including a first color image taken by a first camera and a second color image taken by a second camera; wherein the first camera is set on the robot, and the second camera is set in the field where the robot is located; The color image coding model is used to encode and extract the color image of the robot's environment to obtain color image features, including: The first color image and the second color image are coded and extracted using a color image coding model to obtain color image features.

4. The robot control method according to claim 3, characterized in that: A depth image of the environment in which the robot is located, including a first depth image taken by the first camera and a second depth image taken by the second camera; The depth image encoding model is used to encode and extract the depth image of the robot's environment to obtain the depth image features, including: The first depth image and the second depth image are encoded and extracted using a depth image encoding model to obtain depth image features.

5. A robot control system, characterized in that: include: An instruction processing module, used for receiving robot control instructions and extracting semantic features in the robot control instructions using a large language model; An image processing module is used to obtain a color image of the environment in which the robot is located, perform image segmentation on the color image using an image segmentation model to obtain a mask corresponding to the image content, and determine the semantic category of the mask; A feature fusion module, used for fusing the mask, the semantic category of the mask and the semantic feature to obtain a fused feature; A feature processing module, used for inputting the fusion feature and the preset feature into the multimodal large model to obtain control information; wherein the preset feature includes at least one of the semantic feature, the color image feature of the color image of the robot's environment, the depth image feature of the depth image of the robot's environment, and the robot state feature; An execution module, used to control the robot according to the control information so as to complete the task corresponding to the robot control instruction; The process of generating the fusion feature by the feature fusion module includes: Inputting the mask into a mask feature extractor to obtain mask features; Inputting the semantic category of the mask into the feature extractor of the large language model to obtain a label feature; The mask feature and the label feature corresponding to each of the masks are concatenated to obtain a joint feature; adding a corresponding position code to the joint feature according to the position of the mask in the color image; Processing the joint feature including the position encoding and the semantic feature based on a multi-head cross attention mechanism to obtain an intermediate processing result; The intermediate processing result is processed using the fully connected layer, the pooling layer and the normalization layer of the semantic fusion model to obtain the fusion feature.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the robot control method according to any one of claims 1 to 4 when calling the computer program in the memory.

7. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the robot control method according to any one of claims 1 to 4 are implemented.

Images