Somatosensory robot control method and device based on multi-modal diffusion model

Abstract

The application provides a kind of embodied robot control method and device based on multi-modal diffusion model, it is related to the technical field of intelligent robot, including: obtaining instruction text input by user end, and image feature set collected by robot;Utilize single mode text encoder and multi-modal text encoder, instruction text is carried out feature extraction processing, respectively obtain single mode text feature and multi-modal text feature of instruction text, and by feature fusion processing to single mode text feature and multi-modal text feature, determine target text feature;Determine target image feature based on image feature set, and by default mode screening model, weight distribution and fusion processing are carried out to target text feature and target image feature, determine target image-text fusion feature.The application can significantly improve the accuracy of robot control.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent robots, and in particular to a control method and apparatus for an embodied robot based on a multimodal diffusion model. Background Technology

[0002] Currently, related technologies propose that embodied robots mainly rely on autoregressive algorithms combined with reinforcement learning for perception and decision-making. In addition, interaction schemes can be generated through 3D reconstruction and scene analysis to adjust the robot's behavior decisions. However, the control schemes of the aforementioned embodied robots lack scene generalization, and the accuracy of the models is low in training scenarios with few or no samples. Furthermore, the visual feature information acquired by the models in the aforementioned existing technical solutions is relatively broad, and the precision of their motion space prediction positions is poor, resulting in low accuracy of robot action control. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a embodied robot control method and device based on a multimodal diffusion model, which can significantly improve the accuracy of robot control.

[0004] In a first aspect, embodiments of the present invention provide a control method for an embodied robot based on a multimodal diffusion model. The method includes: acquiring instruction text input by a user terminal and an image feature set collected by the robot, wherein the image feature set includes image features collected by the robot's upper and left / right arms; using a unimodal text encoder and a multimodal text encoder to perform feature extraction processing on the instruction text, obtaining unimodal text features and multimodal text features of the instruction text respectively; and determining target text features by performing feature fusion processing on the unimodal text features and multimodal text features, wherein the unimodal text features include the core information of the instruction text, and the multimodal text features include text-image feature mapping information; determining target image features based on the image feature set, and performing weight allocation and fusion processing on the target text features and target image features through a preset modality screening model to determine target image-text fusion features, so that the embodied robot can interact with the environment based on the target image-text fusion features, and control the robot's actions at the next moment in combination with the robotic arm state information.

[0005] In one implementation, the target text feature includes: a first target text feature. The step of determining the target text feature by performing feature fusion processing on unimodal text features and multimodal text features includes: adding the unimodal text features and multimodal text features to determine the first text feature, and concatenating and processing the unimodal text features and multimodal text features to determine the second text feature; using a cross-attention model to perform deep fusion processing on the first text feature and the second text feature to determine the first target text feature, wherein the first target text feature is used to determine the position corresponding to the noun in the instruction text, so as to select the position using a feature box.

[0006] In one embodiment, the target text feature further includes: a second target text feature, which, after determining the first target text feature, includes: sending the instruction text to a preset text encoder to determine the second target text feature, wherein the second target text feature is used to represent the specific meaning of the instruction text.

[0007] In one embodiment, the step of determining target image features based on an image feature set includes: using an image encoding featurer of a multimodal model and a preset visual feature extractor to perform feature stitching processing on each image feature in the image feature set to determine a first image feature and a second image feature; and using a preset image feature fusion model to perform feature fusion processing on the first image feature and the second image feature to determine the target image feature.

[0008] In one embodiment, a preset image feature fusion model is used to perform feature fusion processing on a first image feature and a second image feature to determine a target image feature. This includes: segmenting the first image feature and the second image feature; inputting the first image feature into a multilayer perceptron to determine a query feature; and inputting the segmentation result of the second image feature into the multilayer perceptron to determine key features and value features. The query feature is used to represent the query matrix of the cross-attention model, and the key features and value features are used to represent the key matrix and value matrix of the cross-attention model, respectively. The target image feature is determined by performing feature fusion processing on the query feature, key features, and value features.

[0009] In one implementation, the step of determining target image features by performing feature fusion processing on query features, key features, and value features includes: multiplying the query features with the key features and value features to determine a first image fusion feature, and concatenating the first image fusion feature with the query features to determine a second image fusion feature; normalizing the first image fusion feature and the second image fusion feature respectively, and adding the normalization results to determine the target image features.

[0010] In one implementation, the preset modal selection model includes a spatial attention module and a global pooling operator. The steps of determining the target image-text fusion features by weighting and fusing target text features and target image features using the preset modal selection model include: performing feature extraction processing on the target image features using the spatial attention module to determine updated target image features; performing feature mapping processing on the target text features using the global pooling operator to determine updated target text features; performing weighting and fusing processing on the updated target image features and updated target text features to determine the initial image-text fusion features; and performing modal interaction processing on the initial image-text fusion features using a cross-attention model to determine the iterative target image-text fusion features.

[0011] In one embodiment, the robotic arm state information includes robotic arm pose information and noisy robotic arm motion information. The step of controlling the robot's action at the next moment by combining the robotic arm state information includes: splicing the robotic arm pose information and noisy robotic arm motion information and inputting them into a multilayer perceptron to determine the robotic arm features, and determining the predicted pose of the robotic arm motion in the next stage based on the robotic arm features and target image-text fusion features, so as to control the robot's action at the next moment based on the predicted pose.

[0012] Secondly, embodiments of the present invention also provide a embodied robot control device based on a multimodal diffusion model. The device includes: an information acquisition module, which acquires instruction text input by a user terminal and an image feature set collected by the robot, wherein the image feature set includes image features collected by the robot's upper and left / right arms; a text feature extraction module, which uses a unimodal text encoder and a multimodal text encoder to perform feature extraction processing on the instruction text, obtaining unimodal text features and multimodal text features of the instruction text respectively, and determines target text features by performing feature fusion processing on the unimodal text features and multimodal text features, wherein the unimodal text features include: core information of the instruction text, and the multimodal text features include: text-image feature mapping information; and an image feature extraction module, which determines target image features based on the image feature set, and performs weight allocation and fusion processing on the target text features and target image features through a preset modality screening model to determine target image-text fusion features, so that the embodied robot can interact with the environment based on the target image-text fusion features, and control the robot's actions at the next moment in combination with the robotic arm state information.

[0013] Thirdly, embodiments of the present invention also provide a server, including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0014] The embodiments of the present invention bring the following beneficial effects: This invention provides a method and apparatus for controlling an embodied robot based on a multimodal diffusion model. After acquiring the command text input from the user and the image feature set collected by the robot, the method uses a unimodal text encoder and a multimodal text encoder to extract features from the command text, obtaining unimodal and multimodal text features respectively. Then, by fusing these unimodal and multimodal text features, target text features are determined. Subsequently, based on the image feature set, target image features are determined. Finally, using a preset modality filtering model, weights are allocated and fused between the target text features and the target image features to determine target image-text fusion features. This allows the embodied robot to interact with the environment based on these target image-text fusion features, and, combined with the robotic arm's state information, controls the robot's actions at the next moment. This invention significantly improves the accuracy of robot control by enhancing the precision of embodied robot feature recognition and model prediction.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a control method for an embodied robot based on a multimodal diffusion model, provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a model training and inference method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a embodied robot control device based on a multimodal diffusion model, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, embodied robots mainly rely on autoregressive algorithms combined with reinforcement learning for perception and decision-making. Additionally, interaction schemes can be generated through 3D reconstruction and scene analysis to adjust the robot's behavior. However, existing technologies suffer from three main problems: First, current embodied algorithms lack scene generalization, resulting in poor accuracy in training scenarios with few or no samples. Second, the models acquire broad visual feature information but lack fine-grained prediction of action space positions, leading to poor implementation performance. Finally, the predicted actions exhibit significant jitter, and the autoregressive model's inability to learn multiple motion trajectories further hinders its accuracy. Therefore, this invention provides an embodied robot control method and device based on a multimodal diffusion model. This method improves the model's accuracy and generalization in new scenarios through the novel Diffusion Action embodied algorithm. Even with limited training samples, the model maintains good performance. The fine-grained image feature extraction process enhances the model's ability to execute refined actions. Furthermore, model jitter constraints address the issue of poor action prediction. This addresses the issue of robotic arm tremors, improves the smoothness of the overall robotic arm movement, and enhances the accuracy of motion prediction under different paths by incorporating a diffusion mechanism.

[0021] See Figure 1 The diagram shows a flowchart of a control method for an embodied robot based on a multimodal diffusion model. The method mainly includes the following steps S102 to S106: Step S102: Obtain the instruction text input by the user terminal and the image feature set collected by the robot, wherein the image feature set includes: image features collected by the robot's upper and left and right arms.

[0022] Step S104: Using a unimodal text encoder and a multimodal text encoder, feature extraction processing is performed on the instruction text to obtain unimodal text features and multimodal text features of the instruction text. The target text features are then determined by feature fusion processing of the unimodal and multimodal text features. The unimodal text features include the core information of the instruction text, and the multimodal text features include text-image feature mapping information. In one embodiment, the unimodal and multimodal text features can be added together to determine the first text feature, and then concatenated and processed to determine the second text feature. A cross-attention model is then used to perform deep fusion processing on the first and second text features to determine the first target text feature. Furthermore, sending the instruction text to a preset text encoder can determine the second target text feature, where the second target text feature represents the specific meaning of the instruction text. The first target text feature is used to determine the position of the noun in the instruction text, so that the position can be selected using a feature box.

[0023] In practical applications, the input command text of the Diffusion Action model is first processed. This command text is used to instruct the robotic arm to perform certain actions, such as "Pour me a glass of water." Then, this command is input into both the unimodal and multimodal text encoders of the BLIP model (i.e., the multimodal model) to obtain the unimodal text features. and multimodal text features Among them, unimodal text features focus more on the information of the command text, while multimodal text features include the mapping information between text and image features.

[0024] Furthermore, in order to better fuse the features of the two information sources, firstly... and The feature is obtained by performing an addition operation. (i.e., the first text feature), will feature and Features are obtained by splicing and manipulating. (i.e., the second text feature), and then a cross-attention model is used to combine the two features mentioned above. and Features are obtained through deep fusion (i.e., the first target text features), where the parameters in the image encoder, unimodal encoder, and multimodal encoder mentioned above are all trainable parameters.

[0025] Simultaneously, the text instructions are input into the YOLO TextEncoder in the YOLO word model to obtain features. (i.e., the second target text features). The parameters of the YOLO text encoder are frozen and do not need to be retrained. Based on the text features and multimodal text features as the input information of the model, the model can still have a certain generalization ability in the training of few samples and zero samples.

[0026] Step S106: Based on the image feature set, determine the target image features, and through a preset modal filtering model, perform weight allocation and fusion processing on the target text features and target image features to determine the target image-text fusion features. This enables the embodied robot to interact with the environment based on the target image-text fusion features, and, combined with the robotic arm state information, control the robot's actions at the next moment. In one embodiment, the target image features can be extracted using a spatial attention module to determine the updated target image features, and the target text features can be mapped using a global pooling operator to determine the updated target text features. Then, the updated target image features and the updated target text features are weighted and fused to determine the initial image-text fusion features. Finally, the initial image-text fusion features are modally interacted using a cross-attention model to determine the iterative target image-text fusion features.

[0027] In practical applications, the acquired text and image feature information can be input into the modality filtering module MSModule. This module then performs multiple self-assigned weights to further weight and fuse the text and image modal information to obtain suitable target text-image fusion features. First, the previously acquired first target text features... and target image features Input into MSModule, for The features are first input into the CBAM channel spatial attention module to obtain the features. Then, this feature is input into the MLP module to obtain the feature. .

[0028] Simultaneously, text features The features are obtained by inputting them into the GAP global pooling operator. Then the features are obtained through the MLP module. Compare this feature with the previously obtained features After splicing and combining, the features are obtained. This feature is then input into the MLP and sigmoid module to obtain the feature. This feature assigns weights to text and visual modalities.

[0029] Then, the distribution features will be assigned and compared with the first target text features respectively. With target image features Multiply them, then enter the MLP module to obtain the k and v features of each modality, and then combine the features obtained from the text modality. The k and v features obtained from the image modality are input into the cross-attention model to obtain the features. Features obtained from visual modalities The k and v features obtained from the text modality are input into the cross-attention model to obtain features. Here, the modal interaction will iterate N times, and the features from the last iteration will be used. as well as splicing and obtaining features .

[0030] Simultaneously, leveraging YOLO World's ability to identify target bounding boxes based on task instruction text and image features, the Vision-LanguagePAN module and its post-processing are used to obtain the target objects mentioned in the instructions. Here, the maximum number of detectable targets per bounding box is set to 6. This results in a total of 18 targets across the three viewpoints. Bounding boxes with fewer than 6 targets are padded with zeros. The targets are then normalized and their features are analyzed. Merging to obtain features This provides more detailed bounding box position information for the overall target, improving the accuracy and precision of the robotic arm's grasping and manipulation of objects. Through the above operations, we can obtain overall visual global and detailed feature information as well as text feature information.

[0031] Furthermore, the robot arm pose information and the noisy robot arm motion information can be spliced ​​together and input into a multilayer perceptron to determine the robot arm features. Based on the robot arm features and the target image-text fusion features, the predicted pose of the robot arm motion in the next stage can be determined, so as to control the robot's action in the next moment according to the predicted pose.

[0032] In practical applications, the state information of the robotic arm is obtained as input to the model to jointly make decisions and predict the robotic arm's actions. Here, ActionInfDit is combined for action prediction. First, it is necessary to obtain the current robotic arm pose (qpos) and the current noisy robotic arm action (actionwithnoise) in each training dataset. The two pieces of information are concatenated to obtain the features. The feature is input into the MLP module to obtain the feature. Simultaneously, the diffusion timesteps and motion frequency are input into their respective MLP modules to obtain features. and features The feature is obtained by concatenating the above two pieces of information. The input is then entered into the Action InfDit module. This module first processes the input... The block-based operation yields Z_t, which is then input into the Inf-DitBlock module. The resulting features are used as q in the cross-attention mechanism and input into the cross-attention model Block. The results obtained through MLP are used as k and v inputs into the cross-attention mechanism cross-attention model Block, and then two Info-DitBlocks are concatenated (Info-DitBlock is a module based on the Diffusion Model and Transformer architecture, mainly used for efficient generation of ultra-high resolution images).

[0033] The above modules will run T times in sequence, and finally input the prediction result of the final Info-DitBlock into the OutputLayer. This layer consists of multiple MLPs and Norm layers to obtain the predicted pose of the robotic arm that needs to move in the next stage.

[0034] The embodied robot control method based on a multimodal diffusion model provided in this invention can significantly improve the accuracy of robot control by enhancing the precision of embodied robot feature recognition and model prediction.

[0035] This invention also provides an implementation method for extracting image features, as detailed in (1) to (2) below: (1) Using the image encoder feature generator of the multimodal model and the preset visual feature extractor respectively, the image features in the image feature set are processed by feature stitching to determine the first image feature and the second image feature. In practical applications, the images of the top of the robot and the left and right mobile phone arms can be input into the BLIP image encoder to obtain the image features. , and features Then, these three image features are concatenated to obtain the feature. (That is, the first image feature), which contains all the robot's visual information. The image features above can help the robot see the overall visual global environment information, while the images on the left and right robotic arms can help the robot capture more detailed information, making it easier for the robotic arms to make model predictions based on this detailed information. At the same time, the above image feature set is input into the visual backbone feature extractor in YOLO world, and the features of the above image feature set are concatenated to obtain the feature. (i.e., the second image feature).

[0036] (2) Using a preset image feature fusion model, feature fusion processing is performed on the first image feature and the second image feature to determine the target image feature. Specifically, the first image feature and the second image feature can be segmented. The first image feature is input into the multilayer perceptron to determine the query feature, and the segmentation processing result of the second image feature is input into the multilayer perceptron to determine the key feature and the value feature. The query feature is used to represent the query matrix of the cross-attention model, and the key feature and the value feature are used to represent the key matrix and the value matrix of the cross-attention model, respectively. Then, the target image feature is determined by performing feature fusion processing on the query feature, the key feature and the value feature. In one embodiment, the first image fusion feature can be determined by multiplying the query feature with the key feature and the value feature, and the second image fusion feature can be determined by concatenating the first image fusion feature with the query feature. Then, the first image fusion feature and the second image fusion feature are normalized respectively, and the normalization results are added together to determine the target image feature.

[0037] In practical applications, the feature obtained by the image feature fusion module is combined with the feature obtained by the BLIP image encoder and input into the IFF Module image feature fusion module to obtain the final feature. (i.e., target image features) Since the image feature encoder in YOLO word is more likely to obtain typical object features, fusing the image features in the BLIP image feature encoder can make the image features obtained by the model richer. The parameters of the YOLO backbone are completely frozen.

[0038] Furthermore, the image feature fusion module first divides the features output by the YOLO backbone into blocks (patchify is a Python library for segmenting an image into smaller blocks or merging blocks into the original image) to obtain YOLO image feature patches, which are then input into two different MLP modules to obtain features. (i.e., the key matrix) and features (That is, the value matrix) Here the features can be... and features This is considered as the k and v matrices in cross-attention (a variant of attention mechanism typically used to handle interactions between two different data modalities, such as text and images); simultaneously, the obtained BLIP image features are first segmented into BLIP image feature patches, which are then input into an MLP module (i.e., a multilayer perceptron) to obtain features. (i.e., query matrix), here Treat it as a q-matrix for cross-attention, then the features... With features Multiplication yields the features (That is, the first image fusion feature). The features... With features Features are obtained by splicing. (i.e., the second image fusion feature), and then the features are processed separately. and features Perform normalization on both sides and then add the results of the normalization to obtain the features. (That is, the target image-text fusion feature), which obtains rich visual representation information.

[0039] See Figure 2 The diagram illustrates a model training and inference method. This embodiment of the invention also provides an implementation method for model training and inference. During model training, the dataset is divided into a training set and a validation set. The training set is used during training, while the validation set is used to observe the convergence of model training. The weights with the lowest decrease in the loss function of the validation set are selected as the final weights of the model. In practical applications, MSE is chosen as the model training loss function, and FID loss function is also added during the diffusion process to accelerate model training convergence. Finally, second-order difference constraints are added to the model prediction results as the model loss, and Savitzky-Golay filters are added during inference to make the model prediction smoother.

[0040] In one implementation, random perturbations are applied to the target objects predicted by YOLO world during training, and masks are used to increase model robustness. At the same time, random perturbations are applied to the images during training (to improve generalization), including color perturbations, mask perturbations, and noise perturbations, to increase the model's adaptability to environmental visual information and improve the accuracy of model prediction results. Finally, in the prediction execution phase, the robotic arm will traverse and execute according to the motion sequence predicted by the model, and then continuously perform model inference until the model completes the overall action.

[0041] In summary, this invention improves the accuracy and generalization of the model in new scenarios through the novel embodied algorithm diffusion model, Diffusion Action. Even with a limited number of training samples, the model still performs well. The fine-grained image feature extraction process enhances the model's ability to execute refined actions. Furthermore, model jitter constraints address the issue of poor action prediction leading to robotic arm jitter, improving the smoothness of the overall robotic arm movement. Finally, the addition of a diffusion mechanism improves the accuracy of action prediction under different paths.

[0042] Regarding the embodied robot control method based on a multimodal diffusion model provided in the foregoing embodiments, this invention provides an embodied robot control device based on a multimodal diffusion model. See [link to relevant documentation]. Figure 3 The diagram shows a structural schematic of an embodied robot control device based on a multimodal diffusion model. The device includes the following parts: The information acquisition module 302 acquires the command text input by the user terminal and the image feature set collected by the robot. The image feature set includes image features collected from the top of the robot and the left and right arms. The text feature extraction module 304 uses a unimodal text encoder and a multimodal text encoder to perform feature extraction processing on the instruction text, obtaining unimodal text features and multimodal text features of the instruction text respectively. The target text features are determined by performing feature fusion processing on the unimodal text features and multimodal text features. The unimodal text features include the core information of the instruction text, and the multimodal text features include the text-image feature mapping information. The image feature extraction module 306 determines the target image features based on the image feature set, and performs weight allocation and fusion processing on the target text features and target image features through a preset modal screening model to determine the target image-text fusion features. This enables the embodied robot to interact with the environment based on the target image-text fusion features, and controls the robot's actions in the next moment by combining the robotic arm's state information.

[0043] The embodied robot control method and apparatus based on a multimodal diffusion model provided in this application can significantly improve the accuracy of robot control by enhancing the accuracy of embodied robot feature recognition and model prediction.

[0044] In one embodiment, the target text feature includes: a first target text feature. When determining the target text feature by performing feature fusion processing on unimodal and multimodal text features, the text feature extraction module 304 is further configured to: add the unimodal and multimodal text features to determine the first text feature; concatenate and process the unimodal and multimodal text features to determine the second text feature; and perform deep fusion processing on the first and second text features using a cross-attention model to determine the first target text feature. The first target text feature is used to determine the position corresponding to a noun in the instruction text, so that the position can be selected using a feature box.

[0045] In one embodiment, the target text feature further includes a second target text feature. After determining the first target text feature, the text feature extraction module 304 is further configured to: send the instruction text to a preset text encoder to determine the second target text feature, wherein the second target text feature is used to represent the specific meaning of the instruction text.

[0046] In one embodiment, when performing the step of determining target image features based on an image feature set, the image feature extraction module 306 is further configured to: use the image encoding featurer of the multimodal model and the preset visual feature extractor to perform feature splicing processing on each image feature in the image feature set to determine the first image feature and the second image feature; and use the preset image feature fusion model to perform feature fusion processing on the first image feature and the second image feature to determine the target image feature.

[0047] In one embodiment, when performing the step of using a preset image feature fusion model to perform feature fusion processing on the first image features and the second image features to determine the target image features, the image feature extraction module 306 is further configured to: perform block processing on the first image features and the second image features, input the first image features into a multilayer perceptron to determine the query features, and input the block processing result of the second image features into the multilayer perceptron to determine the key features and value features, wherein the query features are used to represent the query matrix of the cross-attention model, and the key features and value features are used to represent the key matrix and value matrix of the cross-attention model, respectively; and determine the target image features by performing feature fusion processing on the query features, key features, and value features.

[0048] In one embodiment, when performing the step of determining the target image features by performing feature fusion processing on query features, key features, and value features, the image feature extraction module 306 is further configured to: determine the first image fusion feature by multiplying the query feature with the key feature and the value feature, and concatenate the first image fusion feature with the query feature to determine the second image fusion feature; normalize the first image fusion feature and the second image fusion feature respectively, and add the normalization results to determine the target image features.

[0049] In one embodiment, the preset modal screening model includes a spatial attention module and a global pooling operator. During the step of weighting and fusing target text features and target image features through the preset modal screening model to determine target image-text fusion features, the image feature extraction module 306 is further configured to: perform feature extraction processing on the target image features using the spatial attention module to determine updated target image features; perform feature mapping processing on the target text features using the global pooling operator to determine updated target text features; perform weighting and fusing processing on the updated target image features and updated target text features to determine the initial image-text fusion features; and perform modal interaction processing on the initial image-text fusion features using a cross-attention model to determine the iterative target image-text fusion features.

[0050] In one embodiment, the robotic arm state information includes robotic arm pose information and noisy robotic arm motion information. When performing the step of combining the robotic arm state information to control the robot's action at the next moment, the image feature extraction module 306 is further used to: stitch the robotic arm pose information and the noisy robotic arm motion information together and input them into a multilayer perceptron to determine the robotic arm features, and determine the predicted pose of the robotic arm motion in the next stage based on the robotic arm features and the target image-text fusion features, so as to control the robot's action at the next moment based on the predicted pose.

[0051] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0052] This invention provides a server, specifically, the server includes a processor and a storage device; the storage device stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0053] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0054] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0055] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0056] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0057] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0058] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0059] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0060] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A control method for an embodied robot based on a multimodal diffusion model, characterized in that, The method includes: The system acquires the command text input by the user and the image feature set collected by the robot, wherein the image feature set includes image features collected by the robotic arms above and on the left and right hands of the robot. The instruction text is processed by using a unimodal text encoder and a multimodal text encoder to extract features, thereby obtaining unimodal text features and multimodal text features of the instruction text. The target text features are then determined by performing feature fusion processing on the unimodal text features and the multimodal text features. The unimodal text features include the core information of the instruction text, and the multimodal text features include text-image feature mapping information. Based on the image feature set, target image features are determined, and the target text features and target image features are weighted and fused using a preset modal filtering model to determine target image-text fusion features. This enables the embodied robot to interact with the environment based on the target image-text fusion features, and, in conjunction with the robotic arm's state information, to control the robot's actions in the next moment. The target text features include: a first target text feature. The step of determining the target text feature by performing feature fusion processing on the unimodal text feature and the multimodal text feature includes: adding the unimodal text feature and the multimodal text feature to determine the first text feature; concatenating and processing the unimodal text feature and the multimodal text feature to determine the second text feature; and performing deep fusion processing on the first text feature and the second text feature using a cross-attention model to determine the first target text feature. The first target text feature is used to determine the position corresponding to a noun in the instruction text, so that the position can be selected using a feature box. The step of determining the target image features based on the image feature set includes: using a multimodal model image feature encoder and a preset visual feature extractor to perform feature stitching processing on each image feature in the image feature set to determine the first image feature and the second image feature; and using a preset image feature fusion model to perform feature fusion processing on the first image feature and the second image feature to determine the target image feature. The preset modal selection model includes a spatial attention module and a global pooling operator. The steps of determining the target text-image fusion feature by weighting and fusing the target text features and the target image features using the preset modal selection model include: performing feature extraction on the target image features using the spatial attention module to determine updated target image features; performing feature mapping on the target text features using the global pooling operator to determine updated target text features; performing weighting and fusing on the updated target image features and the updated target text features to determine the initial fused text-image fusion feature; and performing modal interaction processing on the initial fused text-image fusion feature using a cross-attention model to determine the iterative target text-image fusion feature.

2. The embodied robot control method based on a multimodal diffusion model according to claim 1, characterized in that, The target text features further include: second target text features, which, after the step of determining the first target text features, include: The instruction text is sent to a preset text encoder to determine the second target text feature, wherein the second target text feature is used to represent the specific meaning of the instruction text.

3. The embodied robot control method based on a multimodal diffusion model according to claim 1, characterized in that, The step of using a preset image feature fusion model to perform feature fusion processing on the first image features and the second image features to determine the target image features includes: The first image feature and the second image feature are segmented. The first image feature is input into a multilayer perceptron to determine the query feature. The segmentation result of the second image feature is input into the multilayer perceptron to determine the key feature and the value feature. The query feature is used to represent the query matrix of the cross-attention model. The key feature and the value feature are used to represent the key matrix and the value matrix of the cross-attention model, respectively. The target image features are determined by performing feature fusion processing on the query features, the key features, and the value features.

4. The embodied robot control method based on a multimodal diffusion model according to claim 3, characterized in that, The step of determining the target image features by performing feature fusion processing on the query features, the key features, and the value features includes: By multiplying the query feature with the key feature and the value feature, a first image fusion feature is determined, and the first image fusion feature is concatenated with the query feature to determine a second image fusion feature; The first image fusion feature and the second image fusion feature are normalized respectively, and the normalization results are added together to determine the target image features.

5. The embodied robot control method based on a multimodal diffusion model according to claim 1, characterized in that, The robotic arm state information includes: robotic arm pose information and noisy robotic arm motion information. The steps for controlling the robot's next action, based on the robotic arm state information, include: The robot arm pose information and the noisy robot arm motion information are concatenated and input into a multilayer perceptron to determine the robot arm features. Based on the robot arm features and the target image-text fusion features, the predicted pose of the robot arm motion in the next stage is determined so as to control the robot's action in the next moment according to the predicted pose.

6. A control device for an embodied robot based on a multimodal diffusion model, characterized in that, The device includes: The information acquisition module acquires the command text input by the user terminal and the image feature set collected by the robot, wherein the image feature set includes: image features collected by the robotic arms above the robot and the left and right hands; The text feature extraction module uses a unimodal text encoder and a multimodal text encoder to perform feature extraction processing on the instruction text, obtaining unimodal text features and multimodal text features of the instruction text respectively. Then, by performing feature fusion processing on the unimodal text features and the multimodal text features, the target text features are determined. The unimodal text features include: the core information of the instruction text, and the multimodal text features include: text and image feature mapping information. The image feature extraction module determines the target image features based on the image feature set, and performs weight allocation and fusion processing on the target text features and the target image features through a preset modal screening model to determine the target image-text fusion features. This enables the embodied robot to interact with the environment based on the target image-text fusion features, and controls the robot's actions in the next moment by combining the robotic arm's state information. The target text features include: a first target text feature. The step of determining the target text feature by performing feature fusion processing on the unimodal text feature and the multimodal text feature includes: adding the unimodal text feature and the multimodal text feature to determine the first text feature; concatenating and processing the unimodal text feature and the multimodal text feature to determine the second text feature; and performing deep fusion processing on the first text feature and the second text feature using a cross-attention model to determine the first target text feature. The first target text feature is used to determine the position corresponding to a noun in the instruction text, so that the position can be selected using a feature box. The step of determining the target image features based on the image feature set includes: using a multimodal model image feature encoder and a preset visual feature extractor to perform feature stitching processing on each image feature in the image feature set to determine the first image feature and the second image feature; and using a preset image feature fusion model to perform feature fusion processing on the first image feature and the second image feature to determine the target image feature. The preset modal selection model includes a spatial attention module and a global pooling operator. The steps of determining the target text-image fusion feature by weighting and fusing the target text features and the target image features using the preset modal selection model include: performing feature extraction on the target image features using the spatial attention module to determine updated target image features; performing feature mapping on the target text features using the global pooling operator to determine updated target text features; performing weighting and fusing on the updated target image features and the updated target text features to determine the initial fused text-image fusion feature; and performing modal interaction processing on the initial fused text-image fusion feature using a cross-attention model to determine the iterative target text-image fusion feature.

7. A server, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 5.

Images