A multi-modal data processing method supporting multi-task switching and related equipment

Abstract

The application provides a multi-modal data processing method supporting multi-task switching and related equipment, and the method is applied to the technical field of robots, and the method is: acquiring an environment image of an environment currently where a robot is located, historical task instructions, current task instructions, a robot state and robot action noise; using the environment image, the historical task instructions and the current task instructions to generate fusion features and multi-task condition features; extracting robot state features from the robot state; inputting the robot action noise, the fusion features, the multi-task condition features and the robot state features into a preset action expert model to make an action decision, so as to obtain an optimal action sequence of the robot. According to the historical task instructions, the current task instructions and the environment image, the optimal action sequence of the robot is obtained, the complexity problem when the robot switches tasks is effectively solved, and the accuracy of the robot action output when the task is switched is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a multi-modal data processing method supporting multi-task switching and related equipment. BACKGROUND

[0002] In recent years, end-to-end imitation learning has developed rapidly, and with the robustness of large models, methods such as DP (Diffusion Policy) and OpenVLA have been proposed. Most of these existing methods support multi-task training.

[0003] However, in real-world scenarios of robot applications, switching tasks is random and arbitrary; however, existing methods such as DP (Diffusion Policy) and OpenVLA obtain data for each task from the beginning to the end and operate independently and completely during training; when the robot performs a task, these existing methods rely on the successful completion of the previous task and fully return to the initial position for the next task to collect data, and do not consider the complexity of switching tasks, which results in poor accuracy of the robot's actions when switching tasks. SUMMARY

[0004] Therefore, the embodiments of the present application provide a multi-modal data processing method supporting multi-task switching and related equipment to solve the complexity problem when the robot switches tasks.

[0005] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0006] The first aspect of the embodiments of the present application discloses a multi-modal data processing method supporting multi-task switching, comprising:

[0007] obtaining an environment image of an environment in which a robot is currently located, historical task instructions, current task instructions, a robot state, and robot action noise;

[0008] generating fusion features and multi-task conditional features using the environment image, the historical task instructions, and the current task instructions; wherein the fusion features are generated based on the following features: visual features and current environment state features extracted from the environment image, and task instruction features extracted from the historical task instructions and the current task instructions; the multi-task conditional features are generated based on the current environment state features and the task instruction features;

[0009] extracting robot state features from the robot state;

[0010] input the robot action noise, the fusion feature, the multi-task condition feature and the robot state feature into a preset action expert model for action decision to obtain an optimal action sequence of the robot.

[0011] Preferably, the fusion feature and the multi-task condition feature are generated by using the environment image, the historical task instruction and the current task instruction, including:

[0012] A visual feature is extracted from the environment image by using an image encoding model.

[0013] The environment image and a preset prompt word are input into a visual language large model to obtain a current environment state.

[0014] The historical task instruction and the current task instruction are input into a large language model to obtain corresponding task instruction features.

[0015] The current environment state is input into the large language model to obtain corresponding current environment state features.

[0016] The visual feature, the task instruction feature and the current environment state feature are fused by using a specified encoder to obtain a fusion feature.

[0017] The task instruction feature and the current environment state feature are input into a multi-task condition fusion model for information integration to obtain a multi-task condition feature.

[0018] Preferably, the robot state feature is extracted from the robot state, including:

[0019] The robot state feature is extracted from the robot state by using a robot state encoding model.

[0020] Preferably, after obtaining the multi-task condition feature, the process further includes:

[0021] The multi-task condition feature is distilled to obtain a compressed multi-task condition feature.

[0022] Preferably, the process of obtaining the environment image of the environment where the robot is currently located includes:

[0023] A first camera and a second camera are called to obtain the environment image of the environment where the robot is currently located, the first camera has a first visual angle and is mounted on the robot, and the second camera has a global visual angle and is arranged in the environment where the robot is located.

[0024] Preferably, the process of obtaining the robot state includes:

[0025] Call the robot ontology sensor to obtain a robot state, the robot state at least comprising: joint angle of the robot, end effector state.

[0026] Preferably, after obtaining the optimal action sequence of the robot, the method further comprises:

[0027] The optimal action sequence is denoised.

[0028] The second aspect of the embodiment of the application discloses a multi-modal data processing system supporting multi-task switching, the system comprising:

[0029] An acquisition unit is configured to acquire an environment image of an environment in which a robot currently locates, historical task instructions, current task instructions, a robot state and robot action noise;

[0030] A generation unit is configured to generate fusion features and multi-task conditional features by using the environment image, the historical task instructions and the current task instructions; wherein the fusion features are generated based on visual features and current environment state features extracted from the environment image, and task instruction features extracted from the historical task instructions and the current task instructions; the multi-task conditional features are generated based on the current environment state features and the task instruction features;

[0031] An extraction unit is configured to extract robot state features from the robot state;

[0032] A decision unit is configured to input the robot action noise, the fusion features, the multi-task conditional features and the robot state features into a preset action expert model to make an action decision, so as to obtain an optimal action sequence of the robot.

[0033] The third aspect of the embodiment of the application discloses an electronic device, comprising: a processor and a memory, the processor and the memory are connected through a communication bus; wherein the processor is used to call and execute a program stored in the memory; the memory is used to store a program, the program is used to realize the multi-modal data processing method supporting multi-task switching disclosed in the first aspect of the embodiment of the application.

[0034] The fourth aspect of the embodiment of the application discloses a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the multi-modal data processing method supporting multi-task switching disclosed in the first aspect of the embodiment of the application.

[0035] Based on the above, the multi-modal data processing method and related equipment supporting multi-task switching provided by the embodiment of the application obtain the optimal action sequence of the robot according to the historical task instruction, the current task instruction and the environment image, effectively solve the complexity problem when the robot switches tasks, and improve the accuracy of the robot action output when switching tasks. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only a part of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0037] Figure 1 The flowchart of the multi-modal data processing method supporting multi-task switching provided by the embodiment of the present application is shown in the figure.

[0038] Figure 2 The flowchart of generating fusion features and multi-task conditional features provided by the embodiment of the present application is shown in the figure.

[0039] Figure 3 The overall flowchart provided by the embodiment of the present application is shown in the figure.

[0040] Figure 4 The detailed example diagram of each part provided by the embodiment of the present application is shown in the figure.

[0041] Figure 5 The example diagram of the multi-task conditional fusion model provided by the embodiment of the present application is shown in the figure.

[0042] Figure 6 The structural block diagram of the multi-modal data processing system supporting multi-task switching provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] In this application, the terms "comprise", "contain", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not only include those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, the element defined by the phrase "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0045] In recent years, end-to-end imitation learning has developed rapidly, and with the help of the robustness of large models, methods such as DP (Diffusion Policy), RDT, OpenVLA, and Pi0 have been proposed. These existing methods mostly support multi-task training, that is, each task has a corresponding language instruction during training, and the corresponding language instruction is adjusted to switch tasks when the robot executes the task.

[0046] However, the above-mentioned existing methods obtain each task data from the beginning to the end and operate independently and completely during training, so the robot relies on the successful completion of the previous task and fully returns to the initial position of the next task when collecting data during execution. These existing methods do not consider the complexity of switching tasks.

[0047] In real-world scenarios of robot applications, task switching is random and arbitrary, and there may be cases where the user switches to the next task during the execution of the previous task, or the robot switches to the current task while it is not in the position of collecting data for the current task.

[0048] In addition, the state of these tasks during operation and the actions that should be performed between the start of the next task execution also need to be considered. For example: when executing the task of "picking up an apple", it can be divided into the following three states: 1) the robot has not touched the apple, 2) the robot has picked up the apple, 3) the robot has finished picking up the apple. When switching to the task of "opening the drawer" (i.e., switching to the next task), the corresponding operations should be different, which is a problem that existing methods have not considered.

[0049] Therefore, how to handle the previous and next tasks when the user switches tasks, the execution state of the previous task, and model these combinations in some form so that the multi-modal embodied operation model can understand and give the correct action output is a problem that cannot be ignored in the development of robot technology.

[0050] To solve the above problems, the scheme provides a multi-modal data processing method supporting multi-task switching and related equipment. The optimal action sequence of the robot is determined according to historical task instructions, current task instructions and environmental images, effectively solving the complexity problem when the robot switches tasks, and improving the accuracy of the robot action output when switching tasks.

[0051] It should be noted that the multi-modal data processing method supporting multi-task switching and related equipment provided by the scheme can be implemented in a multi-modal embodied intelligent operation framework supporting multi-task switching. The scheme will be described in detail through various embodiments.

[0052] To better understand the content of each embodiment of the scheme, the general idea of the scheme is briefly explained here:

[0053] The scheme uses the encoder in the pre-trained multi-modal large model (such as a visual language large model or any other model) as the backbone network. This can have the ability to fuse multiple modalities, including visual features, task instruction features, and current environmental state features.

[0054] Among them, the multi-view image (hand-eye camera image + third-view camera image) is extracted through the image encoder to obtain the corresponding visual features. A visual language large model (VLM) is used to analyze the multi-view image and return the current environmental state, such as whether it is holding an apple, approaching an apple, and completing the placement of an apple. The current environmental state is input into the large language model (LLM) to extract the current environmental state feature, which is input into the pre-trained "specified encoder".

[0055] The previous task instruction (hereinafter referred to as the historical task instruction) and the current task instruction are extracted into the large language model (LLM) to extract the task instruction feature and input it into the "specified encoder". The "specified encoder" extracts the corresponding fusion features from all inputs.

[0056] The multi-task conditional fusion model further fuses the previous task instruction feature (hereinafter referred to as the historical task instruction feature), the current task instruction feature, and the current environmental state feature extracted by the visual language large model (VLM) and the large language model (LLM), and distills the multi-task conditional feature to obtain the compressed multi-task conditional feature.

[0057] The robot state (such as the angle value of each joint and the state of the gripper) is extracted into the robot state feature through the robot state coding model.

[0058] The action specialist model is a decoder of a visual language model (VLM), which extracts the fusion features extracted by the specified encoder, the multi-task condition features extracted by the multi-task condition fusion model, the robot state features extracted by the robot state encoding model, and the robot action noise, and makes state prediction and robot future action decision through a cross-attention mechanism.

[0059] The robot action noise is Gaussian noise, and the robot future action is decided by the action specialist model and the action decoder. This training method is Diffusion Flow-matching. Finally, the robot can make corresponding and effective action decisions according to the complex conditions of the previous task instruction, the current task instruction, and the current environment image.

[0060] Referring to Figure 1 , a flowchart of a multi-modal data processing method supporting multi-task switching is shown, which includes the following steps:

[0061] Step S101: obtaining an environment image of an environment where a robot is currently located, historical task instructions, current task instructions, a robot state, and robot action noise.

[0062] In the process of implementing step S101, the environment image of the environment where the robot is currently located is obtained, and the robot state and the robot action noise are obtained, and the task instruction input is received, which includes the historical task instructions (previous task instructions) and the current task instructions.

[0063] It should be noted that the received task instruction input (historical task instructions and current task instructions) can provide a multi-level composite condition execution target for the robot; the robot action noise can be Gaussian noise.

[0064] In some embodiments, the specific implementation of obtaining the environment image of the environment where the robot is currently located is to call a first camera and a second camera to obtain the environment image of the environment where the robot is currently located, the first camera having a first perspective and being mounted on the robot, and the second camera having a global perspective and being arranged in the environment where the robot is located.

[0065] That is, the first camera is mounted on the robot, and the second camera is in the environment where the robot is located.

[0066] It can be understood that the robot obtains the environment image (or environment information) through a multi-source sensor, which includes a hand-eye camera image (first perspective) and a third perspective camera image (global perspective or third perspective), thereby fully covering the working scene.

[0067] The multi-source sensor at least includes a first camera and a second camera, the hand-eye camera image is obtained by the first camera with a first visual angle, and the third visual angle camera image is obtained by the second camera with a global visual angle. The first camera and the second camera can be high-resolution depth cameras.

[0068] For example, the first camera and the second camera use a specified model of depth camera with a resolution of “1280*720@30fps”.

[0069] In some other embodiments, the specific implementation of obtaining the robot state is to call a robot body sensor to obtain the robot state, and the robot state at least includes joint angles of the robot and an end effector state.

[0070] That is, the real-time robot state is collected by the robot body sensor.

[0071] Step S102: generating fusion features and multi-task condition features by using the environment image, the historical task instruction and the current task instruction.

[0072] In the process of implementing step S102, the image encoding model, the visual language large model, the large language model and the multi-task condition fusion model are used to process the environment image, the historical task instruction and the current task instruction, so as to generate the fusion features and the multi-task condition features.

[0073] It should be noted that the fusion features are generated based on the visual features and the current environment state features extracted from the environment image, and the task instruction features extracted from the historical task instruction and the current task instruction; and the multi-task condition features are generated based on the current environment state features and the task instruction features.

[0074] Step S103: extracting robot state features from the robot state.

[0075] In the process of implementing step S103, the robot state encoding model is used to extract the robot state features from the robot state.

[0076] For example, the robot state encoding model can be a multi-layer fully connected network (MLP), and the robot state (i.e., the original sensor data) is encoded through the multi-layer fully connected network to convert the robot state into a high-dimensional feature representation, so as to extract the robot state features.

[0077] Step S104: inputting the robot action noise, the fusion features, the multi-task condition features and the robot state features into a preset action expert model for action decision-making to obtain an optimal action sequence of the robot.

[0078] In the process of implementing step S104, the robot action noise, the fusion feature, the multi-task condition feature and the robot state feature are input into a preset action expert model for action decision, so as to obtain an optimal action sequence of the robot (action decision result, i.e., future action of the robot).

[0079] Specifically, the robot action noise, the fusion feature, the multi-task condition feature and the robot state feature are input into the action expert model for action decision, and output features output by the action expert model are obtained; and the output features are decoded by using an action decoder (robot action decoder), so as to obtain the optimal action sequence of the robot.

[0080] In some specific embodiments, after the optimal action sequence of the robot is obtained, the optimal action sequence is denoised, so as to gradually eliminate decision noise and improve smoothness and accuracy of the action trajectory.

[0081] It should be noted that the action expert model adopts a decoder-only transformer architecture (for example, the parameter amount is 500M), and each type of information is weighted by using a cross-attention mechanism, and finally, a future action sequence (only an example, which can be adjusted according to actual requirements) of 30 steps (only an example, which can be adjusted according to actual requirements) and an instant state evaluation are output.

[0082] It should be further noted that when the obtained optimal action sequence is denoised, Diffusion Flow-Matching (DFM) can be used for denoising, and decision noise is gradually eliminated by multiple (for example, 10 times) iterative optimization, so as to improve smoothness and accuracy of the action trajectory.

[0083] The core goal of DFM is to gradually transform noise distribution (such as random Gaussian noise) into target distribution (such as the optimal action sequence of the robot) by learning a probability path, and Flow-Matching directly learns a conversion path from noise to target data by defining a vector field, which is more efficient than a traditional diffusion model.

[0084] In the embodiment of the application, the optimal action sequence of the robot is obtained according to the historical task instruction, the current task instruction and the environment image, which effectively solves the complexity problem of the robot when the task is switched, and improves the accuracy of the output robot action when the task is switched.

[0085] For the above-mentioned embodiments of the application Figure 1 The fusion feature and the multi-task condition feature involved in step S102 are described in detail in the following Figure 2, shows the generation of fusion features and multi-task conditional features provided by the embodiment of the application, including the following steps:

[0086] Step S201: extracting visual features from the environment image using an image encoding model.

[0087] In the process of implementing step S201, the environment image (multi-view image, i.e. hand-eye camera image + third-view camera image) is input into the image encoding model, and the corresponding visual features (also referred to as image features) are extracted by the image encoding model.

[0088] It should be noted that the image encoding model can be a transformer-based image encoder (e.g. with a parameter size of up to 300M), which has been pre-trained based on a large amount of image data and can effectively capture key visual features in the scene.

[0089] Step S202: inputting the environment image and the preset prompt word into a visual language large model to obtain the current environment state.

[0090] In the process of implementing step S202, the environment image and the prompt word (used to obtain the current environment state) are input into the visual language large model (VLM) for processing, so as to extract the semantic information of the current environment state and form a multi-modal understanding of the environment.

[0091] It should be noted that the visual language large model is pre-trained based on large-scale multi-modal (text + image) data, and is based on a transformer architecture. The visual language large model understands the relevance of visual scenes and text prompts through a self-attention mechanism, and outputs a concise environment state description (i.e. the current environment state) of 20 words or less (the number of words is only an example).

[0092] Step S203: inputting the historical task instruction and the current task instruction into a large language model to obtain corresponding task instruction features.

[0093] In the process of implementing step S203, the historical task instruction and the current task instruction are input into the large language model, and the corresponding task instruction features are extracted by the large language model (LLM), wherein the task instruction features include: historical task instruction features corresponding to the historical task instruction (providing historical task context), and current task instruction features corresponding to the current task instruction (clearing immediate task objectives).

[0094] It should be noted that the parameter size of the large language model can be 7B parameter size (only as an example), and the large language model is responsible for parsing the natural language instructions (historical task instructions and current task instructions) input by the user, converting the ambiguous user instructions into structured task representations through semantic understanding capabilities, and providing clear action targets for subsequent decision-making.

[0095] Step S204: inputting the current environment state into the large language model to obtain corresponding current environment state features.

[0096] In the process of implementing step S204, the current environment state is input into the large language model to extract the corresponding current environment state features.

[0097] Step S205: using a specified encoder to fuse the visual features, task instruction features, and current environment state features to obtain fused features.

[0098] In the process of implementing step S205, the specified encoder is used to fuse the visual features, task instruction features, and current environment state features to obtain the fused features.

[0099] It should be noted that the specified encoder can be a transformer encoder; the specified encoder deeply fuses the visual features, task instruction features, and current environment state features to generate fused features containing multi-modal information.

[0100] Step S206: inputting the task instruction features and the current environment state features into a multi-task conditional fusion model for information integration to obtain multi-task conditional features.

[0101] In the process of implementing step S206, the task instruction features and the current environment state features are input into the multi-task conditional fusion model, and the multi-task conditional features are extracted through the multi-task conditional fusion model.

[0102] It should be noted that the multi-task conditional fusion model adopts a structure combining Transformer cross-attention mechanism and multi-layer perceptron (MLP) to realize intelligent integration of robot task-related information.

[0103] In some specific embodiments, after the multi-task conditional features are extracted, the multi-task conditional features are distilled to obtain compressed multi-task conditional features; in this case, the "multi-task conditional features" input into the action specialist model are "compressed multi-task conditional features".

[0104] It should be noted that the specified encoder (transformer encoder) is used for multi-modal fusion (i.e. language + visual fusion), and the multi-task condition fusion model is used to fuse and compress text information (historical task instructions, current task instructions, and current environment state). The following three reasons are set:

[0105] First, the fusion feature is mixed with the visual feature, which may affect the multi-task condition feature;

[0106] Second, the fusion feature is transmitted to the action specialist model, and the multi-task condition feature, the robot state feature, and the robot action noise are calculated by cross-attention mechanism (cross-attention), which is not a parallel input relationship;

[0107] Third, the action specialist model is a Diffusion model, which requires more computing resources (preferably a small number of tokens), and the multi-task condition fusion model is equivalent to compressing information. Directly inputting the multi-task condition feature into the action specialist model can reduce the amount of calculation.

[0108] The above is a description of how to generate fusion features and multi-task condition features.

[0109] It should be noted that the execution order of each step in the above embodiment of the application Figure 2 is only used for example and illustration. In actual execution, one execution order is: sequentially executing steps S201-S206; another execution order is: executing steps S201, S202 and S203 in parallel, then executing step S204, then executing steps S205 and S206 in parallel; another execution order is: executing steps S201, S202 and S203 in parallel, then executing step S204, then executing steps S205 and S206 in sequence. The execution order of each step in Figure 2 is not specifically limited, and can be adjusted according to actual conditions.

[0110] In order to better understand the content of each of the above embodiments, the following will be illustrated from the three aspects of the overall process, the details of each part, and the multi-task condition fusion model. Figures 3 to 5

[0111] I. Illustration of the overall process:

[0112] For example, Figure 3 ​The overall flow example diagram shows that the robot obtains environment images through sensors, including hand-eye camera images (first perspective) and third perspective camera images (global perspective or third perspective), and the sensors include at least a first camera and a second camera, which can be depth cameras with a resolution of "1280*720@30fps".

[0113] Receiving task instruction input, which includes historical task instructions (previous task instructions) and current task instructions.

[0114] The visual language large model (VLM) jointly encodes multi-perspective images (hand-eye camera images + third perspective camera images) and prompt words (used to obtain the current environment state) to extract semantic information of the current environment state and form a multi-modal understanding of the environment.

[0115] The obtained multi-modal data (visual features, task instruction features, and current environment state features, etc.) are uniformly processed by a feature fusion transformer encoder (specified encoder), which uses a self-attention mechanism to dynamically fuse visual features, temporal context (historical task instruction features + current task instruction features), and current environment state features of the robot's current task execution, thereby extracting fusion features. The fusion features are input into the action specialist model to make decisions on the robot's future actions in combination with the robot's state features.

[0116] The above process effectively solves the complex condition training difficulty problem when switching between multiple tasks and enhances the adaptability to complex scene human-computer interaction. Based on the features in the input action specialist model, the action specialist model generates the optimal action sequence (robot's future actions) of the robot through imitation learning strategy, and the robot performs specific operations (such as grabbing, moving, etc.) according to the instruction requirements, thereby forming a "perception-decision-execution" closed loop. In addition, real-time feedback adjustment is also supported to ensure the accuracy and robustness of task execution.

[0117] The above is the specific content of "I. Illustration of the overall flow".

[0118] II. Illustration of details of each part:

[0119] For example Figure 4 As shown in the detailed example diagram, multi-perspective visual acquisition devices are configured in the perception layer to obtain environment images (i.e. hand-eye camera images + third perspective camera images, which are equivalent to visual data) from different angles. These obtained environment images and pre-designed prompt words are jointly input into a large-scale multi-modal pre-trained visual language large model (VLM).

[0120] The visual language model (VLM) is based on a transformer architecture and understands the relevance of visual scenes and text prompts through a self-attention mechanism, outputting concise environment state descriptions (i.e., current environment state) of 20 words or less.

[0121] At the same time, a dedicated transformer-based image encoder (i.e., image encoding model) is used to extract deep features from the original visual data (i.e., environment images), resulting in corresponding visual features. The image encoder is pre-trained on a large amount of image data and can effectively capture key visual features in the scene.

[0122] At the instruction understanding layer, a large language model (LLM) with a 7B parameter size is integrated. The LLM is responsible for parsing user input natural language instructions (historical task instructions and current task instructions) and converting ambiguous user instructions into structured task representations through semantic understanding capabilities, providing clear action targets for subsequent decision-making.

[0123] The robot state (joint angle, end effector state) is obtained through the robot's own sensors and encoded through a multi-layer fully connected network (MLP, i.e., robot state encoding model), converting the original sensor data into high-dimensional feature representations to extract robot state features. Subsequently, these features are processed by a state decoder composed of multiple MLPs and GeLU activation functions, which can restore the encoded features to readable state descriptions.

[0124] At the decision planning layer, the scheme uses a multi-task conditional fusion mechanism to obtain multi-task conditional features; a transformer encoder is used to deeply integrate visual features, task instruction features, and current environment state features to generate fusion features (i.e., task condition representation) containing multi-modal information.

[0125] The fusion features, multi-task conditional features, robot state features, and robot action noise are input into the action specialist model, which uses a decoder-only transformer architecture (with 500M parameters). Through cross-attention mechanisms, each type of information is weighted and processed, and the final output is a 30-step action sequence (i.e., optimal action sequence) and immediate state evaluation. The action decision result is denoised based on DFM through 10 iterations of optimization to gradually eliminate decision noise and improve the smoothness and accuracy of the action trajectory.

[0126] Specifically, the robot action noise, fused features, multi-task condition features, and robot state features are input into the action specialist model for action decision-making, and output features output by the action specialist model are obtained; the output features are decoded by using an action decoder (robot action decoder), so as to obtain an optimal action sequence of the robot.

[0127] For the prediction of the current environment state, the training stage utilizes the accurate state labeling provided by the visual language large model (VLM) to ensure the basic performance of each module.

[0128] In the inference stage, flexible operation path selection is provided: when a completely new environment needs to be processed, the complete language large model (VLM) state evaluation process is enabled; when real-time performance is pursued and the environment is known, the state prediction function of the action specialist model can be directly called, which significantly improves the response speed.

[0129] The above is the specific content of “II. Example illustration for details of each part”.

[0130] III. Example illustration for the multi-task condition fusion model:

[0131] As Figure 5 The example diagram of the multi-task condition fusion model shown can be seen that the multi-task condition fusion model adopts the architecture combining the Transformer cross-attention mechanism and the multi-layer perceptron (MLP), and realizes the intelligent integration of the information related to the robot task.

[0132] The multi-task condition fusion model receives three key inputs: the previous task instruction feature (i.e. the historical task instruction feature, which provides the historical task context) encoded by the 7B parameter scale large language model (LLM), the current task instruction feature (which explicitly indicates the immediate task target), and the current environment state feature.

[0133] Taking the current environment state feature as the query vector, and the current task instruction and the previous task instruction as the key-value pair, the multi-task condition fusion model calculates the correlation weight between the features by using the cross-attention mechanism, realizes the adaptive focusing of the task-related information, and this design can automatically ignore irrelevant information and focus on the “environment state and historical instructions closely related to the current operation”.

[0134] The high-dimensional features after attention fusion are compressed and nonlinearly enhanced by the multi-layer MLP network containing the GeLU activation function, and finally the unified multi-task condition features are output.

[0135] The multi-task condition feature not only retains the semantic integrity of the original information, but also improves the operation efficiency of the subsequent action expert model through dimension reduction processing. As the core information hub, the multi-task condition feature generated by the multi-task condition fusion model will be input into the action expert model together with the fusion feature, the robot state feature and the robot action noise, ensuring that the final generated optimal action sequence meets the alignment of the task context instruction and the robot environment state, providing key support for robot intelligent decision-making in complex scenarios.

[0136] The above is the specific content of "III. Illustration of the Multi-Task Condition Fusion Model".

[0137] In summary, the present scheme aims to further improve the understanding and execution ability of robots for complex tasks through deep language instructions and current task state information, enabling robots to handle complex situations encountered when switching tasks.

[0138] Further, the present scheme effectively solves the complexity problem of robot task switching by integrating the encoding ability of multi-modal large models, state analysis and task instruction fusion, and can dynamically generate adaptive actions according to the previous task instruction, current task instruction and current environment state (such as execution phase, environment feedback). Specifically, through the fusion of multi-view image analysis and language instruction feature extraction, combined with task condition distillation and state encoding, it can accurately identify task phase differences such as "not touched / picked up / placed", and use denoising flow matching technology to decide the optimal action that meets the context, thereby overcoming the limitations of traditional methods relying on independent task execution and fixed initial position, significantly improving the robustness and task continuity of robots in any interruption and non-regular state switching.

[0139] Corresponding to the multi-modal data processing method for supporting multi-task switching provided by the above-mentioned embodiments of the present application, referring to Figure 6 The embodiments of the present application also provide a structural diagram of a multi-modal data processing system for supporting multi-task switching, which comprises an acquisition unit 100, a generation unit 200, an extraction unit 300 and a decision unit 400.

[0140] The acquisition unit 100 is used to acquire the environment image of the environment where the robot is currently located, the historical task instruction, the current task instruction, the robot state and the robot action noise.

[0141] In a specific implementation, the process of the acquisition unit 100 acquiring the environment image of the environment where the robot is currently located includes: calling a first camera and a second camera to acquire the environment image of the environment where the robot is currently located, the first camera having a first view and being mounted on the robot, and the second camera having a global view and being arranged in the environment where the robot is located.

[0142] In another specific implementation, the process in which the acquisition unit 100 acquires the robot state comprises: calling a robot ontology sensor to acquire the robot state, the robot state at least comprising: joint angle of the robot, end effector state.

[0143] The generation unit 200 is configured to generate fusion features and multi-task conditional features by using the environment image, the historical task instruction, and the current task instruction.

[0144] The fusion features are generated based on visual features extracted from the environment image and current environment state features, and task instruction features extracted from the historical task instruction and the current task instruction; and the multi-task conditional features are generated based on the current environment state features and the task instruction features.

[0145] The extraction unit 300 is configured to extract robot state features from the robot state.

[0146] In a specific implementation, the extraction unit 300 is specifically configured to extract the robot state features from the robot state by using a robot state encoding model.

[0147] The decision unit 400 is configured to input the robot action noise, the fusion features, the multi-task conditional features, and the robot state features into a preset action expert model to make an action decision, so as to obtain an optimal action sequence of the robot.

[0148] Preferably, the decision unit 400 is further configured to perform denoising processing on the optimal action sequence.

[0149] In the embodiments of the present application, the optimal action sequence of the robot is obtained according to the historical task instruction, the current task instruction, and the environment image, which effectively solves the complexity problem when the robot switches tasks and improves the accuracy of the output robot action when switching tasks.

[0150] Preferably, in combination with Figure 6 As shown, the generation unit 200 comprises: an extraction subunit, a first input subunit, a second input subunit, a third input subunit, a fusion subunit, and an integration subunit; and the execution principles of the subunits are as follows:

[0151] The extraction subunit is configured to extract visual features from the environment image by using an image encoding model.

[0152] The first input subunit is configured to input the environment image and a preset prompt word into a visual language large model to obtain a current environment state.

[0153] The second input subunit is configured to input the historical task instruction and the current task instruction into a large language model to obtain corresponding task instruction features.

[0154] The third input sub-unit is configured to input the current environment state into the large language model to obtain corresponding current environment state features.

[0155] The fusion sub-unit is configured to fuse the visual features, the task instruction features and the current environment state features by using a designated encoder to obtain fusion features.

[0156] The integration sub-unit is configured to input the task instruction features and the current environment state features into a multi-task conditional fusion model to integrate information to obtain multi-task conditional features.

[0157] Preferably, the integration sub-unit is further configured to distill the multi-task conditional features to obtain compressed multi-task conditional features.

[0158] Preferably, the embodiment of the present application further provides an electronic device, comprising a processor and a memory, the processor and the memory are connected through a communication bus; wherein the processor is configured to call and execute a program stored in the memory; the memory is configured to store the program, and the program is configured to implement the multi-modal data processing method supporting multi-task switching provided by the above method embodiment.

[0159] Preferably, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the multi-modal data processing method supporting multi-task switching provided by the above method embodiment.

[0160] To sum up, the embodiment of the present application provides a multi-modal data processing method supporting multi-task switching and related devices, which obtains the optimal action sequence of the robot according to the historical task instruction, the current task instruction and the environment image, effectively solves the complexity problem when the robot switches tasks, and improves the accuracy of the output robot action when switching tasks.

[0161] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, it is described more simply, and the relevant part can be referred to the part of the method embodiment. The above-described system and system embodiment are only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0162] Those skilled in the art will further appreciate that the units and algorithms described in connection with the examples disclosed herein can be embodied directly in hardware, in software, or in a combination of the two. For the sake of brevity, descriptions of a method of execution of the examples will not be repeated here. To the extent that the steps of the examples are performed by hardware, the steps can be performed by the hardware special purpose machine. To the extent that the steps of the examples are performed by software, the steps can be performed by the software executing on a general purpose computer. To the extent that the steps of the examples are performed by a combination of hardware and software, the steps can be performed by the combination of the general purpose computer executing the software and the hardware special purpose machine.

[0163] The above description of disclosed embodiments provides examples, and is not intended to be limiting. Numerous variations, additions, omissions and other modifications not specifically described can be made to the disclosed embodiments. Specifically, any of the methods described herein can be performed in a different order, can be performed concurrently, can be performed at different times, and / or can be performed by different components, without departing from the scope of the present disclosure. Accordingly, other examples are within the scope of the following claims.

Claims

1. A multi-modal data processing method supporting multi-task switching, characterized by, The method comprises: obtaining an environment image of an environment currently occupied by a robot, historical task instructions, current task instructions, a robot state and robot action noise; generating fusion features and multi-task condition features using the environment image, the historical task instructions and the current task instructions; wherein the fusion features are generated based on visual features and current environment state features extracted from the environment image, and task instruction features extracted from the historical task instructions and the current task instructions; the multi-task condition features are generated based on the current environment state features and the task instruction features; the current environment state features are used to represent an execution stage of the robot in executing the historical task instructions; extracting robot state features from the robot state; inputting the robot action noise, the fusion features, the multi-task condition features and the robot state features into a preset action expert model for action decision-making to obtain an optimal action sequence of the robot.

2. The method of claim 1, wherein, Generating fusion features and multi-task condition features using the environment image, the historical task instructions and the current task instructions comprises: extracting visual features from the environment image using an image encoding model; inputting the environment image and a preset prompt word into a visual language large model to obtain a current environment state; inputting the historical task instructions and the current task instructions into a large language model to obtain corresponding task instruction features; inputting the current environment state into the large language model to obtain corresponding current environment state features; fusing the visual features, the task instruction features and the current environment state features using a designated encoder to obtain fusion features; inputting the task instruction features and the current environment state features into a multi-task condition fusion model for information integration to obtain multi-task condition features.

3. The method of claim 1, wherein, Extracting robot state features from the robot state comprises: extracting robot state features from the robot state using a robot state encoding model.

4. The method of claim 2, wherein, After obtaining the multi-task condition features, further comprising: distilling the multi-task condition features to obtain compressed multi-task condition features.

5. The method of claim 1, wherein, The process of obtaining an environment image of an environment currently occupied by a robot comprises: calling a first camera and a second camera to obtain an environment image of an environment currently occupied by a robot, the first camera having a first perspective and being mounted on the robot, and the second camera having a global perspective and being arranged in an environment occupied by the robot.

6. The method of claim 1, wherein, The process of obtaining a robot state comprises: calling a robot body sensor to obtain a robot state, the robot state at least including joint angles of the robot and an end effector state.

7. The method according to any one of claims 1 to 6, characterized in that, After obtaining the optimal action sequence of the robot, further comprising: performing denoising processing on the optimal action sequence.

8. A multi-modal data processing system supporting multi-task switching, characterized by The system comprises: an obtaining unit configured to obtain an environment image of an environment currently occupied by a robot, historical task instructions, current task instructions, a robot state and robot action noise; The generating unit is configured to generate fusion features and multi-task condition features by using the environment image, the historical task instruction, and the current task instruction; the fusion features are generated based on visual features and a current environment state feature extracted from the environment image and task instruction features extracted from the historical task instruction and the current task instruction; the multi-task condition features are generated based on the current environment state feature and the task instruction features; and the current environment state feature is used to represent an execution stage of the robot in executing the historical task instruction. The extracting unit is configured to extract robot state features from the robot state. The decision unit is configured to input the robot action noise, the fusion features, the multi-task condition features, and the robot state features into a preset action expert model to perform action decision, so as to obtain an optimal action sequence of the robot.

9. An electronic device, comprising: The processor and the memory are connected through a communication bus; the processor is configured to call and execute a program stored in the memory; and the memory is configured to store the program, which is used to implement the multi-modal data processing method supporting multi-task switching according to any one of claims 1 to 7. The computer program stored in the computer readable storage medium is executed by the processor to implement the multi-modal data processing method supporting multi-task switching according to any one of claims 1 to 7. ​ 10. A computer-readable storage medium, characterized in that, ​

Images