Autonomous robot multi-task operation planning method and system driven by body cognition large model

Abstract

The present application relates to a kind of self-robot multi-task operation planning method and system driven by embodied cognitive large model, wherein the method comprises: S1, based on the real-time acquisition RGB diagram and depth diagram are encoded to obtain embodied visual representation;S2, obtain natural language instruction and and embodied visual representation are fused across modalities, obtain fusion feature, and based on fusion feature multi-task decomposition scheme;S3, based on multi-task decomposition scheme, utilize diffusion strategy to generate the continuous action trajectory of robot end effector;S4, obtain the second RGB image and second depth diagram after robot executes according to continuous action trajectory, and it is used as closed loop feedback signal, system is used to realize the above-mentioned method.Compared with prior art, the present application is based on embodied cognitive large model, and the depth of visual and language modalities is deeply fused, and combined with diffusion strategy action decision module, accurate action trajectory is predicted, and the multi-task autonomous planning and accurate execution capability of robot in dynamic complex environment is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous robot operation, and in particular to a method and system for planning multi-task operation of autonomous robots driven by an embodied cognitive large model. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, big data, and deep learning, robotics is gradually shifting from traditional pre-programmed operations to intelligent and autonomous operation modes. Especially in complex and dynamic real-world environments, the ability of robots to autonomously perform multiple tasks has become a research hotspot. However, current traditional robot operation planning methods suffer from the following technical bottlenecks and shortcomings: 1) Limitations in task planning: Traditional robot systems often rely on manually defined or simple rules for task decomposition and planning, lacking flexibility and generalization, making it difficult to achieve efficient and reliable planning when facing complex environments and multi-task collaborative operations; 2) Insufficient modal information fusion: Most existing methods typically rely only on single modal data (such as vision or force perception), resulting in an insufficient and in-depth understanding of environmental information, affecting the stability and accuracy of robot task execution; 3) Lack of accurate real-time motion trajectory prediction, limiting the success rate of robot operations; 4) Lack of real-time adaptability to environmental changes: In dynamic environments, traditional robot systems struggle to perceive environmental changes in real time and dynamically adjust task execution strategies accordingly, leading to poor environmental adaptability and robustness. While the rise of embodied cognitive large models offers potential solutions to the aforementioned problems, such as the Chinese patent application CN118036750A, which provides an embodied intelligent task planning method using multimodal large models and behavior tree structures, combining environmental feedback and multimodal information, although it solves the problem of insufficient planning reliability in existing technologies, it still has the following drawbacks: a) It is more suitable for single-task planning and has limitations for multi-task planning; b) Actions rely on sampling probabilities for generation, resulting in inaccurate prediction results; c) It only performs a one-time environmental assessment during the planning stage and cannot respond to dynamic changes during the execution process in real time.

[0003] Therefore, providing a robot multi-task planning method based on a large embodied cognition model is a technical problem that needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an autonomous robot multi-task operation planning method and system driven by an embodied cognitive large model. It deeply integrates visual and linguistic modalities and combines a diffusion strategy action decision module to predict accurate action trajectories. This effectively solves the problems of insufficient accuracy, poor generalization, and low real-time performance in traditional robot multi-task planning. It significantly improves the robot's multi-task autonomous planning and precise execution capabilities in dynamic and complex environments, and has better generalization ability and real-time adaptability. It can be widely used in multiple fields such as automated production, intelligent assembly, and service robots.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] According to a first aspect of the present invention, a method for multi-task operation planning of an autonomous robot driven by an embodied cognitive large model is provided, comprising:

[0007] S1. Real-time acquisition of the first RGB image and the first depth map when the robot performs multi-task operations, and encoding based on the RGB image and the depth map to obtain an embodied visual representation;

[0008] S2. Obtain the natural language instructions for the multi-task operation, perform cross-modal fusion of the natural language instructions and embodied visual representation to obtain fusion features, and perform multi-task decomposition based on the fusion features to obtain a multi-task decomposition scheme; the multi-task decomposition scheme includes the execution order and priority of a single task.

[0009] S3. Based on the multi-task decomposition scheme, a diffusion strategy is used to generate the continuous motion trajectory of the robot's end effector.

[0010] S4. Obtain the second RGB image and the second depth map after the robot executes the continuous motion trajectory. Compare the first RGB image and the first depth map with the second RGB image and the second depth map respectively. If they are the same, jump to S5; otherwise, generate a prompt message based on the difference and return to S2 to update the multi-task decomposition scheme.

[0011] S5. Determine whether the continuous motion trajectory has been successfully executed. If successful, determine whether the multi-task has been completed. If not completed, return to S1. If the execution is unsuccessful, generate a feedback packet and return to S2 to guide the update of the multi-task decomposition strategy and S4 as an attachment condition.

[0012] As a preferred technical solution, the method for obtaining the embodied visual representation includes:

[0013] The first RGB image and the first depth map are preprocessed;

[0014] The preprocessed first RGB image and first depth map are encoded using a ViT encoder to obtain an embodied visual representation; the visual representation includes semantic features and spatial geometric features.

[0015] As a preferred technical solution, the cross-modal fusion method is as follows: the natural language instruction is encoded to obtain text features, and the text features are fused with the embodied visual representation using a cross-modal feature attention mechanism.

[0016] As a preferred technical solution, the method for obtaining the text features includes:

[0017] The natural language instructions are split into multiple tokens to obtain a discrete index sequence;

[0018] The discrete index sequence is mapped to a low-dimensional dense vector space to form an embedding matrix;

[0019] The discrete index sequence is subjected to sine-cosine position encoding, and the result of the sine-cosine position encoding is added to the embedding matrix to obtain the text feature.

[0020] As a preferred technical solution, the feature fusion method includes:

[0021] Map the embodied visual representation and the text features to a unified dimension;

[0022] The expression for the first attention of text features to embodied visual representation after dimensionality unification is: The second attention of embodied visual representation to text features is calculated, and its expression is: ,in, and These represent the attention weights of text features on the embodied visual representation after normalization, and the attention weights of embodied visual representation on the text features after normalization, respectively. And represent the j-th embodied visual representation, Represents the j-th text feature; Indicates the quantity of embodied visual representations; Indicates the number of text features;

[0023] The intermediate feature is obtained by adding the first attention and the second attention to the text features and the embodied visual representation, respectively.

[0024] The intermediate features are processed by residual processing and normalization to obtain the fused features.

[0025] As a preferred technical solution, the method for calculating the priority includes:

[0026] A learnable task-level CLS summary token is inserted into the fusion features to form a complete input sequence;

[0027] The input sequence is processed using a high-level Transformer to obtain a CLS vector. The task priority is then calculated based on this CLS vector, with the following expression:

[0028] )

[0029] in, Indicates priority; Represents the CLS vector; This indicates the task category header for multi-task decomposition.

[0030] As a preferred technical solution, the method for generating the continuous motion trajectory includes:

[0031] Based on the aforementioned multi-task decomposition scheme, the continuous motion space of the robot's end effector is obtained, and the continuous motion space is discretized into multiple motion units to construct a discrete motion space.

[0032] A continuous initial trajectory is obtained by sampling from a Gaussian distribution, and then each element of the trajectory is mapped to an initial discrete sequence according to discrete intervals;

[0033] The initial discrete sequence is used as the initial value of the diffusion model. Based on the embodied visual representation, the diffusion model's denoising network is used for iterative denoising. In each iteration, the action unit is adjusted to obtain the optimized continuous action trajectory.

[0034] According to a second aspect of the present invention, an autonomous robot multi-task operation planning system driven by an embodied cognitive large model is provided for implementing the above-described method.

[0035] According to a third aspect of the present invention, an autonomous robot multi-task operation planning device driven by an embodied cognitive large model is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0036] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0037] Compared with existing technologies, this invention performs cross-modal fusion of image features from real-time acquired images with text instructions from robot tasks. Based on the fused features, it decomposes multiple tasks to explore the execution order and priority of each sub-task, achieving the optimal execution scheme for multiple tasks in a single inference process. Considering that the continuous motion trajectory obtained by inference may have significant uncertainties and errors, this invention also introduces a diffusion strategy for iterative convergence on the conditional probability flow. The purpose is to significantly reduce the impact of the morning on trajectory generation and achieve online correction of abnormal trajectories. In addition, this invention also collects environmental information after the robot executes the predicted continuous motion trajectory and uses this environmental information as a feedback signal for closed-loop feedback, significantly improving the robot's adaptability and robustness in the face of environmental changes. This provides an efficient, accurate, and robust solution for autonomous robot operation in complex scenarios. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0041] This invention provides a robot multi-task operation planning method that uses a visual Transformer driven by an embodied cognition large model and a GPT-4V model to deeply fuse visual and linguistic modalities, and combines a diffusion strategy action decision module to predict accurate motion trajectories. This method effectively solves the problems of insufficient accuracy, poor generalization, and low real-time performance in traditional robot multi-task planning, significantly improves the robot's ability to autonomously plan and accurately execute multi-task tasks in dynamic and complex environments, and has better generalization ability and real-time adaptability. It can be widely applied in multiple fields such as automated production, intelligent assembly, and service robots.

[0042] Detailed procedures are as follows Figure 1 As shown, it includes:

[0043] S1. Acquire the first RGB image and the first depth map when the robot performs multi-task operations in real time, and encode the embodied visual representation based on the RGB image and the depth map.

[0044] The robot uses RGB-D vision sensors to acquire RGB images and depth information of the environment in real time, forming environmental perception input. Specifically, RGB-D cameras such as Intel RealSense or Azure Kinect installed on the robot capture images of the environment in real time and acquire corresponding depth map data to provide rich environmental information.

[0045] S11. Preprocess the first RGB image and the first depth map.

[0046] S111. Crop the first RGB image and the first depth map to the same size to facilitate subsequent fusion.

[0047] S112. Standardize the first RGB image and the first depth map after S111 to eliminate differences in pixel value distribution.

[0048] S113. Perform noise reduction processing on the standardized first RGB image and the first depth map to reduce sensor noise.

[0049] S12. The preprocessed first RGB image and the first depth map are encoded using the ViT encoder to obtain embodied visual representations; the visual representations include semantic features and spatial geometric features.

[0050] The preprocessed first RGB image and the first depth map are fed into the Vision Transformer (ViT) encoder to obtain a more robot-friendly embodied visual representation, and output a high-dimensional feature vector as the embodied visual representation.

[0051] Step S1 enables the robot to accurately identify obstacles and target objects when performing environmental recognition. For example, when handling the task of recognizing items on a table, the model can accurately identify and locate multiple objects such as cups and books, and extract their spatial location and semantic features.

[0052] S2. Obtain natural language instructions for multi-task operations, perform cross-modal fusion of natural language instructions and embodied visual representations to obtain fusion features, and perform multi-task decomposition based on fusion features to obtain a multi-task decomposition scheme; the multi-task decomposition scheme includes the execution order and priority of individual tasks.

[0053] S21. Obtaining text features:

[0054] S211. Split the natural language instruction into multiple tokens to obtain a discrete index sequence { ,in Indicates the first Discrete indexes of tokens.

[0055] S212. Map the discrete index sequence to a low-dimensional dense vector space to form an embedding matrix.

[0056] S213. Perform sine-cosine positional encoding on the discrete index sequence, and add the result of the sine-cosine positional encoding to the embedding matrix to obtain the text features. The expression is as follows: ,in Representing text features, Represents the embedding matrix. This indicates a sine-prepositional encoding.

[0057] S22, Cross-modal fusion:

[0058] Natural language instructions are encoded to obtain text features, and cross-modal feature attention mechanism is used to fuse text features with embodied visual representations.

[0059] Details include:

[0060] S221. Map embodied visual representations and textual features to a unified dimension, placing them in a unified semantic space to facilitate subsequent interaction.

[0061] S222. The expression for the first attention of text features to embodied visual representation after dimensional unification is:

[0062] ;

[0063] The expression for calculating the second attention of embodied visual representation to text features is as follows:

[0064] ,

[0065] in, and These represent the attention weights of text features on the embodied visual representation after normalization, and the attention weights of embodied visual representation on the text features after normalization, respectively. And represent the j-th embodied visual representation, Represents the j-th text feature; Indicates the quantity of embodied visual representations; Indicates the number of text features.

[0066] The above operations weight each text token across all embodied visual representations, identifying the image region most relevant to its own semantics, such as the specific pixel block on the table corresponding to the cup.

[0067] S223. The first attention and the second attention are added to the text features and the embodied visual representation respectively to obtain the intermediate features, so that the embodied visual representation focuses on the text token in reverse, injecting spatial-geometric information into the text sequence, so that the text understands the spatial relationships in the text instructions.

[0068] S224. After residual processing and normalization of the intermediate features, the fused features are obtained. The two interacting features are then stabilized through residual connections and layer normalization to suppress noise and stabilize gradients. The final fused features are obtained, each element of which carries both linguistic and visual context.

[0069] S3. Based on the multi-task decomposition scheme, a diffusion strategy is used to generate the continuous motion trajectory of the robot's end effector.

[0070] S31. Insert a learnable task-level CLS summary token into the fused features to form a complete input sequence.

[0071] S32. Process the input sequence using a high-level Transformer to obtain a CLS vector. Calculate the task priority based on the CLS vector, with the following expression:

[0072] )

[0073] in, Represents the CLS vector; The task category header represents the multi-task decomposition; The value indicates priority; the larger the value, the higher the priority. Pressing P in descending order will give you the final multi-task execution sequence. For example: pick up the cup, move to the right side of the table, put down the cup, and grab the book.

[0074] S4. Obtain the second RGB image and the second depth map after the robot executes the continuous motion trajectory. Compare the first RGB image and the first depth map with the second RGB image and the second depth map respectively. If they are the same, jump to S5; otherwise, generate prompt information based on the differences and return to S2 to update the multi-task decomposition scheme.

[0075] The trajectory consists of countless continuous control variables, such as position, attitude, joint angle, and velocity, which cannot be enumerated by rules. Furthermore, if the continuous trajectory prediction is performed directly based on the multi-task decomposition scheme obtained by S2, there may be significant uncertainties and errors. Therefore, this invention introduces a diffusion strategy to optimize the continuous trajectory prediction process.

[0076] S41. Obtain the continuous motion space of the robot end effector based on the multi-task decomposition scheme, and discretize the continuous motion space into multiple motion units to construct a discrete motion space.

[0077] S42. Obtain a continuous initial trajectory by sampling from a Gaussian distribution, and then map each element of the trajectory to an initial discrete sequence according to the discrete interval.

[0078] S43. The initial discrete sequence is used as the initial value of the diffusion model. Based on the embodied visual representation, the denoising network of the diffusion model is used for iterative denoising. The action unit is adjusted in each iteration to obtain the optimized continuous action trajectory.

[0079] The diffusion strategy treats the entire trajectory as a random variable and uses conditional probability inference to find the curve that best matches the current state, environment, and task. After processing in step S4, the specific position, optimal angle, and optimal movement path of the robot grabbing the cup are obtained.

[0080] During robot task execution, the embodied cognition big data model analyzes changes in the environment in real time and dynamically updates the task execution strategy. Specifically, this includes real-time detection of changes in objects or states in the environment, such as the movement of items on a table or the sudden appearance of obstacles. Visual feature analysis identifies environmental change cues, and based on these cues, the execution order and priority of sub-tasks are dynamically adjusted, along with subsequent action planning. This guides the model to recalculate the priorities and parameters of relevant sub-tasks (for blocking events, "obstacle clearing" or "repositioning" tasks are inserted and the original task priority is lowered; for optimization events, the order is preserved and only the target pose is updated). The updated plan immediately relies on a diffusion strategy to truncate the already executed trajectory. Starting from the current end state and using the new target constraint g′ as a condition, a hot start denoising process of 10–20 steps yields the remaining trajectory. After rapid collision checking, this trajectory is seamlessly stitched to the historical trajectory, achieving millisecond-level online incremental replanning. By continuously looping "environmental change detection → prompt generation → context refresh → local re-inference → hot start replanning", the robot can adaptively adjust the order of sub-tasks and action trajectories without pausing in constantly evolving real-world scenarios, ensuring the continuity, safety and robustness of multi-task execution.

[0081] S5. Determine whether the continuous motion trajectory has been successfully executed. If successful, determine whether the multi-task has been completed. If not completed, return to S1. If the execution is unsuccessful, generate a feedback packet and return to S2 to guide the update of the multi-task decomposition strategy and S4 as additional conditions.

[0082] In detail, immediately after the completion of each subtask in the multi-task process, a visual feedback loop is initiated. The robot captures the current RGB-D image frame_t and stitches it together with the end effector pose to form observation data. Subsequently, the target detection and pose estimation network regresses the 3D position and pose of the target object from frame_t, denoted as detected_pose_t. The detection result is subtracted from the expected pose in the multi-task decomposition strategy to calculate the error vector. If the magnitude of the error vector is less than a threshold, a state label "success" is generated, and the action sequence continues to advance along the current trajectory; if it exceeds the threshold, it is labeled "fail," and a feedback packet is constructed. The feedback packet is first sent to the diffusion strategy module in step S4, where it is merged with the original target constraint as an additional condition. During the next denoising hot start, the incomplete trajectory is refreshed, and the corrected remaining trajectory is output. Simultaneously, the feedback packet is sent back to S2 to guide the update of the multi-task decomposition strategy, allowing the dynamic task rearrangement logic to determine whether subtasks such as "repositioning" or "obstacle clearing" need to be inserted. The entire process sequentially goes through "observation and acquisition → target detection → error assessment → feedback generation → diffusion replanning → dynamic rearrangement", with end-to-end latency controlled within approximately 20 milliseconds. This allows for continuous correction of placement accuracy and improvement of the robustness and accuracy of overall motion planning while the robot continues to move.

[0083] The present invention also provides an autonomous robot multi-task operation planning system driven by an embodied cognitive large model, for implementing the above method.

[0084] Furthermore, the autonomous robot multi-task operation planning device driven by the embodied cognitive large model provided by this invention includes a central processing unit (CPU), which can execute various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from storage units into random access memory (RAM). The RAM can also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0085] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0086] The processing unit executes the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 by any other suitable means (e.g., by means of firmware).

[0087] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0088] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0089] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0090] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-task operation planning method for autonomous robots driven by an embodied cognitive large model, characterized in that, include: S1. Real-time acquisition of the first RGB image and the first depth map when the robot performs multi-task operations, and encoding based on the RGB image and the depth map to obtain an embodied visual representation; S2. Obtain the natural language instructions for the multi-task operation, perform cross-modal fusion of the natural language instructions and embodied visual representation to obtain fusion features, and perform multi-task decomposition based on the fusion features to obtain a multi-task decomposition scheme; the multi-task decomposition scheme includes the execution order and priority of a single task. S3. Based on the multi-task decomposition scheme, a diffusion strategy is used to generate the continuous motion trajectory of the robot's end effector. The method for generating the continuous motion trajectory includes: Based on the aforementioned multi-task decomposition scheme, the continuous motion space of the robot's end effector is obtained, and the continuous motion space is discretized into multiple motion units to construct a discrete motion space. A continuous initial trajectory is obtained by sampling from a Gaussian distribution, and then each element of the trajectory is mapped to an initial discrete sequence according to discrete intervals; The initial discrete sequence is used as the initial value of the diffusion model. Based on the embodied visual representation, the denoising network of the diffusion model is used for iterative denoising. In each iteration, the action unit is adjusted to obtain the optimized continuous action trajectory. S4. Obtain the second RGB image and the second depth map after the robot executes the continuous motion trajectory. Compare the first RGB image and the first depth map with the second RGB image and the second depth map respectively. If they are the same, jump to S5; otherwise, generate a prompt message based on the difference and return to S2 to update the multi-task decomposition scheme. S5. Determine whether the continuous motion trajectory has been successfully executed. If successful, determine whether the multi-task has been completed. If not completed, return to S1. If the execution is unsuccessful, generate a feedback packet and return to S2 to guide the update of the multi-task decomposition strategy and S4 as additional conditions.

2. The method for multi-task operation planning of autonomous robots driven by an embodied cognitive large model according to claim 1, characterized in that, The method for obtaining the embodied visual representation includes: The first RGB image and the first depth map are preprocessed; The preprocessed first RGB image and first depth map are encoded using a ViT encoder to obtain an embodied visual representation; the visual representation includes semantic features and spatial geometric features.

3. The method for multi-task operation planning of autonomous robots driven by an embodied cognitive large model according to claim 1, characterized in that, The cross-modal fusion method is as follows: the natural language instructions are encoded to obtain text features, and the text features are fused with the embodied visual representation using a cross-modal feature attention mechanism.

4. The method for multi-task operation planning of autonomous robots driven by an embodied cognitive large model according to claim 3, characterized in that, The methods for obtaining the aforementioned text features include: The natural language instructions are split into multiple tokens to obtain a discrete index sequence; The discrete index sequence is mapped to a low-dimensional dense vector space to form an embedding matrix; The discrete index sequence is subjected to sine-cosine position encoding, and the result of the sine-cosine position encoding is added to the embedding matrix to obtain the text feature.

5. The method for multi-task operation planning of autonomous robots driven by an embodied cognitive large model according to claim 3, characterized in that, The feature fusion method includes: Map the embodied visual representation and the text features to a unified dimension; The expression for the first attention of text features to embodied visual representation after dimensionality unification is as follows: The second attention of embodied visual representation to text features is calculated, and its expression is: ,in, and These represent the attention weights of text features on the embodied visual representation after normalization, and the attention weights of embodied visual representation on the text features after normalization, respectively. And represent the j-th embodied visual representation, Represents the j-th text feature; Indicates the quantity of embodied visual representations; Indicates the number of text features; The intermediate feature is obtained by adding the first attention and the second attention to the text features and the embodied visual representation, respectively. The intermediate features are processed by residual processing and normalization to obtain the fused features.

6. The method for multi-task operation planning of autonomous robots driven by an embodied cognitive large model according to claim 1, characterized in that, The methods for calculating the aforementioned priority include: A learnable task-level CLS summary token is inserted into the fusion features to form a complete input sequence; The input sequence is processed using a high-level Transformer to obtain a CLS vector. The task priority is then calculated based on this CLS vector, with the following expression: ) in, Indicates priority; Represents the CLS vector; This indicates the task category header for multi-task decomposition.

7. A multi-task operation planning system for autonomous robots driven by an embodied cognitive large model, characterized in that, The system is used to implement the method described in any one of claims 1 to 6.

8. A device for multi-task operation planning of an autonomous robot driven by an embodied cognitive large model, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images