Multi-modal information fusion body-equipped intelligent robot control method

Abstract

The invention discloses a control method for a multi-modal information fusion intelligent robot with a body. The control method comprises the following steps: initializing a system, and collecting surrounding physical environment and object state information and a natural language instruction of a user; performing scene understanding and task analysis, processing data through a multi-modal information fusion mechanism and a cross-modal attention module, and generating unified multi-modal data; task planning and priority ranking are carried out, complex tasks are decomposed into subtask sequences, and a priority ranking layer dynamically adjusts the execution sequence; performing action execution and feedback adjustment, and generating a control instruction through a self-adaptive operation control algorithm; continuous learning and strategy verification are carried out, integrated execution is realized by using a hybrid AI system, and the robustness of an operation strategy is verified through a simulation environment; and closed-loop iteration is carried out to realize real-time response of the intelligent robot with the body. According to the method, the perception understanding precision and the task execution efficiency of the intelligent robot with the body are improved, the operation precision adaptability and the system robustness flexibility are guaranteed, and the method is suitable for multiple scenes.

Description

Technical Field

[0002] This invention relates to the fields of artificial intelligence and robotics, and in particular to a method for controlling an embodied intelligent robot by multimodal information fusion. Background Technology

[0003] In recent years, with the rapid development of artificial intelligence and robotics, embodied intelligent robots have been increasingly widely used in industrial production, medical care, service industries, and disaster response. The Vision-Language-Motion (VLA) model, by combining computer vision and natural language processing, provides a new technical path for intelligent decision-making in robots. However, existing technologies mainly rely on visual and linguistic information, lack precise control through force feedback, have flat task representations, lack hierarchical decomposition, and struggle to handle complex multi-step tasks.

[0004] In complex scenarios, systems need to simultaneously handle multimodal data, long-term contexts, and dynamically changing environments. Existing methods still have many shortcomings in terms of robustness, adaptability, and safety reliability. Specifically, these include: lack of force perception capability: traditional VLA models mainly rely on visual and linguistic information and cannot perceive physical interaction information such as contact force and friction, which can easily lead to operation failure or object damage in tasks requiring delicate manipulation; limited task understanding capability: existing methods mostly use planar task representations and lack hierarchical task decomposition and planning capabilities, making it difficult to cope with complex multi-step tasks; insufficient adaptability and feedback processing: existing robot execution strategies are fixed and lack adaptive adjustments based on object differences, environmental states, and task requirements, as well as dynamic processing of real-time feedback; lack of safety verification and continuous learning mechanisms: there is a lack of closed-loop verification methods for system robustness, safety, and continuous learning iteration, making it difficult to guarantee long-term reliable operation in complex scenarios.

[0005] This invention proposes a multimodal information fusion-based control method for embodied intelligent robots, aiming to achieve effective control of embodied intelligent robots in complex scenarios. Summary of the Invention

[0006] The purpose of this invention is to overcome the deficiencies in the aforementioned background technology. This invention proposes a multimodal information fusion-based embodied intelligent robot control method, which adopts a four-layer architecture of scene understanding, task planning, priority ranking, and action execution. Through the fusion of visual, linguistic, and force-sensory multimodal information, combined with adaptive operation control, dynamic priority emergency mechanism, hybrid AI reasoning, and continuous learning closed-loop iteration, intelligent control of the embodied robot is achieved.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] The application discloses a kind of multi-modal information fusion embodied intelligent robot control method, it is characterized in that, the control method adopts four-layer architecture and combines multi-modal information fusion mechanism, self-adapting operation control algorithm, dynamic priority and emergency mechanism and hybrid AI system to realize the control of embodied intelligent robot, the four-layer architecture is scene understanding layer, task planning layer, priority sorting layer and action execution layer, the implementation steps of the control method are as follows:

[0009] Step S1, system initialization, after embodied robot starts, vision sensor, six-dimensional force and torque sensor and language understanding module start and collect surrounding physical environment information, object state information and user natural language instruction;

[0010] Step S2, scene understanding and task analysis, scene understanding layer processes acquisition data by multi-modal information fusion mechanism, visual encoder extracts multi-scale visual features from information collected by vision sensor to identify object type and environment state, language encoder analyzes user natural language instruction to extract task semantics, force sensation encoder encodes six-dimensional force and torque sensor data into feature vector to reflect initial physical interaction information, then realizes deep fusion of vision, language and force sensation through cross-modal attention module, generates unified multi-modal data, and completes task analysis;

[0011] Step S3, task planning and priority sorting, task planning layer generates unified multi-modal data based on step S2, decomposes complex task into subtask sequence and constructs task graph;Priority sorting layer real-time perceives environment state, subtask complexity and resource consumption, evaluates subtask based on multi-dimensional evaluation parameters and dynamically adjusts subtask execution order;

[0012] Step S4, action execution and feedback adjustment, action execution layer generates accurate control instruction based on unified multi-modal data generated in step S2 and subtask sorted in step S3, dynamically adjusts force, speed and trajectory to realize action execution, while real-time monitoring execution effect and adjusting operation strategy according to feedback;If fault or emergency is detected, priority sorting layer quickly switches strategy;

[0013] Step S5, continuous learning and strategy verification, unified multi-modal data generated in step S2 is preprocessed, calculated, decided and post-processed by hybrid AI system, and through long context and complex reasoning, multiple rounds of interaction and historical information are processed, to cope with the dynamics and uncertainty of environment;Unified multi-modal data in execution process is uploaded to continuous learning module, to optimize model parameters and planning strategy, and the robustness of operation strategy is verified through fault injection test in simulation environment;

[0014] Step S6, closed loop iteration, repeat steps S2 to S5, realize the real-time response of embodied intelligent robot to dynamic environment and task changes, complete continuous intelligent operation.

[0015] Further, the multi-modal information fusion mechanism in step S2 includes three core encoders and a cross-modal attention module, specifically as follows:

[0016] The visual encoder adopts a multi-scale feature extraction network to process the environmental data collected by the visual sensor, and outputs object contours, positions, material textures and environmental obstacle distribution features;

[0017] The language encoder is based on a pre-trained semantic understanding model to analyze the task target and operation constraints in the user instruction, and outputs a task semantic vector;

[0018] The force encoder filters and normalizes the real-time force value data collected by the six-axis force and torque sensor, maps the six-axis force and torque data into a fixed-dimensional feature vector, and reflects the contact force and impedance state during operation;

[0019] The cross-modal attention module calculates the attention weights of visual features, force features and semantic features guided by the language semantic vector, realizes the dynamic association and information complementation of the three, and generates a unified multi-modal representation.

[0020] Further, the task planning layer in step S3 adopts a two-step method of complex task decomposition and task graph construction, specifically as follows:

[0021] The complex task decomposition decomposes a single complex task into a sequence of subtasks based on task semantics and object attributes; the task graph construction constructs a loop-free task graph with subtasks as nodes and dependency relationships between subtasks as directed edges, and labels the input, output and required resources of each subtask.

[0022] Further, the priority sorting layer in step S3 adopts a multi-dimensional evaluation and dynamic adjustment mechanism, specifically as follows:

[0023] The parameters of the multi-dimensional evaluation include task difficulty, risk coefficient, time constraint and energy consumption;

[0024] The dynamic adjustment rule is: when the environment state changes or the subtask progress is delayed, the priority sorting layer re-evaluates each subtask and adjusts the execution order; when a fault occurs, trigger the emergency strategy to execute the fault troubleshooting subtask or safety protection subtask first.

[0025] Further, the steps of the adaptive operation control algorithm in step S4 are as follows:

[0026] Step S41, object material identification and demand assessment, combining visual features and force sensation features, inferring object material type through classification model, and assessing task refinement demand based on material properties;

[0027] Step S42, adaptive force control and trajectory planning, dynamically adjusting the force threshold, motion speed and trajectory parameters of the robot arm based on the material assessment results; combining the environmental data and object features output by the visual sensor and visual encoder, generating a collision-free optimal trajectory; during execution, real-time force feedback is received, and if the detected force exceeds the threshold or the trajectory deviation is greater than the preset value, the control command is immediately corrected;

[0028] Step S43, effect monitoring and online adjustment, monitoring the execution effect through visual post-processing and force sensation measurement, and adjusting the operation parameters if over-execution or under-execution is found.

[0029] Further, the hybrid AI system in step S5 includes the following modules:

[0030] Long context and complex reasoning module: chain reasoning is used to handle multi-round interactive tasks, tree reasoning is used to deal with multi-branch decision-making tasks, and out-of-order reasoning is supported to call historical task information;

[0031] Environmental dynamics and uncertainty coping module: real-time monitoring of environmental parameters, when the environmental change is greater than the preset threshold, triggering the scene understanding layer to re-perceive and the task planning layer to regenerate sub-task sequences;

[0032] Multi-task, cross-domain collaboration module: supports simultaneous processing of multiple sub-tasks, allocates computing resources through a task scheduler, and realizes parameter migration across domain tasks;

[0033] Safety and robustness verification module: inject faults in the simulation environment to test the system's fault handling capability; use adversarial samples to train visual and force sensation encoders to resist adversarial attacks.

[0034] Further, the continuous learning module in step S5 specifically includes the following units:

[0035] Data collection unit: automatically stores multi-modal data during execution, and labels successful and failed cases;

[0036] Model optimization unit: regularly adjusts the parameters of the visual encoder, language encoder and adaptive control algorithm using labeled data to improve object recognition accuracy, command parsing accuracy and operation success rate.

[0037] Further, the application scenarios of the multi-modal information fusion embodied intelligent robot control method include but are not limited to industrial assembly lines, medical care robots, precision machining, household service robots and rescue robots.

[0038] Compared with the prior art, the present invention, employing the above technical solution, has the following beneficial effects:

[0039] (1) The present invention proposes a multimodal information fusion embodied intelligent robot control method, which breaks through the limitations of single modality by deeply fusing visual, linguistic and force-sensory multimodal information and cross-modal attention mechanism, comprehensively and accurately identifies environmental state and task semantics, provides reliable input for decision-making, and realizes multimodal fusion to improve perception and understanding accuracy.

[0040] (2) The present invention proposes a multimodal information fusion embodied intelligent robot control method. In the four-layer architecture, the task planning layer decomposes complex tasks into ordered sub-tasks and constructs a task graph. The priority sorting layer dynamically adjusts the execution order based on multiple dimensions such as difficulty and risk to ensure that key tasks are processed first, significantly improves the overall execution efficiency, and realizes intelligent planning and dynamic scheduling to improve task efficiency.

[0041] (3) The present invention proposes a multimodal information fusion embodied intelligent robot control method, which matches operation requirements through material recognition and dynamically adjusts trajectory and force in combination with real-time force feedback to achieve compliant operation; at the same time, it corrects deviations in a timely manner through effect monitoring to ensure stable operation quality in dynamic environments and achieves adaptive control to ensure operation accuracy and environmental adaptability.

[0042] (4) The present invention proposes a multimodal information fusion embodied intelligent robot control method, which utilizes a dynamic emergency mechanism to quickly respond to environmental changes, faults or emergencies. The reasoning ability of the hybrid AI system supports multi-round interaction and multi-branch decision-making. Combined with security verification, it significantly improves the system's anti-interference and fault tolerance, and achieves strong robustness and flexibility to cope with complex scenarios.

[0043] (5) The present invention proposes a multimodal information fusion embodied intelligent robot control method, which achieves long-term performance improvement of the system through closed-loop iterative optimization model; the unified architecture and modular design are adapted to multiple scenarios such as industry, medical care, and home, reducing cross-domain application costs, and have strong practical value and promotion potential, achieving continuous evolution and wide adaptation, thus expanding practical value. Attached Figure Description

[0044] Figure 1 This is a flowchart of the control method of the present invention;

[0045] Figure 2 This is an overall structural diagram of the four-layer architecture of the present invention. Detailed Implementation

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort fall within the protection scope of the present application.

[0047] A multi-modal information fusion embodied intelligent robot control method, as shown in Figure 1 , characterized in that the control method adopts a four-layer architecture, as shown in Figure 2 , and realizes the control of the embodied intelligent robot in combination with a multi-modal information fusion mechanism, an adaptive operation control algorithm, a dynamic priority and emergency mechanism and a hybrid AI system, the four-layer architecture being a scene understanding layer, a task planning layer, a priority sorting layer and a motion execution layer, and the implementation steps of the control method being as follows:

[0048] Step S1, system initialization, after the embodied robot is started, the visual sensor, the six-dimensional force and torque sensor and the language understanding module are started and collect surrounding physical environment information, object state information and user natural language instructions;

[0049] Step S2, scene understanding and task analysis, the scene understanding layer processes the collected data through the multi-modal information fusion mechanism, the visual encoder extracts multi-scale visual features from the information collected by the visual sensor to identify object types and environment states, the language encoder analyzes user natural language instructions to extract task semantics, the force sensation encoder encodes six-dimensional force and torque sensor data into a feature vector to reflect initial physical interaction information, and then the cross-modal attention module is used to realize the deep fusion of vision, language and force sensation, generate unified multi-modal data and complete task analysis;

[0050] Step S3, task planning and priority sorting, the task planning layer decomposes complex tasks into a subtask sequence and constructs a task graph based on the unified multi-modal data generated in step S2; the priority sorting layer perceives the environment state, the subtask complexity and the resource consumption in real time, evaluates the subtasks based on multi-dimensional evaluation parameters and dynamically adjusts the execution order of the subtasks;

[0051] Step S4, motion execution and feedback adjustment, the motion execution layer generates accurate control instructions through the adaptive operation control algorithm based on the unified multi-modal data generated in step S2 and the subtasks sorted in step S3, dynamically adjusts the force, speed and trajectory to realize motion execution, simultaneously monitors the execution effect in real time and adjusts the operation strategy according to the feedback; if a fault or an emergency is detected, the priority sorting layer quickly switches the strategy;

[0052] Step S5, continuous learning and strategy verification, uses a hybrid AI system to perform the integration of pre-processing, calculation, decision-making, and post-processing of the unified multi-modal data generated in step S2, and handles multiple rounds of interaction and historical information through long context and complex reasoning to deal with the dynamics and uncertainties of the environment; upload the unified multi-modal data during execution to the continuous learning module to optimize model parameters and planning strategies, and verify the robustness of the operation strategy through fault injection testing in the simulation environment;

[0053] Step S6, closed-loop iteration, repeat steps S2 to S5 to realize real-time response of embodied intelligent robots to dynamic environment and task changes, and complete continuous intelligent operation.

[0054] Further, the multi-modal information fusion mechanism in step S2 includes three core encoders and a cross-modal attention module, as follows:

[0055] The visual encoder uses a multi-scale feature extraction network to process the environmental data collected by the visual sensor, outputting object contours, positions, material textures, and environmental obstacle distribution features;

[0056] The language encoder is based on a pre-trained semantic understanding model to analyze the task objectives and operation constraints in user instructions, outputting a task semantic vector;

[0057] The force encoder filters and normalizes the real-time force value data collected by the six-axis force and torque sensor, mapping the six-axis force and torque data into a fixed-dimensional feature vector to reflect the contact force and impedance state during operation;

[0058] The cross-modal attention module uses the language semantic vector as a guide to calculate the attention weights of visual features, force features, and semantic features, achieving dynamic association and information complementarity among the three, and generating a unified multi-modal representation.

[0059] Further, the task planning layer in step S3 uses a two-step method of complex task decomposition and task graph construction, as follows:

[0060] The complex task decomposition is based on task semantics and object attributes, decomposing a single complex task into a sequence of subtasks; the task graph construction constructs a loop-free task graph with subtasks as nodes and dependency relationships between subtasks as directed edges, labeling the input, output, and required resources for each subtask.

[0061] Further, the priority sorting layer in step S3 uses a multi-dimensional evaluation and dynamic adjustment mechanism, as follows:

[0062] The multi-dimensional evaluation parameters include subtask difficulty, risk coefficient, time constraint, and energy consumption.

[0063] The rule of the dynamic adjustment is that when the environment state changes or the subtask progress is delayed, the priority ranking layer re-evaluates each subtask and adjusts the execution order; when a fault occurs, an emergency strategy is triggered to preferentially execute a fault troubleshooting subtask or a safety protection subtask.

[0064] Further, the steps of the adaptive operation control algorithm in the step S4 are specifically as follows:

[0065] Step S41, object material identification and demand assessment, combining visual features and force sensation features, inferring the object material type through a classification model, and assessing the task fineness demand based on the material properties;

[0066] Step S42, adaptive force control and trajectory planning, dynamically adjusting the force threshold, motion speed and trajectory parameters of the mechanical arm based on the material assessment results; combining the environment data and object features output by the visual sensor and the visual encoder, generating a collision-free optimal trajectory; in the execution process, real-time force sensation feedback is received, and if the detected force exceeds the threshold or the trajectory deviation is greater than the preset value, the control instruction is immediately corrected;

[0067] Step S43, effect monitoring and online adjustment, monitoring the execution effect through visual post-processing and force sensation measurement, and adjusting the operation parameters if over-execution or under-execution is found.

[0068] Further, the hybrid AI system in the step S5 comprises the following modules:

[0069] Long context and complex reasoning module: chain reasoning is used to handle multi-round interactive tasks, tree reasoning is used to deal with multi-branch decision-making tasks, and out-of-order reasoning is used to call historical task information;

[0070] Environment dynamics and uncertainty coping module: real-time monitoring of environmental parameters, when the environmental change is greater than a preset threshold, triggering the scene understanding layer to re-perceive and the task planning layer to regenerate the subtask sequence;

[0071] Multi-task, cross-domain collaboration module: supporting simultaneous processing of multiple subtasks, allocating computing resources through a task scheduler, and realizing parameter migration across cross-domain tasks;

[0072] Safety and robustness verification module: injecting faults in a simulation environment to test the system's fault handling capability; training visual and force sensation encoders using adversarial samples to resist adversarial attacks.

[0073] Further, the continuous learning module in the step S5 specifically comprises the following units:

[0074] Data collection unit: automatically storing multi-modal data in the execution process, and labeling successful and failed cases;

[0075] The model optimization unit adjusts the visual encoder, the language encoder and the adaptive control algorithm parameters by using the labeled data regularly, so as to improve the object recognition accuracy, the instruction analysis accuracy and the operation success rate.

[0076] Further, the application scenarios of the multi-modal information fusion embodied intelligent robot control method include but are not limited to industrial assembly lines, medical care robots, precision machining, home service robots and rescue robots.

[0077] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for controlling an embodied intelligent robot through multimodal information fusion, characterized in that, The control method employs a four-layer architecture, combining a multimodal information fusion mechanism, an adaptive operation control algorithm, a dynamic priority and emergency mechanism, and a hybrid AI system to control the embodied intelligent robot. The four-layer architecture consists of a scene understanding layer, a task planning layer, a priority ranking layer, and an action execution layer. The implementation steps of the control method are as follows: Step S1: System initialization. After the embodied robot is started, the vision sensor, six-dimensional force and torque sensor and language understanding module are activated and collect information on the surrounding physical environment, object status information and user natural language commands. Step S2, Scene Understanding and Task Parsing: The scene understanding layer processes the collected data through a multimodal information fusion mechanism. The visual encoder extracts multi-scale visual features from the information collected by the visual sensor to identify object types and environmental states. The language encoder parses the user's natural language instructions to extract task semantics. The force encoder encodes six-dimensional force and torque sensing data into feature vectors to reflect the initial physical interaction information. Then, the cross-modal attention module achieves deep fusion of vision, language, and force to generate unified multimodal data and complete task parsing. Step S3, Task Planning and Prioritization: The task planning layer decomposes complex tasks into sub-task sequences and constructs a task graph based on the unified multimodal data generated in step S2; the priority ranking layer perceives the environmental status, sub-task complexity and resource consumption in real time, evaluates sub-tasks based on multi-dimensional evaluation parameters, and dynamically adjusts the execution order of sub-tasks. Step S4, Action Execution and Feedback Adjustment: Based on the unified multimodal data generated in step S2 and the sub-tasks sorted in step S3, the action execution layer generates precise control commands through an adaptive operation control algorithm, dynamically adjusts the force, speed, and trajectory to achieve action execution, and monitors the execution effect in real time and adjusts the operation strategy according to feedback; if a fault or sudden event is detected, the priority sorting layer quickly switches the strategy. Step S5, Continuous Learning and Policy Validation: The hybrid AI system is used to perform preprocessing, computation, decision-making, and post-processing of the unified multimodal data generated in step S2. The system handles multi-round interactions and historical information through long context and complex reasoning to cope with the dynamics and uncertainties of the environment. The unified multimodal data during the execution process is uploaded to the continuous learning module to optimize model parameters and planning strategies. The robustness of the operation strategy is verified through fault injection testing in the simulation environment. Step S6: Closed-loop iteration, repeating steps S2 to S5 to achieve real-time response of the embodied intelligent robot to dynamic environment and task changes, and complete continuous intelligent operation.

2. The method for controlling an embodied intelligent robot based on multimodal information fusion according to claim 1, characterized in that, The multimodal information fusion mechanism in step S2 includes three core encoders and a cross-modal attention module, as detailed below: The visual encoder uses a multi-scale feature extraction network to process the environmental data collected by the visual sensor and output the object outline, position, material texture and environmental obstacle distribution features. The language encoder is based on a pre-trained semantic understanding model, which parses the task objectives and operational constraints in user instructions and outputs a task semantic vector. The force encoder filters and normalizes the real-time force data collected by the six-dimensional force and torque sensors, mapping the six-dimensional force and torque data (X, Y, and Z axes) into fixed-dimensional feature vectors to reflect the contact force and impedance state during operation. The cross-modal attention module is guided by language semantic vectors to calculate the attention weights of visual features, force features and semantic features, realize the dynamic association and information complementarity of the three, and generate a unified multimodal representation.

3. The method for controlling an embodied intelligent robot based on multimodal information fusion according to claim 1, characterized in that, The task planning layer in step S3 adopts a two-step method of complex task decomposition and task graph construction, as detailed below: The complex task decomposition is based on task semantics and object attributes, decomposing a single complex task into a sequence of subtasks; the task graph construction uses subtasks as nodes and the dependencies between subtasks as directed edges to construct an acyclic task graph, labeling the input, output, and resources required for execution of each subtask.

4. The method for controlling an embodied intelligent robot based on multimodal information fusion according to claim 1, characterized in that, The priority ranking layer in step S3 employs a multi-dimensional evaluation and dynamic adjustment mechanism, as detailed below: The parameters for the multi-dimensional evaluation include sub-task difficulty, risk coefficient, time constraint, and energy consumption; The rules for dynamic adjustment are as follows: when the environmental state changes or the progress of a subtask is delayed, the priority sorting layer re-evaluates each subtask and adjusts the execution order; when a failure occurs, an emergency strategy is triggered, prioritizing the execution of fault investigation subtasks or safety protection subtasks.

5. The method for controlling an embodied intelligent robot based on multimodal information fusion according to claim 1, characterized in that, The specific steps of the adaptive operation control algorithm in step S4 are as follows: Step S41: Object material identification and requirement assessment. Combining visual and force features, the object material type is inferred through a classification model, and the task refinement requirements are assessed based on material properties. Step S42, Adaptive force control and trajectory planning: Based on the material evaluation results, dynamically adjust the force threshold, movement speed and trajectory parameters of the robotic arm; combine the environmental data and object features output by the vision sensor and vision encoder to generate the collision-free optimal trajectory; during execution, receive force feedback in real time, and if the force exceeds the threshold or the trajectory deviation is greater than the preset value, correct the control command immediately. Step S43: Effect monitoring and online adjustment. The execution effect is monitored through visual post-processing and force measurement. If over-execution or under-execution is found, the operation parameters are adjusted.

6. The method for controlling an embodied intelligent robot based on multimodal information fusion according to claim 1, characterized in that, The hybrid AI system in step S5 includes the following modules: Long context and complex reasoning module: It uses chain reasoning to handle multi-round interactive tasks, tree reasoning to handle multi-branch decision tasks, and supports out-of-order reasoning to call historical task information; Environmental dynamism and uncertainty response module: Real-time monitoring of environmental parameters; when the amount of environmental change exceeds a preset threshold, the scene understanding layer is triggered to re-perceive and the task planning layer regenerates the sub-task sequence. Multi-task, cross-domain collaboration module: Supports simultaneous processing of multiple sub-tasks, allocates computing resources through a task scheduler, and enables parameter migration for cross-domain tasks; Security and robustness verification module: Injects faults into a simulation environment to test the system's fault response capability; uses adversarial examples to train the visual and force encoders to resist adversarial attacks.

7. The method for controlling an embodied intelligent robot based on multimodal information fusion according to claim 1, characterized in that, The continuous learning module in step S5 specifically includes the following units: Data collection unit: Automatically stores multimodal data during execution and labels success and failure cases; Model optimization unit: Regularly adjusts the parameters of the visual encoder, language encoder, and adaptive control algorithm using labeled data to improve object recognition accuracy, instruction parsing accuracy, and operation success rate.

8. A method for controlling an embodied intelligent robot based on multimodal information fusion according to any one of claims 1-7, characterized in that, The application scenarios of the method include, but are not limited to, industrial assembly lines, medical care robots, precision machining, home service robots, and rescue robots.

Images