Complex task-oriented multi-modal data alignment and cross-modal reasoning method and system for body agent

By employing a multimodal data alignment and cross-modal reasoning method based on embodied intelligent agents, the problems of spatiotemporal asynchrony and semantic heterogeneity of multimodal data are solved, optimal action decisions are generated, execution reliability and efficiency are improved, and the system is adapted to complex task scenarios.

CN121480708APending Publication Date: 2026-02-06QINGDAO INSPUR HAIRUO ARTIFICIAL INTELLIGENCE CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202511564587.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-10-30
Publication Date
2026-02-06

AI Technical Summary

Technical Problem

Existing methods for processing multimodal data in embodied intelligent agents lack deep relational modeling, resulting in low data alignment accuracy, severe loss of semantic information, difficulty in handling the spatiotemporal asynchrony and semantic heterogeneity of multimodal data, and insufficient integration of physical execution constraints in cross-modal reasoning processes, leading to decision results that cannot be implemented or low execution efficiency.

Method used

A method is adopted, which involves multi-skill task and resource information collection, raw data preprocessing, three-level multimodal data alignment, and cross-modal reasoning and decision generation, including spatiotemporal calibration, semantic mapping, and linguistic constraint adaptation. This method constructs a cross-modal reasoning engine that integrates physical rules and large models, generates the optimal action decision sequence, and optimizes the model through feedback data.

Benefits of technology

It significantly improves the accuracy of multimodal data alignment, reduces the risk of "semantically correct but physically unexecutable", supports natural language to directly drive complex tasks, expands application boundaries, enables autonomous adaptation to new environments and tasks, and extends the technology life cycle.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121480708A_ABST
    Figure CN121480708A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of multi-modal data processing and intelligent decision making, in particular to a complex-task-oriented multi-modal data alignment and cross-modal reasoning method and system for an agent with a body, and the method comprises the following steps: multi-skill task and resource information collection, original data preprocessing, three-level multi-modal data alignment, cross-modal reasoning, multi-skill task and resource information collection, three-level multi-modal reasoning and cross-modal reasoning. Cross-modal reasoning and decision generation, reasoning result verification and iterative optimization; the method has the beneficial effects that through a three-stage framework of'space-time calibration-semantic mapping-constraint adaptation ', the problems of space-time asynchronization and semantic segmentation of multi-modal data are effectively solved, the alignment accuracy is greatly improved compared with that of a traditional method, and a high-quality data basis is provided for subsequent reasoning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing and intelligent decision-making technology, specifically to a method and system for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks. Background Technology

[0002] Existing multimodal processing methods for embodied intelligent agents mostly adopt a simple fusion model after independent reasoning of a single modality. They lack deep correlation modeling of multimodal data such as vision (e.g., scene images, object outlines), touch (e.g., contact force feedback, object hardness), and language (e.g., user commands, task descriptions), resulting in low data alignment accuracy and serious loss of semantic information.

[0003] In complex task scenarios, traditional methods struggle to handle the spatiotemporal asynchronicity of multimodal data (such as the time difference between real-time acquisition of visual signals and delayed input of language commands) and semantic heterogeneity (such as the difference between the physical quantities of tactile data and the abstract concepts of language commands). Furthermore, the cross-modal reasoning process does not fully incorporate the physical execution constraints of the embodied agent (such as the range of motion of the robotic arm and the threshold of the operating force), resulting in decision results that cannot be implemented or low execution efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for multimodal data alignment and cross-modal reasoning of embodied intelligent agents for complex tasks, so as to solve the problems of inaccurate multimodal data fusion and the disconnect between cross-modal reasoning and physical execution in complex task scenarios, and improve the environmental understanding ability and task completion reliability of embodied intelligent agents.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for multimodal data alignment and cross-modal reasoning of embodied intelligent agents for complex tasks, comprising the following steps: Multi-skill task and resource information collection: The embodied intelligent agent collects multimodal raw data in the task scene in real time through a sensor array. The multimodal raw data includes visual data, tactile data, language data and device status data. Raw data preprocessing: The collected raw data is preprocessed to form a standardized multimodal dataset; Three-level multimodal data alignment: Based on the task scenario and physical attributes of embodied intelligent agents, a three-level alignment framework is constructed to complete multimodal data alignment; Cross-modal reasoning and decision generation: Construct a cross-modal reasoning engine that integrates physical rules and large models to generate optimal action decision sequences; Validation and Iterative Optimization of Reasoning Results: The generated decision sequence is validated, and iterative optimization is performed based on the validation results.

[0006] Preferably, the specific content of the multi-skill task and resource information collection and raw data preprocessing is as follows: Multi-skill task and resource information acquisition: Multimodal raw data is acquired through a sensor array, including visual data such as scene RGB images, depth images, and object target detection box coordinates; tactile data such as end effector contact force values, contact area, and object surface texture features; language data such as user natural language commands and task text descriptions; and device status data such as agent joint angles, movement speed, and battery level. Raw data preprocessing: Visual data is enhanced, target segmented and noise is removed; tactile data is filtered, de-jittered and unit normalized; language data is segmented, semantically labeled and converted into structured text; outlier removal is performed on device status data, and finally a standardized multimodal dataset is formed.

[0007] Preferably, the specific content of the three-level multimodal data alignment is as follows: Spatiotemporal calibration layer: Based on the timestamp in the device status data, it synchronizes the real-time visual and tactile data; combining visual depth information and intelligent agent kinematic model, it maps the contact point coordinates of tactile data to the visual coordinate system to achieve spatial alignment; Semantic mapping layer: Invokes the pre-trained multimodal base model to extract high-level semantic features of each modality; constructs a cross-modal attention mechanism, using the core entities in language instructions as anchors to establish the association weights between visual features, tactile features and language semantics, and achieves semantic alignment; Language constraint adaptation layer: Import the physical constraint parameters and task constraint rules of the embodied intelligent agent, filter and adjust the weights of the aligned multimodal features to ensure that the alignment results meet the requirements of execution feasibility.

[0008] Preferably, the specific content of the cross-modal reasoning and decision generation is as follows: Inference engine initialization: Loading the aligned multimodal feature dataset, task objective, physical constraint parameters, and scene prior knowledge; Large-scale semantic reasoning: invoke the dedicated large-scale model of embodiment, parse the task logic in the language instructions, combine visual-tactile alignment features to identify the target object attributes and environmental state, and generate preliminary action candidate sequences; Physical rule verification: Input the preliminary action sequence into the physical simulation module to simulate the force balance and the rationality of the motion trajectory during the execution process, and verify whether the constraints are met; if there is a conflict, it is fed back to the large model and triggers re-inference; Decision sequence optimization: Based on the simulation verification results, the large model fine-tunes the parameters of the action sequence, and finally generates the optimal action decision sequence that meets the criteria of "semantic accuracy, physical feasibility and task efficiency".

[0009] Preferably, the specific content of the inference result verification and iterative optimization is as follows: Simulation verification: Load the current task scenario and agent model in the digital twin environment, execute the generated decision sequence, and evaluate the core indicators; Human-assisted adjustment: Users correct their needs through natural language input, and the system re-triggers the data alignment and cross-modal inference process; Closed-loop optimization: Store the multimodal feedback data from actual execution into the training dataset, and fine-tune the multimodal alignment model and the cross-modal inference large model regularly to improve long-term inference accuracy and environmental adaptability.

[0010] A system for embodied intelligent agents with multimodal data alignment and cross-modal reasoning for complex tasks includes: The multi-skill task and resource information acquisition module is used to collect multimodal raw data in the task scenario through a sensor array and preprocess it to form a standardized multimodal dataset. The three-level multimodal data alignment module constructs a three-level alignment framework based on the task scenario and physical attributes of the embodied intelligent agent to complete multimodal data alignment; The cross-modal reasoning and decision generation module constructs a cross-modal reasoning engine that integrates physical rules and large models to generate the optimal action decision sequence; The reasoning result verification and iterative optimization module is used to verify the generated decision sequence and perform iterative optimization based on the verification results.

[0011] Preferably, the multi-skill task and resource information acquisition module includes: The data acquisition unit acquires multimodal raw data in the task scenario in real time through a sensor array. The multimodal raw data includes: Visual data: scene RGB image, depth image, object detection box coordinates; Tactile data: end effector contact force value, contact area, and surface texture features of the object; Language data: User natural language commands, task text descriptions; Device status data: joint angles of the intelligent agent, movement speed, and battery level; The data preprocessing unit preprocesses the collected raw data, specifically by: performing image enhancement, target segmentation and noise removal on visual data; filtering, de-jittering and unit normalization on tactile data; word segmentation, semantic annotation and conversion into structured text on language data; and outlier removal on device status data, ultimately forming a standardized multimodal dataset.

[0012] Preferably, the three-level multimodal data alignment module includes: The spatiotemporal calibration layer unit uses the timestamp in the device status data as a reference to synchronize the real-time visual and tactile data; combining visual depth information and the intelligent agent kinematic model, it maps the contact point coordinates of the tactile data to the visual coordinate system to achieve spatial alignment. The semantic mapping layer unit calls the pre-trained multimodal base model to extract high-level semantic features of each modality; it constructs a cross-modal attention mechanism, using the core entities in the language instructions as anchors, to establish the association weights between visual features, tactile features and language semantics, thereby achieving semantic alignment. The language constraint adaptation layer unit imports the physical constraint parameters and task constraint rules of the embodied intelligent agent, and filters and adjusts the weights of the aligned multimodal features to ensure that the alignment results meet the requirements of execution feasibility.

[0013] Preferably, the cross-modal reasoning and decision generation module includes: The inference engine initialization unit loads the aligned multimodal feature dataset, task objective, physical constraint parameters, and scenario prior knowledge. The large-scale semantic reasoning unit calls the dedicated large-scale model to parse the task logic in the language instructions, and combines visual-tactile alignment features to identify the target object attributes and environmental state, generating preliminary action candidate sequences. The physical rule verification unit inputs the preliminary action sequence into the physical simulation module to simulate the force balance and the rationality of the motion trajectory during the execution process, and verifies whether the constraints are met; if there is a conflict, it is fed back to the large model and triggers re-inference; The decision sequence optimization unit, combined with simulation verification results, fine-tunes the parameters of the action sequence in the large model, and finally generates the optimal action decision sequence that meets the criteria of "semantic accuracy, physical feasibility, and task efficiency".

[0014] Preferably, the reasoning result verification and iterative optimization module includes: The simulation verification unit loads the current task scenario and intelligent agent model in the digital twin environment, executes the generated decision sequence, and evaluates core indicators, including multimodal alignment accuracy, task completion success rate, and physical constraint satisfaction rate during execution. The human interaction adjustment unit allows users to correct their needs through natural language input, and the system then re-triggers the data alignment and cross-modal reasoning process. The closed-loop optimization unit stores the multimodal feedback data from actual execution into the training dataset and periodically fine-tunes the multimodal alignment model and the cross-modal inference large model to improve long-term inference accuracy and environmental adaptability.

[0015] Compared with the prior art, the beneficial effects of the present invention are: The proposed method and system for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks effectively solves the problems of spatiotemporal asynchrony and semantic fragmentation of multimodal data through a three-level framework of "spatiotemporal calibration-semantic mapping-constraint adaptation". The alignment accuracy is significantly improved compared with traditional methods, providing a high-quality data foundation for subsequent reasoning.

[0016] By integrating large-scale model semantic reasoning with physical rule verification, the task intent is transformed into an action sequence that conforms to the physical constraints of the agent, significantly reducing the risk of "semantically correct but physically unexecutable" and adapting to demanding scenarios such as precision operations.

[0017] It supports natural language to directly drive processes, allowing users to complete complex task configurations without professional knowledge, greatly expanding the application boundaries of non-professional scenarios such as home services and disability assistance.

[0018] By continuously optimizing the model based on feedback data, it can autonomously adapt to new environments and tasks, achieving performance iteration without manual redevelopment, and significantly extending the technology life cycle and scenario coverage. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: This invention provides a technical solution: a method for multimodal data alignment and cross-modal reasoning of embodied intelligent agents for complex tasks, comprising the following steps: I. Multi-skill tasks and resource information collection The embodied intelligent agent collects multimodal raw data in real time from the task scenario through a sensor array (visual camera, tactile sensor, voice acquisition module, etc.), including: 1. Visual data: scene RGB image, depth image, object detection box coordinates; 2. Tactile data: end effector contact force value, contact area, and surface texture features of the object; 3. Language data: User natural language commands (such as "pick up the red hard glass on the left"), task text description; 4. Device status data: joint angles of the intelligent agent, movement speed, battery level, etc.

[0022] The collected raw data are preprocessed: visual data is enhanced by image enhancement and target segmentation to remove noise; tactile data is filtered to remove jitter and normalized by units; language data is converted into structured text through word segmentation and semantic annotation; and device status data is removed from outliers, ultimately forming a standardized multimodal dataset.

[0023] Level 2 and Level 3 Multimodal Data Alignment Based on the task scenario and physical attributes of embodied intelligent agents, a three-level alignment framework is constructed: 1. Spatiotemporal Calibration Layer: Based on the timestamps in the device status data, this layer synchronizes real-time acquired data such as visual and tactile data. Combining visual depth information with the agent's kinematic model, it maps the contact point coordinates of the tactile data to the visual coordinate system, achieving spatial alignment. For example, when the agent touches an object, the force data detected by the tactile sensor is spatiotemporally bound to the pixel coordinates of the object in the visual image.

[0024] 2. Semantic Mapping Layer: Invoke pre-trained multimodal base models (such as CLIP, ALBEF) to extract high-level semantic features of each modality; construct a cross-modal attention mechanism, using core entities in language instructions (such as "red water cup" and "hard") as anchors to establish association weights between visual features (color, shape), tactile features (hardness) and language semantics, thereby achieving semantic alignment.

[0025] 3. Language Constraint Adaptation Layer: Import the physical constraint parameters (such as the end effector load limit and joint range of motion) and task constraint rules (such as "do not contact high-temperature objects" and "prefer to use idle actuators") of the embodied intelligent agent, filter and adjust the weights of the aligned multimodal features to ensure that the alignment results meet the requirements of execution feasibility.

[0026] III. Cross-modal reasoning and decision generation The specific process for building a cross-modal inference engine that integrates physical rules and large models is as follows: 1. Inference engine initialization: Load the aligned multimodal feature dataset, task objective (e.g., "pick up the water glass and place it on the table"), physical constraint parameters and scenario prior knowledge (e.g., "the water glass is fragile and should be handled gently").

[0027] 2. Large-scale semantic reasoning: The large-scale model dedicated to embodied tasks (pre-trained on the embodied task corpus and physical interaction dataset) is invoked to parse the task logic in the language instructions. Combined with visual-tactile alignment features, the target object attributes (such as material and weight) and environmental state (such as object placement and surrounding obstacles) are identified to generate preliminary action candidate sequences (such as "move to the left side of the target → adjust the end effector posture → contact the target → raise to a height of 5cm → move to the top of the table → release").

[0028] 3. Physical rule verification: Input the preliminary action sequence into the physical simulation module to simulate the force balance and the rationality of the motion trajectory during the execution process, and verify whether the constraints are met (such as contact force ≤ 5N, no obstacle collisions in the motion trajectory); if there is a conflict (such as contact force exceeding the limit), it is fed back to the large model and triggers re-inference.

[0029] 4. Decision Sequence Optimization: Based on the simulation verification results, the large model fine-tunes the parameters of the action sequence (such as adjusting the contact force threshold and optimizing the motion path) to finally generate the optimal action decision sequence that meets the criteria of "semantic accuracy, physical feasibility and task efficiency".

[0030] IV. Verification and Iterative Optimization of Reasoning Results 1. Simulation and verification: Load the current task scenario and agent model in the digital twin environment, execute the generated decision sequence, and evaluate the core indicators: multimodal alignment accuracy (e.g., visual-tactile feature association error ≤ 5%), task completion success rate, and physical constraint satisfaction rate during execution.

[0031] 2. Manual Interaction Adjustment: Users can modify their needs through natural language input (such as "lower the picking height to avoid collision with nearby items"), and the system will re-trigger the data alignment and cross-modal reasoning process.

[0032] 3. Closed-loop optimization: Store the multimodal feedback data from actual execution (such as the deviation between the actual contact force and the simulation value, and the time taken to complete the task) into the training dataset, and fine-tune the multimodal alignment model and the cross-modal inference large model regularly to improve long-term inference accuracy and environmental adaptability.

[0033] Example 2, based on Example 1, proposes an embodied agent multimodal data alignment and cross-modal reasoning system for complex tasks, including: The multi-skill task and resource information acquisition module is used to collect multimodal raw data from the task scene through a sensor array and preprocess it to form a standardized multimodal dataset. It includes: a data acquisition unit, which collects multimodal raw data from the task scene in real time through a sensor array. This multimodal raw data includes: visual data (scene RGB image, depth image, object target detection box coordinates); tactile data (end-effector contact force value, contact area, object surface texture features); language data (user natural language commands, task text description); and device status data (smart agent joint angles, movement speed, battery level). A data preprocessing unit preprocesses the collected raw data, specifically: performing image enhancement and target segmentation to remove noise from the visual data; filtering and de-jittering and unit normalization on the tactile data; word segmentation and semantic annotation to convert the language data into structured text; and outlier removal from the device status data, ultimately forming a standardized multimodal dataset.

[0034] The three-level multimodal data alignment module, based on the task scenario and physical attributes of the embodied agent, constructs a three-level alignment framework to complete multimodal data alignment. It includes: a spatiotemporal calibration layer unit, which uses the timestamp in the device status data as a benchmark to synchronize the real-time visual and tactile data; combining visual depth information and the agent's kinematic model, it maps the contact point coordinates of the tactile data to the visual coordinate system to achieve spatial alignment; a semantic mapping layer unit, which calls a pre-trained multimodal basic model to extract high-level semantic features of each modality; it constructs a cross-modal attention mechanism, using the core entities in the language instructions as anchors to establish the association weights between visual features, tactile features, and language semantics to achieve semantic alignment; and a language constraint adaptation layer unit, which imports the physical constraint parameters and task constraint rules of the embodied agent to filter and adjust the weights of the aligned multimodal features to ensure that the alignment results meet the requirements of execution feasibility.

[0035] The cross-modal reasoning and decision generation module constructs a cross-modal reasoning engine that integrates physical rules and a large model to generate the optimal action decision sequence. This includes: an inference engine initialization unit, which loads the aligned multimodal feature dataset, task objective, physical constraint parameters, and prior scene knowledge; a large model semantic reasoning unit, which calls the embodied large model to parse the task logic in language instructions, and combines visual-tactile alignment features to identify target object attributes and environmental states, generating preliminary action candidate sequences; a physical rule verification unit, which inputs the preliminary action sequences into a physical simulation module to simulate force balance and trajectory rationality during execution, verifying whether the constraints are met; if conflicts exist, feedback is sent to the large model and a re-inference is triggered; and a decision sequence optimization unit, which, based on the simulation verification results, fine-tunes the parameters of the action sequence using the large model, ultimately generating the optimal action decision sequence that meets the criteria of "semantic accuracy, physical feasibility, and task efficiency."

[0036] The inference result verification and iterative optimization module is used to verify the generated decision sequence and iteratively optimize it based on the verification results. It includes: a simulation verification unit, which loads the current task scenario and agent model in the digital twin environment, executes the generated decision sequence, and evaluates core metrics, including multimodal alignment accuracy, task completion success rate, and physical constraint satisfaction rate during execution; a human interaction adjustment unit, where users input corrections via natural language, and the system re-triggers the data alignment and cross-modal inference process; and a closed-loop optimization unit, which stores the multimodal feedback data from actual execution into the training dataset, periodically fine-tunes the multimodal alignment model and the large cross-modal inference model, and improves long-term inference accuracy and environmental adaptability.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks, characterized in that: Includes the following steps: Multi-skill task and resource information collection: The embodied intelligent agent collects multimodal raw data in the task scene in real time through a sensor array. The multimodal raw data includes visual data, tactile data, language data and device status data. Raw data preprocessing: The collected raw data is preprocessed to form a standardized multimodal dataset; Three-level multimodal data alignment: Based on the task scenario and physical attributes of embodied intelligent agents, a three-level alignment framework is constructed to complete multimodal data alignment; Cross-modal reasoning and decision generation: Construct a cross-modal reasoning engine that integrates physical rules and large models to generate optimal action decision sequences; Validation and Iterative Optimization of Reasoning Results: The generated decision sequence is validated, and iterative optimization is performed based on the validation results.

2. The method for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks according to claim 1, characterized in that: The specific content of the multi-skill task and resource information collection and raw data preprocessing is as follows: Multi-skill task and resource information acquisition: Multimodal raw data is acquired through a sensor array, including visual data such as scene RGB images, depth images, and object target detection box coordinates; tactile data such as end effector contact force values, contact area, and object surface texture features; language data such as user natural language commands and task text descriptions; and device status data such as agent joint angles, movement speed, and battery level. Raw data preprocessing: Visual data is enhanced, target segmented and noise is removed; tactile data is filtered, de-jittered and unit normalized; language data is segmented, semantically labeled and converted into structured text; outlier removal is performed on device status data, and finally a standardized multimodal dataset is formed.

3. The method for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks according to claim 2, characterized in that: The specific details of the three-level multimodal data alignment are as follows: Spatiotemporal calibration layer: Based on the timestamp in the device status data, it synchronizes the real-time visual and tactile data; combining visual depth information and intelligent agent kinematic model, it maps the contact point coordinates of tactile data to the visual coordinate system to achieve spatial alignment; Semantic mapping layer: Invokes the pre-trained multimodal base model to extract high-level semantic features of each modality; A cross-modal attention mechanism is constructed, which uses the core entities in language instructions as anchors to establish the association weights between visual features, tactile features and language semantics, thereby achieving semantic alignment. Language constraint adaptation layer: Import the physical constraint parameters and task constraint rules of the embodied intelligent agent, filter and adjust the weights of the aligned multimodal features to ensure that the alignment results meet the requirements of execution feasibility.

4. The method for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks according to claim 3, characterized in that: The specific content of the cross-modal reasoning and decision generation is as follows: Inference engine initialization: Loading the aligned multimodal feature dataset, task objective, physical constraint parameters, and scene prior knowledge; Large-scale model semantic reasoning: invoke the dedicated large-scale model of embodiment, parse the task logic in the language instructions, combine visual-tactile alignment features to identify the target object attributes and environmental state, and generate preliminary action candidate sequences; Physical rule verification: Input the preliminary action sequence into the physical simulation module to simulate the force balance and the rationality of the motion trajectory during the execution process, and verify whether the constraints are met; If a conflict exists, it is fed back to the larger model and triggers a re-inference; Decision sequence optimization: Based on the simulation verification results, the large model fine-tunes the parameters of the action sequence, and finally generates the optimal action decision sequence that meets the criteria of "semantic accuracy, physical feasibility and task efficiency".

5. The method for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks according to claim 4, characterized in that: The specific content of the inference result verification and iterative optimization is as follows: Simulation verification: Load the current task scenario and agent model in the digital twin environment, execute the generated decision sequence, and evaluate the core indicators; Human-assisted adjustment: Users correct their needs through natural language input, and the system re-triggers the data alignment and cross-modal inference process; Closed-loop optimization: Store the multimodal feedback data from actual execution into the training dataset, and fine-tune the multimodal alignment model and the cross-modal inference large model regularly to improve long-term inference accuracy and environmental adaptability.

6. A system for embodied intelligent agents to perform multimodal data alignment and cross-modal reasoning for complex tasks, employing the method described in claim 5, characterized in that: include: The multi-skill task and resource information acquisition module is used to collect multimodal raw data in the task scenario through a sensor array and preprocess it to form a standardized multimodal dataset. The three-level multimodal data alignment module constructs a three-level alignment framework based on the task scenario and physical attributes of the embodied intelligent agent to complete multimodal data alignment; The cross-modal reasoning and decision generation module constructs a cross-modal reasoning engine that integrates physical rules and large models to generate the optimal action decision sequence; The reasoning result verification and iterative optimization module is used to verify the generated decision sequence and perform iterative optimization based on the verification results.

7. A system for embodied intelligent agents with multimodal data alignment and cross-modal reasoning for complex tasks according to claim 6, characterized in that: The multi-skill task and resource information collection module includes: The data acquisition unit acquires multimodal raw data in the task scenario in real time through a sensor array. The multimodal raw data includes: Visual data: scene RGB image, depth image, object detection box coordinates; Tactile data: end effector contact force value, contact area, and surface texture features of the object; Language data: User natural language commands, task text descriptions; Device status data: joint angles of the intelligent agent, movement speed, and battery level; The data preprocessing unit preprocesses the collected raw data, specifically by: performing image enhancement, target segmentation and noise removal on visual data; filtering, de-jittering and unit normalization on tactile data; word segmentation, semantic annotation and conversion into structured text on language data; and outlier removal on device status data, ultimately forming a standardized multimodal dataset.

8. A system for embodied intelligent agents with multimodal data alignment and cross-modal reasoning for complex tasks according to claim 7, characterized in that: The three-level multimodal data alignment module includes: The spatiotemporal calibration layer unit uses the timestamp in the device status data as a reference to synchronize the real-time visual and tactile data; combining visual depth information and the intelligent agent kinematic model, it maps the contact point coordinates of the tactile data to the visual coordinate system to achieve spatial alignment. The semantic mapping layer unit calls the pre-trained multimodal base model to extract high-level semantic features of each modality; it constructs a cross-modal attention mechanism, using the core entities in the language instructions as anchors, to establish the association weights between visual features, tactile features and language semantics, thereby achieving semantic alignment. The language constraint adaptation layer unit imports the physical constraint parameters and task constraint rules of the embodied intelligent agent, and filters and adjusts the weights of the aligned multimodal features to ensure that the alignment results meet the requirements of execution feasibility.

9. A system for embodied intelligent agents with multimodal data alignment and cross-modal reasoning for complex tasks according to claim 8, characterized in that: The cross-modal reasoning and decision generation module includes: The inference engine initialization unit loads the aligned multimodal feature dataset, task objective, physical constraint parameters, and scenario prior knowledge. The large-scale semantic reasoning unit calls the dedicated large-scale model of embodiment to parse the task logic in the language instructions, and combines visual-tactile alignment features to identify the target object attributes and environmental state, and generate preliminary action candidate sequences. The physical rule verification unit inputs the preliminary action sequence into the physical simulation module to simulate the force balance and the rationality of the motion trajectory during the execution process, and verifies whether the constraints are met; if there is a conflict, it is fed back to the large model and triggers re-inference; The decision sequence optimization unit, combined with simulation verification results, fine-tunes the parameters of the action sequence in the large model, and finally generates the optimal action decision sequence that meets the criteria of "semantic accuracy, physical feasibility, and task efficiency".

10. A system for embodied intelligent agents with multimodal data alignment and cross-modal reasoning for complex tasks according to claim 9, characterized in that: The reasoning result verification and iterative optimization module includes: The simulation verification unit loads the current task scenario and intelligent agent model in the digital twin environment, executes the generated decision sequence, and evaluates core indicators, including multimodal alignment accuracy, task completion success rate, and physical constraint satisfaction rate during execution. The human interaction adjustment unit allows users to correct their needs through natural language input, and the system then re-triggers the data alignment and cross-modal reasoning process. The closed-loop optimization unit stores the multimodal feedback data from actual execution into the training dataset and periodically fine-tunes the multimodal alignment model and the cross-modal inference large model to improve long-term inference accuracy and environmental adaptability.

Citation Information

Cited By

  • Intelligent multi-mode perception decision-making method and system

    CN122133090A