Robot multi-modal fusion autonomous decision-making method and system based on large language model

Abstract

The invention relates to the technical field of robot decision making, and provides a robot multi-modal fusion autonomous decision making method and system based on a large language model.The method comprises the steps that a robot obtains multi-modal environment information through a visual sensor, a touch sensor, an auditory sensor and a laser radar which are carried by the robot; performing preliminary filtering and noise reduction processing on the original sensor data, and synchronously recording all the sensor data by timestamps; performing space-time semantic alignment on the preprocessed multi-modal data, mapping pixel coordinates of a target in a visual target coordinate quantization original image to a robot coordinate system, performing uncertainty evaluation on a multi-modal signal through a dynamic Bayesian network, and taking entropy or variance as an uncertainty quantitative evaluation index. According to the method, the information quality is improved from a data fusion source, accurate and reliable basic support is provided for subsequent decision making, and decision making errors caused by data deviation are greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of robot decision-making technology, specifically to a robot multimodal fusion autonomous decision-making method and system based on a large language model. Background Technology

[0002] The rise of autonomous decision-making methods for robots stems from the urgent need to address the complex and unstructured realities of the world. Early robots, operating in strictly controlled factories, relied on pre-programmed repetitive tasks, lacking flexibility. However, as applications expanded to areas such as autonomous driving, space exploration, smart homes, and disaster relief, environments became highly dynamic and unpredictable, rendering remote human control delayed and inefficient. Advances in various technologies have enabled robots to perceive their environment in real time; leaps in computing power have provided the hardware foundation for processing massive amounts of sensory data; and breakthroughs in artificial intelligence, particularly machine learning and large language models, have made core decision engines possible.

[0003] Traditional robot autonomous decision-making technologies often employ modular design, which involves acquiring environmental data through a single or a few sensors, preprocessing and fusing the data using a fixed algorithm, and then generating decision instructions based on preset rules, finite state machines, or traditional machine learning models. Decision verification often relies on fixed threshold judgments, deviation correction uses preset fixed strategies, historical experience reuse often involves matching with manually compiled rule bases, and cross-modal information processing often involves simple splicing after analyzing independent modalities separately, lacking a unified semantic alignment and feature fusion mechanism.

[0004] Traditional technologies struggle to meet the demands of autonomous decision-making in complex scenarios. Their core shortcomings can be deduced from the core benefits of this solution: First, traditional technologies lack sophisticated multimodal fusion mechanisms, often employing fixed weights or simple algorithms to fuse data. This fails to accurately assess the uncertainty of each modality, making them susceptible to interference from low-reliability data and resulting in insufficient decision-making accuracy. Second, their cross-modal adaptability is weak. The lack of a unified semantic mapping space creates a significant modal gap, hindering effective communication between non-textual perception data and decision logic, and making them unsuitable for multi-sensor collaborative scenarios. Third, their decision-making flexibility and adaptability are lacking. Relying on preset rules or simple models, they lack efficient mechanisms for reusing common-sense reasoning and historical experience. Faced with environmental changes, parameter deviations, or hardware failures, they cannot dynamically generate targeted correction strategies, leading to decision failures or task interruptions. Fourth, the standardization of instruction generation and execution is insufficient. Traditional instruction formats are inconsistent, lacking strict field validation mechanisms. Execution failures are easily caused by instruction ambiguity or missing information, and the single decision validation step makes it difficult to guarantee execution security and feasibility. Summary of the Invention

[0005] This invention proposes a multimodal fusion autonomous decision-making method and system for robots based on a large language model, which solves the problems of inaccurate multimodal fusion, weak cross-modal adaptation, and poor decision-making stability of robots in complex scenarios in related technologies.

[0006] The technical solution of the present invention is as follows: A multimodal fusion autonomous decision-making method for robots based on a large language model includes the following steps: Step 1: The robot acquires multimodal environmental information through its onboard visual sensors, tactile sensors, auditory sensors, and LiDAR. It performs preliminary filtering and noise reduction on the raw sensor data, and all sensor data is recorded synchronously with timestamps. Step 2: Perform spatiotemporal semantic alignment on the preprocessed multimodal data. For visual target coordinate quantization, the pixel coordinates of the target in the original image need to be mapped to the robot coordinate system. The specific mapping formula is as follows: The tactile pressure signal is denoised using a moving average filter. The filtering formula is as follows: The sliding window size ranges from 5 to 10. A dynamic Bayesian network is used to assess the uncertainty of the multimodal signal, with entropy or variance as the quantitative evaluation index of uncertainty. Finally, the Softmax function is used to convert the uncertainties of each modality into fusion weights. The weight calculation formula is as follows: ,in, It is the total number of modalities, and the final fusion feature. That is, the modal features Weighted sum: ; Step 3: The fused multimodal features are transformed into structured text representations through a cross-modal attention mechanism. Each modal feature is mapped to the same vector dimension through a modality-specific encoder. Cross-modal attention weights are calculated. The query vector is obtained by concatenating the task instruction encoding and the system state encoding and then performing a linear mapping. The key vector is the feature vector of each modality. The attention weights are obtained by normalization through the softmax function. Finally, the feature vectors are transformed into natural language descriptions through a neural generative model based on an encoder-decoder architecture. Step 4: Input structured text into the large language model to generate robot-executable decisions. Construct an input prompt that includes multimodal translated text, task objectives, environmental constraints, and historical experience. Decompose the high-level task into sub-tasks through task parsing and planning. Select matching action types from the robot action library and generate parameters. Ensure the feasibility of the decision through physical rule verification and hardware constraint verification.

[0007] Preferably, the uncertainty assessment and weight calculation of the dynamic Bayesian network in step 2 specifically includes: the dynamic Bayesian network consists of state variables and observation variables. The state variables represent the robot's internal state and environmental state, and the observation variables correspond to multimodal sensor data. The training data includes multimodal sensor readings and their corresponding real state labels. The network updates the posterior probability distribution of the state variables online through a forward recursive algorithm. The entropy of the posterior probability distribution is used to represent uncertainty. For continuous state variables, the variance is used to represent uncertainty. The fused feature vector is finally used for subsequent cross-modal semantic translation.

[0008] Preferably, the multimodal data alignment in step 3 specifically includes: time alignment using the robot task triggering time as the time origin, timestamping all sensor data, performing sliding window aggregation and interpolation on high-frequency continuous data, using nearest neighbor matching and state preservation on low-frequency instantaneous data, and accumulating and associating asynchronous event data based on triggers. All operations are based on the system master clock. Spatial alignment uses camera calibration to obtain intrinsic and extrinsic parameters to convert pixel coordinates into physical coordinates, and uses the robotic arm kinematic model to convert the local coordinates of the sensors into global coordinates in the base coordinate system. Finally, all spatially related data are uniformly described using the robot base coordinate system. Semantic alignment defines core entity labels, identifies entity categories through auditory commands and visual target detection, establishes a modal attribute mapping table, and binds each modal feature to the attributes of the core entity to ensure that the semantic dimensions of different modal descriptions of the same object can be associated with the multi-dimensional attributes of the same entity.

[0009] Preferably, the cross-modal attention fusion in step 3 specifically includes: feature vector mapping maps each single-modal feature to the same high-dimensional semantic space through a modality-specific encoder; visual features are transformed into 256-dimensional vectors through a visual encoder; tactile features are transformed into 256-dimensional vectors through a tactile encoder; auditory features are transformed into 256-dimensional vectors by embedding instruction text through an auditory encoder; encoder parameters are optimized through cross-modal pre-training; and cross-modal attention weights are calculated to dynamically determine the query vector in the multi-head attention mechanism. This makes it strongly bound to the variable task context, when explicit instructions are present. Composed of instruction-encoded vectors, it drives attention to focus on the modality most relevant to the instruction in autonomous tasks without explicit instructions. This is derived from the system's internal state history information and intrinsic goals, ensuring that attention can dynamically serve implicit goals such as obstacle avoidance and navigation. When calculating similarity, it will... respectively with visual vectors tactile vectors Auditory vectors Calculate the dot product similarity, weight and fuse them to generate a unified feature, and then sum the feature vectors of each modality according to the attention weights to obtain the cross-modal fused feature vector. The formula is .

[0010] Preferably, the task parsing and planning action refinement and parameter generation in step 4 of the large language model decision generation specifically includes: in task parsing and planning, the large language model first understands the core logic of the task objective, and decomposes the high-level task into executable sub-tasks according to the time sequence and causal relationship. The extraction of the core action chain is based on pre-trained physical common sense and task semantics. The robot action library adopts a layered and modular construction, with the bottom layer being indivisible atomic actions and the upper layer being macro actions composed of the bottom layer actions. The robot action library acquires new skills from teaching and reinforcement learning through skill learning, and achieves flexible expansion through parameterized interfaces.

[0011] Preferably, the method further includes steps 5 and 6. Step 5: During the decision execution process, the execution deviation is detected by real-time sensor signals. The deviation detection threshold is calibrated based on the robot's grasping force control accuracy and object stability experiments. The cause of the deviation is analyzed by multimodal signal cross-validation and large language model reasoning. A corrective decision is generated based on the attribution results. The execution results and historical decision data are stored in a structured manner. The text data is converted into vectors by the embedding model and stored in the vector database. When a new task is executed, the Top-N similar historical experience is retrieved by cosine similarity. The retrieved experience is injected with prompt words to assist in decision optimization. Step 6: Transform the verified decision into structured control instructions that the robot can execute. The instruction format includes step ID, action type, target parameters, execution conditions and priority fields. A standard data exchange format is used to ensure parsability. The format conversion module checks the integrity of the required fields. Finally, the instruction is sent to the robot control layer for execution.

[0012] Preferably, the deviation analysis and attribution correction decision generation in the feedback optimization in step 5 specifically includes: deviation analysis and attribution analyzes the causes of deviation through multimodal signal cross-validation and large language model reasoning, and detects deviation by comparing the execution status signal collected by real-time sensors with the expected target.

[0013] Preferably, in step 5, the object position deviation detected by vision and the grasping force deviation detected by touch in the multimodal perception data are used, and the common sense reasoning of the large language model is combined to determine whether the deviation is caused by parameter error, environmental change or hardware failure. The correction decision generation stage will generate targeted correction decisions by the large language model based on the deviation attribution results. If the attribution is parameter error, the action parameters will be adjusted; if the attribution is environmental change, the action will be replanned; if the attribution is hardware failure, the task will be paused and an alarm prompt will be output.

[0014] Preferably, the structured format definition for generating control instructions in step 5 specifically includes: the structured format of the control instructions includes the core fields of step ID, action type, target parameters, execution conditions, and priority. The step ID is used to identify the order of actions in the sequence. The action type is selected from the robot action library. The target parameters include specific values ​​of position, speed, force, and angle. The execution conditions include trigger conditions and termination conditions. The priority is used to handle scheduling conflicts when multiple actions occur concurrently.

[0015] A multimodal fusion autonomous decision-making system for robots based on a large language model includes modules for multimodal perception, data preprocessing and fusion, cross-modal translation, large language model decision-making, feedback optimization, and control command generation. The multimodal perception module acquires timestamped multimodal environmental information through visual, tactile, and auditory sensors and LiDAR. The data preprocessing and fusion module performs filtering and noise reduction, spatiotemporal semantic alignment, and then evaluates uncertainty through a dynamic Bayesian network. The weights are transformed using a Softmax function and weighted summation is used to obtain fused features. The cross-modal translation module maps the features of each modality to a 256-dimensional semantic space, generates a unified feature vector, and converts it into a natural language description. The large language model decision-making module constructs an input prompt to decompose the task, generates and verifies action parameters. The feedback optimization module detects deviations, analyzes the causes, generates corrective decisions, and retrieves historical experience to assist in optimization. The control command generation module converts the decisions into structured instructions containing core fields, which are then sent to the control layer for execution after format verification.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: This invention employs a dynamic Bayesian network to perform refined uncertainty assessment on multimodal signals such as vision, touch, and hearing. Entropy or variance is used as a quantitative indicator to accurately determine the reliability of each modality's perception results. The Softmax function then transforms uncertainty into fusion weights, ensuring that high-confidence modal features dominate the fusion result and effectively reducing interference from low-reliability data. Simultaneously, a cross-modal attention mechanism is combined to dynamically adjust attention weights based on task instructions and system state, strengthening the contribution of features relevant to the core task. This improves information quality from the source of data fusion, providing accurate and reliable foundational support for subsequent decision-making and significantly reducing decision-making errors caused by data bias.

[0017] This invention uses a modality-specific encoder to uniformly map different types of multimodal features to a 256-dimensional semantic space, achieving dimensional alignment and semantic interoperability of features across modalities. Building upon this, a neural generative model with an encoder-decoder architecture transforms non-textual perception data such as visual coordinates and tactile pressure into natural language descriptions understandable by a large language model, completely breaking down information barriers between different modalities. Simultaneously, through temporal, spatial, and semantic three-dimensional alignment processing, it ensures that multimodal data accurately describes the same object or event within the same task scenario, providing a solid guarantee for cross-modal collaborative decision-making and adapting to the needs of multi-sensor collaborative work in complex scenarios.

[0018] In the decision generation stage, this invention utilizes a large language model to construct a complete input prompt based on multimodal translated text, task objectives, environmental constraints, and historical experience. It decomposes high-level tasks according to chronological order and causal relationships, matches and adapts actions from a hierarchical modular action library, and generates parameters. Furthermore, it undergoes dual checks of physical rules and hardware constraints to prevent invalid decisions that are logically feasible but hardware-inoperable. In the execution stage, the feedback optimization module detects execution deviations using real-time sensor signals. The deviation detection threshold is calibrated based on the robot's grasping force control accuracy and object stability experiments to ensure the accuracy of deviation detection. Then, through multimodal signal cross-validation and large language model inference, it accurately analyzes the causes of deviations and generates targeted corrective decisions. This avoids task failures or hardware damage caused by parameter errors, environmental changes, or hardware malfunctions, comprehensively ensuring the safety and stability of decision execution. Attached Figure Description

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1 This is a flowchart illustrating the overall steps of the present invention; Figure 2 This is a flowchart illustrating the robot multimodal fusion decision-making framework based on a large language model, as described in this invention. Figure 3 A semantic alignment diagram of multimodal data and large language models provided by this invention; Figure 4 This is a flowchart of the decision generation method based on a large language model in this invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figures 1-4 As shown in the figure, this embodiment proposes a multimodal fusion autonomous decision-making method for robots based on a large language model. The specific steps are as follows: Step 1: Acquisition of Environmental Information. The robot acquires multimodal local environmental information through its onboard sensors and preprocesses the acquired information.

[0023] Step 2: Assigning values ​​based on uncertainty estimation information. Preprocess the multimodal information sensed by the sensor and assign different weights to uncertain and deterministic information.

[0024] Step 3: Multimodal translation. Establish a cross-modal semantic alignment transformation link. The generated text features must conform to the input format specifications of LLM so that LLM can efficiently parse the core meaning of non-text signals and support cross-modal interaction tasks.

[0025] Step 4: LLM decision generation, based on the semantic understanding and logical reasoning process trained on massive text data. The core is to output decision results that meet the task requirements through multi-stage semantic modeling, and generate "action sequence + parameter" decisions that can be executed by the robot, transforming natural language semantics into machine behavior logic.

[0026] Step 5: Feedback optimization, the "dynamic adjustment closed loop" of the multimodal fusion decision algorithm, the core goal of which is to correct the decision based on real-time feedback signals when the robot makes a deviation, to ensure that the task is eventually completed, and to allow the LLM to "self-correct" based on the actual execution results.

[0027] Step 6: Control command generation. The natural language decisions generated by LLM cannot be directly executed by the robot hardware. They need to be converted into a machine-readable structured format to ensure that the control layer can quickly extract "action type + parameters". The format standardization defines a unified decision output format, which includes core fields such as "step ID, action type, target parameters, execution conditions, and priority" to avoid missing fields.

[0028] Step 2 specifically includes the following steps: Multimodal data preprocessing aims to transform raw sensor signals into "quantifiable, low-noise" features to prepare for subsequent translation. This embodiment takes the preprocessing of multimodal data, including visual and tactile data, as an example: Step 2.1: Visual Target Coordinate Quantization The pixel coordinates of the target in the original image need to be mapped to the robot coordinate system: (1) in: Represents the pixel coordinates of the target in the image, where u is the horizontal pixel and v is the vertical pixel; Represents the camera intrinsic parameter matrix; This represents the coordinate offset, the positional deviation between the camera and the robot's base coordinate system; Represents the two-dimensional coordinates of the target in the robot coordinate system (unit: m); Step 2.2: Filtering the tactile pressure signal. The pressure signal output from the original tactile sensor contains noise and needs to be reduced by moving average filtering. (2) in: This is the original pressure signal at time t (unit: N); N represents the sliding window size, which was determined through experimental optimization based on the typical sampling frequency and signal smoothness requirements of the tactile sensor. The experiment compared the effects of different window sizes on the smoothing effect and real-time performance of the pressure signal, finding that N=5~10, while ensuring the filtering effect, could control the delay within a certain range. Within the specified range, it meets the real-time control requirements; Indicates the sensor sampling interval (unit: s); This represents the filtered effective pressure signal. Step 2.3: Uncertainty assessment of the preprocessed multimodal sensing signals is performed using a dynamic Bayesian network. This network uses the preprocessed visual, acoustic, and other sensor signals as synchronous input observation variables and establishes a hidden system state variable for them. By training on a multivariate dataset containing real state labels, the network can learn the probabilistic relationship between sensor readings and system state and its temporal dynamics. In actual inference, the network uses an online filtering algorithm to fuse the continuous multimodal signal sequences, calculates the posterior probability distribution of the system state in real time, and finally uses the entropy or variance of this distribution as a quantitative evaluation index of uncertainty, thereby dynamically judging the reliability of the sensing results. Step 2.4: Obtain the uncertainty of each mode Then, the Softmax function is used to convert them into fusion weights. Specifically, the uncertainty is first negativeened to obtain the confidence score, which is then calculated using Softmax to ensure that modes with high uncertainty are automatically assigned lower weights. (3) in, This represents the total number of modalities. The final fusion features. That is, the modal features Weighted sum: (4) Step 3 specifically includes the following steps: Multimodal data translation is the core component of multimodal fusion decision algorithms. Its core objective is to transform non-textual perception data such as visual, auditory, and tactile data into structured text representations that can be understood by the large language model (LLM), thereby eliminating the modal gap. The specific steps can be broken down into four stages: data alignment, modal feature extraction, cross-modal attention fusion, and text generation and optimization. Step 3.1: Multimodal data alignment. Multimodal data often comes from different sensors and has problems such as asynchronous time, inconsistent spatial coordinate systems, and differences in semantic dimensions. Alignment is required to ensure that the data describes the same object / event in the same task scenario, so as to provide a foundation for subsequent fusion.

[0029] (1) Time Alignment: Different sensors have different sampling frequencies, which may result in a time lag where the visual grasping action is observed, but the tactile data has not yet provided feedback on the pressure. Operation: The "robot task trigger moment" is used as the time origin. For example, the moment when the user issues the "hand over the apple" command is recorded as _____. Timestamp all sensor data, such as visual frames. tactile signals auditory instructions To unify the temporal granularity of multimodal data, a differentiated alignment strategy based on data characteristics is adopted: sliding window aggregation and interpolation are used for high-frequency continuous data; nearest neighbor matching and state preservation are used for low-frequency instantaneous data; and trigger-based accumulation and correlation are performed for asynchronous event data. All operations are based on the system master clock to ensure that the final fused modal data can accurately describe the scene state at the same moment, thereby avoiding information distortion or errors caused by simple uniform interpolation, such as matching all data to... The time window ensures that the scene state is described at the same moment.

[0030] (2) Spatial alignment: The intrinsic and extrinsic parameters are obtained through camera calibration, and the pixel coordinates are aligned. Convert to physical coordinates The local coordinates of the sensors are transformed into global coordinates in the base coordinate system through the kinematic model of the robotic arm; the final output is that all spatially related data are uniformly described using the "robot base coordinate system".

[0031] (3) Semantic alignment: Different modalities describe the same object with different semantic dimensions, and need to be associated as "multi-dimensional attributes of the same entity". Operation: Define "core entity label": such as determining the core entity as "apple" through the auditory instruction "pass apple", or identifying the entity category as "apple" through visual object detection; establish "modality-attribute mapping table": bind the features of each modality to the attributes of the core entity.

[0032] Step 3.2: Single-modal feature extraction. Aligned multimodal data still contains redundant noise. It is necessary to filter out the core information related to the task through "single-modal feature extraction" to reduce the computational load of subsequent fusion and improve semantic accuracy. Specific operations for feature extraction of each modality: Modality type, original data format, core feature extraction target, commonly used techniques / tools / methods, output results, examples. Visual RGB image, depth map, object category, position, shape, state - Object detection YOLOv8, Faster R-CNN: extract object category, bounding box; - Depth map parsing: calculate object 3D coordinates, size; - Image segmentation Mask R-CNN: distinguish object from background; - State recognition: determine whether the object is damaged / open. Category: Apple; Position: (0.8m, 0.6m, 0.5m); Size: diameter 8cm; State: complete. Tactile pressure matrix, torque signal, contact pressure, hardness, grasping stability - Signal filtering: remove noise; - Feature quantization: convert continuous pressure signal into "average pressure" and "maximum pressure"; - Stability judgment: if pressure fluctuation < ±0.5N, it is judged as "stable grasping". Average pressure: 3N; Maximum pressure: 4.2N; Hardness: 50ShoreA; Grasping Status: Stable auditory speech waveform, ambient sound user commands, key semantics, environmental cues - Speech-to-text: Converts speech to text; - Command extraction: Filters task-related commands; - Ambient sound recognition: Determines if there is interference. User command: "Hand me the apple"; Keywords: apple, hand; Ambient sound: No interference. LiDAR point cloud, infrared obstacle position, ambient temperature, distance - Point cloud clustering: Distinguishes obstacles from the background; - Distance calculation: LiDAR measures the shortest distance between the obstacle and the robot; - Infrared temperature measurement: Outputs ambient temperature / object temperature. Obstacle position: (1.2m, 0.3m, 0); Distance to robot: 0.8m; Ambient temperature: 25℃. Key principles: Feature "task-oriented": Only extract features relevant to the current task; Feature "structured": Avoid outputting unstructured data, convert it into LLM-understandable forms such as "key-value pairs" and "category labels".

[0033] Step 3.3: Cross-modal attention fusion. Single-modal features are still "isolated information fragments," "pressure 3N," and "instructions passed around like apples." Cross-modal attention fusion is needed to calculate the correlation between features from different modalities, determine which features are more important to the task, and ultimately form a "unified fused feature vector"—this is the core technical step in translation. Specific steps: Feature vector mapping problem: Features from different modalities have different dimensions, making direct correlation calculation impossible. Operation: A "modal-specific encoder" maps each single-modal feature to the same high-dimensional semantic space, ensuring dimensional consistency: Visual features → Visual encoder: Converts "category, location, and size" into a 256-dimensional vector. ; Tactile features → Tactile encoder: Transforms "pressure, hardness, and stability" into 256-dimensional vectors Auditory features → Auditory encoder: Transforms "instruction text embedding" into a 256-dimensional vector. Technical basis: Encoder parameters are optimized through "cross-modal pre-training" to ensure that the vector distance of the same semantics in different modalities is as close as possible. The core logic of cross-modal attention weight calculation is: different tasks have different importance in each modality. Operation: To dynamically determine the query vector Query in the multi-head attention mechanism, making it strongly bound to the variable "task context" rather than fixed to a single modality; Specifically: When explicit instructions exist, the Query is composed of the encoded vector of that instruction, driving attention to focus on the modality most relevant to the instruction; in autonomous tasks without explicit instructions, the Query is derived from the system's internal state, historical information, and intrinsic goals, ensuring that attention dynamically serves implicit goals such as obstacle avoidance and navigation. This design allows the fusion strategy to adaptively adjust according to actual task requirements, possessing true versatility; Similarity calculation: The Query is compared with the visual vector V, tactile vector T, and auditory vector A (itself) using "dot product similarity" to measure relevance: where... (Query) (key), (Vector dimension) This is a normalization term to prevent excessively high similarity values ​​due to excessively high dimensionality; softmax normalization: converts similarity into weights between 0 and 1, ensuring that the sum of the weights is 1. , , Example output: (Instructions have the highest weight) (Position weight is secondary). (Stability weight is lowest). Weighted fusion to generate unified features: The feature vectors of each modality are summed according to attention weights to obtain the "cross-modal fusion feature vector". : . It contains the core information of all modalities and highlights important task-related features, providing a "semantically condensed feature base" for subsequent text generation.

[0034] Step 3.4: Text Generation and Optimization of Cross-Modal Fusion Feature Vectors The data is still a "numerical vector" and needs to be converted into natural language text through "text generation"—this step is the final output of the translation and directly determines whether LLM can accurately understand multimodal scene information. The specific operation steps are as follows: Feature-Text Mapping Core Idea: A neural generative model based on an encoder-decoder architecture is adopted. This model takes the fused multimodal features as input and "translates" them into natural language descriptions through a learned language model. To ensure the accuracy and flexibility of the generation, a dynamic concept vocabulary mechanism is introduced. That is, when generating each word, the general vocabulary is combined with entity and attribute candidate words decoded in real time from the current scene features. This method enables the system to creatively combine known concepts to describe unforeseen objects and scenes, thereby achieving powerful generalization ability. Template design principles: It includes the three elements of "entity + attribute + task"; the language is concise and conforms to the text understanding habits of LLM.

[0035] Step 4 specifically includes the following steps: LLM decision generation is the "core brain" of the multimodal fusion decision algorithm. Its core objective is to generate "action sequence + parameters" decisions that can be executed by the robot based on the structured text, task objectives and historical experience after multimodal translation. In essence, it is to "convert natural language semantics into machine behavioral logic". The specific steps can be broken down into five stages: input construction → task parsing and planning → action refinement and parameter generation → decision verification and optimization → structured output.

[0036] Step 4.1: Constructing Decision Input. Since LLM cannot directly perceive the robot's state, it's necessary to first integrate "multimodal translated text, task objectives, environmental constraints, and historical experience" into a structured input prompt to ensure comprehensive information—the more complete the input, the more accurate the decision. This is a crucial prerequisite for avoiding "illusionary decision-making" in LLM. Core information integration involves logically sorting the multimodal translated text and user / system-specified task objectives according to the sequence "scenario → objective → constraint," forming the basic input framework.

[0037] Step 4.2: Task Analysis and Planning. LLM first needs to understand the "core logic of the task objective" and break down the high-level task into executable sub-tasks according to "time sequence + causal relationship"—this step is the "framework building" for decision-making, ensuring the coherence of action logic. Task core logic identification: LLM extracts the "core action chain" of the task through pre-trained physical common sense and task semantics. For example, the core logic of "passing an apple" is: "maintain stable grasping → move to the target position → release the object," excluding irrelevant actions.

[0038] The subtasks are broken down according to the time sequence of "preparation → execution → confirmation", and the core action chain is broken down into subtasks, with each subtask corresponding to a "stage goal".

[0039] Step 4.3: Action Refinement and Parameter Generation. The subtask remains a "qualitative description" and needs further refinement into quantitative "action type + parameters"—this is the key transformation from "semantic" to "executable" decision-making. Action type matching selects a uniquely matching action type from the robot's "action library" based on the subtask objective. The "action library" is a layered and modular robot skill system. Its bottom layer consists of indivisible actions, while the upper layer comprises macro-actions that combine the bottom-layer actions to achieve the sub-objective. This library can acquire new skills through teaching or reinforcement learning and can be flexibly expanded through parameterized interfaces. All newly added actions must undergo rigorous simulation and real-world environment verification to ensure that the system maintains the determinism and reliability of its behavior while its capabilities continuously improve. The robot action library must be defined in advance.

[0040] Step 4.4: Decision Verification and Optimization. LLM may generate decisions that are "logically correct but not hardware-executable." The feasibility and safety of these decisions must be ensured through "physical rule verification + risk assessment + feedback correction"—this is crucial to preventing robot hardware damage or task failure. The physical rule and hardware constraint verification calls the "robot hardware parameter library" to verify each motion parameter generated by the LLM, filtering out unexecutable decisions.

[0041] For highly complex tasks, if the initial decision fails the validation, the "validation error message" is re-entered into the LLM and the decision is generated again by combining the multimodal translated text until the validation is passed.

[0042] Step 5 specifically includes the following steps: Feedback optimization is the "dynamic adjustment closed loop" of the multimodal fusion decision algorithm. Its core objective is to correct the decision based on real-time feedback signals when the robot makes a deviation, so as to ensure the task is eventually completed. The specific steps can be broken down into five stages: deviation detection and signal acquisition, deviation analysis and attribution, corrected decision generation, corrected decision verification, and execution and secondary feedback. Each stage needs to combine the reasoning capabilities of multimodal perception and LLM to ensure the accuracy of the correction.

[0043] Step 5.1: Deviation detection and signal acquisition. The premise of decision correction is to accurately identify the "deviation between the execution result and the expected goal" - the execution status signal needs to be collected by the robot's real-time sensors, compared with the expected goal, and it needs to be determined whether there is a deviation. This step should avoid misjudging the deviation. Step 5.2: Deviation Analysis and Attribution. Merely detecting a deviation is insufficient for accurate correction; the cause of the deviation must be analyzed through "multimodal signal cross-validation + LLM inference." Step 5.3: Corrective Decision Generation. Based on the bias attribution results, the LLM needs to generate "targeted corrective decisions"—if the attribution is "parameter error", then adjust the action parameters; if the attribution is "environmental change", then replan the action; if the attribution is "hardware failure", then output "pause task + alarm prompt". The corrective decision should avoid repeating the original error and adapt to the current environmental state. Step 5.4: Correction and Decision Verification. Correction of decisions still needs to go through "physical rule verification + risk assessment" to avoid "introducing new problems after correction". This step is consistent with the original decision verification logic, but needs to focus more on "the special characteristics of the deviation scenario". Hardware constraints and physical rules verification calls the robot's hardware parameter library and physical common sense to verify whether the parameters for corrective decisions are executable; Step 5.5: Correct Decision Execution and Secondary Feedback. The verified corrected decision is converted into a structured format that the robot can execute. After the drive is executed, feedback signals are collected again to confirm whether the deviation has been resolved. If it is resolved, the task is completed. If the deviation still exists, the closed loop of "deviation detection → correction" is repeated until the task is successful. The structured output of the corrected decision refers to the structured format of the original decision and converts the corrected decision into instructions that the robot control layer can parse, ensuring that the fields are compatible with the original decision.

[0044] After the secondary feedback and task closure are completed, the final execution result is collected, compared with the correction target, and it is determined whether the task is completed. If completed, the "original decision, deviation information, correction decision, and execution result" are stored in the historical decision database as experience for subsequent similar tasks. Specific Solution: The core of the closed-loop implementation of experience learning is "structured storage + RAG-linked retrieval": Historical data is stored in a structured format of "core fields + multimodal attachments". It must store task meta-information, decision-making full-link data, execution result data and associated feature labels. Text data is converted into high-dimensional vectors by the embedding model and stored in the vector database. Multimodal attachments are stored synchronously as cross-modal vectors. When a new task is received, LLM first extracts the core features of the task and converts them into a unified vector. It then retrieves the Top-N similar historical tasks in the vector database using the cosine similarity algorithm, recalls the corresponding effective experience and pitfall avoidance cases, and injects these historical data into prompt words. This allows LLM to optimize decisions in combination with new scenarios. At the same time, through mechanisms such as dynamic label updates, adjustment of experience weights, and multimodal cross-validation, the accuracy of experience reuse is improved, forming a complete learning closed loop of "storage-retrieval-utilization-optimization". If not completed: "stage 1 to stage 5" are repeated to generate secondary correction decisions until the task is successful. If multiple corrections still fail: the task is paused and a fault alarm is output, prompting manual intervention.

[0045] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Since the above embodiments are fundamentally similar to the method embodiments, they are described relatively simply; relevant parts can be referred to the descriptions of the method embodiments.

[0047] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal fusion autonomous decision-making method for robots based on a large language model, characterized in that, Includes the following steps: Step 1: The robot acquires multimodal environmental information through its onboard visual sensors, tactile sensors, auditory sensors, and LiDAR. It performs preliminary filtering and noise reduction on the raw sensor data, and all sensor data is recorded synchronously with timestamps. Step 2: Perform spatiotemporal semantic alignment on the preprocessed multimodal data. For visual target coordinate quantization, the pixel coordinates of the target in the original image need to be mapped to the robot coordinate system. The tactile pressure signal is denoised by moving average filtering. The uncertainty of the multimodal signal is evaluated by dynamic Bayesian network, with entropy or variance as the quantitative evaluation index of uncertainty. Finally, the uncertainty of each modality is converted into fusion weights by the Softmax function. The final fusion feature is the weighted sum of the features of each modality. Step 3: The fused multimodal features are transformed into structured text representations through a cross-modal attention mechanism. Each modal feature is mapped to the same vector dimension through a modality-specific encoder. Cross-modal attention weights are calculated. The query vector is obtained by concatenating the task instruction encoding and the system state encoding and then performing a linear mapping. The key vector is the feature vector of each modality. The attention weights are obtained by normalization through the softmax function. Finally, the feature vectors are transformed into natural language descriptions through a neural generative model based on an encoder-decoder architecture. Step 4: Input structured text into the large language model to generate robot-executable decisions. Construct an input prompt that includes multimodal translated text, task objectives, environmental constraints, and historical experience. Decompose the high-level task into sub-tasks through task parsing and planning. Select matching action types from the robot action library and generate parameters. Ensure the feasibility of the decision through physical rule verification and hardware constraint verification.

2. The multimodal fusion autonomous decision-making method according to claim 1, characterized in that, The uncertainty assessment and weight calculation of the dynamic Bayesian network in step 2 specifically includes: the dynamic Bayesian network consists of state variables and observation variables. The state variables represent the robot's internal state and environmental state, and the observation variables correspond to multimodal sensor data. The training data includes multimodal sensor readings and their corresponding real state labels. The network updates the posterior probability distribution of the state variables online through a forward recursive algorithm. The entropy of the posterior probability distribution is used to represent uncertainty. For continuous state variables, the variance is used to represent uncertainty. The fused feature vector is finally used for subsequent cross-modal semantic translation.

3. The multimodal fusion autonomous decision-making method according to claim 1, characterized in that, The multimodal data alignment in step 3 specifically includes: time alignment using the robot task triggering time as the time origin, timestamping all sensor data, performing sliding window aggregation and interpolation on high-frequency continuous data, using nearest neighbor matching and state preservation on low-frequency instantaneous data, and accumulating and associating asynchronous event data based on triggers. All operations are based on the system master clock. Spatial alignment uses camera calibration to obtain intrinsic and extrinsic parameters to convert pixel coordinates into physical coordinates, and uses the robotic arm kinematic model to convert the local coordinates of the sensors into global coordinates in the base coordinate system. Finally, all spatially related data are uniformly described using the robot base coordinate system. Semantic alignment defines core entity labels, identifies entity categories through auditory commands and visual target detection, establishes a modal attribute mapping table, and binds each modal feature to the attributes of the core entity to ensure that the semantic dimensions of different modal descriptions of the same object can be associated with the multi-dimensional attributes of the same entity.

4. The multimodal fusion autonomous decision-making method according to claim 1, characterized in that, The cross-modal attention fusion in step 3 specifically includes: feature vector mapping maps each single-modal feature to the same high-dimensional semantic space through a modality-specific encoder; visual features are transformed into 256-dimensional vectors through a visual encoder; tactile features are transformed into 256-dimensional vectors through a tactile encoder; auditory features are transformed into 256-dimensional vectors by embedding instruction text through an auditory encoder; encoder parameters are optimized through cross-modal pre-training; and cross-modal attention weights are calculated to dynamically determine the query vector in the multi-head attention mechanism. This makes it strongly bound to the variable task context, when explicit instructions are present. Composed of instruction-encoded vectors, it drives attention to focus on the modality most relevant to the instruction in autonomous tasks without explicit instructions. This is derived from the system's internal state history information and intrinsic goals, ensuring that attention can dynamically serve implicit goals such as obstacle avoidance and navigation. When calculating similarity, it will... respectively with visual vectors tactile vectors Auditory vectors Calculate the dot product similarity, weight and fuse them to generate a unified feature, and then sum the feature vectors of each modality according to the attention weights to obtain the cross-modal fused feature vector. The formula is .

5. The multimodal fusion autonomous decision-making method according to claim 1, characterized in that, The task parsing and planning action refinement and parameter generation in step 4 of the large language model decision generation specifically include: In task parsing and planning, the large language model first understands the core logic of the task objective and decomposes the high-level task into executable sub-tasks according to the time sequence and causal relationship. The extraction of the core action chain is based on pre-trained physical common sense and task semantics. The robot action library adopts a layered and modular construction. The bottom layer consists of indivisible atomic actions, and the upper layer consists of macro actions composed of the bottom layer actions. The robot action library acquires new skills from teaching and reinforcement learning through skill learning and achieves flexible expansion through parameterized interfaces.

6. The multimodal fusion autonomous decision-making method according to claim 1, characterized in that, The method further includes steps 5 and 6. Step 5: During the decision execution process, the execution deviation is detected by real-time sensor signals. The deviation detection threshold is based on the experimental calibration of robot grasping force control accuracy and object stability. The cause of the deviation is analyzed by multimodal signal cross-validation and large language model reasoning. Correction decisions are generated based on the attribution results. The execution results and historical decision data are stored in a structured manner. The text data is converted into vectors by the embedding model and stored in the vector database. When a new task is executed, the Top-N similar historical experience is retrieved by cosine similarity. The retrieved experience is injected with prompt words to assist in decision optimization. Step 6: Transform the verified decision into structured control instructions that the robot can execute. The instruction format includes step ID, action type, target parameters, execution conditions and priority fields. A standard data exchange format is used to ensure parsability. The format conversion module checks the integrity of the required fields. Finally, the instruction is sent to the robot control layer for execution.

7. The multimodal fusion autonomous decision-making method according to claim 6, characterized in that, The deviation analysis and attribution correction decision generation in step 5 of the feedback optimization specifically includes: deviation analysis and attribution analyze the causes of deviation through multimodal signal cross-validation and large language model reasoning, and compare the execution status signals collected by real-time sensors with the expected target to detect deviation.

8. The multimodal fusion autonomous decision-making method according to claim 6, characterized in that, In step 5, the object position deviation detected by vision and the grasping force deviation detected by touch in the multimodal perception data are used. Combined with the common sense reasoning of the big language model, it is determined whether the deviation is caused by parameter error, environmental change or hardware failure. The correction decision generation stage will generate targeted correction decisions by the big language model based on the deviation attribution results. If the attribution is parameter error, the action parameters will be adjusted. If the attribution is environmental change, the action will be replanned. If the attribution is hardware failure, the task will be paused and an alarm prompt will be output.

9. The multimodal fusion autonomous decision-making method according to claim 6, characterized in that, The structured format definition for generating control instructions in step 5 specifically includes: the structured format of the control instructions contains the core fields of step ID, action type, target parameters, execution conditions, and priority. The step ID is used to identify the order of actions in the sequence. The action type is selected from the robot action library. The target parameters include specific values ​​of position, speed, force, and angle. The execution conditions include trigger conditions and termination conditions. The priority is used to handle scheduling conflicts when multiple actions occur concurrently.

10. A robot multimodal fusion autonomous decision-making system based on a large language model, used to apply the multimodal fusion autonomous decision-making method as described in any one of claims 1-9, characterized in that, The system includes modules for multimodal perception, data preprocessing and fusion, cross-modal translation, large language model decision-making, feedback optimization, and control command generation. The multimodal perception module acquires timestamped multimodal environmental information through visual, tactile, and auditory sensors and LiDAR. The data preprocessing and fusion module performs filtering and noise reduction, spatiotemporal semantic alignment, and then evaluates uncertainty through a dynamic Bayesian network. The weights are transformed using the Softmax function and weighted summation is used to obtain the fused features. The cross-modal translation module maps the features of each modality to a 256-dimensional semantic space, generates a unified feature vector, and converts it into a natural language description. The large language model decision-making module constructs an input prompt to decompose the task, generates and verifies action parameters. The feedback optimization module detects deviations, analyzes the causes, generates corrective decisions, and retrieves historical experience to assist in optimization. The control command generation module converts the decisions into structured instructions containing core fields, which are then sent to the control layer for execution after format verification.

Images