Multi-modal fusion body-equipped intelligent system in complex scene and use method of multi-modal fusion body-equipped intelligent system

Abstract

The invention discloses a multi-modal fusion body-equipped intelligent system in a complex scene and a use method, and belongs to the technical field of artificial intelligence and robots. Aiming at the problems of perception lag, task splitting and incapability of understanding an open instruction of an existing intelligent agent in a dynamic complex environment, a complete perception-decision-execution closed loop is constructed by integrating core modules such as multi-modal perception, open vocabulary target detection, dynamic semantic mapping, natural language instruction analysis and multi-modal decision. According to the system, vision-language embedding, cross-modal attention fusion and an intelligent obstacle avoidance mechanism are adopted, multi-dimensional understanding and self-adaptive path planning of the environment are achieved, and the performance of the model is optimized through a two-stage training strategy. According to the method, the accurate response, multi-task cooperative execution and dynamic obstacle avoidance capability of the intelligent agent to a natural language instruction in an unknown scene are remarkably improved, and the method can be widely applied to the fields of military reconnaissance, disaster rescue, urban security and protection and the like and has high practicability, adaptability and expandability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robotics technology, specifically relating to an embodied intelligence system and its usage method for multimodal fusion in complex scenarios. Background Technology

[0002] With the rapid development of artificial intelligence subfields such as intelligent robot technology, computer vision, and natural language processing, robots have gradually expanded from traditional industrial applications to complex scenarios such as military, urban security, and disaster relief. In military scenarios, especially in individual soldier operations, complex environments, changing tasks, and unexpected events place higher demands on the perception, decision-making, and execution capabilities of intelligent agents. Most existing intelligent robot systems still rely on single-modal data during task execution, making it difficult to achieve unified perception and fusion processing of multi-source environmental data, resulting in problems such as perception lag, task fragmentation, and poor coordination.

[0003] Furthermore, in dynamic environments, robots need to understand and respond to unstructured natural language commands, quickly perceive environmental changes, and make decisions. This poses challenges to their information processing capabilities, task planning abilities, and execution stability. Traditional systems built on preset rules or narrow vocabulary sets lack the ability to handle open-ended language commands and undefined scenarios, making them unsuitable for dynamic battlefield tasks. Especially in urban scenarios or indoor buildings, where the environment is complex and obstructions are frequent, traditional methods have significant shortcomings in target recognition and path planning. Summary of the Invention

[0004] To address the problems of existing intelligent agents exhibiting lag in perception, fragmented tasks, and inability to understand open instructions in dynamic and complex environments, this invention provides a multimodal fusion embodied intelligence system and its usage method for complex scenarios.

[0005] To achieve the above objectives, the present invention employs the following technical solutions:

[0006] A multimodal fusion embodied intelligence system for complex scenarios, the system comprising:

[0007] Multimodal perception module, open semantic detection module, dynamic map construction module, language instruction parsing module, multimodal decision-making module, intelligent obstacle avoidance mechanism, and cross-modal attention fusion mechanism;

[0008] The multimodal perception module consists of a high-resolution RGB visual sensor, a depth measurement sensor, and a high-sensitivity environmental sound collector, and is used to synchronously collect and preprocess key perception data in complex environments.

[0009] The open semantic detection module employs an open vocabulary-based target detection neural network to detect objects of any category in RGB images in real time without prior category setting; by introducing a language model that supports text embedding and an image region feature matching mechanism, it achieves the joint output of object category labels and bounding box coordinates.

[0010] The dynamic map construction module uses an inverse pinhole camera projection algorithm to back-project the depth image, constructs a spatial point cloud, and then generates a two-dimensional map structure readable by the embodied intelligent agent through high compression and rasterization processing.

[0011] The language instruction parsing module is used to process natural language instructions input by the user, extract semantic fragments through natural language processing technology and match them with the current visual scene information;

[0012] The multimodal decision module integrates map features, language features, current sensor observation information, and historical action states to output the next optimal action decision.

[0013] The intelligent obstacle avoidance mechanism is an obstacle avoidance strategy based on image texture analysis built into the embodied intelligent system;

[0014] The cross-modal attention fusion mechanism is used for feature fusion and employs a three-level processing flow.

[0015] The key sensing data used in the multimodal sensing module include RGB image data, depth image data, and sound waveform data;

[0016] RGB image data is used for target recognition and visual scene analysis, depth image data is used for 3D spatial structure modeling, and sound waveform data is used for voice-assisted scene understanding or abnormal event capture.

[0017] The open semantic detection module employs a text encoder enhancement strategy, using the text encoder in the CLIP model as a pre-training foundation and introducing a learnable weight matrix to enhance the adaptability of the open vocabulary target detection neural network to open vocabulary; its text embedding calculation process is as follows:

[0018]

[0019] in, This represents the enhanced text embedding vector; This represents the pre-trained text encoder, specifically the text encoder of the CLIP model. A sequence of markers representing the input text; This indicates the original text to be entered. Mapping to a feature space with semantic information yields the basic text embedding. This represents a learnable weight matrix, which is a sequence obtained after preprocessing such as word segmentation for user commands or query targets. This represents the adapter output vector.

[0020] First, the 2D image bounding boxes output by open semantic detection are converted to 3D world coordinates through inverse perspective transformation; then, the 3D coordinates are projected onto the corresponding positions on the 2D raster map; finally, raster annotation conflicts are handled, and the coordinate projection function is as follows:

[0021]

[0022] in, Represents the grid coordinates in a two-dimensional grid map. Represents the horizontal coordinate in the world coordinate system. Represents the vertical coordinate in the world coordinate system. This represents the actual physical distance represented by each grid cell on the map along the X-axis (horizontal) direction. This represents the actual physical distance represented by each grid cell on the map along the Y-axis (vertical direction).

[0023] The dynamic map construction module includes the following steps:

[0024] First: The raw data collected by the depth measurement sensor is converted into a 3D point cloud using the inverse pinhole projection algorithm. The formula is as follows:

[0025]

[0026] in, This represents a set of 3D point clouds, which is a set of discrete 3D spatial points obtained by processing the depth image using a depth camera through an inverse pinhole projection algorithm. Let represent the three-dimensional coordinates of the i-th point, where This represents the coordinates of the i-th point along the X-axis. This represents the coordinates of the i-th point along the Y-axis. This represents the coordinates of the i-th point along the Z-axis. This represents the index of a point in the point cloud. This represents the total number of points in the point cloud.

[0027] Secondly, intelligent height compression processing is performed to retain only the effective point cloud data near the working plane of the carrier. The height compression processing is adaptive. By setting the height threshold parameter Δz, only the point cloud data that meets the height constraint is retained. Δz is a configurable parameter with a value range of 0.5 meters to 1.5 meters. This range can adapt to the point cloud filtering needs in most indoor scenarios, effectively removing interfering point clouds from the ceiling and the ground, while completely retaining the key environmental information in the workspace.

[0028] Then: Construct a binary occupancy raster map to clearly distinguish between feasible and obstacle areas in the environment;

[0029] Finally: Construct a semantically labeled raster map to provide category labels for key objects in the environment;

[0030] The semantic annotation raster is generated using the highest confidence principle. When multiple semantic labels of different categories are projected onto the same raster cell, the system automatically selects the semantic category with the highest detection confidence as the annotation result for that raster, ensuring the accuracy and uniqueness of the semantic icon annotation. The specific determination logic is as follows:

[0031]

[0032] in, This represents the semantic label ultimately assigned at raster coordinates (i, j). This is the system's output, indicating what kind of object or category the physical region represented by that raster cell is classified as. Let S represent a candidate semantic label, which is a variable that varies within the candidate set S. Indicates projection onto the current grid. The set of all candidate semantic labels on the grid is a collection of data. Since a single grid may correspond to multiple points in three-dimensional space, and each point may come from detection results at different times and from different perspectives, multiple different semantic labels may be projected onto the same grid. This represents the detection confidence of the candidate semantic label s.

[0033] The language instruction parsing module includes the following steps:

[0034] First, natural language processing techniques are used to segment the input long text instruction into a sequence of semantically independent sub-segments;

[0035] Second: Calculate the matching probability distribution vector between each sub-segment and the current visual scene;

[0036] Then: the degree of ambiguity of the instruction is determined by information entropy analysis, and the optimal instruction fragment is dynamically selected; the Shannon entropy value of the matching probability distribution is calculated, and when the entropy value exceeds the preset threshold of 1.5, it is determined to be a high ambiguity scene. At this time, the complete instruction text is output to ensure accuracy; when the entropy value is lower than the threshold, the sub-instruction fragment with the highest matching probability is output to improve execution efficiency.

[0037] Finally: If the entropy value is higher than the set threshold of 1.5, the current scene is determined to be semantically ambiguous, and the complete instruction is returned to ensure robustness; if the entropy value is lower, the sub-instruction with the highest confidence is executed directly to improve response efficiency.

[0038] The multimodal decision-making module includes the following steps:

[0039] First, a deep convolutional network is used to extract joint spatial features of the local occupancy map and the semantic map;

[0040] Secondly: the timing characteristics of instructions in the bidirectional long short-term memory network coding language are adopted;

[0041] Then: A cross-modal attention mechanism is designed to integrate visual, linguistic, and spatial features. This mechanism employs a three-level processing structure: First, the spatial features of multiple consecutive frames of map data, such as the combined features of the local occupancy map and semantic map from the past five frames, are input into a temporal modeling unit—this unit is implemented using a Long Short-Term Memory (LSTM) network. Through dynamic modeling of historical feature sequences, it outputs a long-term visual memory state containing the temporal changes in the environment. Second, the attention weight distribution between linguistic features and the long-term visual memory state is calculated—the linguistic features are low-dimensional vectors obtained by encoding natural language instructions using a Bi-LSTM network, containing key semantic information such as the action type and target object in the instructions. Finally, the weighted visual memory, depth image information, and linguistic encoding are fused to generate the final decision vector. Its core state update formula is:

[0042]

[0043] in, Indicates at time step The decision state vector encapsulates current perception, instruction, and historical state information, and is used to generate the final action instruction. This refers to a cross-modal attention mechanism, responsible for the deep fusion of features from different sources and modalities. Indicates time step The visual memory state. It is obtained by processing historical map spatial features by a temporal modeling unit, and represents the system's long-term visual memory and understanding of the environment. Indicates at time step The fused feature vector, This represents a vector concatenation operation, which joins two or more vectors along their feature dimensions to form a new vector that contains all the input information. This represents the attention calculation function. It is used to calculate the relevance weights between the query vector and the key-value pairs. This represents the decision state vector at time step t-1, which represents the historical decision state of the system at the previous time step.

[0044] Finally: The output contains the probability distribution of four basic action commands: go straight for a fixed distance, turn left at a fixed angle and proceed in parallel, turn right at a fixed angle and proceed in parallel, and stop completely.

[0045] The multimodal decision-making module integrates an intelligent obstacle avoidance mechanism, which includes: real-time analysis of the texture features of the current RGB image and calculation of the grayscale variance of the entire image; when the grayscale variance of the entire image is detected to be lower than an empirical threshold of 10, it is determined that the system has entered a low-texture dead zone region; then, the image is segmented into two symmetrical regions, and the average grayscale value of each region is calculated and compared; based on the comparison result, a direction decision command is automatically triggered to allow the system to escape a potential dead state. The formula for the direction decision command is:

[0046]

[0047] in, This represents the average grayscale value of the left half of the current RGB image. This represents the average grayscale value of the right half of the current RGB image.

[0048] The embodied intelligence system adopts a two-stage training process, as follows:

[0049] Phase 1: Train the base model using expert demonstration data through supervised learning to minimize action prediction error; the supervised learning loss function is:

[0050]

[0051] in, This represents the behavior cloning loss value. The smaller the value, the smaller the difference between the model's predicted action and the expert's action, and the better the cloning effect. This represents the total number of time steps, indicating the total number of steps in a complete task from start to finish. This represents a single time step index, ranging from 1 to T, used to traverse every execution step of the task. This represents the action vector predicted by the model at step t, which contains the specific parameters of the action. This represents the optimal action vector demonstrated by the expert at step t, which is the "standard answer" learned by the model. It can be derived from human expert operation data or high-quality preset trajectory data.

[0052] Phase Two: Fine-tuning parameters in a simulated environment using reinforcement learning to maximize overall performance. This reinforcement learning phase employs a proximal policy optimization algorithm and designs a multi-objective reward function to comprehensively consider path efficiency, trajectory matching degree, and safety. The multi-objective reward function is as follows:

[0053]

[0054] in, This represents the total reward value, which comprehensively evaluates the performance of the agent in a single navigation attempt; a higher value indicates better performance. Indicator of path efficiency, a combined metric measuring "navigation success" and "path simplicity". This represents standardized dynamic time warping, used to evaluate the similarity between the agent's actual trajectory and the "expert ideal trajectory." This indicates the number of collisions between the agent and environmental obstacles during navigation.

[0055] The embodied intelligent system is deployed on a quadrupedal mobile robot platform, which includes a high-torque servo drive system, an embedded AI computing unit Jetson AGX Orin, a multimodal sensor board, and a high-speed wireless communication module.

[0056] A method for using a multimodal fusion embodied intelligence system in complex scenarios, the method comprising the following steps:

[0057] Step 1: Simultaneously acquire environmental visual and depth image data using multiple sensors;

[0058] Step 2: Construct a dynamically updated 2D occupancy map and semantic annotation map in real time;

[0059] Step 3: Parse the input natural language instructions and output the best matching sub-instruction;

[0060] Step 4: Fuse multi-source sensory features to generate an action probability distribution;

[0061] Step 5: Execute the action command with the highest probability.

[0062] Step 6: Update the internal state of the system and return to Step 1 to form a closed-loop control.

[0063] Compared with the prior art, the present invention has the following advantages:

[0064] 1) Enhanced perception capabilities: By employing RGB vision, depth perception, and environmental audio in conjunction with open semantic detection and semantic map construction, it achieves real-time understanding and adaptation to unknown environments and new targets, breaking through the dependence of traditional robots on predefined environments.

[0065] 2) More accurate command understanding: By introducing natural language segmentation and image-text matching mechanisms, combined with information entropy analysis, the system can accurately parse and semantically match complex natural language commands, enabling it to execute unstructured, fuzzy, or open-ended complex passwords.

[0066] 3) More intelligent decision-making mechanism: The combination of multimodal attention mechanism and RNN policy network realizes deep integration of language, vision and spatial features, improves the rationality of decision-making and the coherence of task planning; at the same time, the built-in intelligent obstacle avoidance strategy effectively avoids the problem of environmental deadlock.

[0067] 4) More advanced training strategy: The model adopts a two-stage hybrid optimization strategy of "supervised learning + reinforcement learning", which enables the model to not only learn from expert experience, but also continuously optimize itself in the simulation environment, achieving efficient generalization ability and policy robustness.

[0068] 5) High scalability: Thanks to the introduction of an open vocabulary detection framework and semantic graphing mechanism, the system can continuously expand its ability to recognize categories and adapt to different scenarios, supports flexible access to multiple languages, multiple tasks and multiple modal inputs, and has broad engineering application value. Attached Figure Description

[0069] Figure 1 This is a diagram illustrating the overall system architecture of the present invention.

[0070] Figure 2 This is an overall flowchart of the present invention;

[0071] Figure 3 A diagram illustrating the specific implementation process of fusing RGB image features and depth image features;

[0072] Figure 4 The diagram illustrates the specific processing steps for obtaining the language segment that best matches the current scene by processing RGB images and language modal features using the CLIP model. Detailed Implementation

[0073] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.

[0074] A multimodal fusion embodied intelligence system for complex scenarios, the system comprising:

[0075] Multimodal perception module, open semantic detection module, dynamic map construction module, language instruction parsing module, multimodal decision-making module, intelligent obstacle avoidance mechanism, and cross-modal attention fusion mechanism;

[0076] The multimodal perception module consists of a high-resolution RGB visual sensor, a depth measurement sensor, and a high-sensitivity environmental sound collector, and is used to synchronously collect and preprocess key perception data in complex environments.

[0077] The open semantic detection module employs an open-vocabulary target detection neural network to perform real-time detection of objects of any category in RGB images without prior category setting. By introducing a language model supporting text embedding and an image region feature matching mechanism, it achieves joint output of object category labels and bounding box coordinates. The specific implementation process is as follows: Figure 3 As shown;

[0078] The dynamic map construction module uses an inverse pinhole camera projection algorithm to back-project the depth image, constructs a spatial point cloud, and then generates a two-dimensional map structure readable by the embodied intelligent agent through high compression and rasterization processing.

[0079] The language instruction parsing module is used to process natural language instructions input by the user, extract semantic fragments through natural language processing technology and match them with the current visual scene information;

[0080] The multimodal decision module integrates map features, language features, current sensor observation information, and historical action states to output the next optimal action decision.

[0081] The intelligent obstacle avoidance mechanism is an obstacle avoidance strategy based on image texture analysis built into the embodied intelligent system;

[0082] The cross-modal attention fusion mechanism is used for feature fusion and employs a three-level processing flow.

[0083] The key sensing data used in the multimodal sensing module include RGB image data, depth image data, and sound waveform data;

[0084] RGB image data is used for target recognition and visual scene analysis, depth image data is used for 3D spatial structure modeling, and sound waveform data is used for voice-assisted scene understanding or abnormal event capture.

[0085] The open semantic detection module employs a text encoder enhancement strategy, using the text encoder in the CLIP model as a pre-training foundation and introducing a learnable weight matrix to enhance the adaptability of the open vocabulary target detection neural network to open vocabulary; its text embedding calculation process is as follows:

[0086]

[0087] in, This represents the enhanced text embedding vector; This represents the pre-trained text encoder, specifically the text encoder of the CLIP model. A sequence of markers representing the input text; This indicates the original text to be entered. Mapping to a feature space with semantic information yields the basic text embedding. This represents a learnable weight matrix, which is a sequence obtained after preprocessing such as word segmentation for user commands or query targets. This represents the adapter output vector.

[0088] First, the 2D image bounding boxes output by open semantic detection are converted to 3D world coordinates through inverse perspective transformation; then, the 3D coordinates are projected onto the corresponding positions on the 2D raster map; finally, raster annotation conflicts are handled, and the coordinate projection function is as follows:

[0089]

[0090] in, Represents the grid coordinates in a two-dimensional grid map. Represents the horizontal coordinate in the world coordinate system. Represents the vertical coordinate in the world coordinate system. This represents the actual physical distance represented by each grid cell on the map along the X-axis (horizontal) direction. This represents the actual physical distance represented by each grid cell on the map along the Y-axis (vertical direction).

[0091] The dynamic map construction module includes the following steps:

[0092] First: The raw data collected by the depth measurement sensor is converted into a 3D point cloud using the inverse pinhole projection algorithm. The formula is as follows:

[0093]

[0094] in, This represents a set of 3D point clouds, which is a set of discrete 3D spatial points obtained by processing the depth image using a depth camera through an inverse pinhole projection algorithm. Let represent the three-dimensional coordinates of the i-th point, where This represents the coordinates of the i-th point along the X-axis. This represents the coordinates of the i-th point along the Y-axis. This represents the coordinates of the i-th point along the Z-axis. This represents the index of a point in the point cloud. This represents the total number of points in the point cloud.

[0095] Secondly, intelligent height compression processing is performed to retain only the effective point cloud data near the working plane of the carrier. The height compression processing is adaptive. By setting the height threshold parameter Δz, only the point cloud data that meets the height constraint is retained. Δz is a configurable parameter with a value range of 0.5 meters to 1.5 meters. This range can adapt to the point cloud filtering needs in most indoor scenarios, effectively removing interfering point clouds from the ceiling and the ground, while completely retaining the key environmental information in the workspace.

[0096] Then: Construct a binary occupancy raster map to clearly distinguish between feasible and obstacle areas in the environment;

[0097] Finally: Construct a semantically labeled raster map to provide category labels for key objects in the environment;

[0098] The semantic annotation raster is generated using the highest confidence principle. When multiple semantic labels of different categories are projected onto the same raster cell, the system automatically selects the semantic category with the highest detection confidence as the annotation result for that raster, ensuring the accuracy and uniqueness of the semantic icon annotation. The specific determination logic is as follows:

[0099]

[0100] in, This represents the semantic label ultimately assigned at raster coordinates (i, j). This is the system's output, indicating what kind of object or category the physical region represented by that raster cell is classified as. Let S represent a candidate semantic label, which is a variable that varies within the candidate set S. Indicates projection onto the current grid. The set of all candidate semantic labels on the grid is a collection of data. Since a single grid may correspond to multiple points in three-dimensional space, and each point may come from detection results at different times and from different perspectives, multiple different semantic labels may be projected onto the same grid. This represents the detection confidence of the candidate semantic label s.

[0101] The language instruction parsing module includes the following steps:

[0102] First, natural language processing techniques are used to segment the input long text instruction into a sequence of semantically independent sub-segments;

[0103] Second: Calculate the matching probability distribution vector between each sub-segment and the current visual scene;

[0104] Then: the degree of ambiguity of the instruction is determined by information entropy analysis, and the optimal instruction fragment is dynamically selected; the Shannon entropy value of the matching probability distribution is calculated, and when the entropy value exceeds the preset threshold of 1.5, it is determined to be a high ambiguity scene. At this time, the complete instruction text is output to ensure accuracy; when the entropy value is lower than the threshold, the sub-instruction fragment with the highest matching probability is output to improve execution efficiency.

[0105] Finally: If the entropy value is higher than the set threshold of 1.5, the current scene is considered semantically ambiguous, and the complete instruction is returned to ensure robustness; if the entropy value is low, the sub-instruction with the highest confidence is executed directly to improve response efficiency. The specific processing procedure is as follows: Figure 4 As shown.

[0106] The multimodal decision-making module includes the following steps:

[0107] First, a deep convolutional network is used to extract joint spatial features of the local occupancy map and the semantic map;

[0108] Secondly: the timing characteristics of instructions in the bidirectional long short-term memory network coding language are adopted;

[0109] Then: A cross-modal attention mechanism is designed to integrate visual, linguistic, and spatial features. This mechanism employs a three-level processing structure: First, the spatial features of multiple consecutive frames of map data, such as the combined features of the local occupancy map and semantic map from the past five frames, are input into a temporal modeling unit—this unit is implemented using a Long Short-Term Memory (LSTM) network. Through dynamic modeling of historical feature sequences, it outputs a long-term visual memory state containing the temporal changes in the environment. Second, the attention weight distribution between linguistic features and the long-term visual memory state is calculated—the linguistic features are low-dimensional vectors obtained by encoding natural language instructions using a Bi-LSTM network, containing key semantic information such as the action type and target object in the instructions. Finally, the weighted visual memory, depth image information, and linguistic encoding are fused to generate the final decision vector. Its core state update formula is:

[0110]

[0111] in, Indicates at time step The decision state vector encapsulates current perception, instruction, and historical state information, and is used to generate the final action instruction. This refers to a cross-modal attention mechanism, responsible for the deep fusion of features from different sources and modalities. Indicates time step The visual memory state. It is obtained by processing historical map spatial features by a temporal modeling unit, and represents the system's long-term visual memory and understanding of the environment. Indicates at time step The fused feature vector, This represents a vector concatenation operation, which joins two or more vectors along their feature dimensions to form a new vector that contains all the input information. This represents the attention calculation function. It is used to calculate the relevance weights between the query vector and the key-value pairs. This represents the decision state vector at time step t-1, which represents the historical decision state of the system at the previous time step.

[0112] Finally: The output contains the probability distribution of four basic action commands: go straight for a fixed distance, turn left at a fixed angle and proceed in parallel, turn right at a fixed angle and proceed in parallel, and stop completely.

[0113] The multimodal decision-making module integrates an intelligent obstacle avoidance mechanism, which includes: real-time analysis of the texture features of the current RGB image and calculation of the grayscale variance of the entire image; when the grayscale variance of the entire image is detected to be lower than an empirical threshold of 10, it is determined that the system has entered a low-texture dead zone region; then, the image is segmented into two symmetrical regions, and the average grayscale value of each region is calculated and compared; based on the comparison result, a direction decision command is automatically triggered to allow the system to escape a potential dead state. The formula for the direction decision command is:

[0114]

[0115] in, This represents the average grayscale value of the left half of the current RGB image. This represents the average grayscale value of the right half of the current RGB image.

[0116] The embodied intelligence system adopts a two-stage training process, as follows:

[0117] Phase 1: Train the base model using expert demonstration data through supervised learning to minimize action prediction error; the supervised learning loss function is:

[0118]

[0119] in, This represents the behavior cloning loss value. The smaller the value, the smaller the difference between the model's predicted action and the expert's action, and the better the cloning effect. This represents the total number of time steps, indicating the total number of steps in a complete task from start to finish. This represents a single time step index, ranging from 1 to T, used to traverse every execution step of the task. This represents the action vector predicted by the model at step t, which contains the specific parameters of the action. This represents the optimal action vector demonstrated by the expert at step t, which is the "standard answer" learned by the model. It can be derived from human expert operation data or high-quality preset trajectory data.

[0120] Phase Two: Fine-tuning parameters in a simulated environment using reinforcement learning to maximize overall performance. This reinforcement learning phase employs a proximal policy optimization algorithm and designs a multi-objective reward function to comprehensively consider path efficiency, trajectory matching degree, and safety. The multi-objective reward function is as follows:

[0121]

[0122] in, This represents the total reward value, which comprehensively evaluates the performance of the agent in a single navigation attempt; a higher value indicates better performance. Indicator of path efficiency, a combined metric measuring "navigation success" and "path simplicity". This represents standardized dynamic time warping, used to evaluate the similarity between the agent's actual trajectory and the "expert ideal trajectory." This indicates the number of collisions between the agent and environmental obstacles during navigation.

[0123] The embodied intelligent system is deployed on a quadrupedal mobile robot platform, which includes a high-torque servo drive system, an embedded AI computing unit Jetson AGX Orin, a multimodal sensor board, and a high-speed wireless communication module.

[0124] A method for using a multimodal fusion embodied intelligence system in complex scenarios, the method comprising the following steps:

[0125] Step 1: Simultaneously acquire environmental visual and depth image data using multiple sensors;

[0126] Step 2: Construct a dynamically updated 2D occupancy map and semantic annotation map in real time;

[0127] Step 3: Parse the input natural language instructions and output the best matching sub-instruction;

[0128] Step 4: Fuse multi-source sensory features to generate an action probability distribution;

[0129] Step 5: Execute the action command with the highest probability.

[0130] Step 6: Update the internal state of the system and return to Step 1 to form a closed-loop control.

[0131] like Figure 1 The diagram shown illustrates the overall system architecture of this invention. Deployed on a quadrupedal mobile robot platform, the system collects environmental data in real time via a multimodal sensor array. This data is then processed collaboratively by modules such as open semantic detection, dynamic map construction, language command parsing, and multimodal decision-making to ultimately generate robot action commands, enabling autonomous navigation and multi-task execution.

[0132] In the multimodal sensing module, the RGB image sensor is a global shutter camera with a resolution of at least 1920×1080 and a frame rate of at least 30Hz; the depth sensor is an active infrared structured light depth camera with a depth resolution of at least 640×480 and an effective ranging range of 0.2–5 meters; the ambient sound collector is a MEMS microphone array with a sampling rate of at least 16kHz. All sensors are synchronously triggered via hardware to ensure data time alignment accuracy within milliseconds.

[0133] Furthermore, in the open semantic detection module, the YOLO-World model is pre-trained on the COCO dataset and a custom open vocabulary dataset. The input image size is adjusted to 640×640, and the text encoder uses the CLIP text encoding branch. During inference, query text can be generated by combining dynamic categories appearing in user commands, enabling instant recognition of newly added targets. The detection results output bounding box coordinates, category labels, and confidence scores. The confidence threshold is set to 0.5, retaining only targets with a confidence score higher than this threshold.

[0134] In the dynamic map construction module, the inverse pinhole projection model converts the depth image into a 3D point cloud, with a point cloud resolution set to 0.05 meters per pixel. The 2D raster map has a grid size of 0.1 meters × 0.1 meters, and a height filtering threshold Δz of 1.0 meter to filter out ground points and excessively high obstacles. During semantic annotation, if multiple candidate labels exist for the same raster, the label with the highest confidence level is selected for assignment, and the update timestamp is recorded.

[0135] In the language instruction parsing module, long instruction segmentation uses a Transformer-based sentence segmentation model to break down complex instructions into independent sub-instructions. The CLIP model uses ViT-B / 32 as the image encoder, and the text encoder outputs a 512-dimensional vector. The Shannon entropy threshold is set to 1.5. If the calculated entropy value exceeds this threshold, it is determined to be semantically ambiguous, and the user needs to be asked to clarify or re-enter the instruction.

[0136] In the multimodal decision-making module, the CNN spatial feature extraction network adopts a ResNet-18 structure, with the input being a concatenated tensor of the local semantic map and the occupancy map. The hidden layer dimension of the Bi-LSTM language encoder is set to 256. The cross-modal attention mechanism employs a three-level processing, where the query vector is the language feature, and the key-value pairs are the concatenation of visual features and map features. The final output layer is the probability distribution of four types of actions, which is normalized using the Softmax function, and the action with the highest probability is selected as the execution instruction.

[0137] In the intelligent obstacle avoidance mechanism, the calculation of image grayscale variance uses a sliding window of 30×30 pixels, and the empirical threshold for the overall image variance is set to 500. The left and right regions are divided by the vertical line in the image, and the average grayscale value of the left and right halves is calculated separately. The turning decision prioritizes the side with the higher average grayscale value, and the turning angle is fixed at 15°.

[0138] It should be noted that key parameters in this invention, such as the height threshold Δz, confidence threshold, and Shannon entropy threshold, can be adjusted according to actual task requirements and environmental conditions, and are not specifically limited in this application.

[0139] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A multi-modal fusion embodied intelligent system in a complex scene, characterized in that, The system comprises: A multi-modal perception module, an open semantic detection module, a dynamic map construction module, a language instruction analysis module, a multi-modal decision module, an intelligent obstacle avoidance mechanism, and a cross-modal attention fusion mechanism. The multi-modal perception module is composed of a high-resolution RGB visual sensor, a depth measurement sensor, and a high-sensitivity environmental sound collector, and synchronously collects and pre-processes key perception data in a complex environment. The open semantic detection module uses a target detection neural network with an open vocabulary set to detect objects of any category in an RGB image in real time without prior category setting; by introducing a language model supporting text embedding and an image region feature matching mechanism, the module realizes the joint output of object category labels and bounding box coordinates. The dynamic map construction module uses an inverse pinhole camera projection algorithm to perform back-projection on depth images to construct a spatial point cloud, and then performs height compression and rasterization to generate a two-dimensional map structure readable by the embodied intelligent agent. The language instruction analysis module is used to process natural language instructions input by the user, extract semantic fragments through natural language processing technology, and match them with current visual scene information. The multi-modal decision module fuses map features, language features, current sensor observation information, and historical action states to output the next optimal action decision. The intelligent obstacle avoidance mechanism is an obstacle avoidance strategy based on image texture analysis built into the embodied intelligent system. The cross-modal attention fusion mechanism is used for feature fusion and adopts a three-level processing procedure.

2. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 1, characterized in that, The key perception data used in the multi-modal perception module includes RGB image data, depth image data, and sound waveform data. The RGB image data is used for target recognition and visual scene analysis, the depth image data is used for three-dimensional space structure modeling, and the sound waveform data is used for voice-assisted scene understanding or abnormal event capture.

3. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 2, characterized in that, The open semantic detection module uses a text encoder enhancement strategy, which uses the text encoder in the CLIP model as the pre-training basis and introduces a learnable weight matrix to enhance the adaptability of the open semantic detection module to open vocabularies. The text embedding calculation process is as follows: wherein, denotes the enhanced text embedding vector; denotes a pre-trained text encoder, denotes a token sequence of the input text; denotes mapping the input raw text to a feature space with semantic information, resulting in a base text embedding, denotes a learnable weight matrix, denotes the adapter output vector; First, convert the two-dimensional image bounding box output by the open semantic detection into three-dimensional world coordinate system coordinates through perspective inverse transformation; then project the three-dimensional coordinates onto the corresponding positions of the two-dimensional grid map; finally, handle the grid label conflicts, and the coordinate projection function is: wherein, represents a grid coordinate in a two-dimensional grid map, represents a lateral coordinate in a world coordinate system, represents a longitudinal coordinate in a world coordinate system, represents is the real physical distance represented by each grid of the map in the X-axis direction, represents is the real physical distance represented by each grid of the map in the Y-axis direction.

4. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 3, characterized in that, The dynamic map construction module specifically includes: First, convert the original data collected by the depth measurement sensor into a three-dimensional spatial point cloud through the inverse pinhole projection algorithm, and the formula is: ,in, Represents a set of three-dimensional point clouds. Indicates the first The three-dimensional coordinates of the points, where Indicates the first The coordinates of a point along the X-axis. Indicates the first The coordinates of the point along the Y-axis. Indicates the first The coordinates of the point along the Z-axis. This represents the index of a point in the point cloud. This represents the total number of points in the point cloud; Second, perform intelligent height compression processing to retain only the effective point cloud data near the carrier working plane; the height compression processing is adaptive, and by setting a height threshold parameter Δz, only the point cloud data that meets the height constraint is retained, where Δz is a configurable parameter and its value range is 0.5 meters to 1.5 meters; Then, construct a binary occupancy grid map to clearly distinguish between the feasible region and the obstacle region in the environment; Finally, construct a semantic annotation grid map to provide category annotations for key objects in the environment. The semantic labeling grid map is generated by using a highest confidence principle, when semantic labels of multiple different categories are projected to the same grid cell, the system automatically selects a semantic category with the highest detection confidence as the labeling result of the grid, and the specific determination logic is: wherein, denotes the semantic label finally assigned at grid coordinates , denotes a candidate semantic label, denotes the set of all candidate semantic labels projected onto the current grid , denotes the detection confidence of the candidate semantic label .

5. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 4, characterized in that, The language instruction analysis module specifically has: First, the natural language processing technology is used to divide the input long text instruction into a sequence of semantic independent sub-segments; Second, the matching probability distribution vector of each sub-segment and the current visual scene is calculated; Then, the instruction ambiguity degree is judged by information entropy analysis, and the optimal instruction segment is dynamically selected; the entropy value of the matching probability distribution is calculated, and when the entropy value exceeds the preset threshold 1.5, it is judged as a high ambiguity scene; when the entropy value is lower than the threshold, the sub-instruction segment with the highest matching probability is output; Finally, if the entropy value is higher than the set threshold 1.5, it is judged that the current scene is semantically ambiguous, and the complete instruction is returned.

6. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 5, wherein, The multi-modal decision module specifically has: First, the joint spatial features of the local occupancy map and the semantic map are extracted by a deep convolutional network; Second, the time sequence features of the language instruction are encoded by using a bidirectional long short-term memory network; Then, a cross-modal attention fusion mechanism is designed to fuse visual, language and spatial features, and a three-level processing structure is adopted: 1) first, the continuous multi-frame map spatial features are input into the time sequence modeling unit, and the long-term visual memory state containing the environmental time sequence change rule is output by dynamically modeling the historical feature sequence; 2) the attention weight distribution between the language features and the long-term visual memory state is calculated, and the language features are low-dimensional vectors obtained by encoding natural language instructions by using a bidirectional long short-term memory network Bi-LSTM; 3) the weighted visual memory, deep image information and language encoding are fused to generate the final decision vector; Its state update formula is: wherein, denotes the decision state vector at time step denotes the decision state vector at time step denotes the cross-modal attention fusion mechanism, denotes the visual memory state at time step denotes the visual memory state at time step denotes the fusion feature vector at time step denotes the fusion feature vector at time step denotes the vector concatenation operation, denotes the attention computation function, denotes the decision state vector at time step denotes the decision state vector at time step Finally, the probability distribution of the four basic action instructions is output, and the four basic action instructions include: straight fixed distance, left turn fixed angle and forward, right turn fixed angle and forward, and complete stop.

7. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 6, characterized in that, The intelligent obstacle avoidance mechanism is integrated in the multi-modal decision module, and the intelligent obstacle avoidance mechanism specifically has: the texture features of the current RGB image are analyzed in real time, and the gray variance value of the whole image is calculated; when it is detected that the gray variance value of the whole image is lower than the empirical threshold 10, it is judged that it enters the low-texture dead angle area; then the image is segmented into two symmetrical regions, and the average gray values of the regions are calculated and compared; according to the comparison result, the direction decision instruction is automatically triggered to make the system escape from the potential dead state, and the formula of the direction decision instruction is: wherein, represents the average gray value of the left half region of the current RGB image, represents the average gray value of the right half region of the current RGB image.

8. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 7, characterized in that, The embodied intelligent system adopts a two-stage training process, which is specifically as follows: Stage one: using expert demonstration data to train the basic model by supervised learning to minimize the action prediction error; the supervised learning loss function is: wherein, represents the behavior cloning loss value, represents the total number of time steps, representing the total steps from the beginning to the end of a complete task, represents a single time step index, with a value range of 1 to T, represents the model's predicted action vector at the step, represents the optimal action vector demonstrated by the expert at the step; Stage two: fine-tuning the parameters in the simulation environment by reinforcement learning to maximize the comprehensive performance index; the reinforcement learning stage adopts a proximal policy optimization algorithm, and a multi-objective reward function is designed to comprehensively consider the path efficiency, trajectory matching degree and safety, and the multi-objective reward function is: wherein, denotes the total reward value, denotes the path efficiency, denotes the normalized dynamic time warping, denotes the number of collisions of the agent with the environmental obstacles during the navigation.

9. The embodied intelligent system with multi-modal fusion in complex scenes according to claim 8, wherein, The body intelligence system is deployed on a four-legged mobile robot platform, and the platform has a high-torque servo drive system, an embedded AI computing unit Jetson AGX Orin, a multi-modal sensor board and a high-speed wireless communication module.

10. A method of using the embodied intelligent system based on multi-modal fusion in complex scenes according to claims 1-9, characterized in that, The method comprises the following steps: Step one: synchronously collecting environment visual and depth image data through multiple sensors; Step two: constructing a dynamically updated two-dimensional occupancy map and a semantic annotation map in real time; Step three: analyzing the input natural language instruction and outputting the optimal matching sub-instruction; Step four: fusing multi-source perception features to generate an action probability distribution; Step five: executing the action instruction with the highest probability; Step six: updating the internal state of the system and returning to step one to form a closed-loop control.

Images