Robust multi-modal perception and cross-modal fusion system for robots

Abstract

The application discloses a kind of robot multimodal robust perception and cross-modal fusion system, belong to robot multimodal perception and environmental fusion technical field.The system includes multimodal sensor module, uncertainty evaluation module, dynamic modal gating module, multimodal fusion module and factor graph optimization module.Uncertainty evaluation module calculates the uncertainty parameter of each perception mode, and dynamic modal gating module judges mode validity accordingly;Multimodal fusion module generates fusion scene representation using mask training based Transformer fusion network and cross-modal attention mechanism;Factor graph optimization module combines environmental observation information and robot ontology motion information to optimize robot state.The system can maintain environmental perception continuity when part of the perception mode is degraded or fails, and improve the perception stability in complex environment.

Description

Technical Field

[0001] This invention relates to the field of robot environmental perception technology, and in particular to a robot multimodal robust perception and cross-modal fusion system. Background Technology

[0002] With the development of robotics technology, robots are increasingly being applied to autonomous perception and task execution in complex environments. In complex environments, robots typically need to utilize vision, lidar, inertial measurement units, and other environmental sensors to acquire information about the external environment and their own motion state in order to achieve environmental understanding and motion control.

[0003] In existing technologies, multimodal fusion robot systems typically collect environmental information through multiple sensors and perform preprocessing, spatiotemporal registration, and fusion analysis on the data from different sensors. For example, Chinese invention patent application CN121808550A (application number 202511941311.2, publication date April 7, 2026) discloses a robot environment recognition system based on multimodal perception. It uses visual, lidar, and inertial measurement data for cross-modal feature alignment and adaptive weight fusion, and dynamically adjusts the initial weights of different modalities according to the perception quality. When the inertial data is abnormal, the inertial modal weights are temporarily set to zero.

[0004] In addition, Chinese invention patent application CN120839843A (application number 202510934805.1, publication date October 28, 2025) discloses a multimodal fusion real-time environmental monitoring visual robot system. It uses multiple sensors such as binocular stereo camera, lidar, gas sensor and microphone array to collect environmental data. It achieves multi-source information fusion through heterogeneous data preprocessing, spatiotemporal alignment fusion and environmental semantic understanding, and uses cross-modal attention mechanism to dynamically allocate the weights of different modal features.

[0005] However, in complex environments such as earthquake rescue, disaster sites, and high-smoke, high-humidity environments where perception quality changes significantly, the reliability of data from different sensors dynamically changes after being affected by environmental factors. While existing technologies can dynamically adjust modal weights through perception quality indicators or attention mechanisms, and can reduce the participation of individual modalities or switch modal combinations when they are abnormal, they have not yet established a unified failure judgment and gating mechanism based on the uncertainty of prediction results for each modality. Furthermore, the adaptability of fusion networks to combinations lacking different modalities, and the tight coupling between environmental observation and robot motion information, still have room for improvement.

[0006] Therefore, how to dynamically adjust the fusion process based on the reliable state of different perception modalities in the above-mentioned complex environment, and maintain the robot's environmental perception capability in the event of partial modal failure, has become an urgent technical problem to be solved. Summary of the Invention

[0007] The main objective of this invention is to provide a robust multimodal perception and cross-modal fusion system for robots, which aims to dynamically adjust the fusion process based on the reliability status of different perception modes and maintain the robot's environmental perception capability even when some modes fail.

[0008] To achieve the above objectives, this invention proposes a robot multimodal robust perception and cross-modal fusion system, comprising: A multimodal sensor module is used to collect information about the robot's surrounding environment and the robot's own motion. It also includes: an uncertainty assessment module, connected to the multimodal sensor module, used to establish uncertainty estimation models for different sensing modes and output uncertainty parameters corresponding to each mode; The dynamic modal gating module, connected to the uncertainty assessment module, is used to determine the modal validity based on the comparison results of the uncertainty parameters of each modality with the corresponding threshold, and to control modal data that do not meet the reliability conditions from participating in subsequent fusion processing. A multimodal fusion module, connected to the dynamic modal gating module, is used to convert effective modal data into feature tokens and input them into a multimodal Transformer fusion network trained based on a mask. The network generates a fused scene representation through modal encoding and a cross-modal attention mechanism, wherein the cross-modal attention mechanism adjusts the attention weights based on modal uncertainty. The factor graph optimization module, connected to the multimodal fusion module, is used to construct a factor graph optimization model by using the fused scene representation as an environmental observation factor and the robot's motion information as a motion prior factor, so as to jointly optimize the robot's state information and environmental information.

[0009] Preferably, the multimodal sensor module includes a visual sensor group, a geometric information acquisition group, a life environment information acquisition group, and a body motion information acquisition group, wherein the visual sensor group includes a depth camera, an anti-fog visible light camera, an infrared thermal imaging camera, and a polarized light camera.

[0010] Preferably, the uncertainty assessment module uses a Bayesian neural network or the Monte Carlo Dropout method to obtain cognitive uncertainty and accidental uncertainty, and according to... Calculate the total modal uncertainty, where, This represents the total modal uncertainty. This indicates cognitive uncertainty. This represents random uncertainty, with α and β being weighting coefficients.

[0011] Preferably, the dynamic modal gating module generates a modal failure flag based on the comparison result between the total modal uncertainty and the preset modal uncertainty threshold, and prevents modal data with failure flags from participating in the forward computation of the fusion network.

[0012] Preferably, the multimodal fusion module includes a Transformer encoder and a Transformer decoder. The Transformer encoder receives feature tokens with modality type encoding and obtains fused features through a multi-head self-attention mechanism.

[0013] Preferably, the multimodal fusion module employs a random masking strategy during the training phase, randomly discarding some modal tokens with a preset probability, and uses data augmentation methods combining multiple modal missing elements for training.

[0014] Preferably, the cross-modal attention mechanism in the multimodal fusion module uses modal features with lower uncertainty parameters relative to other effective modal features as the basis for enhancement, and uses the modal uncertainty parameters to modulate the attention weights to enhance the expression of effective modal features with relatively high uncertainty parameters but not exceeding the corresponding threshold.

[0015] Preferably, the factor graph optimization module uses the fused scene representation output by the multimodal fusion module as the environmental observation factor, the robot motion state obtained by the inertial measurement unit and the joint encoder as the motion prior factor, and jointly optimizes the robot state variables and environmental landmarks within a sliding window.

[0016] Preferably, the factor graph optimization module uses the iSAM2 incremental smoothing and mapping algorithm for real-time state estimation.

[0017] Compared with the prior art described above, the present invention has the following advantages: Compared to the existing technologies, this invention reduces the impact of abnormal perception data on environmental understanding results by setting up an uncertainty assessment module and a dynamic modal gating module to determine the modal reliability state based on the uncertainty of different perception modalities and control the way unreliable modal data participates in fusion. By adopting a multimodal Transformer fusion network trained based on masks, the fusion model can adapt to the fusion scene representation under different modal combinations. The cross-modal attention mechanism combined with modal reliability information enhances the complementary effect between effective modalities. The factor graph optimization module fuses environmental observation information and robot body motion information to improve the continuity of robot state estimation under perception degradation. Attached Figure Description

[0018] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein: Figure 1This is a schematic diagram of the overall workflow of the present invention; Figure 2 This is a schematic diagram of the multimodal Transformer fusion network structure. Detailed Implementation

[0019] Reference Figure 1 and Figure 2 The specific embodiments described below are based on the technical solution of this application. These embodiments are used to illustrate the structural composition, component mating relationships, and working process of this application, enabling those skilled in the art to understand and implement the technical solution of this application, and are not intended to impose additional limitations on the scope of protection of this application.

[0020] Example 1 This embodiment illustrates a specific implementation of a robot multimodal robust perception and cross-modal fusion system. This embodiment is only used to explain the technical solution of this application and is not intended to limit the scope of protection of this application. Those skilled in the art can adjust the system structure, data processing method, and implementation process according to actual application needs without departing from the technical concept of this application.

[0021] In this embodiment, the robot is deployed in a complex environment with smoke, low light, high humidity, and numerous obstacles. The robot collects information about the external environment and its own motion through a multimodal sensor module, and achieves stable environmental perception through uncertainty assessment, dynamic modal gating, multimodal Transformer fusion, and factor graph optimization.

[0022] The multimodal sensor module includes a vision sensor group, a geometric information acquisition group, a life environment information acquisition group, and a body motion information acquisition group.

[0023] The visual sensor group includes a depth camera, an anti-fog visible light camera, an infrared thermal imaging camera, and a polarized light camera. The depth camera is used to acquire spatial depth information of the environment, the anti-fog visible light camera is used to acquire texture and color information in normal scenes, the infrared thermal imaging camera is used to acquire heat source information in smoke and dust environments, and the polarized light camera is used to enhance feature information in low-texture scenes.

[0024] The geometric information acquisition group includes a LiDAR system for acquiring three-dimensional geometric information about the robot's surrounding environment. The life environment information acquisition group includes a millimeter-wave radar, a microphone array, and temperature, humidity, and gas sensors. The millimeter-wave radar detects target motion information in the environment, the microphone array acquires sound information, and the temperature, humidity, and gas sensors obtain environmental state parameters. The body motion information acquisition group includes an inertial measurement unit, tactile / force sensors, and joint encoders for acquiring the robot's own motion and interaction states.

[0025] Each sensor collects data synchronously during operation. The collected data from different modalities are then timestamped to ensure that the data acquired by different sensors correspond to the same time base, thus forming multimodal input data.

[0026] The uncertainty assessment module receives data output from the multimodal sensor module and establishes uncertainty estimation models for different sensing modes.

[0027] In one implementation, the uncertainty assessment module employs a Bayesian neural network or the Monte Carlo Dropout method to calculate the uncertainty of prediction results for different modalities. Cognitive uncertainty represents the degree of uncertainty inherent in the model itself, while accidental uncertainty represents the degree of uncertainty arising from environmental factors affecting the sensor data.

[0028] For each sensing mode, the corresponding uncertainty parameters are calculated, and based on... The total uncertainty of this mode is obtained.

[0029] in, This represents the total modal uncertainty. This indicates cognitive uncertainty. This represents random uncertainty, where α and β are weighting coefficients. α and β can be determined through validation using conventional parameters on training or calibration data.

[0030] The above methods can convert the output results of different sensors into information with reliability evaluation, providing a basis for subsequent mode selection.

[0031] The dynamic modal gating module judges the validity of each modal data based on the total modal uncertainty output by the uncertainty assessment module.

[0032] Specifically, corresponding uncertainty thresholds are set for different modes, and the uncertainty thresholds for each mode are determined through experimental calibration. During experimental calibration, a threshold that can distinguish between valid and failed states can be selected based on the uncertainty output of the corresponding mode under normal and degraded operating conditions. When the uncertainty parameter of a certain mode exceeds the corresponding threshold, the dynamic mode gating module determines that the current state of the mode is unreliable and generates a corresponding mode failure flag; when the mode uncertainty does not exceed the corresponding threshold, the mode remains valid.

[0033] For modal data that is judged to be invalid, the dynamic modal gating module controls it to not enter the forward computation process of the subsequent fusion network, thereby reducing the impact of abnormal data on the fusion result.

[0034] For example, in a smoky environment, the depth estimation error of a visual sensor may increase due to smoke obstruction, thus raising the uncertainty parameter of the visual modality. When this parameter exceeds a preset threshold, the dynamic modality gating module generates a visual modality failure flag and prevents the visual modality from entering the subsequent fusion network, allowing other valid modalities to continue participating in environmental understanding. When the uncertainty of the visual modality increases but does not exceed the corresponding threshold, the visual modality remains valid, and its feature influence is reduced by a cross-modal attention mechanism.

[0035] The multimodal fusion module receives the valid modal data output by the dynamic modal gating module and performs unified feature representation on different modal data.

[0036] Specifically, each valid modality data is first converted into a corresponding feature token, and a modality type code is attached. After input embedding processing, the feature tokens of different modalities form a token sequence with a unified dimension.

[0037] The multimodal fusion module employs a mask-based trained multimodal Transformer fusion network. For example... Figure 2 As shown, the network includes a Transformer encoder and a Transformer decoder.

[0038] During the training phase, a random masking strategy is employed to randomly discard some modality tokens with a preset probability, enabling the network to learn environmental representation capabilities under different modality combinations. The preset probability can be determined through conventional verification based on the coverage of different missing modality combinations in the training data. Through this training method, even with missing sensor data, the fusion network can still generate a fused scene representation based on the remaining valid modality information.

[0039] During the runtime phase, the Transformer encoder performs feature extraction and correlation analysis on the input multimodal token sequence, and the Transformer decoder outputs a unified fused scene representation based on the encoding results. This fused scene representation includes environmental geometric information, semantic information, and reliability information corresponding to different modalities.

[0040] In the multimodal fusion process, a cross-modal attention mechanism is introduced. This mechanism calculates attention weights based on the correlation between different modalities and adjusts the attention weights using modal uncertainty parameters. Specifically, effective modal features with lower uncertainty parameters relative to other effective modalities are used as queries, and all effective modal features are used as keys and values. The attention score is modulated using uncertainty parameters.

[0041] During the modulation process, the attention weights of each effective mode are adjusted according to its uncertainty parameter: modes with relatively low uncertainty parameters receive higher weights, while modes with relatively high uncertainty parameters but not exceeding the corresponding threshold receive lower weights. This allows effective modes with higher reliability to enhance the feature representation of other modes that are still in an effective state, achieving information complementarity between different perceptual modes.

[0042] The factor graph optimization module is connected to the multimodal fusion module to further fuse robot motion information and environmental perception information.

[0043] Specifically, the fused scene representation output by the multimodal fusion module is used as an environmental observation factor, and the robot motion state obtained by the inertial measurement unit and joint encoder is used as a motion prior factor to construct a factor graph optimization model.

[0044] The factor graph optimization module performs joint optimization of robot state variables and external environmental landmarks within a sliding window, so that the robot state estimation results are simultaneously constrained by environmental observation information and its own motion information.

[0045] When the data quality of some environmental perception modes deteriorates due to the aforementioned complex environmental factors, the factor graph optimization module can utilize the robot's motion information to maintain the continuity of state estimation, enabling the robot to maintain stable operation.

[0046] In this embodiment, a robust robot perception process for the aforementioned complex environment is achieved through data interaction between the multimodal sensor module, the uncertainty assessment module, the dynamic modal gating module, the multimodal Transformer fusion module, and the factor graph optimization module.

[0047] The specific work process is as follows: First, the multimodal sensor module simultaneously collects visual, geometric, environmental, and robot motion data.

[0048] Then, the uncertainty assessment module performs reliability analysis on the data of each mode to obtain the uncertainty parameters corresponding to each mode.

[0049] Subsequently, the dynamic modal gating module filters valid modal data based on the uncertainty judgment results and inputs the valid data into the multimodal fusion module.

[0050] The multimodal fusion module uses a mask-trained Transformer network to fuse effective modalities and enhances information between different modalities through a cross-modal attention mechanism.

[0051] Finally, the factor graph optimization module combines the environmental perception results and the robot's own motion information to jointly optimize the robot's state and environmental information, and outputs stable environmental perception results.

[0052] Through the above implementation methods, the robot can still understand the environment by utilizing the remaining effective modes and its own motion information even when some sensors are affected by the environment, the data quality deteriorates, or some modes are unavailable, thereby improving the stability and adaptability of the robot's perception system in complex environments.

[0053] Example 2 This embodiment illustrates a multimodal fusion enhancement scheme based on Embodiment 1. This embodiment primarily addresses the issue of varying reliability of some sensing modes under different environmental conditions, and further optimizes the multimodal fusion process.

[0054] In this embodiment, the robot still collects environmental and motion information through a multimodal sensor module. Each sensor in the multimodal sensor module outputs a corresponding data stream, which is then processed for time synchronization and input into the uncertainty assessment module.

[0055] For different types of sensor data, the uncertainty assessment module establishes corresponding uncertainty evaluation methods based on sensor characteristics.

[0056] For the visual mode, the uncertainty of the visual perception result is determined by analyzing the quality of image information; for the lidar mode, the uncertainty of the geometric information is determined based on the quality of point cloud data; for the millimeter-wave radar mode, the uncertainty of the target information is determined based on the target detection result; for the thermal imaging mode, the uncertainty of the heat source information is determined based on the heat source detection result; and for the force mode, the uncertainty of the contact information is determined based on the change of force signal.

[0057] After the uncertainty assessment module outputs the uncertainty parameters corresponding to each mode, the dynamic mode gating module determines the current state of each mode based on the preset threshold.

[0058] When a modality is affected by the environment, resulting in reduced data reliability, but its uncertainty does not exceed the corresponding failure threshold, the modality remains valid, and its participation in the fusion process is reduced by the cross-modal attention mechanism. When its uncertainty exceeds the corresponding threshold, the dynamic modality gating module generates a failure flag and prevents the modality from participating in the forward computation of the fusion network. When the data of a modality recovers and its uncertainty falls back to within the corresponding threshold, the modality is restored to participate in the fusion.

[0059] Through the above methods, the system can dynamically adjust the data usage status of different modalities according to environmental changes.

[0060] In this embodiment, the multimodal fusion module adopts a Transformer fusion network with an encoder and decoder structure.

[0061] The data corresponding to the valid modalities are first converted into feature tokens. Each feature token includes feature information of the corresponding modality and modality type information. The feature tokens of different modalities are linearly mapped to form an input sequence of uniform dimension.

[0062] The Transformer encoder performs feature association calculations on the input sequence. It analyzes the relationships between features from different modalities using a self-attention mechanism and generates fused intermediate features.

[0063] Subsequently, the Transformer decoder generates a unified fused scene representation based on the encoder output.

[0064] During training, the multimodal fusion module employs modality missing data augmentation. Different combinations of missing modalities are randomly included in the training samples, enabling the fusion network to learn environmental understanding under different sensor combinations.

[0065] For example, when visual modal data is unavailable, the fusion network can perform environmental structure inference based on lidar, millimeter-wave radar, and body motion information; when lidar data is interfered with, the fusion network can use visual information, thermal imaging information, and other effective modalities to perform environmental information compensation.

[0066] During the cross-modal fusion process, the system further utilizes the cross-modal attention mechanism to achieve information complementarity between effective modalities.

[0067] Specifically, the cross-modal attention mechanism calculates the degree of correlation between different modal features and adjusts the attention weights based on the modal uncertainty parameters output by the uncertainty assessment module.

[0068] When the uncertainty of a certain effective mode is lower than that of other effective modes, the feature weight of that mode is increased, enabling it to provide more effective information to other modes; when the uncertainty of a certain effective mode is relatively high but does not exceed the corresponding threshold, the system reduces its feature influence to avoid abnormal information from affecting the overall fusion result.

[0069] The factor graph optimization module performs joint optimization based on the fused scene representation output by the multimodal fusion module and the robot's own motion information.

[0070] During the optimization process, environmental observation information is used as an input factor graph model for environmental constraints, and the motion state output by the inertial measurement unit and joint encoder is used as an input factor graph model for motion constraints.

[0071] When the robot is in a smoke, high temperature, or obstructed environment, and some external sensors cannot continuously provide stable data, the factor graph optimization module maintains the state estimation based on the remaining effective environmental observation information and the robot's own motion constraints.

[0072] In this embodiment, dynamic modal gating, modal missing training, and body motion constraint fusion enable the robot perception system to adapt to different degrees of environmental changes.

[0073] Example 3 This embodiment illustrates a specific application method based on a multimodal fusion system.

[0074] In this embodiment, the robot enters the interior of a building with a dense smoky environment to perform an environmental detection task.

[0075] After the robot starts, the multimodal sensor module synchronously collects environmental data. Among them, the vision sensor acquires environmental image information, the infrared thermal imaging camera acquires heat source distribution information, the lidar acquires spatial geometric information, the millimeter-wave radar acquires target motion information, and the inertial measurement unit acquires the robot's motion state.

[0076] After the collected multimodal data undergoes time synchronization processing, it enters the uncertainty assessment module.

[0077] Due to the influence of dense smoke, the image quality acquired by the visual sensor is reduced, and the uncertainty parameters of the visual modality increase. When the visual modality uncertainty exceeds a preset threshold, the dynamic modality gating module generates a visual modality failure flag and controls the visual modality data from entering the fusion network.

[0078] Meanwhile, the infrared thermal imaging mode is less affected by smoke and dust, and its uncertainty parameters remain within the threshold range, so it continues to participate in the fusion process.

[0079] When the lidar generates some abnormal point clouds due to smoke and dust interference, the dynamic mode gating module filters the lidar features based on the lidar data reliability judgment results, and only retains reliable information to participate in the fusion.

[0080] Millimeter-wave radar, due to its ability to penetrate smoke and dust, can continuously provide target-related information, thus maintaining its effectiveness.

[0081] The effective modal data, after dynamic modal gating, enters the multimodal Transformer fusion network.

[0082] The fusion network generates a fused scene representation based on the current available modal combinations. Since the data features for different modalities are learned during the training phase, the environmental structure can still be inferred from the remaining modal information even if some sensor data is unavailable.

[0083] The module optimizes the input factor graph for fusion scene representation.

[0084] The factor graph optimization module utilizes motion information provided by the inertial measurement unit and effective environmental observation information to optimize the robot's state. When the robot traverses complex areas that degrade the quality of environmental observations, the system mitigates the impact of abnormal observation information and maintains the continuity of state estimation using the robot's own motion information.

[0085] Finally, the system outputs the robot's current environmental perception results, including environmental spatial information, target location information, and robot status information.

[0086] Through this embodiment, the system can automatically adjust the fusion strategy when external perception conditions change, enabling the robot to maintain continuous environmental perception capabilities.

[0087] The above embodiments illustrate specific implementation methods of the technical solution of this application. Those skilled in the art will understand that, without altering the technical concept of this application, appropriate adjustments can be made to the sensor combination, data processing flow, and fusion method according to the actual application environment.

Claims

1. A robot multimodal robust perception and cross-modal fusion system, comprising: A multimodal sensor module is used to collect information about the robot's surrounding environment and the robot's own motion. The feature is that it further includes: an uncertainty assessment module, connected to the multimodal sensor module, used to establish uncertainty estimation models for different sensing modes and output uncertainty parameters corresponding to each mode; The dynamic modal gating module, connected to the uncertainty assessment module, is used to determine the modal validity based on the comparison results of the uncertainty parameters of each modality with the corresponding threshold, and to control modal data that do not meet the reliability conditions from participating in subsequent fusion processing. A multimodal fusion module, connected to the dynamic modal gating module, is used to convert effective modal data into feature tokens and input them into a multimodal Transformer fusion network trained based on a mask. The network generates a fused scene representation through modal encoding and a cross-modal attention mechanism, wherein the cross-modal attention mechanism adjusts the attention weights based on modal uncertainty. The factor graph optimization module, connected to the multimodal fusion module, is used to construct a factor graph optimization model by using the fused scene representation as an environmental observation factor and the robot's motion information as a motion prior factor, so as to jointly optimize the robot's state information and environmental information.

2. The robot multimodal robust perception and cross-modal fusion system according to claim 1, characterized in that, The multimodal sensor module includes a vision sensor group, a geometric information acquisition group, a life environment information acquisition group, and a body motion information acquisition group. The vision sensor group includes a depth camera, an anti-fog visible light camera, an infrared thermal imaging camera, and a polarized light camera.

3. The robot multimodal robust perception and cross-modal fusion system according to claim 1, characterized in that, The uncertainty assessment module uses a Bayesian neural network or Monte Carlo Dropout method to obtain cognitive uncertainty and accidental uncertainty, and based on... Calculate the total modal uncertainty, where, This represents the total modal uncertainty. This indicates cognitive uncertainty. This represents random uncertainty, with α and β being weighting coefficients.

4. The robot multimodal robust perception and cross-modal fusion system according to claim 1, characterized in that, The dynamic modal gating module generates a modal failure flag based on the comparison between the total modal uncertainty and the preset modal uncertainty threshold, and prevents modal data with failure flags from participating in the forward computation of the fusion network.

5. The robot multimodal robust perception and cross-modal fusion system according to claim 1, characterized in that, The multimodal fusion module includes a Transformer encoder and a Transformer decoder. The Transformer encoder receives feature tokens with modality type encoding and obtains fused features through a multi-head self-attention mechanism.

6. The robot multimodal robust perception and cross-modal fusion system according to claim 5, characterized in that, The multimodal fusion module employs a random masking strategy during the training phase, randomly discarding some modal tokens with a preset probability, and uses data augmentation methods with multiple modality missing combinations for training.

7. The robot multimodal robust perception and cross-modal fusion system according to claim 5, characterized in that, The cross-modal attention mechanism in the multimodal fusion module uses modal features with lower uncertainty parameters relative to other effective modal features as the basis for enhancement, and uses the modal uncertainty parameters to modulate the attention weights to enhance the expression of effective modal features with relatively high uncertainty parameters but not exceeding the corresponding threshold.

8. The robot multimodal robust perception and cross-modal fusion system according to claim 1, characterized in that, The factor graph optimization module uses the fused scene representation output by the multimodal fusion module as the environmental observation factor, and the robot motion state obtained by the inertial measurement unit and joint encoder as the motion prior factor, and jointly optimizes the robot state variables and environmental landmarks within a sliding window.

9. The robot multimodal robust perception and cross-modal fusion system according to claim 8, characterized in that, The factor graph optimization module uses the iSAM2 incremental smoothing and mapping algorithm for real-time state estimation.

Images