A robot complex scene perception system and method based on environment adaptive multi-modal fusion

Abstract

The application discloses a robot complex scene perception system and method based on environment adaptive multi-modal fusion, relates to the technical field of robot perception, and comprises four modules of a flexible sensing matrix, environment feature self-calibration, cross-modal causal reasoning and scene prediction.The flexible sensing matrix collects environment parameters and multi-modal data of vision, inertia and touch.The self-calibration module corrects data errors through environment-error mapping and gradient descent algorithm.The causal reasoning module realizes data fusion and failure completion based on a Bayesian network.The scene prediction module outputs a prediction result through a pre-trained LSTM model and feeds back an optimized collection strategy.The application breaks through the bottleneck of traditional perception technology, improves the environment adaptability, system fault tolerance and decision initiative of complex unstructured scenes, is suitable for scenes such as disaster rescue and outdoor inspection, and helps robots to work autonomously and safely.

Description

Technical Field

[0001] This invention relates to the field of robot perception technology, and more specifically to a robot complex scene perception system and method based on environment adaptive multimodal fusion. Background Technology

[0002] With the widespread application of robotics in complex and unstructured scenarios such as disaster relief, outdoor inspection, and planetary exploration, environmental perception, as the core foundation for autonomous decision-making and safe operation of robots, faces increasingly stringent performance requirements. Complex scenarios present extreme conditions such as drastic changes in lighting, large fluctuations in temperature and humidity, rugged and varied terrain, and interference from dust / smoke, making it difficult for traditional robot perception technologies to meet practical application needs.

[0003] Existing visual sensors rely on stable lighting environments. In scenes with strong direct sunlight or dense smoke and dust, the image signal-to-noise ratio decreases significantly, and feature extraction is prone to failure. Most mainstream tactile sensors use a single sensitive material, capable of detecting only pressure or temperature, failing to simultaneously acquire crucial multi-dimensional information on pressure, temperature, and humidity in complex scenarios. Furthermore, the sensitive elements of tactile sensors are susceptible to environmental temperature and humidity, with detection accuracy dropping drastically in high-temperature and high-humidity environments. Many sensors employ fixed-weight fusion or statistical correlation algorithms, failing to establish a causal relationship between perceived data and environmental conditions. In dynamic scenarios, fixed weights cannot respond to real-time changes in data features, resulting in low fusion accuracy. Statistical correlation algorithms rely on the assumption of data stationarity; when environmental parameters change abruptly, the data distribution deviates from the preset model, significantly reducing fusion stability and even leading to misjudgments. In complex scenarios, sensors are prone to failure due to collisions, dust accumulation, extreme temperature and humidity, etc. In such cases, the system cannot use remaining sensor information to fill in the missing data, causing a break in the perception link, which in turn leads to robot instability or forced task termination, severely impacting operational reliability.

[0004] The aforementioned technical deficiencies result in significant shortcomings in the adaptability, reliability, and decision-making efficiency of traditional robot perception systems in complex and unstructured scenarios. These deficiencies make it difficult to meet the core requirements for autonomous robot operation in scenarios such as disaster relief and outdoor inspection. There is an urgent need to develop a multimodal fusion perception system with environmental adaptability, high fault tolerance, and dynamic prediction capabilities. Summary of the Invention

[0005] In view of this, the present invention provides a robot complex scene perception system and method based on environment adaptive multimodal fusion to solve the problems existing in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A robot complex scene perception system based on environment adaptive multimodal fusion includes: a flexible sensing matrix module, an environmental feature self-calibration module, a cross-modal causal inference module, and a scene prediction module. The flexible sensing matrix module collects multi-source sensing data; the environmental feature self-calibration module dynamically corrects the data errors of the multi-source sensing data through environmental parameters to obtain calibration data; the cross-modal causal inference module fuses and completes the calibration data through a causal model to generate a cross-modal fusion feature vector; the scene prediction module outputs scene prediction probabilities based on a pre-trained LSTM model combined with the cross-modal fusion feature vector, selects the scene with the highest probability as the prediction result, and applies it to the flexible sensing matrix module to optimize the next round of data acquisition strategy.

[0007] Preferably, the multi-sensory data collected by the flexible sensing matrix module includes illumination, temperature, humidity, ground stiffness data, as well as visual modal data, inertial modal data, and tactile modal data.

[0008] Preferably, the environmental feature self-calibration module specifically includes: acquiring a light intensity sequence. Temperature sequence , Humidity sequence: Ground stiffness sequence: The acquired data is preprocessed by using the 3σ criterion to remove outliers, linear interpolation to fill in missing data, and then smoothing the data using a moving average filter to obtain the preprocessed data sequence. , , , Based on the environment-error mapping database, the gradient descent algorithm is used to calculate the compensation coefficient and adjust the signal amplification factor and filtering parameters of the sensing unit.

[0009] Preferably, the calculation of the compensation coefficient specifically includes: defining a loss function. ,in These are compensation coefficients, corresponding to signal amplification factor correction, low-pass filter cutoff frequency correction, and high-pass filter cutoff frequency correction, respectively. Based on the current Calculated prediction error For systematic error; Take the partial derivative to obtain the gradient. ,in Iterative updates , Let be the learning rate, and let be the iteration termination condition. Or the number of iterations reaches the preset number; signal amplification factor is adjusted: ,in This is the initial magnification, 100 ≤ ≤2000; Filter parameter adjustment: Low-pass filter cutoff frequency High-pass filter cutoff frequency ,in ∈[10Hz,100Hz], ∈[0.1Hz, 1Hz], and must satisfy < .

[0010] Preferably, the cross-modal causal reasoning module specifically includes: Causal structure learning subunits are used to construct a Bayesian network. The Peter-Clark algorithm is used to initialize the causal network topology, and redundant edges are removed through conditional independence checks to obtain the initial topology. Based on Bayesian information criteria Optimize, ,in, L The likelihood function value, k To determine the number of model parameters, the topology that minimizes the Bayesian information criterion is selected as the final causal structure G. The expectation-maximization algorithm is used to estimate the conditional probability table of the Bayesian network. Let the node variables in the network be... , ,..., conditional probability The parameters follow a Dirichlet distribution. The optimal estimate is obtained iteratively using the expectation-maximization algorithm. The convergence condition is that the difference in likelihood functions between two iterations ≤ The converged Bayesian network is used as a causal model. The dynamic fusion subunit employs a dynamic time warping algorithm to align the time axes of different modalities, finding the optimal alignment path through dynamic programming to minimize the total path distance; it also utilizes node influence weights based on causal networks. The formula for fusing aligned modal data is as follows: ; ; Where F is the fused data feature vector; a fusion error feedback factor is introduced. , For standard fusion samples, when β is greater than a preset threshold, the node weights are recalculated. Iterative optimization of the fusion results; The fault-tolerant completion subunit employs an anomaly detection algorithm to calculate the posterior probability of each modality's data. ,like < , If the posterior threshold is used, the modal sensor is determined to be faulty; based on the causal structure G, the failure node is extracted. parent node and child nodes ; Construct a completion model ,in, For the completed node data; calculate the completion error. , The imputation error is the average of historical normal data. If the imputation error exceeds the error threshold, then L1 regularization is used to optimize the imputation model. λ is the penalty coefficient.

[0011] Preferably, the scene prediction module specifically includes: training the LSTM model using a historical dataset, the historical dataset including historical modal data and corresponding scene labels; performing principal component analysis to reduce the dimensionality of the completed modal data, obtaining a dimensionality-reduced feature vector, extracting time-domain features and frequency-domain features, and constructing a joint feature vector; inputting the joint feature vector into the trained LSTM model, outputting the scene prediction probability, selecting the scene with the highest probability as the prediction result, and applying it to the flexible sensing matrix module to optimize the next round of data acquisition strategy.

[0012] A robot complex scene perception method based on environment adaptive multimodal fusion includes: collecting environmental parameter data and core modal data, wherein the environmental parameter data includes light intensity, temperature, humidity, and ground stiffness data, and the core modal data includes visual modal data, inertial modal data, and tactile modal data; preprocessing the environmental parameter data, calculating compensation coefficients based on an environment-error mapping database using a gradient descent algorithm, and dynamically adjusting the signal amplification factor and filtering parameters of the sensing unit; constructing a causal model based on a Bayesian network, performing causal structure learning and dynamic fusion on the core modal data, and generating a cross-modal fusion feature vector; simultaneously detecting sensor failure states and completing missing data through causal chain back reasoning; performing dimensionality reduction and spatiotemporal feature extraction on the cross-modal fusion feature vector, constructing a joint feature vector, inputting it into a pre-trained LSTM model to output scene prediction probabilities, and selecting the scene with the highest probability as the prediction result; and adjusting the sampling frequency and key perception areas of the sensing unit based on the prediction result to optimize the data acquisition strategy for the next round.

[0013] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a robot complex scene perception system and method based on environment adaptive multimodal fusion. Through the closed-loop collaborative design of flexible sensing matrix, environment adaptive self-calibration, cross-modal causal reasoning and scene prediction, it has many significant advantages over the prior art: At the technical level, it effectively breaks through the environmental adaptability bottleneck of traditional robot perception systems, maintaining stable perception performance in extreme unstructured scenarios such as drastic changes in lighting, large fluctuations in temperature and humidity, and complex and varied terrain. It solves the problems of weak anti-interference capability of single sensors and rigidity of multimodal data fusion. Through the causal reasoning mechanism, it realizes the completion of perception data in the case of sensor failure, which greatly improves the fault tolerance and operational reliability of the system and avoids task interruption due to local sensor failure. The scene prediction function enables the robot to shift from passive data collection to active perception, respond to dynamic changes in the environment in advance, reduce decision lag, and optimize movement and operation strategies.

[0014] From an economic perspective, the modular and hot-swappable flexible sensor matrix design can be adapted to various types of robot platforms, reducing the R&D and adaptation costs of the sensing system; the environmental adaptive self-calibration mechanism reduces the manual calibration process, lowering maintenance difficulty and operation and maintenance investment; the independent innovation of core sensing technologies and fusion algorithms can be extended to multiple fields such as industrial inspection and medical monitoring, forming diversified industrial value and driving the development of upstream and downstream related industries.

[0015] At the societal level, this will significantly enhance robots' autonomous operation capabilities in high-risk and complex scenarios such as disaster relief, outdoor inspection, and planetary exploration, reducing the risk of personnel exposure to dangerous environments and providing stronger protection for life safety; it will also promote the localization of high-end robot sensing equipment, break foreign technological monopolies, enhance my country's technological self-reliance and controllability in the fields of emergency equipment and intelligent robots, and improve the response and handling efficiency of the public emergency system. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure provided by the present invention; Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention discloses a robot complex scene perception system based on environment adaptive multimodal fusion, such as... Figure 1 As shown, the system includes: a flexible sensing matrix module, an environmental feature self-calibration module, a cross-modal causal inference module, and a scene prediction module. The flexible sensing matrix module collects multi-source sensing data; the environmental feature self-calibration module dynamically corrects the data errors of the multi-source sensing data through environmental parameters to obtain calibration data; the cross-modal causal inference module fuses and completes the calibration data through a causal model to generate a cross-modal fusion feature vector; the scene prediction module outputs scene prediction probabilities based on a pre-trained LSTM model combined with the cross-modal fusion feature vector, selects the scene with the highest probability as the prediction result, and applies it to the flexible sensing matrix module to optimize the next round of data acquisition strategy.

[0020] Flexible sensing matrix module: Hydrogel composite sensing units (8 electrodes + 36 measurement paths) are arranged at the foot / actuator, and transparent tactile and visual integrated sensors (double-ring key line markings) are arranged in key parts of the body to simultaneously collect tactile, visual, and proprioceptive data (proprioceptive data includes temperature, humidity, and ground stiffness data).

[0021] Detailed calculation and implementation process of the environmental characteristic self-calibration module

[0022] Sensor data acquisition: Environmental parameters are acquired in real time using a light sensor (accuracy ±5 lux), a temperature and humidity sensor (temperature ±0.1℃, humidity ±2%RH), and a ground stiffness sensor (pressure strain type, range 0-200MPa). The acquisition period is set to T (configurable from 10ms to 1s), and the acquired data sequence is as follows: Light intensity sequence: L=[L1,L2,...,L] n (Unit: lux) Temperature sequence: T=[T1,T2,...,T] n (Unit: °C) Humidity sequence: H=[H1,H2,...,H] n (Unit: %RH) Ground stiffness sequence: K=[K1,K2,...,K] n (Unit: MPa) Outliers were removed using the 3σ criterion, missing data were filled in by linear interpolation, and then the data were smoothed by moving average filtering (window size N=5) to obtain the preprocessed data sequences L', T', H', and K'.

[0023] Environment-Error Mapping Database Construction: The storage structure uses (L0, T0, H0, K0) as an index, associating it with the corresponding sensing unit system error ε (including zero-point drift, gain error, and nonlinearity error). The database format is: D = {(L... 0i ,T 0i H 0i ,K0i )→ε i |i=1,2,...,M}, where M is the number of samples (≥10000).

[0024] Error fitting: Multivariate Gaussian regression was used to fit the database samples to establish a mapping model between environmental parameters and errors: ε=f(L,T,H,K)+δ, where δ is the fitting residual (|δ|≤10). -6 ).

[0025] Compensation coefficient calculation (gradient descent algorithm)

[0026] Objective function: Define the loss function J(θ) = ½∑( -ε)², where θ=[θ1,θ2,θ3] are compensation coefficients (corresponding to signal amplification correction, low-pass filter cutoff frequency correction, and high-pass filter cutoff frequency correction, respectively). This represents the prediction error calculated based on the current θ.

[0027] Gradient calculation: Take the partial derivative of J(θ) to obtain the gradient. J(θ)=[ J / θ1, J / θ2, J / θ3] T ,in J / θ i =∑( -ε)· / θ i .

[0028] Iterative update: θ k+1 =θ k -η· J(θ k ), where η is the learning rate (0.001≤η≤0.01, adaptively adjusted), and the iteration termination condition is ||θ k+1 -θ k ||≤10 -8 Or the number of iterations reaches 500.

[0029] Signal amplification factor adjustment: A new =A0·(1+θ1), where A0 is the initial magnification (100≤A0≤1000), A new It must satisfy 100≤A ne w≤2000.

[0030] Filter parameter adjustment: Low-pass filter cutoff frequencyf lp =f l0 • (1+θ2), the high-pass filter cutoff frequency f hp = f h0 ·(1+θ3), where f l0 ∈[10Hz,100Hz], f h0 ∈[0.1Hz, 1Hz], and must satisfy f hp < f lp .

[0031] Detailed calculation and implementation process of the cross-modal causal reasoning module

[0032] Causal Structure Learning Subunit

[0033] Input data: Collect multimodal sensor data, including visual modal data V (image feature vector, dimension D). v =2048), tactile modal data T (pressure matrix, dimension D) t =1024), inertial modal data I (acceleration / angular velocity vector, dimension D) i =6), the data matrix is ​​X=[V,T,I) T (Dimension (D) v +D t +D i ()×N, where N is the number of data samples).

[0034] Bayesian network structure construction: The causal network topology is initialized using the PC algorithm (Peter-Clark algorithm), and redundant edges are removed by conditional independence test (χ² test, significance level α=0.05) to obtain the initial topology G0.

[0035] G0 is optimized based on the Bayesian Information Criterion (BIC), where BIC = -2lnL + klnN, and L is the likelihood function value and k is the number of model parameters. The topological structure with the smallest BIC is selected as the final causal structure G.

[0036] Causal parameter learning: The expectation-maximization (EM) algorithm is used to estimate the conditional probability table (CPT) of a Bayesian network. Let the node variables in the network be X1, X2, ..., X... k Conditional probability P(X) i |Pa(X i The parameters follow a Dirichlet distribution. The optimal estimate is obtained through EM iteration. The convergence condition is that the difference in likelihood between two iterations is ≤10. -5 .

[0037] Dynamic fusion subunit

[0038] Non-rigid data alignment: Dynamic Time Warping (DTW) algorithm is used to align the time axes of different modalities. Let the data sequence of modality A be A=[a1,a2,...,a...]. m The modal B data sequence is B=[b1,b2,...,b] n Define the distance matrix d(i,j) = ||a i -b j ||2, Find the optimal alignment path W={(i1,j1),(i2,j2),...,(i k ,j k This minimizes the total path distance.

[0039] Weighted Fusion Calculation: Node Influence Weights ω Based on Causal Networks i (Calculated by node degree centrality, ω) i =deg(X i ) / ∑deg(X j The fusion formula is: F=∑(ω i ·X i ·P(X i |Pa(X i ))), where F is the fused data feature vector (dimension D=512).

[0040] Fusion Adaptive Adjustment: Introduce a fusion error feedback factor β=||F-F0||2 / ||F0||2 (F0 is the standard fusion sample). When β>0.1, recalculate the node weights ω. i The fusion results are iteratively optimized.

[0041] Fault-tolerant completion subunit

[0042] Sensor failure detection: A Bayesian network-based anomaly detection algorithm is used to calculate the posterior probability P(X) of each modality data. i |F), if P(X) i |F)<θ th (threshold θ) th If the value is less than 0.05, then the modal sensor is considered to be faulty.

[0043] Failure information completion: 1. Extract the failure node X based on the causal structure G. i Parent node Pa(X) i ) and child node Ch(X) i ).

[0044] 2. Construct the completion model: i =∑(P(X i |Pa(X i ))·X j )+∑(P(Ch(X j )|X i )·Ch(X j )) / 2, where i This is the completed node data.

[0045] Completion verification: Calculate the completion error e = || i - i ||2, ( i (where e is the historical normal data mean). If e > 0.02, then combine L1 regularization to optimize and complete the model: min || i -∑(w j X j )||2+λ||w||1(λ=0.01).

[0046] Detailed calculation and implementation process of the scene prediction module

[0047] Historical data storage: collecting data from the past T h (T) h A historical dataset S={(F1,C1),(F2,C2),...,(F3,C1),(F4,C2),...,(F5,C2),(F6,C7,C8,C9 ... m C m )}, where F is the cross-modal fusion feature vector (from the output of module 3), and C is the scene label (such as "indoor flat road surface", "outdoor rugged road surface", etc., for a total of K scene categories).

[0048] Feature enhancement: Principal component analysis (PCA) is performed on F to reduce dimensionality, retaining principal components with a cumulative variance contribution rate of ≥95%, resulting in a dimensionality-reduced feature vector F' (dimension D'=128); at the same time, time-domain features (mean, variance, peak value) and frequency-domain features (spectral peak value, center frequency) are extracted to construct a joint feature vector F'' (dimension D''=256).

[0049] LSTM prediction model structure: The input layer receives the joint feature vector F'', with an input dimension of 256. The hidden layer consists of three LSTM units, each with 256 hidden nodes. The activation function is tanh, while the activation functions for the forget gate, input gate, and output gate are sigmoid. To prevent overfitting, a dropout layer with a dropout rate of 0.2 is added after the LSTM layers. The output layer uses the softmax activation function and outputs the predicted probability distributions for K scene classes: P(C1|F''), P(C2|F''), ..., P(C k |F'').

[0050] Model training: Loss function: The cross-entropy loss function is used: L=-∑C·lnP(C|F'').

[0051] Optimizer: Adam optimizer is used, with learning rate η = 0.0001, decay rates β1 = 0.9, β2 = 0.999, and ε = 10. -8 .

[0052] Training process: Divide the dataset S into training set, validation set and test set in a ratio of 7:2:1, and iterate for 50 rounds. Calculate the validation set accuracy after each round. Stop training and save the optimal model parameters when the validation set accuracy improves by ≤0.1% for 5 consecutive rounds.

[0053] Scene prediction and feedback adjustment

[0054] Scene prediction: Input the joint feature vector F'' at the current time step t The trained LSTM model outputs scene prediction probabilities, and the scene with the highest probability is selected as the prediction result. t+1 (Scenario prediction for the next 1-5 data collection cycles).

[0055] Sensor matrix parameter adjustment: Sampling frequency adjustment: Preset the reference sampling frequency f_C for different scenarios (e.g., f=10Hz for indoor scenarios, f=50Hz for outdoor rugged scenarios), and adjust the sampling frequency according to the predicted scenario. t+1 Adjust the sampling frequency f new =f C ·(1+γ), where γ is the scene complexity coefficient (calculated based on historical data, 0≤γ≤0.5).

[0056] Adjustment of key sensing areas: Based on the contribution ω of each sensing unit in the causal model to the predicted scene. j (ω) j =P( t+1 |X j ) / ∑P( t+1 |X j The top 30% of sensor units in terms of contribution will be designated as key sensing areas, and their data acquisition priority will be increased (e.g., by shortening the acquisition delay and increasing the number of sampling points).

[0057] Dynamic updates: Every 100 data collection cycles, the output layer parameters of the LSTM model are retrained and the latest scene data is incorporated to ensure prediction accuracy (test set accuracy is required to be ≥92%).

[0058] A robot complex scene perception method based on environment adaptive multimodal fusion includes: collecting environmental parameter data and core modal data. The environmental parameter data includes light intensity, temperature, humidity, and ground stiffness data, while the core modal data includes visual modal data, inertial modal data, and tactile modal data. The environmental parameter data is preprocessed, and compensation coefficients are calculated using a gradient descent algorithm based on an environment-error mapping database to dynamically adjust the signal amplification factor and filtering parameters of the sensing unit. A causal model is constructed based on a Bayesian network, and causal structure learning and dynamic fusion are performed on the core modal data to generate a cross-modal fusion feature vector. Simultaneously, sensor failure states are detected, and missing data is filled in through causal chain back-reasoning. The cross-modal fusion feature vector is dimensionality reduced and spatiotemporal feature extracted to construct a joint feature vector, which is input into a pre-trained LSTM model to output scene prediction probabilities. The scene with the highest probability is selected as the prediction result. Based on the prediction result, the sampling frequency and key perception areas of the sensing unit are adjusted to optimize the data acquisition strategy for the next round.

[0059] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The methods disclosed in the embodiments are described simply because they correspond to the methods disclosed in the embodiments; relevant parts can be found in the method section.

[0060] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot complex scene perception system based on environment adaptive multimodal fusion, characterized in that, include: The system comprises a flexible sensing matrix module, an environmental feature self-calibration module, a cross-modal causal reasoning module, and a scene prediction module. The flexible sensing matrix module collects multi-source sensing data; the environmental feature self-calibration module dynamically corrects data errors in the multi-source sensing data using environmental parameters to obtain calibration data; and the cross-modal causal reasoning module fuses and completes the calibration data using a causal model to generate a cross-modal fusion feature vector. The scene prediction module outputs scene prediction probabilities based on the pre-trained LSTM model and the cross-modal fusion feature vector. It selects the scene with the highest probability as the prediction result and applies it to the flexible sensing matrix module to optimize the next round of data acquisition strategy.

2. The robot complex scene perception system based on environment adaptive multimodal fusion according to claim 1, characterized in that, The flexible sensing matrix module collects multi-sensory data including illumination, temperature, humidity, ground stiffness data, as well as visual modal data, inertial modal data, and tactile modal data.

3. The robot complex scene perception system based on environment adaptive multimodal fusion according to claim 2, characterized in that, The environmental feature self-calibration module specifically includes: acquiring a light intensity sequence. Temperature sequence , Humidity sequence: Ground stiffness sequence: The acquired data is preprocessed by using the 3σ criterion to remove outliers, linear interpolation to fill in missing data, and then smoothing the data using a moving average filter to obtain the preprocessed data sequence. , , , Based on the environment-error mapping database, the gradient descent algorithm is used to calculate the compensation coefficient and adjust the signal amplification factor and filtering parameters of the sensing unit.

4. The robot complex scene perception system based on environment adaptive multimodal fusion according to claim 3, characterized in that, The calculation of the compensation coefficient specifically includes: defining the loss function. ,in These are compensation coefficients, corresponding to signal amplification factor correction, low-pass filter cutoff frequency correction, and high-pass filter cutoff frequency correction, respectively. Based on the current Calculated prediction error For systematic error; Take the partial derivative to obtain the gradient. ,in Iterative updates , Let be the learning rate, and let be the iteration termination condition. Or the number of iterations reaches the preset number; signal amplification factor is adjusted: ,in This is the initial magnification, 100 ≤ ≤2000; Filter parameter adjustment: Low-pass filter cutoff frequency High-pass filter cutoff frequency ,in ∈[10Hz,100Hz], ∈[0.1Hz, 1Hz], and must satisfy < .

5. A robot complex scene perception system based on environment adaptive multimodal fusion according to claim 2, characterized in that, The cross-modal causal reasoning module specifically includes: Causal structure learning subunits are used to construct a Bayesian network. The Peter-Clark algorithm is used to initialize the causal network topology, and redundant edges are removed through conditional independence checks to obtain the initial topology. Based on Bayesian information criteria Optimize, Where L is the likelihood function value, k is the number of model parameters, and the topology that minimizes the Bayesian information criterion is selected as the final causal structure G; the expectation-maximization algorithm is used to estimate the conditional probability table of the Bayesian network, assuming the node variables in the network are... , ,..., conditional probability The parameters follow a Dirichlet distribution. The optimal estimate is obtained iteratively using the expectation-maximization algorithm. The convergence condition is that the difference in likelihood functions between two iterations ≤ The converged Bayesian network is used as a causal model. The dynamic fusion subunit employs a dynamic time warping algorithm to align the time axes of different modalities, finding the optimal alignment path through dynamic programming to minimize the total path distance; it also utilizes node influence weights based on causal networks. The formula for fusing aligned modal data is as follows: ； ； Where F is the fused data feature vector; a fusion error feedback factor is introduced. , For standard fusion samples, when β is greater than a preset threshold, the node weights are recalculated. Iterative optimization of the fusion results; The fault-tolerant completion subunit employs an anomaly detection algorithm to calculate the posterior probability of each modality's data. ,like < , If the posterior threshold is used, the modal sensor is determined to be faulty; based on the causal structure G, the failure node is extracted. parent node and child nodes ; Construct a completion model ,in, For the completed node data; calculate the completion error. , The imputation error is the average of historical normal data. If the imputation error exceeds the error threshold, then L1 regularization is used to optimize the imputation model. λ is the penalty coefficient.

6. A robot complex scene perception system based on environment adaptive multimodal fusion according to claim 5, characterized in that, The scene prediction module specifically includes: training the LSTM model using a historical dataset, which includes historical modal data and corresponding scene labels; performing principal component analysis to reduce the dimensionality of the completed modal data, obtaining dimensionality-reduced feature vectors, extracting time-domain and frequency-domain features, and constructing a joint feature vector; inputting the joint feature vector into the trained LSTM model, outputting scene prediction probabilities, selecting the scene with the highest probability as the prediction result, and applying it to the flexible sensing matrix module to optimize the next round of data acquisition strategy.

7. A robot complex scene perception method based on environment adaptive multimodal fusion, applied to the robot complex scene perception system based on environment adaptive multimodal fusion as described in claims 1-6, characterized in that, include: Environmental parameter data and core modal data are collected. The environmental parameter data includes light intensity, temperature, humidity, and ground stiffness data. The core modal data includes visual modal data, inertial modal data, and tactile modal data. The environmental parameter data is preprocessed, and the compensation coefficient is calculated based on the environment-error mapping database using the gradient descent algorithm. The signal amplification factor and filtering parameters of the sensing unit are dynamically adjusted. A causal model is constructed based on a Bayesian network. Causal structure learning and dynamic fusion are performed on the core modal data to generate cross-modal fusion feature vectors. Simultaneously, sensor failure states are detected, and missing data is filled in through reverse reasoning of causal chains. Dimensionality reduction and spatiotemporal feature extraction are performed on the cross-modal fusion feature vectors to construct joint feature vectors. These joint feature vectors are input into a pre-trained LSTM model to output scene prediction probabilities, and the scene with the highest probability is selected as the prediction result. Based on the prediction result, the sampling frequency and key sensing areas of the sensing units are adjusted to optimize the data acquisition strategy for the next round.

Images