A multi-modal fusion perception method based on error state Kalman network

Abstract

The application belongs to the technical field of multi-modal fusion perception, and discloses a multi-modal fusion perception method based on an error state Kalman network, which comprises the following steps: 1, collecting multi-modal data of an intelligent device, and preprocessing the multi-modal data to generate a global state matrix; 2, constructing a state space model based on the global state matrix, defining an error state matrix and an error dynamics equation; 3, calculating the uncertainty of the state space model; 4, dynamically solving Kalman gain by using an RNN model; 5, cyclically checking whether the data of an image, text, laser radar and global navigation satellite system meet the fusion perception requirement, if yes, updating the error state matrix, and if no, not performing fusion processing; and 6, calibrating the error state matrix by using the uncertainty and Kalman gain, and realizing multi-modal fusion perception. The application significantly improves the effect of multi-modal fusion perception and provides support for multi-modal fusion perception.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal fusion perception, and in particular relates to a multi-modal fusion perception method based on an error state Kalman network. BACKGROUND

[0002] Multi-modal fusion perception technology is a core driving force for promoting the high-quality development of key fields such as autonomous driving, intelligent medical treatment, and industrial automation. This technology can significantly improve the decision robustness and environmental adaptability of the motion target detection, tracking, and positioning links by fully integrating multi-source heterogeneous data such as vision, laser radar, global navigation satellite system (GNSS), inertial measurement unit (IMU), and the like, thereby creating a more efficient and reliable solution for complex business scenarios.

[0003] Multi-modal fusion perception methods based on Kalman filtering have received a high degree of attention. Kalman filtering is a classic recursive filtering algorithm that can estimate the state of a dynamic system in real time from a series of measurement values that contain random noise and uncertainties. It has derived variants such as extended Kalman filter (EKF), unscented Kalman filter (UKF), and error state Kalman filter (ESKF). For multi-modal fusion perception tasks, these algorithms are all dedicated to mapping the data of heterogeneous sensors to a unified information model to achieve more comprehensive cognition of the environment.

[0004] Chinese patent application ZL2025103449379 discloses a navigation method for an intelligent pseudo-ginseng harvester based on LSTM and GNSS / INS multi-modal fusion. This method can effectively improve the precision and robustness of the navigation system and adapt to complex environments in hilly and mountainous areas. However, this method only uses the Kalman filtering algorithm for preliminary data fusion, and the LSTM model involved is designed to learn historical INS and GNSS data to generate pseudo-information, without considering the complexity and nonlinearity of the underlying target system. Moreover, this method cannot effectively encapsulate the noise statistical characteristics and domain knowledge dependency, thereby seriously affecting the effect of multi-modal fusion perception.

[0005] The Chinese patent application ZL2020111903854 discloses a robot multi-sensor fusion perception and spatial positioning method, system and device, although this method can effectively deal with multi-factor comprehensive adverse interference and realize high-precision environmental perception and spatial positioning of the robot, but this method does not fully consider the recursion of Kalman gain calculation, cannot estimate the state of the nonlinear system from the perspective of long sequence modeling, and also cannot capture the long-term dependence relationship in multi-source heterogeneous time series data, which not only may lead to a significant decrease in the robustness of the model in processing noisy observations, but also further limits its effective application in target detection, motion planning, trajectory tracking and other backend tasks.

[0006] In summary, although the existing method has achieved certain results, the existing method does not consider the complexity and nonlinear characteristics of the target system, which makes it difficult to be accurately described as a processable state space model, and the existing method generally has the problem of insufficient long sequence modeling capability, which directly leads to a significant decrease in the robustness of the model in processing noisy observations. SUMMARY

[0007] To solve the above technical problems, the present application proposes a multi-modal fusion perception method based on error state Kalman network, which effectively encapsulates the noise statistical characteristics and domain knowledge dependence, significantly improves the effect of multi-modal fusion perception, and also designs a state estimator to realize long sequence modeling and capture long-term dependence, further improving the robustness of the model in processing noisy observations.

[0008] To achieve the above purpose, the present application is realized by the following technical scheme:

[0009] The present application is a multi-modal fusion perception method based on error state Kalman network, which is applied to intelligent devices, and specifically the multi-modal fusion perception method includes the following steps:

[0010] Step 1, collect multi-modal data of the intelligent device, and pre-process the collected multi-modal data to generate a global state matrix related to the target task of the intelligent device, wherein the multi-modal data includes image data, text data, laser radar data, global navigation satellite system (GNSS) data, and inertial measurement unit (IMU) sensor data;

[0011] Step 2, based on the global state matrix generated in step 1, a nonlinear, Gaussian and continuous state space model is constructed, a Taylor expansion method is used to implement local linearization processing on the state space model, and an error state matrix and an error dynamics equation are defined;

[0012] Step 3, calculating the uncertainty of the state space model using inertial measurement unit sensor data in the multi-modal data;

[0013] Step 4, dynamically solving the Kalman gain using a compact RNN model with limited complexity;

[0014] Step 5, cyclically checking whether the image data, text data, lidar data, and global navigation satellite system (GNSS) data of Step 1 meet the fusion perception requirements, and if so, updating the error state matrix, and if not, not performing fusion processing, i.e., not calculating the error observation value;

[0015] Step 6, calibrating the error state matrix using the uncertainty of the state space model obtained in Step 3 and the Kalman gain obtained in Step 4, to achieve multi-modal fusion perception.

[0016] A further improvement of the present application is that Step 2 specifically includes the following steps:

[0017] Step 2.1, considering the complexity of the intelligent device, the state space model is usually nonlinear, Gaussian, and continuous, and is therefore formalized as:

[0018]

[0019] wherein, and respectively represent the global state matrix of the intelligent device at time slot and time slot , represents a nonlinear evolution function, represents a noise-driven matrix at time slot , represents an additive Gaussian white noise, represents a Gaussian distribution, represents a state noise covariance matrix, represents an observation matrix at time slot , represents a nonlinear observation function, represents an additive Gaussian white noise with an observation noise covariance matrix ;

[0020] Step 2.2, defining an error state matrix for measuring the state deviation:

[0021]

[0022] wherein, represents a nominal state matrix, used to represent the additive Gaussian white noise The status of smart devices;

[0023] Step 2.3: Use the Taylor expansion method to analyze the nonlinear evolution function. and nonlinear observation function Perform local linearization:

[0024]

[0025] in, Indicates in time slot The evolution matrix at that point, i.e., the nonlinear evolution function In the time slot Jacobian matrix at the location; Indicates in time slot The observation matrix at that point, i.e., the nonlinear observation function In the time slot If we ignore the second-order and higher-order terms of the Taylor expansion, the above formula is simplified to:

[0026]

[0027] The error state matrix from step 2.2 Substituting into the above equation, we can then obtain the error dynamics equation:

[0028]

[0029] in, Represents the error observation matrix. This represents the nominal observation matrix.

[0030] A further improvement of this application is that step 3 specifically includes the following steps:

[0031] Step 3.1: Initialize the nominal state matrix Based on inertial measurement unit sensor data, the nominal state matrix is ​​recursively predicted. With the prediction error state matrix ;

[0032] Step 3.2: Calculate the uncertainty of the state-space model. :

[0033]

[0034] in, Indicates the transpose symbol. Indicates in time slot The error covariance matrix at that point, Also referred to as the prediction error covariance matrix, denotes the evolution matrix at time slot , denotes the noise driven matrix at time slot , denotes the state noise covariance matrix at time slot .

[0035] A further improvement of the present application is that step 4 specifically comprises the following steps:

[0036] Step 4.1, design four trackable features:

[0037] Feature is the difference degree between the error state matrix at time slot and the error state matrix at time slot ; Feature is the difference degree between the error state matrix at time slot

[0038] and the predicted error state matrix at time slot ; Feature is the difference degree between the observation matrix at time slot and the observation matrix at time slot ;

[0039] Feature is the difference degree between the state matrix set of a group of discrete time slots , wherein denotes the total length of the selected time slots;

[0040] Feature is the difference degree between the state matrix set of a group of discrete time slots , wherein denotes the total length of the selected time slots;

[0041] Step 4.2, establish a compact RNN model to track features , feature , feature and feature , design a state estimator based on a hybrid structure of Transformer-RNN, and use the state estimator as an internal storage unit of the intelligent device to realize long sequence modeling and capture the time sequence dependency of multi-modal data;

[0042] Step 4.3, train the state estimator designed in step 4.2 in a supervised manner in an end-to-end manner, and the training loss function is the square error between the nominal state matrix and the predicted nominal state matrix : ​​

[0043] .

[0044] wherein the nominal state matrix The calibration will be completed in step 6.

[0045] A further improvement of the present application is that the state estimator in step 4.2 comprises an LSTM model, a Transformer model, an embedding layer, a position encoding layer in the RNN model, and the process of solving the Kalman gain by using the state estimator is:

[0046] Step 4.2.1, using an independent RNN model to track the features , the feature , the feature , and the feature ;

[0047] Step 4.2.2, considering factors such as long sequence modeling and parameter quantity, an LSTM model in the RNN model is selected to learn the noise-driven matrix The additive white Gaussian noise , the state noise covariance matrix , and the state noise covariance matrix

[0048] Step 4.2.3, the input state noise covariance matrix of the LSTM model is cascaded with the Transformer model, and the embedding layer and the position encoding are used to pass the logical information in the time dimension, and then the Kalman gain is solved:

[0049]

[0050] wherein, represents the position encoding, represents the embedding layer, represents the Transformer model, represents the model.

[0051] A further improvement of the present application is that step 5 is specifically: cyclically checking whether the image data, the text data, the lidar data, and the GNSS data are available, if available, then calculating whether the credibility between the image data, the text data, the lidar data, and the GNSS data and the IMU sensor data is greater than a threshold value, if the credibility is greater than the threshold value, then calculating the error observation value , otherwise, the error observation value is not calculated.

[0052] The further improvement of the present application is that step 6 specifically comprises the following steps:

[0053] Step 6.1, utilizing Kalman gain Dynamically updating error state matrix :

[0054]

[0055] Step 6.2, updating error covariance matrix at time slot : :

[0056]

[0057] wherein, denotes a unit matrix, denotes a transposed observation matrix;

[0058] Step 6.3, updating nominal state matrix by integrating predicted nominal state matrix and error state matrix :

[0059] .

[0060] The beneficial effects of the present application are:

[0061] The error state Kalman network designed in the present application fully considers the complexity and nonlinearity of the target system at the bottom layer, can dynamically calculate the Kalman gain, avoids matrix inversion operation, realizes effective encapsulation of the dependence of noise statistical characteristics and domain knowledge, and significantly improves the fusion perception effect of multi-modal data such as image, text, laser radar, GNSS, IMU, etc.

[0062] The state estimator based on the hybrid structure of Transformer-RNN designed in the present application further considers the recursion of Kalman gain calculation, can realize long sequence modeling in a supervised training manner, capture long-term dependencies, overcome sequence length limitations, and improve the robustness of the model in processing noisy observations, thereby providing strong support for multi-modal fusion perception. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a flowchart of the present application.

[0064] Figure 2 is a schematic diagram of the state estimator of the present application.

[0065] Figure 3 is a flowchart of an embodiment of the present application. DETAILED DESCRIPTION

[0066] Embodiments of the present application will be described below with reference to drawings, and many practical details will be described in the following description for the purpose of providing a thorough understanding of the present application. It will be apparent, however, that the present application can be practiced without many of the details. In other instances, well-known structures and components are not described in detail or are shown in block diagram form in order to avoid obscuring the present application. Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0067] As shown in Figure 1 The present application is a multi-modal fusion perception method based on error state Kalman network, which is applied to intelligent devices. Specifically, the multi-modal fusion perception method comprises the following steps:

[0068] Step 1, collect multi-modal data of the intelligent device, and pre-process the collected multi-modal data to generate a global state matrix related to the target task of the intelligent device, wherein the multi-modal data includes image data, text data, laser radar data, global navigation satellite system (GNSS) data, and inertial measurement unit (IMU) sensor data.

[0069] Step 2, based on the global state matrix generated in step 1, a nonlinear, Gaussian, and continuous state space model is constructed, a local linearization process is performed on the state space model using Taylor expansion method, and an error state matrix and an error dynamics equation are defined.

[0070] Step 3, the inertial measurement unit sensor data in the multi-modal data is used to calculate the uncertainty of the state space model.

[0071] Step 4, a compact RNN model with limited complexity is used to dynamically solve the Kalman gain, and effectively encapsulate the noise statistical characteristics and domain knowledge dependence.

[0072] Step 5, cyclically check whether the image data, text data, laser radar data, and global navigation satellite system (GNSS) data of step 1 meet the fusion perception requirements, if yes, update the error state matrix, if not, do not perform fusion processing, that is, do not calculate the error observation value.

[0073] Step 6, using the uncertainty of the state space model obtained in step 3 and the Kalman gain obtained in step 4, calibrate the error state matrix to realize multi-modal fusion perception.

[0074] The following takes an intelligent robot on a certain domestic science and technology island as an example to illustrate the scheme of the application. A series of implementation steps are required to construct the model proposed in the application. All the models are programmed by Python language and trained on a server configured with Intel i7-12700KF CPU and GeForce RTX 4090 GPU.

[0075] The detailed process is shown in a multi-modal fusion perception method based on error state Kalman network as shown in Figure 3 The method comprises the following steps:

[0076] Step 1, the intelligent robot moves in the physical world and interacts with the open environment, collecting lidar data, global navigation satellite system (GNSS) data and inertial measurement unit (IMU) sensor data. Specifically:

[0077] Step S11, clean the above data by coordinate transformation, uniform sampling, timing dynamic positioning and other technologies to avoid signal shielding, noise interference, measurement drift and other problems;

[0078] Step S12, the goal of the intelligent robot is to move according to the established trajectory, so set the global state matrix , wherein , and represent the position, velocity and attitude of the intelligent robot at time slot .

[0079] Step 2, construct a nonlinear, Gaussian and continuous state space model.

[0080] Step S21, establish the motion model of the intelligent robot, including position, velocity and attitude equations:

[0081]

[0082] wherein, denotes the time step, denotes the quaternion rotation matrix, denotes the gravitational acceleration, denotes the Hadamard product, and denote the dynamic and angular velocity of the intelligent robot at time slot .

[0083] Step S22, use Taylor expansion method to perform local linearization on the above motion model, and define the error state matrix and error dynamics equation:

[0084]

[0085] in, , and These represent the position error state matrix, velocity error state matrix, and attitude error state matrix, respectively.

[0086] Step S23: Based on lidar and GNSS data, establish an observation model for the intelligent robot:

[0087]

[0088] in, and Indicates having a covariance matrix and Additive white Gaussian noise. Generally, using... Simplified representation.

[0089] Step 3: Calculate the uncertainty of the state-space model using the inertial measurement unit sensor data from the multimodal data. This specifically includes the following steps:

[0090] Step S31: Initialize the nominal state matrix Based on inertial measurement unit sensor data, the nominal state matrix is ​​recursively predicted. With the prediction error state matrix ;

[0091] Step S32: Calculate the uncertainty of the state-space model. :

[0092]

[0093] in, Indicates the transpose symbol. Indicates in time slot The error covariance matrix at that point, Also known as the prediction error covariance matrix, Indicates in time slot The evolution matrix at that point, Indicates in time slot The noise driving matrix at that location, Indicates in time slot The state noise covariance matrix at that location.

[0094] Step 4: Employ a compact RNN model with finite complexity to dynamically solve for the Kalman gain, effectively encapsulating the statistical characteristics of noise and its dependence on domain knowledge. This includes the following steps:

[0095] Step S41, four trackable features are designed:

[0096] Features Error state matrix at time slot Difference degree between error state matrix at time slot Error state matrix at time slot Features

[0097] Features Error state matrix at time slot Difference degree between predicted error state matrix at time slot Error state matrix at time slot Features

[0098] Features Observation matrix at time slot Difference degree between observation matrix at time slot Observation matrix at time slot Features

[0099] Features State matrix set of a group of discrete time slots , wherein represents the total length of the selected time slot;

[0100] Step S42, a compact RNN model is established to track features , feature , feature and feature , a state estimator based on a hybrid structure of Transformer-RNN is designed, and the state estimator is taken as an internal storage unit of the intelligent device to realize long sequence modeling and capture the time sequence dependency of multi-modal data.

[0101] As shown in Figure 2 , the state estimator includes an LSTM model in an RNN model, a Transformer model, an embedding layer and a position encoding layer, and a process for solving Kalman gain by using the state estimator is as follows:

[0102] An independent RNN model is used to track features , feature , feature and feature , considering factors such as long sequence modeling and parameter quantity, an LSTM model in an RNN model is selected to learn additive white Gaussian noise closely related to noise-driven matrix Here, the state noise covariance matrix ​​​As an additional input of the LSTM model, the input state noise covariance matrix of the LSTM model is concatenated with the Transformer model, passes the logical information in the time dimension through the embedding layer and the position encoding, and then solves the Kalman gain :

[0103]

[0104] wherein, represents the position encoding, represents the embedding layer, represents the Transformer model, represents the model.

[0105] Step S43, training the state estimator designed in step 4.2 in a supervised manner end-to-end, and the training loss function is the square error between the nominal state matrix and the predicted nominal state matrix :

[0106] .

[0107] wherein, the nominal state matrix will be calibrated in step 6.

[0108] Step 5, check whether the image data, text data, lidar data, global navigation satellite system (GNSS) data of step 1 meet the fusion perception requirements in a loop, if they meet, update the error state matrix, if they do not meet, do not perform fusion processing, that is, do not calculate the error observation value. In this step, the loop check specifically means: check whether the image data, text data, lidar data, global navigation satellite system (GNSS) data are available in a loop, if they are available, calculate whether the credibility between the image data, text data, lidar data, global navigation satellite system (GNSS) data and the inertial measurement unit (IMU) sensor data is greater than a threshold value, if the credibility is greater than the threshold value, calculate the error observation value , otherwise do not calculate the error observation value.

[0109] Step 6, calibrate the error state matrix using the uncertainty of the state space model obtained in step 3 and the Kalman gain obtained in step 4, and realize multi-modal fusion perception, which specifically includes the following steps:

[0110] Step 6.1, dynamically update the error state matrix using the Kalman gain :

[0111]

[0112] According to the fusion perception requirement of image data, text data, lidar data, global navigation satellite system (GNSS) data, the error state matrix Multiple updates are implemented, at this time, the Kalman gain is maintained Unchanged.

[0113] Step 6.2, update the error covariance matrix at time slot :

[0114]

[0115] wherein, denotes a unit matrix, denotes the observation matrix after transposition;

[0116] Step 6.3, update the nominal state matrix by integrating the predicted nominal state matrix and the error state matrix :

[0117] .

[0118] In order to evaluate the performance of the present application, a data set is generated. The reference trajectory is generated using the MATLAB rate driven trajectory generator, which uses random initial position, speed, heading and angular velocity as input. A total of 100 trajectories are generated, each with a length of 100 seconds. The simulator runs at a frequency of 200Hz, and the sampling rate of the IMU measurement is set to 10Hz.

[0119] In order to ensure the fairness of the experiment, three methods are selected as the comparison. These methods are UKF, EKF and ES-EKF, all of which are related to fusion perception. The average results of the comparison are shown in Table 1.

[0120] Table 1

[0121]

[0122] By observation, it can be seen that the RMSE value of the present application is significantly lower than that of the comparison method, which fully proves the superiority and effectiveness of the present application.

[0123] ​The application integrates heterogeneous data such as vision, lidar, GNSS, IMU, etc. through ESKF, adopts a compact RNN with limited complexity to dynamically calculate the Kalman gain of ESKF, and realizes effective encapsulation of the dependence of noise statistical characteristics and domain knowledge. In addition, considering the recursion of Kalman gain calculation, a state estimator based on a hybrid structure of Transformer-RNN is further designed, and it is used as an internal storage unit to realize long sequence modeling, capture long-term dependencies, overcome sequence length limitations, and further improve the robustness of the model in processing noisy observations.

[0124] The above merely describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A multi-modal fusion perception method based on error-state Kalman network, characterized in that: The multi-modal fusion perception method is applied to an intelligent device, and specifically the multi-modal fusion perception method comprises the following steps: Step 1, collecting multi-modal data of the intelligent device, and preprocessing the collected multi-modal data to generate a global state matrix related to a target task of the intelligent device, wherein the multi-modal data includes image data, text data, laser radar data, global navigation satellite system data, and inertial measurement unit sensor data; Step 2, based on the global state matrix generated in step 1, a state space model is constructed, a local linearization process is performed on the state space model, an error state matrix and an error dynamics equation are defined; Step 3, the inertial measurement unit sensor data in the multi-modal data is used to calculate the uncertainty of the state space model; Step 4, a compact RNN model with limited complexity is used to dynamically solve the Kalman gain; Step 5, the image data, text data, laser radar data, and global navigation satellite system data of step 1 are checked to see if they meet the fusion perception requirements, if they do, the error state matrix is updated, if they do not, no fusion processing is performed; Step 6, using the uncertainty of the state space model obtained in step 3 and the Kalman gain obtained in step 4, the error state matrix is calibrated to achieve multi-modal fusion perception, wherein Step 2 specifically comprises the following steps: Step 2.1, the state space model is formalized as: ； ； wherein, and denote the global state matrix of the smart device at time slot and time slot respectively, denotes a nonlinear evolution function, denotes a noise driven matrix at time slot denotes an additive white Gaussian noise, denotes a Gaussian distribution, denotes a state noise covariance matrix, denotes an observation matrix at time slot denotes a nonlinear observation function, denotes an additive white Gaussian noise with an observation noise covariance matrix ​​​ Step 2.2, defining an error state matrix for measuring state bias : ； wherein, denotes a nominal state matrix, used to characterize the state of the intelligent device without considering the additive white Gaussian noise under the intelligent device. Step 2.

3. Apply Taylor expansion to the nonlinear evolution function and the nonlinear observation function Implement a local linearization process: ； ； wherein denotes the evolution matrix at time slot denotes the observation matrix at time slot ;​ The error state matrix of step 2.2 is Substituting into the above equation, the error dynamics equation is obtained as follows: ； ； wherein, denotes the error observation matrix, denotes the nominal observation matrix.

2. The multi-modal fusion perception method based on the error-state Kalman network according to claim 1, characterized in that: Step 3 specifically comprises the following steps: Step 3.1, initialize nominal state matrix and recursively predict the nominal state matrix based on inertial measurement unit sensor data and the predicted error state matrix ; Step 3.2, Compute uncertainty of the state space model : ； wherein, denotes a transpose symbol, denotes an error covariance matrix at time slot denotes an evolution matrix at time slot denotes a noise driven matrix at time slot denotes a state noise covariance matrix at time slot denotes a state noise covariance matrix at time slot denotes a noise driven matrix at time slot denotes a state noise covariance matrix at time slot denotes a state noise covariance matrix at time slot 3. The multi-modal fusion perception method based on the error-state Kalman network according to claim 2, characterized in that: Step 4 specifically comprises the following steps: Step 4.1, four trackable features are designed: Features Error state matrix at time slot Error state matrix at time slot Difference between error state matrices at time slots Error state matrix at time slot Difference between error state matrices at time slots Features Error state matrix at time slot Error state matrix at time slot Error state matrix at time slot Error state matrix at time slot Difference between Features is a matrix of observations for a time slot is a matrix of observations for a time slot is a difference degree between Features a set of state matrices for a set of discrete time slots wherein denotes the total length of the selected time slot; Step 4.2, establish compact RNN model, track features , features , features and features , design a state estimator based on a hybrid structure of Transformer-RNN, and use the state estimator as an internal storage unit of the intelligent device to realize long sequence modeling and capture the time sequence dependency of multi-modal data; Step 4.

3. Train the state estimator designed in Step 4.2 end-to-end in a supervised manner, with the training loss function being the squared error between the predicted nominal state matrix and the true nominal state matrix ​ 。 4. The multi-modal fusion perception method based on the error-state Kalman network according to claim 3, characterized in that: The state estimator in step 4.2 includes an LSTM model, a Transformer model, an embedding layer, and a position encoding layer, and the process of solving the Kalman gain using the state estimator is: Step 4.2.1, Tracking features using an RNN model , features , features and features ; Step 4.2.2, Selecting the LSTM model in the RNN model to learn additive white Gaussian noise The state noise covariance matrix As an additional input to the LSTM model; Step 4.2.3, input state noise covariance matrix of the LSTM model Cascade connection with the Transformer model, pass through the embedding layer and the position encoding, pass the logical information in the time dimension, solve the Kalman gain : ； wherein, represents a position encoding, represents an embedding layer, represents a Transformer model, represents model.

5. The multi-modal fusion perception method based on error-state Kalman network according to claim 4, characterized in that: The step 5 specifically comprises: cyclically checking whether the image data, the text data, the lidar data and the GNSS data are available, if available, calculating whether the credibility between the image data, the text data, the lidar data, the GNSS data and the inertial measurement unit sensor data is greater than a threshold value, if the credibility is greater than the threshold value, calculating the error observation value , otherwise, not calculating the error observation value.

6. The multi-modal fusion perception method based on the error-state Kalman network according to claim 5, characterized in that: Step 6 specifically comprises the following steps: Step 6.1, Utilizing Kalman Gain Dynamic updating of error state matrix : ； Step 6.2, update the error covariance matrix at time slot :​ ； wherein, denotes the identity matrix, denotes the transposed observation matrix; Step 6.3, updating the nominal state matrix by incorporating the prediction with the error state matrix , the nominal state matrix : 。

Images