Multi-modal data fusion method based on modal missing domain noise uncertainty

Abstract

The invention relates to a multi-modal data fusion method based on modal missing domain noise uncertainty, and the method comprises the steps: carrying out the fusion of data after time-space online calibration through combining a joint embedded uncertain multi-modal data fusion strategy and a multi-modal stable representation strategy under uncertainty; according to the joint embedding uncertain multi-modal data fusion strategy, decoupling representation learning is introduced, probability embedding obeying multivariate Gaussian distribution is used, back propagation is achieved through a re-parameterization technique, meanwhile, a regularization item is introduced, and the approximation degree of probability embedding and normal Gaussian distribution is constrained through KL divergence; according to the multi-modal stable representation strategy under the uncertainty, quantitative modeling is carried out for single-modal noise uncertainty, KL divergence is introduced, semantic consistency and identical distribution of different modals are constrained, meanwhile, a self-supervised contrast learning strategy is introduced, and single-modal stable representation learning of uncertainty gains is achieved. Compared with the prior art, the accuracy and robustness of multi-modal data fusion can be further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data fusion, and in particular relates to a multi-modal data fusion method based on the uncertainty of noise in the modal missing domain. Background Art

[0002] As an important technical support for ocean resource development, environmental monitoring, and shipping safety assurance, intelligent ships are facing an increasing demand for high-performance perception and decision-making. Multi-modal data fusion technology is a key link in realizing ship intelligence. By integrating multi-source data from different sensors (such as inertial measurement units, global positioning systems, radars, sonars, cameras, and lidars, etc.), more comprehensive and accurate environmental perception information can be obtained, thereby effectively improving the performance of functions such as ship autonomous navigation, fault diagnosis, and safety warning.

[0003] However, due to the limitations of the sensors themselves (such as measurement accuracy, sampling frequency, etc.), the interference of complex marine environments (such as sea waves, sea fog, electromagnetic interference, etc.), and errors in the data transmission process, multi-modal sensor data often has uncertainties, mainly manifested as data missing and noise interference. These uncertainty factors seriously affect the accuracy and robustness of data fusion, and limit the application efficiency of intelligent ships in complex sea area environments. Traditional data fusion methods, such as fusion strategies based on deterministic embedding, although can process multi-modal data to a certain extent, often show obvious limitations when facing the noise uncertainty in the modal missing domain. For example, the Chinese patent application with the publication number CN119088034A proposes a ship obstacle avoidance and automatic berthing method based on a deep learning algorithm, which uses a method combining distributed deep deterministic policy gradient and game theory to train the obstacle avoidance and path planning decision-making model of the ship. This method is completely based on deterministic assumptions, has limitations in environmental dynamics and adversarial generation, and the model generalization ability is insufficient.

[0004] When the prior art deals with modal missing and noise uncertainty, it usually assumes that the quality of each modal data is relatively high and the consistency is good. Therefore, the specificities of each modal data cannot be fully exploited during the fusion process, resulting in insufficient accuracy and robustness of the fusion result. Especially in complex marine environments, the diversity and uncertainty of sensor data further exacerbate this problem, making it difficult for traditional data fusion methods to meet the requirements of intelligent ships for high-performance perception and decision-making. Therefore, it is urgent to design a new intelligent ship multi-modal data fusion method to further improve the accuracy and robustness of multi-modal data fusion, so that intelligent ships can achieve more efficient and reliable information acquisition and processing in complex marine environments. Summary of the Invention

[0005] The object of the present invention is to overcome the defects existing in the above-mentioned prior art and provide a multi-modal data fusion method based on the uncertainty of modal missing domain noise, so as to further improve the accuracy and robustness of multi-modal data fusion.

[0006] The object of the present invention can be achieved by the following technical solutions:

[0007] The present invention provides a multi-modal data fusion method based on the uncertainty of modal missing domain noise, including the following steps:

[0008] Obtain data from multiple sensors and perform spatio-temporal online calibration;

[0009] Combine the joint embedding uncertain multi-modal data fusion strategy and the multi-modal stable representation strategy under uncertainty to fuse the calibrated data;

[0010] Among them, the joint embedding uncertain multi-modal data fusion strategy introduces decoupled representation learning, uses probability embeddings that follow a multivariate Gaussian distribution, and realizes backpropagation through the reparameterization trick; at the same time, a regularization term is introduced, and the KL divergence is used to constrain the approximation degree of the probability embedding and the normal Gaussian distribution;

[0011] The multi-modal stable representation strategy under uncertainty quantifies and models the single-modal noise uncertainty, introduces the KL divergence, and constrains the semantic consistency and the same distribution of different modalities; at the same time, a self-supervised contrast learning strategy is introduced to realize the single-modal stable representation learning with uncertainty gain.

[0012] Furthermore, the types of sensors include inertial measurement units, global positioning systems, radars, sonars, cameras, and lidars. A unique identifier is assigned to each sensor, and its installation position and orientation information are recorded.

[0013] Furthermore, the specific process of the spatio-temporal online calibration is as follows:

[0014] Time calibration: Using the inertial measurement unit as the reference sensor for time calibration, extract the motion features of each sensor, and estimate the time offset through three-dimensional motion correlation analysis. Calculate the motion feature correlation under different time offsets, and obtain the optimal time offset value that maximizes the correlation;

[0015] Spatial calibration: Select three orthogonal directions in the coordinate system of the inertial measurement unit as the reference directions, calculate the correlation between the projections of the motion features of the inertial measurement unit in these directions and the projections of the motion features of the target sensor in the corresponding directions, and calculate the external parameter rotation matrix between the sensors through the singular value decomposition method;

[0016] Observability check: Check the condition number and minimum eigenvalue of the covariance matrix to determine whether the motion data meets the observability conditions;

[0017] Calibration result verification: Verify the accuracy of the calibration result by comparing the sensor data before and after calibration; if the calibration result does not meet the requirements, readjust the sensor installation position or sampling frequency, and re-collect and calibrate the data.

[0018] Furthermore, in the joint embedding uncertain multimodal data fusion strategy, the probability embedding z is defined i ~p(z i |x i ) obeys the multivariate Gaussian distribution Among them, x i is the data of the i-th mode, μ i For expectations, is the variance; define the probability embedding z i The reparameterized form is z i =μ i +σ i ∈ i ,∈ i From the standard normal distribution The random variables sampled in the training process are used with the help of the reparameterization technique τ i =μ i +εσ i , Implement back propagation.

[0019] Furthermore, a regularization term is introduced in the optimization process of back propagation Using KL divergence to show constraints and normal Gaussian distribution Force two distributions to approach one another asymptotically:

[0020]

[0021] Furthermore, during the training process, τ i As input, to ensure that the sampled embeddings share the same learning direction; during inference, use μ i as input to relax the directional constraints on the reasoning representation.

[0022] Furthermore, the forward propagation loss during training and inference is and Rewrite them into the following forms:

[0023]

[0024] Furthermore, in the multimodal stable representation strategy under uncertainty, for the deterministic single-modal representation Introduce a linear mapping module and obtain the mean value and variance Define the modal representation according to the uncertainty estimation knowledge theory of the latent space representation which follows a multivariate Gaussian distribution Thus, uncertainty quantification modeling is realized.

[0025] Furthermore, introduce the KL divergence and construct the constraint objective function as follows:

[0026]

[0027] where the first term is the semantic consistency constraint for different modalities, and the second term is the same distribution constraint.

[0028] Furthermore, for the probability distribution representation of modality s According to the reparameterization trick randomly sample two sample representations and regard the former as the reference point and the latter as the matching point to construct a positive sample pair; meanwhile, randomly draw a set of unmatched points j≠i from other distribution representations to construct a negative sample pair; then, introduce a self-supervised contrast learning strategy to achieve optimization:

[0029]

[0030] where and cosim represent the temperature factor and cosine similarity respectively.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention proposes a multi-modal data fusion method based on the uncertainty of modal missing domain noise, which combines the joint embedding uncertain multi-modal data fusion strategy and the multi-modal stable representation strategy under uncertainty to achieve high-accuracy and high-robustness data fusion. Specifically, in the joint embedding uncertain multi-modal data fusion strategy of the present invention, probability embedding is used to replace traditional deterministic embedding to cope with the intertwined constraints of single-modal noise uncertainty in the modal missing domain. At the same time, the reparameterization technique is introduced to achieve backpropagation, which reduces the directional constraint on the inference representation, and the KL divergence regularization term is introduced to constrain the approximation degree of the probability embedding to the normal Gaussian distribution, enhancing the generalization ability of multi-modal specific information. In the multi-modal stable representation strategy under uncertainty of the present invention, the single-modal noise uncertainty is quantified and modeled, and the KL divergence is introduced to constrain the semantic consistency and the same distribution of different modalities. At the same time, the self-supervised contrast learning strategy is introduced to achieve the single-modal stable representation learning with uncertainty gain, thereby improving the stability of each modality data under uncertain conditions. The above method can fully explore the specificity of each modality data under the conditions of modal missing and noise uncertainty, improve the accuracy and robustness of the multi-modal data fusion of intelligent ships, enable intelligent ships to achieve more efficient and reliable information acquisition and processing in complex marine environments, and thus significantly improve their autonomous navigation ability, fault diagnosis accuracy and safety warning efficiency, providing strong technical support for the implementation of the marine power strategy.

[0033] 2. The present invention assigns a unique identifier to each sensor and records its installation position and orientation information, performs spatio-temporal online calibration before multi-modal data fusion, uses an inertial measurement unit with a high sampling rate and high precision as a reference sensor for time calibration, calculates the motion feature correlation under different time offsets, and obtains the optimal time offset value that maximizes the correlation. On the basis of time calibration, three orthogonal directions in the coordinate system of the inertial measurement unit are selected as reference directions, and the external parameter rotation matrix between sensors is calculated by the singular value decomposition method for spatial calibration between sensors. To ensure the uniqueness of the estimation result of the external parameter rotation matrix, the observability condition of the motion data is judged by checking the condition number and the minimum eigenvalue of the covariance matrix. Finally, the accuracy of the calibration result is verified by comparing the sensor data before and after calibration. The above steps can ensure the spatio-temporal consistency of multi-modal data and provide a high-quality data basis for subsequent data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of the method of the present invention;

[0035] Figure 2 is a schematic diagram of the data fusion process combining the joint embedding uncertain multi-modal data fusion strategy and the multi-modal stable representation strategy under uncertainty. Detailed implementation mode

[0036] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and gives a detailed implementation manner and a specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0037] Embodiment:

[0038] This embodiment provides a multi-modal data fusion method based on modal missing domain noise uncertainty, which can be used to improve the accuracy and robustness of the fusion of multi-modal data of intelligent ships in complex marine environments, such as Figure 1 shown, including the following steps:

[0039] S1. Data acquisition and calibration of multi-sensors of intelligent ships.

[0040] First, a variety of sensors are arranged on the intelligent ship and initialized to ensure their normal operation. By synchronizing the sensor clocks and recording the installation position and orientation information of each sensor, a foundation is laid for subsequent data calibration and fusion. After starting the sensors, multi-modal data is collected and stored in a local storage device, and at the same time, it is transmitted to the monitoring center or cloud server in real time through a wireless communication module; in addition, the ship's navigation status information, such as speed, heading, and rudder angle, is recorded for subsequent data calibration and fusion. In the data calibration stage, a certain sensor is used as a reference sensor for time calibration, and the time offset between the sensors is estimated through three-dimensional motion correlation analysis, and the optimal time offset value is found to achieve time synchronization. Then, the external parameter rotation matrix between the sensors is estimated, and the external parameter rotation matrix is calculated by the singular value decomposition (SVD) method to achieve spatial calibration. In order to ensure the uniqueness of the estimation result of the external parameter rotation matrix, it is checked whether the motion data satisfies the observability condition. Finally, by comparing the sensor data before and after calibration, the accuracy of the calibration result is verified. If the calibration result does not meet the requirements, the installation position or sampling frequency of the sensor is readjusted, and data acquisition and calibration are performed again. The specific process is as follows:

[0041] S11. Sensor arrangement and initialization.

[0042] (1) Install a variety of sensors on the ship, including Inertial Measurement Unit (IMU), Global Positioning System (GPS), Radar (RADAR), Sonar (SONAR), Camera (CAMERA), and Light Detection and Ranging (LiDAR). The IMU and GPS are installed at the center position of the ship to measure the motion and position information of the ship; the radar and sonar are installed at the bottom and sides of the ship to measure the distance to surrounding obstacles; the cameras are installed at the bow, stern, and sides to collect visual information of the surrounding environment; the LiDAR is installed on the deck to obtain high-precision three-dimensional environment information. Assign a unique identifier to each sensor and record its installation position and orientation information for subsequent data calibration and fusion.

[0043] (2) Initialize the sensors to ensure their normal operation. Perform warm-up and initialization operations on the IMU and GPS to eliminate initial biases. Synchronize the sensor clocks, using the Pulse Per Second (PPS) signal of the GPS as the clock synchronization signal to ensure that all sensor timestamps are consistent. Set the sampling frequencies of different sensors.

[0044] S12. Multi-sensor data acquisition and transmission.

[0045] Start the sensors to collect multi-modal data, including the acceleration and angular velocity data of the IMU, the position and velocity data of the GPS, the distance and velocity data of the radar, the depth data of the sonar, the image data of the camera, and the point cloud data of the LiDAR. Store the collected data in a local storage device and transmit it to the monitoring center or cloud server in real time through a wireless communication module. At the same time, record the ship's navigation status information, such as speed, heading, rudder angle, etc., for subsequent data calibration and fusion.

[0046] S13. Spatio-temporal online calibration of sensor data.

[0047] (1) Time calibration.

[0048] Use the IMU as the reference sensor for time calibration, and use its high sampling rate and high-precision timestamp as the benchmark. Extract the motion features of each sensor, such as the angular velocity data of the IMU, the distance change rate data of the radar, the optical flow data of the camera, etc., and perform time offset estimation through three-dimensional motion correlation analysis. Set the enumeration range of the time offset to -1000 ms to 1000 ms, with a step size of 2.5 ms, calculate the motion feature correlation at different time offsets, and find the optimal time offset value that maximizes the correlation. Let the angular velocity data of the IMU be w I (t), and the motion feature of the target sensor be w G (t). For the time offset t d , calculate the motion feature correlation after time alignment:

[0049]

[0050] Among them, Δt is the sampling period of the target sensor.

[0051] Calculate its correlation:

[0052]

[0053] Among them, Cov and Var represent covariance and variance respectively. By enumerating t d , find the best time offset value with the maximum correlation:

[0054]

[0055] To improve the accuracy of time offset estimation, in a preferred embodiment, a quadratic fitting method can be used to fit three sampling points near the correlation peak to obtain a more accurate time offset value.

[0056] (2) Spatial calibration.

[0057] Based on the time calibration, perform spatial calibration between sensors to estimate the external reference rotation matrix between sensors. Select three orthogonal directions in the IMU coordinate system as the reference directions, and calculate the correlation between the projections of the IMU motion characteristics in these directions and the projections of the target sensor motion characteristics in the corresponding directions. Assume that the three orthogonal directions in the IMU coordinate system are d1, d2, d3, and the motion characteristics of the target sensor are ω G , then there is:

[0058]

[0059] Among them, R IG is the external reference rotation matrix from the IMU to the target sensor. Calculate the external reference rotation matrix between sensors by the singular value decomposition (SVD) method:

[0060] R IG = UV T

[0061] Among them, U and V are the left singular matrix and the right singular matrix of the covariance matrix . The calculation formula of the covariance matrix is:

[0062]

[0063] Among them, N is the number of samples, and are the means of the IMU and target sensor motion characteristics respectively.

[0064] (3) Observability check.

[0065] To ensure the uniqueness of the estimated result of the extrinsic rotation matrix, it is necessary to check whether the motion data satisfies the observability condition. If there is symmetry in the motion data, it may lead to non-uniqueness of the estimated result of the extrinsic rotation matrix. The observability condition of the motion data can be judged by checking the condition number and the minimum eigenvalue of the covariance matrix:

[0066]

[0067] where ζ u and ζ b are preset thresholds for balancing the estimation accuracy and robustness.

[0068] (4) Verification of calibration results.

[0069] Verify the accuracy of the calibration results by comparing the sensor data before and after calibration. For example, fuse the calibrated radar data with the camera data and observe whether the obstacles detected by the radar are consistent with the positions of the obstacles in the camera images; fuse the calibrated GPS data with the IMU data and observe whether the trajectory of the ship is continuous and smooth. If the calibration results do not meet the requirements, readjust the installation position or sampling frequency of the sensors and re-collect data and perform calibration.

[0070] S2. Construct a joint embedding uncertain multi-modal data fusion strategy.

[0071] For the multi-modal data that has been spatio-temporally calibrated, conventional multi-modal data fusion is to learn in the existing subspace. Taking the classification task (cross-entropy loss) of multi-sensor fusion features as an example, assume is a ship multi-sensor calibration data set with a sample set size of N, and x i contains information of S modalities corresponding to the number of classification categories y i = {1, 2,..., K}, then the forward propagation equation of the model implementation process is:

[0072]

[0073] where, and are the encoded embedding representation, encoding function and Bernoulli index (simulating random modality missing) of modality S respectively; z i , f U and ψ represent the traditional common subspace deterministic joint embedding representation of the modality set , the function for calculating the joint representation z and the pooling operator respectively. Since the joint representation in the training and inference processes is the deterministic embedding z i , all belonging to the category y iAll input modal combinations are forced to learn ψ(z in the same direction as i ), and the specificity of different input modalities cannot be exploited, resulting in insufficient representation learning and poor generalization representation ability. This limitation is difficult to be compatible with the rich and diverse characteristics of multi-sensor heterogeneous data in complex oceans.

[0074] To solve the above problems, this embodiment introduces a decoupled representation learning method, using probabilistic embeddings to replace traditional deterministic embeddings to cope with the intertwined constraints of single-modal noise uncertainty in the missing modality domain. During the training process, the reparameterization trick is used to enable backpropagation, enabling the model to introduce uncertainty during training, while using the mean as the input during the inference process to relieve the direction constraint on the inference representation and enhance the model's ability to capture specific information of different modal combinations. To balance the generalization ability and uncertainty of the model, a regularization term is introduced, and the Kullback-Leibler (KL) divergence is used to constrain the approximation degree of the probabilistic embedding to the normal Gaussian distribution, thereby optimizing the model's generalization ability for multi-modal specific information. The specific design is as follows:

[0075] S21. Introduce decoupled representation learning and reparameterization to achieve backpropagation.

[0076] As Figure 2 shown, this embodiment introduces decoupled representation learning, using probabilistic embeddings to replace deterministic embeddings. First, define the probabilistic embedding z i ~p(z i |x i ) to follow a multivariate Gaussian distribution μ i and σ i are respectively predicted and estimated through the feature mapping methods f μ (z i ), f σ (z i ) (compared with the feature pooling for estimating μ i and σ i ). Secondly, since the sampling process of the joint representation of each sample is a non-differentiable operation, the reparameterization trick τ i =μ i +εσ i is used to achieve backpropagation.

[0077] During the training process, τ i is used as the input for prediction (ensuring that the sampled embedding ψ(τ i ) shares the same learning direction as ), while during the inference process, μ iAs the input (relax the directional constraints on the inference representation and enhance the model's ability to capture specific information of different modality combinations). Therefore, the forward propagation equation for implementing the training and inference processes can be rewritten as:

[0078]

[0079] S22. Constraint relaxation and Chengdu regularization representation.

[0080] In the above method for jointly representing uncertainty probabilities, the value of σ i controls the degree of relaxation of the learning direction. When σ i = 0, it degenerates into the deterministic embedding representation method of the ordinary common subspace (i.e., τ i = μ i , without relaxation); when σ i is larger, the model has a stronger ability to capture specific information of modality combinations (better generalization ability), but at the same time, it also means a larger sampling range, which may introduce higher uncertainty (the directional constraint from τ i to μ i is weaker), hindering the optimization of the parameter matrix . Therefore, in the optimization process of the above method, the present invention further introduces a regularization term, and uses the Kullback-Leibler (KL) divergence to explicitly constrain and the normal Gaussian distribution to force the two distributions to gradually approach:

[0081]

[0082] S3. Construct a multi-modal stable representation strategy under uncertainty.

[0083] Since ship multi-sensors collect data features in a complex marine environment, the single-modal noise has arbitrary uncertainty. In this embodiment, considering the uncertainty of single-modal noise, a linear mapping module is introduced to obtain the mean and variance respectively, realize the quantization modeling of modality representation, and use the KL divergence to constrain the approximation degree of the latent space representation of modality representation and the normal Gaussian distribution to ensure the semantic consistency of different modalities. Further, using the arbitrary uncertainty of single-modal noise to generate unknown sample data, positive sample pairs and negative sample pairs are constructed, and a self-supervised contrast learning strategy is introduced. By contrastive learning, the modality representation is optimized, the stability of single-modal data under uncertainty is enhanced, so that the learned single-modal representation pairs are resistant to different single-modal inputs with similar semantics, thereby improving the stability of each modality data under uncertainty. The specific design is as follows:

[0084] S31. Modality semantic consistency and same-distribution constraint.

[0085] To ensure the semantic consistency of different modalities in the same sample; meanwhile, to meet the approximation degree between the multivariate Gaussian distribution and the normal Gaussian distribution of modality s, KL divergence is introduced for constraint, and the form of the constraint objective function is constructed as follows:

[0086]

[0087] S32. Design of single-modal stable representation based on uncertainty gain.

[0088] Due to the complexity of the marine environment and the natural ambiguity of semantics, any uncertainty in single-modal noise is inevitable, but this also means the diversity of single-modal data. Therefore, in this embodiment, the arbitrary uncertainty of single-modal noise is used to generate unknown sample data, so that the learned single-modal representation is resistant (i.e., insensitive) to different single-modal inputs with similar semantics, thereby benefiting the single-modal stable representation. Specifically, for the probability distribution representation of modality s According to the reparameterization trick in the above steps Randomly sample two sample representations and (the former is used as a reference point and the latter is regarded as a matching point), and construct positive sample pairs; meanwhile, randomly select a set of mismatched points from other distribution representations j≠i, and construct negative sample pairs; further, introduce a self-supervised contrast learning strategy to achieve optimization:

[0089]

[0090] where τ and cosim represent the temperature factor and cosine similarity respectively.

[0091] S4. Multi-modal data fusion of intelligent ships.

[0092] Combining the fusion strategy of joint representation embedding and the optimization objective strategy of single-modal representation with noise uncertainty, low-quality multi-modal data fusion based on Gaussian probability embedding and noise uncertainty quantification is realized. Through the joint representation strategy of probability embedding, the limitation of a single low-quality constraint is broken through, and the challenges brought by the difficulty of expressing modal diversity by deterministic embedding and the dynamic change of single-modal noise are solved. By combining the two strategies, high-robustness ship multi-sensor data fusion perception can be realized under the constraints of data missing and noise interweaving, thereby providing strong support for the efficient information acquisition and processing of intelligent ships in complex marine environments.

[0093] The optimization objective strategy of single-modal representation with fusion and noise uncertainty based on joint representation embedding designed in this embodiment is specifically as follows:

[0094]

[0095] Through the joint representation strategy of probabilistic embedding, a single-modal stable representation method using noise uncertainty is adopted to break through the limitation of a single low-quality constraint and solve the challenges brought by the difficulty of deterministic embedding in expressing modal diversity and the dynamic changes of single-modal noise. The goal is to achieve highly robust multi-sensor data fusion perception for ships under the constraints of data loss and noise interweaving.

[0096] Compared with the prior art, the method of the present invention has the following advantages:

[0097] 1. When dealing with modal loss and noise uncertainty, the prior art usually assumes that the quality of each modal data is high and the consistency is good, which results in its inability to fully exploit the specificity of each modal data, thus affecting the accuracy and robustness of the fusion result. In contrast, the present invention can effectively cope with the noise uncertainty in the modal loss domain by introducing probabilistic embedding and decoupled representation learning. With the help of spatio-temporal calibration, uncertainty modeling and self-supervised contrast learning strategies, the present invention can fully exploit the specificity of each modal data, thereby significantly improving the accuracy and robustness of the fusion result. Especially in complex marine environments, the diversity and uncertainty of sensor data further highlight the advantages of the present invention, enabling it to better meet the data fusion requirements in complex scenarios.

[0098] 2. Traditional methods rely on deterministic embedding during training and inference, which forces all input modal combinations belonging to the same class to learn embeddings in the same direction. This processing method limits the model's ability to capture specific information of different modal combinations, resulting in poor generalization representation ability. The present invention introduces probabilistic embedding and reparameterization techniques to introduce uncertainty during training, alleviating the direction constraint on the inference representation. This not only enhances the model's ability to capture specific information of different modal combinations but also further optimizes the model's generalization ability through KL divergence regularization and self-supervised contrast learning. The model can maintain stable performance under different modal data combinations, effectively improving the adaptability and reliability of intelligent ships in complex marine environments.

[0099] 3. In a complex marine environment, the diversity and uncertainty of sensor data pose great challenges to data fusion. Traditional methods often struggle to effectively handle data missing and noise interference, resulting in insufficient robustness of the fusion results. Through steps such as spatio-temporal calibration, decoupled representation learning, uncertainty modeling, and joint optimization, the present invention constructs a data fusion method that can effectively cope with the uncertainty of modal missing domain noise. Using the joint representation strategy of probabilistic embedding and the single-modal stable representation method of noise uncertainty, the present invention can achieve highly robust data fusion perception under the constraints of data missing and noise interweaving. This enables intelligent ships to achieve more efficient and reliable information acquisition and processing in complex marine environments, significantly improving their autonomous navigation capabilities, fault diagnosis accuracy, and safety warning effectiveness, providing strong technical support for the application of intelligent ships in fields such as marine resource development, environmental monitoring, and shipping safety assurance.

[0100] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0101] The above description of the embodiments is for the convenience of those of ordinary skill in the art to understand and use the invention. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative labor. Therefore, the present invention is not limited to the above embodiments, and the improvements and modifications made by those skilled in the art without departing from the scope of the present invention should be within the protection scope of the present invention.

Claims

1. A multi-modal data fusion method based on the uncertainty of modal missing domain noise, characterized in that It includes the following steps: Obtain data from multiple sensors and perform spatio-temporal online calibration; Combine the joint embedded uncertain multi-modal data fusion strategy and the multi-modal stable representation strategy under uncertainty to fuse the calibrated data; Among them, the joint embedded uncertain multi-modal data fusion strategy introduces decoupled representation learning, uses probability embeddings that follow a multivariate Gaussian distribution, and realizes backpropagation through the reparameterization trick; at the same time, a regularization term is introduced, and the KL divergence is used to constrain the approximation degree of the probability embedding and the normal Gaussian distribution; The multi-modal stable representation strategy under uncertainty quantifies and models the uncertainty of single-modal noise, introduces the KL divergence, and constrains the semantic consistency and the same distribution of different modalities; at the same time, a self-supervised contrast learning strategy is introduced to realize the single-modal stable representation learning with uncertainty gain.

2. The multimodal data fusion method based on modal missing domain noise uncertainty according to claim 1, wherein The types of sensors include inertial measurement units, global positioning systems, radars, sonars, cameras, and lidars. A unique identifier is assigned to each sensor, and its installation position and orientation information are recorded.

3. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 2, characterized in that The specific process of the spatio-temporal online calibration is as follows: Time calibration: Use the inertial measurement unit as the reference sensor for time calibration, extract the motion features of each sensor, and estimate the time offset through three-dimensional motion correlation analysis. Calculate the motion feature correlation under different time offsets to obtain the optimal time offset value that maximizes the correlation; Spatial calibration: Select three orthogonal directions in the coordinate system of the inertial measurement unit as the reference directions, calculate the correlation between the projection of the motion features of the inertial measurement unit in these directions and the projection of the motion features of the target sensor in the corresponding directions, and calculate the extrinsic rotation matrix between the sensors through the singular value decomposition method; Observability check: Judge whether the motion data meets the observability conditions by checking the condition number and the minimum eigenvalue of the covariance matrix; Calibration result verification: Verify the accuracy of the calibration result by comparing the sensor data before and after calibration; if the calibration result does not meet the requirements, readjust the installation position or sampling frequency of the sensor, and re-collect and calibrate the data.

4. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 1, characterized in that In the combined embedding uncertain multi-modal data fusion strategy, the probability embedding z i ~p(z i |x i ) follows a multivariate Gaussian distribution where x i is the data of the i-th modality, μ i is the expectation, is the variance; the reparameterized form of the probability embedding z i is defined as z i = μ i + σ i ∈ i , ∈ i is a random variable sampled from the standard normal distribution . During the training process, with the help of the reparameterization trick τ i = μ i + ∈σ i backpropagation is implemented.

5. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 4, characterized in that Introduce a regularization term in the optimization process of backpropagation Use KL divergence for explicit constraint and the normal Gaussian distribution Force the two distributions to gradually approximate:

6. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 4, characterized in that, During training, τ i is used as the input to ensure that the sampled embeddings share the same learning direction; During the inference process, μ i is used as the input to reduce the directional constraints on the inference representation.

7. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 6, characterized in that Forward propagation loss in the training and inference processes and are rewritten into the following forms respectively:

8. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 1, characterized in that, In the multi-modal stable representation strategy under uncertainty, for the deterministic single-modal representation introduce a linear mapping module and respectively obtain the mean and variance Define the latent space representation of the modal representation according to the uncertainty estimation knowledge theory to follow a multivariate Gaussian distribution Thus, uncertainty quantification modeling is realized.

9. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 8, characterized in that, Introduce the KL divergence and construct the constraint objective function as follows: Among them, the first term is the semantic consistency constraint of different modalities, and the second term is the same distribution constraint.

10. A multimodal data fusion method based on modal missing domain noise uncertainty according to claim 8, characterized in that Probability distribution representation for modality s According to the reparameterization trick Randomly sample two sample representations and Use the former as the reference point and the latter as the matching point to construct positive sample pairs; meanwhile, randomly draw a set of unmatched points from other distribution representations Construct negative sample pairs; then, introduce a self-supervised contrastive learning strategy to achieve optimization: wherein, and cosim represent a temperature factor and a cosine similarity, respectively.

Images