Multi-modal data fusion method and device based on dynamic self-adaptive correction

Abstract

This invention provides a multimodal data fusion method and apparatus based on dynamic adaptive bias correction, comprising: inputting audio and video data into a single-modal encoder and a classification head respectively to obtain audio and video prediction probability vectors; concatenating audio and video features and inputting them into a multimodal fusion network via feature-level stopping gradients to obtain a multimodal joint prediction probability vector; determining the guiding modality, the guided modality, and the confidence difference using an entropy determination module; calculating the cross-modal guidance loss using a divergence weighting module, updating the parameters of the single-modal encoder and classification head based on this loss and hard labels during the single-modal update stage, and freezing the fusion network; obtaining sample-level fusion soft modulation weights through a soft divergence modulation module, constructing a weighted fusion loss by combining hard labels and the multimodal joint prediction probability vector, updating the fusion network parameters, and outputting the joint classification result. This invention can improve the bias correction efficiency of low-reliability modalities and the single-modal discrimination capability, suppress noise-induced negative transfer in fusion, and improve training stability.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and multimodal information processing technology, and in particular to a multimodal data fusion method and apparatus based on dynamic adaptive correction. Background Technology

[0002] With the rapid development of information technology, multimodal data such as audio, video, and text are experiencing explosive growth on the internet. How to effectively integrate this multi-source heterogeneous information to improve the decision-making accuracy of deep learning models in complex scenarios has become a core challenge in the fields of artificial intelligence and multimodal information processing. Multimodal fusion technology is currently widely used in key tasks such as emotion recognition, action recognition, and human-computer interaction. Its core value lies in constructing more robust feature representations than single-modal models by exploring the complementarity and correlation between different modalities.

[0003] However, existing multimodal fusion methods still have significant limitations in practical applications. Existing methods typically assume that the guidance relationship between modalities remains fixed during training, lacking a mechanism to dynamically guide the direction based on the real-time state of the samples. When the reliability of different modalities changes on different samples, using a fixed dominant modality or a fixed fusion strategy can easily lead to guidance mismatch, limiting the effective correction of weaker modalities and resulting in low correction efficiency for low-reliability modalities and low single-modal discrimination ability. Existing fusion strategies usually only directly concatenate or apply fixed weights to modal features, lacking the ability to continuously characterize and flexibly modulate the degree of conflict in modal prediction probability vectors. When there are significant semantic discrepancies between different modalities in a sample, existing multimodal fusion methods often fail to implement differentiated fusion suppression for conflicting samples, causing local noise to be directly amplified and transmitted to the fusion layer, leading to negative transfer. Furthermore, in existing training processes, single-modal guidance learning and multimodal fusion optimization are often intertwined, lacking an effective gradient cutoff mechanism, leading to mutual interference between different optimization objectives and reducing the stability of model training. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal data fusion method and apparatus based on dynamic adaptive bias correction, which solves the problems of low bias correction efficiency and low single-modal discrimination capability of existing technologies, the direct amplification of local noise and its transmission to the fusion layer to cause negative transfer phenomenon, and the low stability of model training.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a multimodal data fusion method based on dynamic adaptive bias correction, the method comprising: Audio and video data are input into a single-modal encoder and a single-modal classifier, respectively, for feature extraction and classification, resulting in audio and video prediction probability vectors. The extracted audio and video features are concatenated and a feature-level gradient stopping operation is performed before inputting them into a multimodal fusion network to obtain a multimodal joint prediction probability vector. The audio and video prediction probability vectors are then input into a cross-modal guidance module, where their information entropy is calculated. Based on the comparison of information entropy, the guiding modality, the guided modality, and the confidence difference are dynamically determined at the sample level. In the cross-modal guidance module, gradient truncation is performed on the predicted probability vector of the guiding mode to obtain a fixed constant vector, and the divergence between the fixed constant vector and the predicted probability vector of the guided mode is weighted according to the confidence difference to construct the cross-modal guidance loss. The parameter gradients of the multimodal fusion network are set to zero, and single-modal audio loss and single-modal video loss are constructed based on the real hard labels, audio prediction probability vector, video prediction probability vector and cross-modal guided loss respectively. The coefficients of single-modal audio loss and single-modal video loss are scaled to obtain single-modal total loss. The single-modal total loss is backpropagated to update the parameters of single-modal encoder and single-modal classification head. The audio prediction probability vector and the video prediction probability vector are input into the soft divergence modulation module to calculate the bidirectional symmetric divergence composed of the positive Kullback-Leibler divergence and the reverse Kullback-Leibler divergence, and generate sample-level fusion soft modulation weights through the negative exponential mapping of temperature hyperparameter modulation. Based on the real hard labels, the multimodal joint prediction probability vector, and the sample-level fusion soft modulation weights, a weighted fusion loss is constructed, and the multimodal fusion network parameters are updated by backpropagation based on the weighted fusion loss, outputting the joint classification result. A second aspect of this invention provides a multimodal data fusion device based on dynamic adaptive bias correction, the device comprising: The prediction probability vector determination module is used to input audio data and video data into a single-modal encoder and a single-modal classifier, respectively, to perform feature extraction and classification processing to obtain audio prediction probability vectors and video prediction probability vectors. After concatenating the extracted audio features and video features and performing feature-level stopping gradient operation, the concatenation is input into a multimodal fusion network to obtain a multimodal joint prediction probability vector. The dynamic determination module is used to input the audio prediction probability vector and the video prediction probability vector into the cross-modal guidance module, calculate the information entropy of the two respectively, and dynamically determine the guiding modality, the guided modality and the confidence difference at the sample level based on the comparison result of the information entropy. The gradient truncation and construction module is used in the cross-modal guidance module to perform gradient truncation on the predicted probability vector of the guiding mode to obtain a fixed constant vector, and to construct the cross-modal guidance loss by weighting the divergence between the fixed constant vector and the predicted probability vector of the guided mode according to the confidence difference. The loss determination module is used to set the parameter gradients of the multimodal fusion network to zero, and construct single-modal audio loss and single-modal video loss respectively based on the real hard labels, audio prediction probability vector, video prediction probability vector and cross-modal guided loss. The coefficients of single-modal audio loss and single-modal video loss are scaled to obtain single-modal total loss. The single-modal total loss is backpropagated to update the parameters of single-modal encoder and single-modal classification head. The calculation and generation module is used to input the audio prediction probability vector and the video prediction probability vector into the soft divergence modulation module, calculate the bidirectional symmetric divergence composed of the positive Kullback-Leibler divergence and the reverse Kullback-Leibler divergence, and generate sample-level fusion soft modulation weights through the negative exponential mapping of temperature hyperparameter modulation. The build and update module is used to construct a weighted fusion loss based on the real hard labels, the multimodal joint prediction probability vector, and the sample-level fusion soft modulation weights, and to update the multimodal fusion network parameters based on the weighted fusion loss through backpropagation, and output the joint classification result.

[0006] Compared to existing technologies, the multimodal data fusion method and apparatus based on dynamic adaptive bias correction provided by this invention inputs audio and video data into a single-modal encoder and a single-modal classifier, respectively, for feature extraction and classification processing to obtain audio and video prediction probability vectors. The extracted audio and video features are concatenated and subjected to feature-level gradient stopping operation before being input into a multimodal fusion network to obtain a joint multimodal prediction probability vector. The audio and video prediction probability vectors are then input into a cross-modal guidance module, where their information entropy is calculated. Based on the information entropy comparison results, the guiding modality, the guided modality, and the confidence difference are dynamically determined at the sample level. In the cross-modal guidance module, gradient truncation is performed on the prediction probability vector of the guiding modality to obtain a fixed constant vector. The divergence between the fixed constant vector and the prediction probability vector of the guided modality is weighted according to the confidence difference to construct a cross-modal guidance loss. The multimodal data is then processed... The gradient of the fusion network parameters is set to zero. Single-modal audio loss and single-modal video loss are constructed based on the real hard labels, audio prediction probability vector, video prediction probability vector, and cross-modal guided loss, respectively. The coefficients of the single-modal audio loss and single-modal video loss are scaled to obtain the single-modal total loss. This single-modal total loss is backpropagated to update the parameters of the single-modal encoder and single-modal classifier. The audio prediction probability vector and video prediction probability vector are input into the soft divergence modulation module to calculate the bidirectional symmetric divergence composed of the forward and reverse Kullback-Leibler divergences. Sample-level fusion soft modulation weights are generated through a negative exponential mapping of temperature hyperparameter modulation. A weighted fusion loss is constructed based on the real hard labels, the multimodal joint prediction probability vector, and the sample-level fusion soft modulation weights. Backpropagation based on the weighted fusion loss updates only the parameters of the multimodal fusion network, outputting the joint classification result.Based on the above technical process, by dynamically determining the difference between the guidance mode and the confidence level, and performing gradient truncation on the predicted probability vector of the guidance mode to construct a cross-modal guidance loss, the guidance relationship between different modes in different samples is no longer fixed through a dynamic guidance mechanism based on confidence differences. Instead, it can adaptively switch with changes in modality reliability, enabling high-reliability modes to accurately guide low-reliability modes, thereby improving the correction efficiency of low-reliability modes and the single-modality discrimination capability. Sample-level fusion soft-modulation weights are generated using bidirectional symmetric divergence to weight and modulate the fusion loss. This mechanism continuously quantifies the degree of conflict between predicted probability vectors of different modes and applies sample-level soft modulation to the fusion loss accordingly. This prevents highly conflicting samples from participating in joint optimization with the same intensity as non-conflicting samples, effectively suppressing negative fusion transfer caused by local noise. When updating single-modal parameters, the multi-modal fusion network parameters remain unchanged, and single-modal and fusion optimizations are updated alternately and independently. This training isolation mechanism ensures the update boundaries of dynamic guidance learning and fusion learning during backpropagation, improving overall training stability. Attached Figure Description

[0007] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein: Figure 1 A flowchart illustrating a multimodal data fusion method based on dynamic adaptive bias correction is shown schematically. Figure 2 A schematic diagram of the overall architecture of a multimodal fusion network is shown. Figure 3 A schematic diagram of a multimodal data fusion device based on dynamic adaptive correction is shown. Detailed Implementation

[0008] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0009] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0010] The methods described in the embodiments of the present invention will be explained in detail below.

[0011] Figure 1 The flowchart illustrating a multimodal data fusion method based on dynamic adaptive bias correction in an embodiment of the present invention is shown schematically. See [link to flowchart illustration]. Figure 1 As shown, this multimodal data fusion method based on dynamic adaptive correction may include: S101. Input the audio data and video data into the single-modal encoder and single-modal classifier respectively, and perform feature extraction and classification processing to obtain the audio prediction probability vector and the video prediction probability vector respectively. After concatenating the extracted audio features and video features and performing feature-level stopping gradient operation, input them into the multimodal fusion network to obtain the multimodal joint prediction probability vector.

[0012] Specifically, S101 includes: Step A1: Input the audio data into the audio encoder to extract the audio features. The audio features are then input into the audio classification head and the Softmax activation function to obtain the audio prediction probability vector.

[0013] Step A2: Input the video data into the video encoder to extract video features. The video features are then input into the video classification head and the Softmax activation function to obtain the video prediction probability vector.

[0014] The audio encoder and video encoder are single-mode encoders, and the audio classification head and video classification head are single-mode classification heads. Each single-mode classification head consists of a single-layer fully connected network.

[0015] Step A3: Concatenate the audio features with the video features to obtain preliminary multimodal features.

[0016] Step A4: Perform a feature-level stopping gradient operation on the initial multimodal features to obtain the gradient-trunculated multimodal features.

[0017] Step A5: Input the gradient-trimmed multimodal features into the multimodal fusion network for classification to obtain the multimodal joint prediction probability vector.

[0018] Perform feature-level stopping gradient operation on the initial multimodal features, i.e. The data is then input into a multimodal fusion network to obtain a joint multimodal prediction probability vector. .

[0019] Figure 2 A schematic diagram of the overall architecture of a multimodal fusion network is shown below. Figure 2As shown, the overall architecture of the multimodal fusion network includes: an audio encoder, a video encoder, an audio classification head, a video classification head, a multimodal fusion network, a cross-modal guidance module, and a soft divergence modulation module. In this process, audio and video data are input to the audio encoder and video encoder, respectively, to extract audio and video features. These features are then input to the audio and video classification heads, respectively, and after Softmax activation, output audio and video prediction probability vectors. They are also concatenated and subjected to a feature-level gradient stopping operation before being input to the multimodal fusion network, outputting a joint multimodal prediction probability vector. The cross-modal guidance module dynamically determines the guiding and guided modes based on the information entropy of the audio and video prediction probability vectors, and constructs a cross-modal guidance loss using the gradient-truncated guiding mode prediction probability vector, which is used to update the single-modal encoder and classification head. The soft divergence modulation module calculates the bidirectional symmetric divergence composed of the forward and reverse Kullback-Leibler divergences, and generates sample-level fusion soft modulation weights through a negative exponential mapping modulated by temperature hyperparameters. These weights are then used to modulate the multimodal fusion loss and update the multimodal fusion network.

[0020] S102. Input the audio prediction probability vector and the video prediction probability vector into the cross-modal guidance module, calculate the information entropy of the two respectively, and dynamically determine the guiding modality, the guided modality and the confidence difference at the sample level based on the comparison result of the information entropy.

[0021] Information entropy includes first information entropy and second information entropy. The information entropy corresponding to the audio prediction probability vector and the video prediction probability vector is used to characterize the prediction confidence of the corresponding modality on the current sample.

[0022] Specifically, step S102 includes: Step B1: Calculate the audio prediction probability vectors according to information theory. First information entropy and video prediction probability vector Second information entropy .

[0023] The information entropy corresponding to the prediction probability vector, namely the first information entropy of the audio prediction probability vector and the second information entropy of the video prediction probability vector, is expressed as follows: ; in, The information entropy corresponding to the predicted probability vector, For the prediction probability vector, the th The predicted probabilities of each category, The total number of categories, For category indexing.

[0024] Step B2: Compare the magnitudes of the first information entropy and the second information entropy to dynamically designate the mode corresponding to the minimum information entropy as the guiding mode and the mode corresponding to the maximum information entropy as the guided mode.

[0025] Compare the first information entropy With the second information entropy The size of the value determines whether a mode with lower information entropy is a high-reliability mode, and the high-reliability mode is dynamically designated as the guiding mode; the mode with higher information entropy is determined as a low-reliability mode, and the low-reliability mode is designated as the mode to be corrected, i.e., the guided mode.

[0026] Step B3: Calculate the difference between the information entropy of the guiding mode and the information entropy of the guided mode, and use the difference as the confidence difference.

[0027] The difference is used as the confidence level difference for assessing performance differences.

[0028] S103. In the cross-modal guidance module, gradient truncation is performed on the prediction probability vector of the guidance mode to obtain a fixed constant vector, and the divergence between the fixed constant vector and the prediction probability vector of the guided mode is weighted according to the confidence difference to construct the cross-modal guidance loss.

[0029] The predicted probability vector of the guiding modality is used as soft guidance information. After performing gradient truncation on the soft guidance information, it is used to guide the guided modality in cross-modal guidance learning, and the cross-modal guidance loss is calculated. The relationship corresponding to cross-modal guidance learning is determined at the sample level by comparing the information entropy of the predicted probability vectors of each modality. The guidance direction can be dynamically switched with different samples at different training stages.

[0030] Specifically, step S103 includes: Step C1: Use the predicted probability vector of the guiding mode as a soft label.

[0031] Soft tags contain category association information.

[0032] Step C2: Perform gradient truncation on the soft label to cut off the backpropagation path of the soft label and obtain a fixed constant vector.

[0033] Perform gradient truncation on soft labels on the physical computation graph.

[0034] Step C3: Calculate the divergence between the fixed constant vector and the predicted probability vector of the guided mode, and use the confidence difference to weight the divergence to construct the cross-modal guidance loss.

[0035] The expression for the cross-modal guidance loss is: ; ; in, The cross-modal guidance loss is dynamically determined by the information entropy corresponding to the prediction probability vector of each modality. For a single sample, only one form of cross-modal guidance loss is used to calculate the loss. This is the audio prediction probability vector. The first information entropy of the audio prediction probability vector. This is the probability vector for video prediction. The second information entropy of the video prediction probability vector. The total number of categories, For category indexing, The first in the audio prediction probability vector The predicted probabilities of each category, The first in the video prediction probability vector The predicted probabilities of each category, For the first probability vector in the video prediction probability vector The soft labels corresponding to the predicted probabilities of each category undergo gradient truncation, resulting in a fixed constant vector after the gradient truncation operation. For the first probability vector in the audio prediction vector The soft labels corresponding to the predicted probabilities of each category undergo gradient truncation, resulting in a fixed constant vector after the gradient truncation operation.

[0036] S104. Set the parameter gradient of the multimodal fusion network to zero, and construct single-modal audio loss and single-modal video loss according to the real hard label, audio prediction probability vector, video prediction probability vector and cross-modal guidance loss respectively. Scale the coefficients of the single-modal audio loss and single-modal video loss to obtain the single-modal total loss. Backpropagate the single-modal total loss to update the parameters of the single-modal encoder and single-modal classification head.

[0037] The parameters of the single-modal encoder and single-modal classifier are updated based on the single-modal total loss. During the update process, the gradient of the parameters of the multimodal fusion network is cleared to zero. The parameters of the multimodal fusion network remain unchanged during the update process to isolate the interference of single-modal optimization on the parameter update of the fusion network.

[0038] Specifically, step S104 includes: Step D1: Calculate the audio basic cross-entropy loss based on the true hard labels and the audio prediction probability vector. Calculate the video's basic cross-entropy loss based on the true hard labels and the video prediction probability vector. .

[0039] Cross-entropy loss functions include audio-based cross-entropy loss. and video basic cross-entropy loss .

[0040] Step D2: Based on the audio-based cross-entropy loss, video-based cross-entropy loss, and cross-modal guidance loss, obtain the single-modal audio loss and single-modal video loss respectively, and obtain the single-modal total loss by weighted summation.

[0041] Using the balance coefficient With guidance intensity coefficient The audio-based basic cross-entropy loss, video-based basic cross-entropy loss, and cross-modal guidance loss are linearly weighted and added together to obtain the single-modal audio loss. and single-modal video loss Finally, for and The total loss of a single mode is obtained by scaling the coefficients. .

[0042] The expressions for single-modal audio loss, single-modal video loss, and single-modal total loss are as follows: ; ; ; in, For single-modal audio loss, For single-modal video loss For the total loss of a single mode, For balance coefficient, This is the audio prediction probability vector. For real hard labels, This is the probability vector for video prediction. Let cross-entropy be the loss function. Audio prediction probability vector With real hard labels The audio-based cross-entropy loss between them Video prediction probability vector With real hard labels The video's basic cross-entropy loss between them, To guide the strength coefficient, This indicates taking the average. In For cross-modal guidance loss when video is the guiding modality, In The cross-modal guidance loss is given when the audio is the guiding modality.

[0043] Step D3: Backpropagate the total loss of the single mode to update the parameters of the single mode encoder and the single mode classifier. During the backpropagation process, set the gradient of the parameters of the multimodal fusion network to zero to keep the parameters of the multimodal fusion network unchanged.

[0044] Total loss of single mode Backpropagation is performed to calculate the gradients of the underlying unimodal encoder and classification head parameters and update the parameters; the network layers are traversed, and the gradients of all parameters involved in the multimodal network are forcibly set to zero during backpropagation to completely isolate the cross-interference caused by unimodal optimization.

[0045] S105. Input the audio prediction probability vector and the video prediction probability vector into the soft divergence modulation module, calculate the bidirectional symmetric divergence composed of the positive Kullback-Leibler divergence and the reverse Kullback-Leibler divergence, and generate sample-level fusion soft modulation weights through negative exponential mapping of temperature hyperparameter modulation.

[0046] Two-way symmetric divergence is used to characterize the degree of semantic conflict between different modalities, and the fusion soft modulation weights at the fusion sample level are the monotonic mapping results of the two-way symmetric divergence.

[0047] Specifically, S105 includes: Step 5.1: Predict the probability vector based on the audio. and video prediction probability vector Calculate the Kullback-Leibler divergence of the two modes on the predicted probabilities of each class.

[0048] Kullback-Leibler divergence includes forward Kullback-Leibler divergence and reverse Kullback-Leibler divergence.

[0049] Step 5.2: Combine the forward Kullback-Leibler divergence and the reverse Kullback-Leibler divergence to calculate the bidirectional symmetric divergence, in order to assess the degree of semantic conflict between modalities.

[0050] Step 5.3: Apply temperature hyperparameter to the two-way symmetric divergence The adjusted bidirectional symmetric divergence is mapped to sample-level fusion soft-modulation weights for adaptive adjustment using a negative exponential function. .

[0051] The expression for the two-way symmetric divergence is: ; in, It is a two-way symmetric divergence. This is the audio prediction probability vector. This is the probability vector for video prediction. The total number of categories, For category indexing, The first in the audio prediction probability vector The predicted probabilities of each category, The first in the video prediction probability vector The predicted probabilities of each category; The expression for the sample-level fusion soft modulation weights is: ; in, For sample-level fusion soft modulation weights, This is a temperature hyperparameter used to control the decay rate.

[0052] S106. Based on the real hard labels, the multimodal joint prediction probability vector, and the sample-level fusion soft modulation weights, construct a weighted fusion loss, and update the multimodal fusion network parameters based on backpropagation of the weighted fusion loss, and output the joint classification result.

[0053] Specifically, S106 includes: Step E1: Calculate the multimodal fusion basic cross-entropy loss based on the true hard labels and the multimodal joint prediction probability vector. .

[0054] Step E2: Multiply the multimodal fusion basic cross-entropy loss by the sample-level fusion soft modulation weights to obtain the weighted fusion loss. .

[0055] By using sample-level fusion soft modulation weights, the multimodal fusion basic cross-entropy loss corresponding to the joint prediction is continuously modulated to obtain the weighted fusion loss.

[0056] The expression for the weighted fusion loss is: ; in, For weighted fusion loss, For sample-level fusion soft modulation weights, This is the joint prediction probability vector for multiple modalities. Let be the basic cross-entropy loss for multimodal fusion, and let represent the joint prediction probability vector of the multimodal modes. With real hard labels The basic cross-entropy loss between them.

[0057] Step E3: Backpropagate the weighted fusion loss to update the parameters of the multimodal fusion network and output the joint classification result.

[0058] Backpropagation of the weighted fusion loss can also update the parameters of the classification head.

[0059] This invention comprises two core mechanisms. The first is a dynamic guidance mechanism, which dynamically determines the guiding and guided modes at the sample level based on the confidence differences of the output vectors of different modes, enabling the guidance direction to adaptively switch as the reliability of the samples changes. The second is a soft divergence modulation mechanism, which constructs soft modulation weights reflecting the degree of semantic conflict based on the bidirectional symmetric divergence between the prediction probability vectors of different modes, and continuously adjusts the fusion loss to reduce the adverse effects of conflicting samples on the fusion layer. Furthermore, gradient truncation operations are set in the feature fusion stage, the cross-modal guidance stage, and the single-modal backpropagation stage to ensure that the single-modal guidance optimization and the multi-modal fusion optimization do not interfere with each other during backpropagation.

[0060] This invention inputs audio and video data into corresponding single-modal encoders to extract features, and then inputs these features into corresponding single-modal classification heads to obtain audio and video prediction probability vectors. The features are concatenated and a feature-level gradient stopping operation is performed before inputting them into a multimodal fusion network to obtain multimodal prediction probability vectors. Modal confidence is calculated based on the information entropy of each modal prediction probability vector, and the guiding and guided modalities are dynamically determined at the sample level based on the confidence differences, enabling the high-reliability modality to provide switchable soft guidance to the low-reliability modality. Furthermore, the bidirectional symmetric divergence between different modal prediction probability vectors is calculated to characterize the degree of semantic conflict between modalities, and the conflict degree is mapped to soft divergence modulation weights acting on the fusion loss, thereby continuously suppressing the fusion intensity of conflicting samples. Based on this, a gradient truncation mechanism is set up distributed in the feature fusion stage, the cross-modal guidance stage, and the single-modal backpropagation stage to ensure that single-modal guidance learning and fusion learning are decoupled. This method can simultaneously improve the overall performance of multimodal fusion in complex environments and the representation quality of weak modalities.

[0061] Simulation experiments were conducted on the multimodal data fusion method based on dynamic adaptive bias correction of the present invention, and the results were compared with existing multimodal fusion methods to further illustrate the effectiveness of the present invention.

[0062] The specific simulation experimental conditions are as follows: operating system: Ubuntu 20.04, Python 3.8, experimental platform: PyTorch-1.11.0, processor: Intel Core i9-12900KF, graphics card: NVIDIA RTX 3090Ti, memory: 24GB.

[0063] The simulation experiment specifically includes: Simulation Experiment 1: Accuracy comparison experiment of multimodal fusion classification task.

[0064] It should be noted that all the following experiments were conducted in the same experimental environment. The datasets used are the CREMA-D emotion recognition dataset and the KineticSound action recognition dataset, both multimodal classification tasks. The baseline methods and the multimodal data fusion method based on dynamic adaptive correction proposed in this invention are both multimodal classification methods based on audio and video features.

[0065] Table 1. Accuracy comparison between the baseline method and the method proposed in this invention.

[0066] Table 1 shows the experimental results, comparing the overall accuracy and single-modal accuracy of the baseline method and the method proposed in this invention. The experimental results are as follows: The method proposed in this invention outperforms the baseline method in all performance metrics across different tasks. In the emotion recognition task, the multimodal accuracy is improved by 8.46%, audio accuracy by 4.07%, and video accuracy by 30.61%. In the action recognition task, the multimodal accuracy is improved by 8.62%, audio accuracy by 6.23%, and video accuracy by 16.35%. These experimental results verify the effectiveness of dynamic bias correction and soft divergence modulation in improving multimodal fusion classification results.

[0067] Simulation Experiment 2: Ablation Experiment of Main Modules of Multimodal Data Fusion Method Based on Dynamic Adaptive Correction.

[0068] It should be noted that all the following experiments were conducted in the same experimental environment. To investigate the role of each module in this invention, each module was removed and ablation experiments were performed.

[0069] Table 2 Ablation experiments of the main modules of the multimodal data fusion method based on dynamic adaptive correction.

[0070] Table 2 shows the experimental results, comparing the performance of the proposed method and its two variants in multimodal and single-modal accuracy. Removing dynamic bias correction means not introducing cross-modal soft label guidance loss based on stopping gradients in single-modal backpropagation, and removing soft divergence modulation means not using sample-level soft weights for cross-entropy modulation in the fusion classification head. The experimental results are as follows: On different datasets, removing dynamic bias correction consistently leads to a decrease in single-modal accuracy, demonstrating the importance of dynamic bias correction in mitigating the degradation of weak modality representations, and that it is independent of specific data distributions. The proposed method outperforms the method removing soft divergence modulation in multiple scenarios, verifying the crucial role of symmetric divergence, which reflects the degree of sample-level conflict, in isolating instantaneous noise and preventing negative migration in the fusion layer.

[0071] Based on the above Figure 1As can be seen from the implementation, in this embodiment of the invention, audio data and video data are input into a single-modal encoder and a single-modal classifier, respectively, for feature extraction and classification processing, to obtain audio prediction probability vectors and video prediction probability vectors. The extracted audio features and video features are concatenated and a feature-level gradient stopping operation is performed before inputting them into a multimodal fusion network to obtain a multimodal joint prediction probability vector. The audio and video prediction probability vectors are input into a cross-modal guidance module, where their information entropy is calculated, and the guiding modality, the guided modality, and the confidence difference are dynamically determined at the sample level based on the information entropy comparison results. In the cross-modal guidance module, gradient truncation is performed on the prediction probability vector of the guiding modality to obtain a fixed constant vector, and the divergence between the fixed constant vector and the prediction probability vector of the guided modality is weighted according to the confidence difference to construct the cross-modal guidance loss. The parameter gradients of the multimodal fusion network are set to zero. The system constructs single-modal audio and video losses based on real hard labels, audio prediction probability vectors, video prediction probability vectors, and cross-modal guidance loss. It then scales the coefficients of these losses to obtain the total single-modal loss, which is backpropagated to update the parameters of the single-modal encoder and classifier. The audio and video prediction probability vectors are input into a soft divergence modulation module to calculate a bidirectional symmetric divergence consisting of forward and reverse Kullback-Leibler divergences. Sample-level fusion soft modulation weights are generated through a negative exponential mapping of temperature hyperparameter modulation. A weighted fusion loss is constructed based on real hard labels, the multimodal joint prediction probability vector, and the sample-level fusion soft modulation weights. Backpropagation based on this weighted fusion loss updates only the multimodal fusion network parameters, outputting the joint classification result. Based on the above technical process, by dynamically determining the difference between the guidance mode and the confidence level, and performing gradient truncation on the predicted probability vector of the guidance mode to construct a cross-modal guidance loss, the guidance relationship between different modes in different samples is no longer fixed through a dynamic guidance mechanism based on confidence differences. Instead, it can adaptively switch as the reliability of the mode changes, enabling high-reliability modes to accurately guide the learning of low-reliability modes, thereby improving the correction efficiency of low-reliability modes and the single-mode discrimination capability. Sample-level fusion soft-modulation weights are generated using bidirectional symmetric divergence to weight and modulate the fusion loss. This continuously quantifies the degree of conflict between predicted probability vectors of different modes through a soft divergence modulation mechanism based on bidirectional symmetric divergence, and accordingly performs sample-level soft modulation on the fusion loss. This prevents highly conflicting samples from participating in joint optimization with the same intensity as non-conflicting samples, effectively suppressing negative fusion transfer caused by local noise. When updating single-modal parameters, the multi-modal fusion network parameters remain unchanged, and single-modal and fusion optimizations are updated alternately and independently. This training isolation mechanism ensures the update boundary of dynamic guidance learning and fusion learning during backpropagation, improving overall training stability.

[0072] Based on the same inventive concept, as an implementation of the above-mentioned multimodal data fusion method based on dynamic adaptive correction, this embodiment of the invention also provides a multimodal data fusion device based on dynamic adaptive correction. Figure 3 This is a structural diagram of the multimodal data fusion device based on dynamic adaptive bias correction in an embodiment of the present invention. See also... Figure 3 As shown, the device may include: The prediction probability vector determination module 301 is used to input audio data and video data into a single-modal encoder and a single-modal classification head, respectively, to perform feature extraction and classification processing to obtain audio prediction probability vectors and video prediction probability vectors; after concatenating the extracted audio features and video features and performing feature-level stopping gradient operation, the concatenation is input into a multimodal fusion network to obtain a multimodal joint prediction probability vector. The dynamic determination module 302 is used to input the audio prediction probability vector and the video prediction probability vector into the cross-modal guidance module, calculate the information entropy of the two respectively, and dynamically determine the guiding modality, the guided modality and the confidence difference at the sample level based on the comparison result of the information entropy. The gradient truncation and construction module 303 is used in the cross-modal guidance module to perform gradient truncation on the prediction probability vector of the guidance mode to obtain a fixed constant vector, and to construct the cross-modal guidance loss by weighting the divergence between the fixed constant vector and the prediction probability vector of the guided mode according to the confidence difference. The loss determination module 304 is used to set the parameter gradient of the multimodal fusion network to zero, and construct single-modal audio loss and single-modal video loss respectively based on the real hard label, audio prediction probability vector, video prediction probability vector and cross-modal guided loss. The coefficients of the single-modal audio loss and single-modal video loss are scaled to obtain the single-modal total loss. The single-modal total loss is backpropagated to update the parameters of the single-modal encoder and single-modal classification head. The calculation and generation module 305 is used to input the audio prediction probability vector and the video prediction probability vector into the soft divergence modulation module, calculate the bidirectional symmetric divergence composed of the positive Kullback-Leibler divergence and the reverse Kullback-Leibler divergence, and generate sample-level fusion soft modulation weights through the negative exponential mapping of temperature hyperparameter modulation. The construction and update module 306 is used to construct a weighted fusion loss based on the real hard labels, the multimodal joint prediction probability vector and the sample-level fusion soft modulation weights, and update the multimodal fusion network parameters based on the weighted fusion loss through backpropagation, and output the joint classification result.

[0073] The prediction probability vector determination module 301 is specifically used to input audio data into an audio encoder, extract audio features, and input the audio features into an audio classification head and a Softmax activation function to obtain an audio prediction probability vector; input video data into a video encoder, extract video features, and input the video features into a video classification head and a Softmax activation function to obtain a video prediction probability vector; concatenate the audio features and video features to obtain preliminary multimodal features; perform a feature-level stopping gradient operation on the preliminary multimodal features to obtain gradient-truncated multimodal features; and input the gradient-truncated multimodal features into a multimodal fusion network for classification to obtain a multimodal joint prediction probability vector.

[0074] The dynamic determination module 302 is specifically used to calculate the first information entropy of the audio prediction probability vector and the second information entropy of the video prediction probability vector according to the information theory mechanism; compare the magnitudes of the first information entropy and the second information entropy to dynamically designate the mode corresponding to the minimum information entropy as the guiding mode and the mode corresponding to the maximum information entropy as the guided mode; calculate the difference between the information entropy of the guiding mode and the information entropy of the guided mode, and use the difference as the confidence difference. The information entropy includes the first information entropy and the second information entropy.

[0075] The gradient truncation and construction module 303 is specifically used to take the predicted probability vector of the guiding mode as a soft label; perform gradient truncation operation on the soft label to cut off the backpropagation path of the soft label and obtain a fixed constant vector; calculate the divergence between the fixed constant vector and the predicted probability vector of the guided mode, and use the confidence difference to weight the divergence to construct the cross-modal guidance loss.

[0076] In the gradient cutoff and construction module 303, the expression for the cross-modal guided loss is: ; ; in, The cross-modal guidance loss is dynamically determined by the information entropy corresponding to the prediction probability vector of each modality. For a single sample, only one form of cross-modal guidance loss is used to calculate the loss. This is the audio prediction probability vector. The first information entropy of the audio prediction probability vector. This is the probability vector for video prediction. The second information entropy of the video prediction probability vector. The total number of categories, For category indexing, The first in the audio prediction probability vector The predicted probabilities of each category, The first in the video prediction probability vector The predicted probabilities of each category, For the first probability vector in the video prediction probability vector Gradient truncation is performed on the soft labels corresponding to the predicted probabilities of each category. For the first probability vector in the audio prediction vector Gradient truncation is performed on the soft labels corresponding to the predicted probabilities of each category.

[0077] The loss determination module 304 is specifically used to calculate the audio basic cross-entropy loss based on the real hard labels and the audio prediction probability vector; calculate the video basic cross-entropy loss based on the real hard labels and the video prediction probability vector; obtain the single-modal audio loss and single-modal video loss based on the audio basic cross-entropy loss, the video basic cross-entropy loss and the cross-modal guidance loss respectively, and obtain the single-modal total loss by weighted summation; backpropagate the single-modal total loss to update the parameters of the single-modal encoder and the single-modal classification head, and set the parameter gradient of the multimodal fusion network to zero during the backpropagation process to keep the parameters of the multimodal fusion network unchanged.

[0078] In the loss determination module 304, the expressions for single-modal audio loss, single-modal video loss, and single-modal total loss are as follows: ; ; ; in, For single-modal audio loss, For single-modal video loss For the total loss of a single mode, For balance coefficient, This is the audio prediction probability vector. For real hard labels, This is the probability vector for video prediction. Let cross-entropy be the loss function. Audio prediction probability vector With real hard labels The audio-based cross-entropy loss between them Video prediction probability vector With real hard labels The video's basic cross-entropy loss between them, To guide the strength coefficient, This indicates taking the average. In For cross-modal guidance loss when video is the guiding modality, In The cross-modal guidance loss is given when the audio is the guiding modality.

[0079] In the calculation and generation module 305, the expression for the two-way symmetric divergence is: ; in, It is a two-way symmetric divergence. This is the audio prediction probability vector. This is the probability vector for video prediction. The total number of categories, For category indexing, The first in the audio prediction probability vector The predicted probabilities of each category, The first in the video prediction probability vector The predicted probabilities of each category; The expression for the sample-level fusion soft modulation weights is: ; in, For sample-level fusion soft modulation weights, This is a temperature hyperparameter used to control the decay rate.

[0080] The construction and update module 306 is specifically used to calculate the multimodal fusion basic cross-entropy loss based on the real hard labels and the multimodal joint prediction probability vector; multiply the multimodal fusion basic cross-entropy loss by the sample-level fusion soft modulation weights to obtain the weighted fusion loss; backpropagate the weighted fusion loss to update the parameters of the multimodal fusion network and output the joint classification result.

[0081] It should be noted that the above description of the multimodal data fusion device based on dynamic adaptive bias correction is similar to the description of the above-described multimodal data fusion method based on dynamic adaptive bias correction, and has similar beneficial effects. For any technical details not disclosed in the embodiments of the multimodal data fusion device based on dynamic adaptive bias correction of this invention, please refer to the description of the multimodal data fusion method based on dynamic adaptive bias correction of this invention for understanding.

[0082] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-modal data fusion method based on dynamic adaptive correction, characterized in that, include: Audio and video data are input into a single-modal encoder and a single-modal classifier, respectively, for feature extraction and classification, resulting in audio and video prediction probability vectors. The extracted audio and video features are concatenated and a feature-level gradient stopping operation is performed before inputting them into a multimodal fusion network to obtain a multimodal joint prediction probability vector. The audio and video prediction probability vectors are then input into a cross-modal guidance module, where their information entropy is calculated. Based on the comparison of the information entropy, the guiding modality, the guided modality, and the confidence difference are dynamically determined at the sample level. In the cross-modal guidance module, gradient truncation is performed on the predicted probability vector of the guidance mode to obtain a fixed constant vector, and the divergence between the fixed constant vector and the predicted probability vector of the guided mode is weighted according to the confidence difference to construct the cross-modal guidance loss; The parameter gradients of the multimodal fusion network are set to zero, and single-modal audio loss and single-modal video loss are constructed based on the real hard labels, the audio prediction probability vector, the video prediction probability vector, and the cross-modal guided loss, respectively. The single-modal audio loss and the single-modal video loss are scaled by coefficients to obtain the single-modal total loss. The single-modal total loss is backpropagated to update the parameters of the single-modal encoder and the single-modal classification head. The audio prediction probability vector and the video prediction probability vector are input into the soft divergence modulation module to calculate the bidirectional symmetric divergence composed of the positive Kullback-Leibler divergence and the reverse Kullback-Leibler divergence, and generate sample-level fusion soft modulation weights through negative exponential mapping of temperature hyperparameter modulation. Based on the real hard labels, the multimodal joint prediction probability vector, and the sample-level fusion soft modulation weights, a weighted fusion loss is constructed, and the multimodal fusion network parameters are updated by backpropagation based on the weighted fusion loss, and the joint classification result is output.

2. The method of claim 1, wherein, The audio and video data are respectively input into a single-modal encoder and a single-modal classification head for feature extraction and classification processing, respectively, to obtain audio prediction probability vectors and video prediction probability vectors. The extracted audio and video features are concatenated, and a feature-level stopping gradient operation is performed before inputting them into a multimodal fusion network to obtain a multimodal joint prediction probability vector, including: The audio data is input into an audio encoder to extract audio features, and the audio features are then input into an audio classification head and a Softmax activation function to obtain the audio prediction probability vector. The video data is input into a video encoder to extract video features, and the video features are then input into a video classification head and the Softmax activation function to obtain the video prediction probability vector. The audio features are concatenated with the video features to obtain preliminary multimodal features; Perform the feature-level stop gradient operation on the preliminary multimodal features to obtain gradient-trunculated multimodal features; The gradient-trimmed multimodal features are input into the multimodal fusion network for classification to obtain the multimodal joint prediction probability vector.

3. The method of claim 1, wherein, The information entropy includes a first information entropy and a second information entropy. The step of inputting the audio prediction probability vector and the video prediction probability vector into the cross-modal guidance module, calculating their information entropies respectively, and dynamically determining the guiding modality, the guided modality, and the confidence difference at the sample level based on the comparison result of the information entropies includes: Based on information theory mechanisms, the first information entropy of the audio prediction probability vector and the second information entropy of the video prediction probability vector are calculated respectively. The first information entropy is compared with the second information entropy, so that the mode corresponding to the minimum information entropy is dynamically designated as the guiding mode, and the mode corresponding to the maximum information entropy is dynamically designated as the guided mode; Calculate the difference between the information entropy of the guiding mode and the information entropy of the guided mode, and use the difference as the confidence difference.

4. The method of claim 3, wherein, In the cross-modal guidance module, gradient truncation is performed on the predicted probability vector of the guided modality to obtain a fixed constant vector, and the divergence between the fixed constant vector and the predicted probability vector of the guided modality is weighted according to the confidence difference to construct the cross-modal guidance loss, including: The predicted probability vector of the guided mode is used as a soft label; The gradient truncation operation is performed on the soft label to cut off the backpropagation path of the soft label, thereby obtaining the fixed constant vector; The divergence between the fixed constant vector and the predicted probability vector of the guided mode is calculated, and the divergence is weighted using the confidence difference to construct the cross-modal guidance loss.

5. The method of claim 4, wherein, The expression for the cross-modal guidance loss is: ； ； in, The cross-modal guidance loss is dynamically determined by the information entropy corresponding to the prediction probability vector of each modality. For a single sample, only one form of cross-modal guidance loss is used to calculate the loss. Let be the audio prediction probability vector. The first information entropy of the audio prediction probability vector. Let be the video prediction probability vector. The second information entropy is the video prediction probability vector. The total number of categories, For category indexing, The first in the audio prediction probability vector The predicted probabilities of each category, The first in the video prediction probability vector The predicted probabilities of each category, For the video prediction probability vector, the first... Gradient truncation is performed on the soft labels corresponding to the predicted probabilities of each category. For the first probability vector in the audio prediction vector Gradient truncation is performed on the soft labels corresponding to the predicted probabilities of each category.

6. The multimodal data fusion method based on dynamic adaptive correction according to claim 1, characterized in that, The steps include: setting the parameter gradients of the multimodal fusion network to zero, constructing single-modal audio loss and single-modal video loss based on the real hard labels, the audio prediction probability vector, the video prediction probability vector, and the cross-modal guided loss; scaling the single-modal audio loss and the single-modal video loss to obtain the single-modal total loss; and backpropagating the single-modal total loss to update the parameters of the single-modal encoder and the single-modal classification head. Calculate the audio basic cross-entropy loss based on the real hard labels and the audio prediction probability vector; calculate the video basic cross-entropy loss based on the real hard labels and the video prediction probability vector. Based on the audio-based cross-entropy loss, the video-based cross-entropy loss, and the cross-modal guidance loss, the single-modal audio loss and the single-modal video loss are obtained respectively, and the total single-modal loss is obtained by weighted summation; The total loss of the single modality is backpropagated to update the parameters of the single modality encoder and the single modality classification head. During the backpropagation process, the parameter gradient of the multimodal fusion network is set to zero to keep the parameters of the multimodal fusion network unchanged.

7. The multimodal data fusion method based on dynamic adaptive correction according to claim 6, characterized in that, The expressions for the single-modal audio loss, the single-modal video loss, and the single-modal total loss are as follows: ； ； ； in, For the single-modal audio loss, For the single-modal video loss, The total loss for the single mode is given. For balance coefficient, Let be the audio prediction probability vector. For the actual hard label, Let be the video prediction probability vector. Let cross-entropy be the loss function. The audio prediction probability vector With the real hard label The audio-based cross-entropy loss between them The video prediction probability vector With the real hard label The video's basic cross-entropy loss between them, To guide the strength coefficient, This indicates taking the average. In The cross-modal guidance loss is defined as the video used as the guiding modality. In The cross-modal guidance loss is defined as the audio when it is the guiding modality.

8. The multimodal data fusion method based on dynamic adaptive correction according to claim 1, characterized in that, The expression for the two-way symmetric divergence is: ； in, For the two-way symmetric divergence, Let be the audio prediction probability vector. Let be the video prediction probability vector. The total number of categories, For category indexing, The first in the audio prediction probability vector The predicted probabilities of each category, The first in the video prediction probability vector The predicted probabilities of each category; The expression for the sample-level fusion soft modulation weights is: ； in, The sample-level fusion soft modulation weights, This is a temperature hyperparameter used to control the decay rate.

9. The multimodal data fusion method based on dynamic adaptive correction according to claim 1, characterized in that, The process involves constructing a weighted fusion loss based on the true hard labels, the multimodal joint prediction probability vector, and the sample-level fusion soft modulation weights, and updating the multimodal fusion network parameters through backpropagation based on the weighted fusion loss, outputting the joint classification result, including: Calculate the multimodal fusion basic cross-entropy loss based on the real hard label and the multimodal joint prediction probability vector; The weighted fusion loss is obtained by multiplying the multimodal fusion basic cross-entropy loss with the sample-level fusion soft modulation weights. The weighted fusion loss is backpropagated to update the parameters of the multimodal fusion network, and the joint classification result is output.

10. A multimodal data fusion device based on dynamic adaptive correction, characterized in that, include: The prediction probability vector determination module is used to input audio data and video data into a single-modal encoder and a single-modal classifier, respectively, to perform feature extraction and classification processing to obtain audio prediction probability vectors and video prediction probability vectors. After concatenating the extracted audio features and video features and performing feature-level stopping gradient operation, the concatenation is input into a multimodal fusion network to obtain a multimodal joint prediction probability vector. The dynamic determination module is used to input the audio prediction probability vector and the video prediction probability vector into the cross-modal guidance module, calculate the information entropy of the two respectively, and dynamically determine the guiding modality, the guided modality and the confidence difference at the sample level based on the comparison result of the information entropy. The gradient truncation and construction module is used in the cross-modal guidance module to perform gradient truncation on the prediction probability vector of the guidance mode to obtain a fixed constant vector, and to construct the cross-modal guidance loss by weighting the divergence between the fixed constant vector and the prediction probability vector of the guided mode according to the confidence difference. The loss determination module is used to set the parameter gradient of the multimodal fusion network to zero, and construct single-modal audio loss and single-modal video loss according to the real hard label, the audio prediction probability vector, the video prediction probability vector and the cross-modal guided loss respectively. The single-modal audio loss and the single-modal video loss are scaled by coefficients to obtain the single-modal total loss. The single-modal total loss is backpropagated to update the parameters of the single-modal encoder and the single-modal classification head. The calculation and generation module is used to input the audio prediction probability vector and the video prediction probability vector into the soft divergence modulation module, calculate the bidirectional symmetric divergence composed of the positive Kullback-Leibler divergence and the reverse Kullback-Leibler divergence, and generate sample-level fusion soft modulation weights through negative exponential mapping of temperature hyperparameter modulation. The construction and update module is used to construct a weighted fusion loss based on the real hard labels, the multimodal joint prediction probability vector and the sample-level fusion soft modulation weights, and update the multimodal fusion network parameters based on the weighted fusion loss through backpropagation, and output the joint classification result.

Images