A multimodal feature dynamic fusion method and system based on uncertainty estimation

Abstract

The present invention proposes a multimodal feature dynamic fusion method and system based on uncertainty estimation, which relates to the technical field of multimodal data processing. The method comprises acquiring multimodal data; extracting features from each modal data using different pre-trained models to obtain visual feature vectors, audio feature vectors, and text feature vectors, respectively; inputting the visual modal data into a feature enhancement module in the multimodal feature dynamic fusion model to obtain enhanced feature vectors; inputting the feature vectors, audio feature vectors, and text feature vectors into an uncertainty measurement module to calculate the uncertainty of each modal feature vector; and inputting each modal feature vector and its corresponding uncertainty value into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result. This method solves the static weight defects and noise perception problems of the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a multimodal feature dynamic fusion method and system based on uncertainty estimation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In today's big data landscape, multimodal data is experiencing explosive growth and a high degree of modal heterogeneity. This data, typically collected from distributed environments, contains multi-dimensional information such as visual, auditory, and text. Its effective processing is crucial for applications such as intelligent storage management and personalized service recommendations.

[0004] Currently, some research has been conducted on multimodal data processing tasks, mainly in two ways: treating dynamic data as a spatiotemporal sequence, extracting spatial dependencies from it, and representing them in a form that can be used for subsequent processing; and using a fusion framework in which the main modality guides the auxiliary modality, leveraging the complementarity between modalities to enhance representation capabilities. However, traditional research methods have the following technical problems:

[0005] The problem of inconsistent semantic strength contained in multimodal data of different samples is not taken into account.

[0006] The first is the static weight defect: in the feature fusion stage, fixed modal weights are preset based on prior knowledge, which makes the proportion of different modal information in the fusion different, and cannot flexibly handle the importance of different modal information between different samples.

[0007] The second is the lack of noise perception: it is impossible to capture the dynamic noise existing in different samples and different modal information.

[0008] These defects restrict the data processing efficiency in complex scenarios, especially in resource-constrained environments such as edge computing. There is an urgent need to develop dynamic and adaptive multimodal processing methods to adapt to different scenarios. Summary of the Invention

[0009] The purpose of the present invention is to provide a method and system for dynamic fusion of multimodal features based on uncertainty estimation. By introducing an uncertainty estimation mechanism, dynamic weight allocation of different samples and different modal information is achieved to solve the technical problem that the existing technology does not take into account the inconsistent semantic strength contained in the multimodal data of different samples.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] A first aspect of the present invention provides a multimodal feature dynamic fusion method based on uncertainty estimation, comprising:

[0012] Acquiring multimodal data, wherein the multimodal data includes visual modal data, audio modal data, and text modal data;

[0013] Different pre-trained models are used to extract features from different modal data to obtain visual feature vectors, audio feature vectors, and text feature vectors respectively;

[0014] Input all feature vectors into the trained multimodal feature dynamic fusion model and output the final multimodal feature dynamic fusion result;

[0015] The multimodal feature dynamic fusion model inputs the visual modality data into the feature enhancement module to obtain the enhanced feature vector , the feature vector , the audio feature vector and the text feature vector are input into the uncertainty measurement module to calculate the uncertainty size of each modal feature vector, and each modal feature vector and its corresponding uncertainty size are input into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result; the uncertainty size corresponding to each modal feature vector includes the data uncertainty size and the model uncertainty size.

[0016] As a further technical solution, the feature extraction of different modal data using different pre-trained models is specifically as follows:

[0017] The pre-trained model resnet18 is used to obtain the visual feature vector of the visual modality data, the pre-trained model AST is used to obtain the audio feature vector of the audio modality data, and the pre-trained model Bert is used to obtain the text feature vector of the text modality data.

[0018] As a further technical solution, the feature enhancement module includes three-stage processing of channel-time-space, specifically including:

[0019] Input the visual feature vector to the module, perform global average pooling and global maximum pooling operations, input the pooled feature vector into a multi-layer perceptron with shared parameters, sum the feature vectors processed by the multi-layer perceptron and generate a channel attention map after activation, and then perform channel-by-channel multiplication with the input visual feature vector to obtain the feature vector ;

[0020] The feature vector Perform average pooling on the spatial dimensions H and W to obtain time step features ; Through the serial multi-scale expansion of one-dimensional convolution to the time step feature Modeling is performed, and a temporal attention map is generated by activation, and then combined with the feature vector Perform channel-by-channel multiplication to obtain the feature vector ;

[0021] Pool the feature vector along the channel and time dimensions, splice them, and then generate a spatial attention map through a convolution operation followed by activation. Then, perform a per-channel multiplication operation with the feature vector to generate the final feature vector .

[0022] As a further technical solution, the process of predicting the data uncertainty size is specifically as follows:

[0023] Input the feature vectors of each modality, and introduce a set containing M induced points, where M << N and N is the number of samples;

[0024] Construct a joint Gaussian distribution, including the kernel matrix between feature vectors, the kernel matrix between feature vectors and induced points, and the kernel matrix between induced points;

[0025] Introduce a variational distribution q( ):

[0026]

[0027] where is the function value at the induced point, is the variational mean of[[ID=4[3]] and is the variational covariance of ; is a multivariate Gaussian distribution;<000008[1]]

[0028] Calculate the prediction variance to represent the data uncertainty:

[0029] <00000

[88] ]where is the calculation result of the kernel function for the input, is the input feature vector, ∈{1,2,3,…,N}; <0

[000] 093>As a further technical solution, the process of predicting the model uncertainty size is specifically as follows:

[0032] Introduce a variational distribution to approximate the true posterior distribution of the neural network parameters;

[0033] Perform Monte Carlo sampling on the variational distribution[[ID=

[79] ] Perform T sampling, each sampling extracts a set of weight parameters from the variational distribution;

[0034] Calculate the predicted probability of the t-th sampling weight parameter, t∈[1,T];

[0035] Calculate the average of all predicted probabilities to obtain the predicted mean;

[0036] Calculate the prediction variance;

[0037] Model uncertainty is defined as the diagonal elements of the covariance matrix , that is, the sample variance of the “probability of class c”, is defined as:

[0038]

[0039] in, Defined as model uncertainty, Indicates input The predicted probability distribution of For the input data under the tth dropout mask The probability value predicted as class c; is the predicted mean of the probability of class c.

[0040] As a further technical solution, each modal feature vector and its corresponding uncertainty size are input into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result, specifically:

[0041] The total uncertainty of each modal eigenvector is calculated by the uncertainty of each modal eigenvector data and the uncertainty of the model;

[0042] The reciprocal of the total uncertainty of each modal eigenvector is taken as its original weight;

[0043] The weights of all modal eigenvectors are normalized, and the final weight of each modal eigenvector during dynamic fusion is obtained by calculating the ratio of the original weight of each modal eigenvector to the weight of all modal eigenvectors.

[0044] The final weight is combined with the mean of the classifiers of each modality Perform weighted fusion and output the result of dynamic fusion of multimodal features.

[0045] As a further technical solution, the total uncertainty of each modal eigenvector is calculated by the uncertainty size of each modal eigenvector data and the uncertainty size of the model, specifically:

[0046]

[0047]

[0048] in, is the total uncertainty of the mth mode; is the data uncertainty of the mth mode; is the model uncertainty size of the mth mode; is a hyperparameter used to maintain data stability.

[0049] A second aspect of the present invention provides a multimodal feature dynamic fusion system based on uncertainty estimation, comprising:

[0050] The data acquisition module is configured to: acquire multimodal data, wherein the multimodal data includes visual modal data, audio modal data, and text modal data;

[0051] The feature extraction module is configured to: use different pre-trained models to extract features from different modal data to obtain visual feature vectors, audio feature vectors, and text feature vectors respectively;

[0052] The feature fusion module is configured to: input all feature vectors into the trained multimodal feature dynamic fusion model and output the final multimodal feature dynamic fusion result;

[0053] The multimodal feature dynamic fusion model inputs the visual modality data into the feature enhancement module to obtain the enhanced feature vector , the feature vector , the audio feature vector and the text feature vector are input into the uncertainty measurement module to calculate the uncertainty size of each modal feature vector, and each modal feature vector and its corresponding uncertainty size are input into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result; the uncertainty size corresponding to each modal feature vector includes the data uncertainty size and the model uncertainty size.

[0054] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a multimodal feature dynamic fusion method based on uncertainty estimation as described in the first aspect of the present invention.

[0055] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, it implements the steps in a multimodal feature dynamic fusion method based on uncertainty estimation as described in the first aspect of the present invention.

[0056] The technical solution of the present invention has the following beneficial effects:

[0057] The proposed multimodal feature dynamic fusion model includes feature enhancement, uncertainty estimation, feature fusion, and loss function calculation. By dynamically evaluating the strength of semantic information contained in different samples and modalities and assigning them different weights, it fully utilizes high-quality modal information while reducing the impact of low-quality modalities, achieving dynamic fusion of multimodal features with high confidence and strong interpretability.

[0058] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0060] Figure 1 Schematic diagram of the multimodal feature dynamic fusion process in Example 1;

[0061] Figure 2 Schematic diagram of the multimodal feature dynamic fusion system framework in Example 1. DETAILED DESCRIPTION

[0062] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0063] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0064] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0065] Example 1

[0066] This embodiment discloses a multimodal feature dynamic fusion method based on uncertainty estimation; Figure 1 As shown, including:

[0067] S1: Acquire multimodal data, where the multimodal data includes visual modal data, audio modal data, and text modal data;

[0068] S2: Use different pre-trained models to extract features from different modal data to obtain visual feature vectors, audio feature vectors, and text feature vectors respectively;

[0069] S3: Input all feature vectors into the trained multimodal feature dynamic fusion model and output the final multimodal feature dynamic fusion result;

[0070] Among them, the multimodal feature dynamic fusion model inputs the visual modality data into the feature enhancement module to obtain the enhanced feature vector , the feature vector , the audio feature vector and the text feature vector are input into the uncertainty measurement module to calculate the uncertainty size of each modal feature vector, and each modal feature vector and its corresponding uncertainty size are input into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result; the uncertainty size corresponding to each modal feature vector includes the data uncertainty size and the model uncertainty size.

[0071] In step S1, different multimodal data can be acquired based on the multimodal data contained in different downstream tasks. Here, visual modal data, audio modal data, and text modal data are used as examples. The acquired multimodal data can come from data sources such as video files, audio streams, and image-text pairs. Due to various issues with the acquired multimodal data, such as inconsistent temporal information, missing modalities, and the presence of significant noise, preprocessing of the multimodal data is necessary.

[0072] The acquired multimodal data consists primarily of three parts: visual, audio, and text. Different preprocessing methods are employed to address the distinct characteristics of the three modalities. Keyframe extraction is used to extract keyframes from the multimodal data and serve as visual modal information. Audio extraction is used to extract the mel-spectrogram of the audio information in the multimodal data and serve as audio modal information. Textual data, such as copywriting and comments, is used as textual modal information.

[0073] In step S2, different methods are used to extract features from different modal data according to the different information characteristics contained in the different modal data.

[0074] The 20 keyframes of each sample in the preprocessed visual modality data are used as the time step information for that sample. This data is flattened along the batch_size dimension and input into the pretrained resnet18 model to obtain a visual feature vector. The preprocessed audio modality data (one-dimensional waveform data) is divided into short time windows. A short-time Fourier transform is performed on each window to obtain the frequency components within each short window, which in turn generates a mel-spectrogram. This is then input into the pretrained AST model to obtain an audio feature vector. The preprocessed text modality data is input into the pretrained BERT model. After multiple processing steps, such as layering and embedding, the contextual representation vector for each token is output, resulting in a text feature vector.

[0075] In step S3, the constructed multimodal feature dynamic fusion model includes a feature enhancement module, an uncertainty measurement module and a multimodal dynamic fusion module. Specifically:

[0076] Feature Enhancement Module

[0077] The traditional CBAM (Convolutional Block Attention Module, channel space attention mechanism) consists of two parts, namely the channel attention mechanism and the spatial attention mechanism. It processes static image data and the dimension of the received feature vector is However, in multimodal data, the extracted key frames are input into the pre-trained model as temporal information, and the dimension of the input feature vector is B represents the batch size, T represents the number of time steps, C represents the number of channels, H represents the height, and W represents the width. Because traditional CBAM can only capture attention in the channel and spatial dimensions and cannot perceive the temporal information between frames, a temporal attention mechanism is introduced to improve CBAM.

[0078] S311: Channel Attention Mechanism

[0079] Input the visual feature vector X to the module, perform global average pooling and global maximum pooling operations, input the pooled feature vector into a multi-layer perceptron MLP with shared parameters, sum the feature vectors processed by the multi-layer perceptron MLP and generate a channel attention map after Sigmoid activation, and then perform channel-by-channel multiplication with the input visual feature vector X to obtain the feature vector ; The formula is as follows:

[0080]

[0081]

[0082] in, Represents the channel attention map, AvgPool and MaxPool are the average pooling layer and the maximum pooling layer, It is the Sigmoid activation function, which changes the pooling dimension to global pooling of T, H, and W; Represents a channel-by-channel multiplication operation.

[0083] S312: Temporal Attention Mechanism

[0084] The feature vector Perform average pooling on the spatial dimensions H and W to obtain time step features ;

[0085]

[0086] The time step features are processed by serially dilating multi-scale one-dimensional convolutions Modeling is performed, and a temporal attention graph is generated through sigmoid activation, and then combined with the feature vector Perform channel-by-channel multiplication to obtain the feature vector , complete the time step dynamic modeling; the formula is as follows:

[0087]

[0088]

[0089] in, Represents the temporal attention map; Conv1D is a one-dimensional convolutional layer.

[0090] S313: Spatial Attention Mechanism

[0091] The feature vector Pooling along the channel and time dimensions, after splicing, a 7×7 convolution operation and sigmoid activation are performed to generate a spatial attention map, which is then combined with the feature vector Perform channel-by-channel multiplication operations to accurately focus on the salient areas in the sample and generate the final feature vector ; The formula is as follows:

[0092]

[0093]

[0094] in, represents the spatial attention map; It is a 7×7 convolution kernel.

[0095] In this embodiment, the feature enhancement module outputs a global feature vector with stronger discriminability and robustness through three-stage processing of channel-time-space, which effectively improves the model's ability to express discriminative regions, key frames and global semantics.

[0096] Uncertainty Measurement Module

[0097] In artificial intelligence, uncertainty is divided into aleatoric uncertainty and epistemic uncertainty, also known as data uncertainty and model uncertainty. Data uncertainty is caused by noise inherent in the data itself and does not change with increases or decreases in the amount of data. It objectively exists in the real world. Model uncertainty is caused by an inadequate fit between model parameters or data structure and the data distribution, reflecting the model's lack of knowledge about unknown inputs.

[0098] In this embodiment, let Represents the entire training set, the enhanced feature vector , audio feature vector, and text feature vector are processed in the same way. The following only uses feature vector For example.

[0099] If the input is X, the predicted target is Y, and the model parameter is θ, then the Bayesian prediction distribution is:

[0100]

[0101] Define the total uncertainty as , can be divided into:

[0102]

[0103] in, represents the uncertainty of the data ( ), is the intrinsic variance of the output prediction for each set of model parameters θ, and then takes the expectation of the distribution of θ; represents the model uncertainty ( ), is the variance between the predicted means of the model for the same input X under different parameters θ.

[0104] S321: Calculation of data uncertainty

[0105] S3211: Feature vector after feature enhancement As a parameter input to the module, define the input sample set as X={ , , ...... }.

[0106] S3212: To improve the computational efficiency, instead of directly performing calculations on all data points, a set containing M induced points is introduced to reduce the amount of computation; where M << N and N is the number of samples.

[0107] For each induced point in the set Z, calculate its corresponding function value to represent the function output at the induced point . Efficiently approximate the posterior distribution of the Gaussian process, where the posterior distribution describes the probability distribution of the function output given the data and model parameters.

[0108] S3213: Use the induced points to construct a joint Gaussian distribution, including the kernel matrix between feature vectors , the kernel matrix between the feature vectors and the induced points , and the kernel matrix between the induced points ;

[0109]

[0110] S3214: Since the true posterior distribution is difficult to solve, introduce a variational distribution q( ):

[0111]

[0112] where, is the function value at the induced point , is the function value of the input x; is 's variational mean, is 's variational covariance; is a multivariate Gaussian distribution.

[0113] S3215: Define the approximate posterior distribution through the variational distribution:

[0114]

[0115] S3216: Calculate the predictive distribution of each input feature vector, including the predictive mean and the predictive variance :

[0116]

[0117] where,

[0118]

[0119]

[0120] in, Represents the calculation result of the kernel function on the input, is the input feature vector, is the function value of the input eigenvector, ∈{1,2,3,…,N}; The calculation results are different depending on the kernel function used. If the RBF kernel function is used, Equal to 1.

[0121] Using prediction variance Indicates the uncertainty of the data being sought.

[0122] S322: Calculation of model uncertainty

[0123] For visual feature vectors , whose true Bayesian prediction It is obtained by integrating all possible neural network parameters:

[0124]

[0125] Among them, w is all the parameters of the neural network, is the conditional probability given the parameter w; is the posterior distribution after referring to the dataset D.

[0126] S3221: Since the above integral cannot be calculated directly, the variational distribution is introduced To approximate the true posterior distribution of the neural network parameters In a neural network with Dropout, if the weights of each layer are regarded as the result of random "discarding", it is equivalent to selecting a special variational distribution:

[0127]

[0128] is the i-th “base” weight value obtained by standard optimization during the model training phase; is the Dirac indicator function; For each weight The probability of being "set to zero" is maintained as the training probability if it is not set to zero. .

[0129] S3222: For the formula:

[0130]

[0131] Sampling; Variational distribution Perform T sampling, ,Right now:

[0132]

[0133] Each sampling draws a set of weight parameters from the variational distribution;

[0134] in, Indicates that in the tth sampling from the variational distribution A set of weight parameters extracted from .

[0135] S3223: Calculate the predicted probability of the t-th sampling weight parameter, t∈[1,T].

[0136] By definition, the predicted probability of the t-th weighted sample is:

[0137]

[0138] S3224: Calculate the average of all predicted probabilities to obtain the predicted mean; calculate the predicted covariance.

[0139] The predicted mean is:

[0140]

[0141] The forecast covariance is:

[0142]

[0143]

[0144] in, is the outer product matrix under the t-th sampling, It is the expectation of external accumulation. is the outer product with respect to the mean.

[0145] S3225: Model uncertainty is defined as the diagonal elements of the covariance matrix , that is, the sample variance of the “cth class probability”,

[0146]

[0147] in, Indicates input The predicted probability distribution of For the input data under the tth dropout mask The probability value predicted as class c; is the predicted mean of the probability of class c.

[0148] This embodiment dynamically evaluates the data uncertainty of different models for different samples, measuring the noise of the input data itself and the strength of the semantic information it contains; it also evaluates model uncertainty, measuring the model's own confidence in the prediction results under different parameters. Combining the two uncertainty measures provides dynamic weights for multimodal feature fusion, ensuring the reliability of multimodal fusion.

[0149] Multimodal dynamic fusion module

[0150] S331: For the mth mode, the total uncertainty of each modal eigenvector is calculated based on the uncertainty of each modal eigenvector data and the model uncertainty prediction results:

[0151]

[0152]

[0153] in, is the total uncertainty of the mth mode; is the data uncertainty of the mth mode; is the model uncertainty size of the mth mode; is a hyperparameter used to maintain data stability.

[0154] Step S332: According to the data uncertainty and model uncertainty, obtain the initial weights of different modes. Take the reciprocal as the initial weight of the mth mode :

[0155]

[0156] Step S333: Normalize the weights of all modalities and calculate the ratio of the original weight of the mth modality to the weights of all modalities to obtain the final weight of the mth modality during dynamic fusion. :

[0157]

[0158]

[0159] Step S334: The final weight is added to the mean of each modality classifier. Perform weighted fusion and output the result of dynamic fusion of multimodal features:

[0160]

[0161] In this embodiment, when performing feature fusion on different modalities of different samples, unlike traditional methods, there is no fixed fusion weight for each modality. Instead, a dynamic weight is obtained based on the uncertainty of each modality under different samples.

[0162] Calculate the loss function for model training

[0163] Step S341: Calculate cross entropy loss

[0164] After completing the uncertainty weighted fusion and obtaining the final fused output, the fused output and each single modality output are supervised using weighted cross entropy, and then they are added together to obtain the total loss. Specifically:

[0165] Let the fused output for:

[0166]

[0167] The predicted probability after Softmax for:

[0168]

[0169] 、 Both represent the probability of classification results, respectively indicating the probability of the classification result being the cth category and the probability of the classification result being the kth category, and C indicates that there are C categories of classification results in total;

[0170] The true label is a one-hot vector , the category weight is . Then the cross entropy loss of the fusion branch is for:

[0171]

[0172] For the fusion output of the mth modality, the respective cross entropy loss is also calculated :

[0173]

[0174]

[0175] Sum the cross entropy losses of each part to get the total cross entropy loss:

[0176]

[0177]

[0178]

[0179]

[0180] 、 、 denote the cross entropy losses of visual modality, audio modality, and text modality respectively.

[0181] Step S342: Calculate ranking loss

[0182] In a pair of samples with "ranking priority", the confidence of high-quality samples should be higher than that of low-quality samples. Therefore, the ranking relationship between sample pairs is modeled and the ranking loss is calculated. :

[0183]

[0184] Among them, (i, j) is the index pair of all constructed sample pairs, is the sort direction label, is the confidence level of the prediction of the i-th sample, The offset confidence of the introduced margin.

[0185] The ranking loss is calculated by combining historical information, comparing adjacent sample pairs, and dynamically adjusting the intervals, thereby optimizing the model's ability to sort sample pairs and ensuring that the confidence of the model output can correctly reflect the relative order between samples.

[0186] Step S343: Maximize the lower bound of evidence

[0187] By maximizing the lower bound of evidence, the variational distribution is made close to the true posterior distribution, thereby optimizing indicators such as kernel function parameters, induction point locations, and variational distribution parameters.

[0188] For the mth mode, its ELBO (Variational Evidence Lower Bound) is:

[0189]

[0190] in, is the variational induced variable distribution of mode m, is the function value of the mode at the training point, is the prior induced distribution, is the supervision target of the modality; [] is the expected data fitting term, which means calculating the degree of fit of the model to the true value under the current variational distribution; KL() is the KL regularization term.

[0191] Maximizing the evidence lower bound is equivalent to minimizing the negative evidence lower bound. The overall optimization goal is:

[0192]

[0193] Here, m represents each mode.

[0194] Under limited computing resources, the prediction performance is as close as possible to the complete Gaussian distribution to ensure efficient inference.

[0195] Step S344: Network training

[0196] In the cross entropy loss , ranking loss and minimize the negative evidence lower bound Perform weighted summation to obtain the final loss function :

[0197]

[0198] in, and are two hyperparameters responsible for adjusting the proportion of ranking loss and minimum negative evidence lower bound in the total loss.

[0199] The backpropagation algorithm continuously optimizes network parameters, terminating training when the overall loss function converges to a preset threshold or reaches a plateau. The model parameters corresponding to the optimal performance are saved as the final training result, at which point the network reaches the optimal state for the current configuration.

[0200] like Figure 2 As shown in the figure, the system within the dashed box represents the main system modules. The feature extraction module uses ResNet-18 for visual modality data, AST for audio modality data, and BERT for text modality data. Visual features are input to the feature enhancement module described in S3. Each modality's features are then fed into the uncertainty measurement module to obtain the uncertainty of each modality. These features are then fed into the multimodal dynamic fusion module for dynamic fusion, and the network is trained using a loss function. Appropriate network parameters are determined, and the results are finally tested.

[0201] It should be noted that the core of the dynamic fusion method of multimodal features proposed in this embodiment is to solve the problem of inconsistent semantic strength in multimodal data from different sources, dynamically evaluate the strength of semantic information contained in different modalities under different samples, and dynamically generate respective weights belonging to different modalities. By introducing an uncertainty measurement module, the model uncertainty under different modalities is dynamically evaluated and combined with data uncertainty, a multimodal feature representation with high reliability is obtained, so that the weight of each modal feature in the fusion process can be more accurately and dynamically adjusted.

[0202] It can be subsequently applied to different fields: for example, tasks involving temporal information (audio-visual emotion recognition), multiple keyframe images and audio information of multimodal data (such as video) are input, and emotion recognition is performed based on facial expressions in the image and voice intonation in the audio; for example, in the field of industrial monitoring, the results of dynamic fusion of multimodal features can be applied to spatiotemporal anomaly detection, and ultimately make maintenance decisions; for example, in intelligent driving, the results of dynamic fusion of multimodal features can generate road topology maps and input them into the decision-making network; for example, multimodal data in medical diagnosis, such as imaging and genetic data, have different importance in different cases, and uncertainty assessment can improve diagnostic accuracy.

[0203] In these fields, the idea of ​​multimodal dynamic fusion and uncertainty measurement of each modality has an important impact on improving the performance and accuracy of tasks.

[0204] Example 2

[0205] This embodiment discloses a multimodal feature dynamic fusion system based on uncertainty estimation, including:

[0206] The data acquisition module is configured to: acquire multimodal data, wherein the multimodal data includes visual modal data, audio modal data, and text modal data;

[0207] The feature extraction module is configured to: use different pre-trained models to extract features from different modal data to obtain visual feature vectors, audio feature vectors, and text feature vectors respectively;

[0208] The feature fusion module is configured to: input all feature vectors into the trained multimodal feature dynamic fusion model and output the final multimodal feature dynamic fusion result;

[0209] The multimodal feature dynamic fusion model inputs the visual modality data into the feature enhancement module to obtain the enhanced feature vector , the feature vector , the audio feature vector and the text feature vector are input into the uncertainty measurement module to calculate the uncertainty size of each modal feature vector, and each modal feature vector and its corresponding uncertainty size are input into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result; the uncertainty size corresponding to each modal feature vector includes the data uncertainty size and the model uncertainty size.

[0210] Example 3

[0211] The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal feature dynamic fusion method based on uncertainty estimation as described in the first embodiment of the present disclosure.

[0212] Example 4

[0213] The purpose of this embodiment is to provide an electronic device. The electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for dynamic fusion of multimodal features based on uncertainty estimation as described in the first embodiment of the present disclosure are implemented.

[0214] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0215] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0216] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A multimodal feature dynamic fusion method based on uncertainty estimation, characterized in that: include: Acquiring multimodal data, wherein the multimodal data includes visual modal data, audio modal data, and text modal data; Different pre-trained models are used to extract features from different modal data to obtain visual feature vectors, audio feature vectors, and text feature vectors respectively; Input all feature vectors into the trained multimodal feature dynamic fusion model and output the final multimodal feature dynamic fusion result; The multimodal feature dynamic fusion model inputs the visual modality data into the feature enhancement module to obtain the enhanced feature vector ; The feature vector , the audio feature vector and the text feature vector are input into the uncertainty measurement module to calculate the uncertainty size of each modal feature vector, and each modal feature vector and its corresponding uncertainty size are input into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result; The uncertainty size corresponding to each modal eigenvector includes the data uncertainty size and the model uncertainty size; The data uncertainty prediction process is specifically as follows: Input the feature vectors of each modality and introduce a set containing M induced points , where M << N and N is the number of samples; Construct a joint Gaussian distribution, including a kernel matrix between eigenvectors , the kernel matrix between the eigenvectors and the induced points 、 and the nuclear matrix between the induced points ; Introducing the variational distribution q( ): in, is the function value at the induction point, for The variational mean of for The variational covariance of is a multivariate Gaussian distribution; Calculating prediction variance Expressing data uncertainty: in, is the calculation result of the kernel function on the input, is the input feature vector, ∈{1,2,3,…,N}; The model uncertainty size prediction process is specifically as follows: Introducing variational distribution Approximate the true posterior distribution of neural network parameters; Perform Monte Carlo sampling on the variational distribution Perform T sampling, each sampling extracts a set of weight parameters from the variational distribution; Calculate the predicted probability of the t-th sampling weight parameter, t∈[1,T]; Calculate the average of all predicted probabilities to obtain the predicted mean; Calculate the prediction variance; Model uncertainty is defined as the diagonal elements of the covariance matrix , that is, the sample variance of the "probability of class c", is defined as: in, Defined as model uncertainty, Indicates input The predicted probability distribution of For the input data under the tth dropout mask The probability value predicted as class c; is the predicted mean of the probability of category c; The total uncertainty of each modal eigenvector is calculated by the uncertainty of each modal eigenvector data and the uncertainty of the model; in, is the total uncertainty of the mth mode; is the data uncertainty of the mth mode; is the model uncertainty size of the mth mode; It is a hyperparameter used to maintain data stability; The reciprocal of the total uncertainty of each modal eigenvector is taken as its original weight; is the initial weight of the mth mode; The weights of all modal eigenvectors are normalized, and the final weight of each modal eigenvector during dynamic fusion is obtained by calculating the ratio of the original weight of each modal eigenvector to the weight of all modal eigenvectors. is the final weight of the mth modality dynamic fusion; The final weight is combined with the mean of the classifiers of each modality Perform weighted fusion and output the result of dynamic fusion of multimodal features.

2. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that: The method of using different pre-trained models to extract features from different modal data is as follows: The pre-trained model resnet18 is used to obtain the visual feature vector of the visual modality data, the pre-trained model AST is used to obtain the audio feature vector of the audio modality data, and the pre-trained model Bert is used to obtain the text feature vector of the text modality data.

3. The multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that: The feature enhancement module includes three-stage processing: channel-time-space, specifically including: Input the visual feature vector to the module, perform global average pooling and global maximum pooling operations, input the pooled feature vector into a multi-layer perceptron with shared parameters, sum the feature vectors processed by the multi-layer perceptron and generate a channel attention map after activation, and then perform channel-by-channel multiplication with the input visual feature vector to obtain the feature vector ; The feature vector Perform average pooling on the spatial dimensions H and W to obtain the time step features ; Through the serial multi-scale expansion of one-dimensional convolution to the time step feature Modeling is performed, and a temporal attention map is generated by activation, and then combined with the feature vector Perform channel-by-channel multiplication to obtain the feature vector ; The feature vector Pooling along the channel and time dimensions, concatenating, and generating a spatial attention map through convolution and activation, which is then combined with the feature vector Perform channel-by-channel multiplication to generate the final feature vector .

4. A multimodal feature dynamic fusion system based on uncertainty estimation, using a multimodal feature dynamic fusion method based on uncertainty estimation according to claim 1, characterized in that: include: The data acquisition module is configured to: acquire multimodal data, wherein the multimodal data includes visual modal data, audio modal data, and text modal data; The feature extraction module is configured to: use different pre-trained models to extract features from different modal data to obtain visual feature vectors, audio feature vectors, and text feature vectors respectively; The feature fusion module is configured to: input all feature vectors into the trained multimodal feature dynamic fusion model and output the final multimodal feature dynamic fusion result; The multimodal feature dynamic fusion model inputs visual modality data into a feature enhancement module to obtain an enhanced feature vector; The feature vector , the audio feature vector and the text feature vector are input into the uncertainty measurement module to calculate the uncertainty size of each modal feature vector, and each modal feature vector and its corresponding uncertainty size are input into the multimodal dynamic fusion module to obtain the multimodal feature dynamic fusion result; the uncertainty size corresponding to each modal feature vector includes the data uncertainty size and the model uncertainty size.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multimodal feature dynamic fusion method based on uncertainty estimation as described in any one of claims 1 to 3 are implemented.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the multimodal feature dynamic fusion method based on uncertainty estimation as described in any one of claims 1 to 3 are implemented.

Images