A multi-modal learning method, device and equipment under different perception sources and a medium

Abstract

The application discloses a multimodal learning method and device under different perception sources, equipment and medium, relates to the technical field of machine learning, and utilizes a learnable temperature parameter to adjust the semantic energy score of different modes, and utilizes the semantic energy weight score to weight the interaction features. In this process, the semantic quality of each mode is dynamically evaluated through the energy score, noise modes are suppressed, key information is highlighted, a temperature adjustment and dynamic gating mechanism are introduced to eliminate the quality difference and semantic asymmetry between modes, and adaptive feature fusion is realized. Then, a gradient adjustment factor is obtained according to the confidence ratio, and a direction consistency adjustment factor is generated according to the cosine similarity. In this process, the gradient score perception represented by the gradient adjustment factor obtained based on the confidence ratio and the gradient direction alignment represented by the direction consistency adjustment factor generated according to the cosine similarity are used to adjust the mode gradient from two dimensions, so that the overall performance of the multimodal learning is finally improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a multi-modal learning method under different perception sources, device, equipment and medium. BACKGROUND

[0002] Multi-modal learning integrates information from different perception sources (such as vision, audio, text, etc.) to improve the performance of models in video understanding, emotion recognition, human-computer interaction and other tasks; multi-modal data often leads to asymmetric information between modalities due to differences in perception mechanisms, semantic abstraction levels and data quality; the non-alignment and cross-modal long-distance dependence proposed by existing researchers are key problems in multi-modal learning; some researchers emphasize the noise, incompleteness, imbalance and quality fluctuations of low-quality data, which all weaken the fusion effect; at the present stage, traditional static fusion methods (such as splicing and summation) are difficult to adapt to modal heterogeneity, and may introduce redundant or even interfering information; and dynamic fusion methods learn modal weights, which are prone to training instability and other problems.

[0003] At the present stage, the overall performance of multi-modal models on MEAD and CREMA-D datasets is even lower than that of single modal in some scenarios, reflecting significant differences in learning dynamics between modalities; some researchers point out that multi-modal models that optimize unified learning goals and adopt joint training strategies may not perform as well as single modal models in some cases, indicating that multi-modal learning often ignores the learning dynamic differences between modalities during the optimization phase; during joint training, different modalities show obvious inconsistency in convergence speed, gradient amplitude and directionality; among them, the dominant modal gradient is strong and updates quickly, while the non-dominant modal may not be effectively optimized due to weak or inconsistent gradient direction; this imbalance in the optimization process often leads to excessive dependence on the dominant modal, which weakens the potential of modal complementarity and limits the expressive power of the fusion mechanism.

[0004] Therefore, the current multi-modal learning method has poor fusion effect when fusing different modal information due to the difficulty in adapting to the modal heterogeneity between modalities, i.e., the difficulty in adapting to quality differences and semantic asymmetry, and at the same time, only focuses on gradient amplitude balance during multi-modal training, ignoring gradient direction consistency, resulting in poor overall performance of multi-modal learning. SUMMARY

[0005] The embodiments of the present application provide a multi-modal learning method under different perception sources, device, equipment and medium, which can solve the problems in the prior art.

[0006] The embodiments of the present application provide a multi-modal learning method under different perception sources, which includes the following steps:

[0007] The perception data of different perception sources are acquired, and modal feature extraction is performed on the perception data to obtain different modal features; wherein the different modal features include audio modal features and visual modal features;

[0008] The different modal features are aligned in a semantic space to capture complementary information between different modalities, to obtain interaction features of different modalities; information expression strength of the interaction features of different modalities is calculated to obtain semantic energy scores of different modalities; a learnable temperature parameter is used to adjust the semantic energy scores of different modalities to control energy sensitivity, to obtain semantic energy weight scores of different modalities, and the interaction features are weighted using the semantic energy weight scores to obtain weighted interaction features of different modalities; weights of the interaction features of different modalities are obtained according to the weighted interaction features of different modalities, and the interaction features of different modalities are weighted based on the weights to obtain fusion features;

[0009] The learnable temperature parameter is an annealing temperature, which corresponds to a critical point of a system energy function in a simulated annealing algorithm, and is gradually reduced in a cosine annealing manner in learning;

[0010] Confidence of perception results predicted by the different modal features alone and confidence of perception results predicted by the fusion features are calculated respectively, and a ratio of the confidence of the single prediction to the confidence of the fusion prediction is taken as a confidence ratio; a gradient adjustment factor is calculated from a hyperbolic tangent function and a temperature parameter according to the confidence ratio; a cosine similarity between different modal gradient vectors is obtained to generate a direction consistency adjustment factor; the modal gradient in the fusion features is adjusted according to the gradient adjustment factor and the direction consistency adjustment factor to obtain final features, and multi-modal learning is completed.

[0011] Preferably, the interaction features of different modalities are obtained by:

[0012] The multi-modal features are , , both of which have a dimension of 512;

[0013] Two layers of linear transformation and normalization operation are used to map the features of the two modalities into a unified space, and the formula is represented as:

[0014] ;

[0015] Wherein: represents a LayerNorm layer; represents an activation function; W x , W y represents linear transformation parameters; , represents the mapped modal features;

[0016] Take attention as an example, define the query key value mapping, build cross attention alignment mechanism, and the formula is expressed as:

[0017] ;

[0018] Where: , , ; Mutual information alignment is performed on and using residual enhancement, and the formula is expressed as:

[0019] ;

[0020] Where: is used to calculate the attention distribution; Similarly, symmetric operation is performed on to obtain the updated features of the interaction modal .

[0021] Preferably, the obtained weight interaction features of different modalities include:

[0022] The interaction features of different modalities are expressed as:

[0023] ;

[0024] Where: is expressed as the feature after visual modal interaction; is expressed as the feature after audio modal interaction; The combination of and is input into the energy network to generate energy score, and the formula is expressed as:

[0025] ;

[0026] Where: is the function of the energy network, which is transformed by two layers of nonlinear transformation and outputs the energy score corresponding to each modality, and the formula is expressed as:

[0027] ;

[0028] Where: ; and are expressed as bias terms;

[0029] The temperature parameter is used to adjust the semantic energy score of different modalities to control the energy sensitivity, and the formula is expressed as: ​​

[0030] ;

[0031] wherein: represents a function; and represents a weight; a temperature parameter is an annealing temperature;

[0032] The semantic energy weight score is utilized to weight the interaction features of different modalities to obtain weighted interaction features of different modalities. and The interaction features of different modalities are weighted to obtain weighted interaction features of different modalities.

[0033] Preferably, the obtaining of the fusion features comprises:

[0034] The interaction features of different modalities and the weighted interaction features are input into a two-dimensional neural gating fusion network, and a formula representation of an input vector of the two-dimensional neural gating fusion network is:

[0035] ;

[0036] The two-dimensional neural gating fusion network maps the input vector into a modal fusion weight , and a formula representation of a gating fusion weight calculation is:

[0037] ;

[0038] wherein: and are weights calculated by a two-dimensional gating neural network ; is a normalization constraint; the weights and are preliminarily weighted and fused to the interaction features of different modalities, and a formula representation of the preliminary fusion is:

[0039] ;

[0040] The fusion features are spliced with a modal residual vector to obtain a 2 d dimensional vector, and the spliced 2 d dimensional vector is mapped back to a 3 d dimensional space through a feedforward neural network to output a final enhanced fusion feature , and a formula representation is:

[0041] ;

[0042] wherein: represents splicing; represents a learning function.

[0043] Preferably, the gradient adjustment factor is obtained by:

[0044] Let the audio modality prediction of the model output be , the visual modality prediction be , the fusion prediction be , and the corresponding true label be ; define the confidence score of each modality, which is expressed as:

[0045] ;

[0046] wherein: denotes the number of samples in the mini-batch; the Softmax function is used to convert the vector into a probability distribution; denotes the probability value corresponding to the true label selected from the Softmax output;

[0047] The confidence score of the fusion prediction is obtained as s y , and the confidence ratio is defined as p v , p a , and the gradient adjustment factor is obtained based on the hyperbolic function compression, the gradient adjustment factor of the audio modality and the gradient adjustment factor of the visual modality are respectively expressed as:

[0048] ;

[0049] ;

[0050] wherein: denotes the temperature control hyperparameter.

[0051] Preferably, the direction consistency adjustment factor is generated by:

[0052] The cosine similarity of the two modality gradients is calculated as , which is expressed as:

[0053] ;

[0054] wherein: denotes the dot product of the two gradients, denotes the norm of the two gradients, respectively;

[0055] The calculated cosine similarity is set as the direction consistency adjustment factor.

[0056] Preferably, the modal gradient in the fusion feature is adjusted, represented as:

[0057] The gradient adjustment factor and the direction consistency adjustment factor are combined to adjust the modal gradient, and the formula is represented as:

[0058] ;

[0059] Wherein: represents the gradient of the loss function of the model ; represents a coefficient for adjusting the intensity; represents an optional Gaussian disturbance term;

[0060] represents an angle adjustment factor, and the formula is:

[0061] g .

[0062] The embodiment of the application also provides a multi-modal learning device under different perception sources, comprising:

[0063] A feature extraction module is configured to acquire perception data of different perception sources, perform modal feature extraction on the perception data, and obtain different modal features; wherein the different modal features include audio modal features and visual modal features.

[0064] A fusion module is configured to perform semantic space alignment on the different modal features, capture complementary information between different modalities, obtain interaction features of different modalities, calculate information expression strength of the interaction features of different modalities, obtain semantic energy scores of different modalities, adjust the semantic energy scores of different modalities by using a learnable temperature parameter to control energy sensitivity, obtain semantic energy weight scores of different modalities, weight the interaction features by using the semantic energy weight scores to obtain weighted interaction features of different modalities, obtain weights of the interaction features of different modalities according to the weighted interaction features of different modalities, and weight the interaction features of different modalities based on the weights to obtain fusion features.

[0065] Wherein, the learnable temperature parameter is an annealing temperature, which corresponds to a critical point of a system energy function in a simulated annealing algorithm, and gradually decreases in a cosine annealing manner in learning.

[0066] The optimization learning module is used for calculating confidence degrees of the perception results predicted by the different modal features respectively and the perception results predicted by the fused features respectively, and taking a ratio of the confidence degrees of the individual prediction to the confidence degrees of the fused prediction as a confidence ratio.

[0067] The embodiment of the present application further provides an electronic device comprising a memory and a processor.

[0068] The memory is used for storing a computer program.

[0069] The processor is used for executing the computer program stored in the memory, so as to realize the steps of the multi-modal learning method under different perception sources.

[0070] The embodiment of the present application further provides a computer readable storage medium used for storing a computer program, wherein the computer program is executed by a processor to realize the steps of the multi-modal learning method under different perception sources.

[0071] The embodiment of the present application provides a multi-modal learning method under different perception sources, a device, an apparatus and a medium.

[0072] The embodiment of the present application obtains semantic energy scores of different modalities by calculating information expression strengths of interactive features of different modalities, adjusts the semantic energy scores of different modalities by using a learnable temperature parameter to control energy sensitivity, obtains semantic energy weight scores of different modalities, weights the interactive features by using the semantic energy weight scores to obtain weighted interactive features of different modalities, and adaptively fuses the interactive features and the weighted interactive features. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 This is a schematic diagram illustrating the overall performance of the existing multimodal model provided in this embodiment of the invention on the MEAD and CREMA-D datasets;

[0074] Figure 2 A schematic diagram of the overall framework of DAGMP for a multimodal learning method under different sensing sources provided in an embodiment of the present invention;

[0075] Figure 3 This is a schematic diagram of the EEDF policy processing flow architecture of a multimodal learning method under different sensing sources provided in an embodiment of the present invention;

[0076] Figure 4 This is a schematic diagram of the DEA mechanism processing flow architecture of a multimodal learning method under different sensing sources provided in an embodiment of the present invention;

[0077] Figure 5 A schematic diagram illustrating the t-SNE feature visualization results of a multimodal learning method under different perceptual sources provided in an embodiment of the present invention;

[0078] Figure 6 This diagram illustrates the performance comparison of a gradient modulation method on the CREMA-D and AVE datasets for a multimodal learning method under different sensing sources provided in this embodiment of the invention; where (a) is the result on the CREMA-D dataset and (b) is the result on the AVE dataset. Detailed Implementation

[0079] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0080] Despite the progress made in many fields of multimodal learning, effectively fusing information from different modalities to improve the performance of downstream tasks still faces challenges. Multimodal data often leads to information asymmetry between modalities due to differences in perceptual mechanisms, semantic abstraction levels, and data quality. Tsai et al. pointed out that non-alignment and long-distance cross-modal dependencies are key challenges in multimodal learning. Zhang et al. emphasized that noise, incompleteness, imbalance, and quality fluctuations in low-quality data can all weaken the fusion effect. Traditional static fusion methods (such as splicing and summation) are difficult to adapt to modal heterogeneity and may introduce redundant or even interfering information. Dynamic fusion methods learn modal weights, which can easily cause problems such as training instability.

[0081] Recent researches further reveal that there is a significant imbalance problem in the training optimization phase of multi-modal learning; as shown in Figure 1 the overall performance of multi-modal models is even lower than that of single modal in some scenarios, reflecting the significant difference in learning dynamics between modalities; Wang et al. pointed out that multi-modal models that optimize unified learning objectives and adopt joint training strategies may not perform as well as single modal models in some cases; indicating that multi-modal learning often ignores the learning dynamic differences between modalities in the optimization phase; during joint training, different modalities show obvious inconsistency in convergence speed, gradient amplitude and directionality; among them, the dominant modal gradient is strong and updated quickly, while the non-dominant modal may not be effectively optimized due to weak or inconsistent gradient; this imbalance in the optimization process often leads to excessive dependence on the dominant modal, which weakens the potential of modal complementarity and limits the expressive ability of the fusion mechanism.

[0082] Multi-modal learning improves the model's expressive ability and robustness by fusing information from different perception sources; although different modalities have complementarity, they are often accompanied by noise, redundancy or missing, making it difficult to improve fusion; early methods mostly use static fusion, which is simple in structure but cannot dynamically adjust the contribution of modalities, making it difficult to deal with inconsistent modal quality; to enhance modal interaction, some studies introduce feature reconstruction and regularization mechanisms, which are efficient but lack flexibility; dynamic fusion strategies are based on attention mechanisms; they can improve semantic coordination but have problems such as large parameter overhead, difficulty in distinguishing key and noise modalities, and training instability, which are particularly evident in low light resource scenarios; from the perspective of modeling modal quality, some methods use energy functions or uncertainty measures to evaluate modal importance and introduce temperature adjustment to improve robustness to modal imbalance; although this alleviates some conflict problems, this type of method generally has the problem of disconnection between "modal weight estimation" and "feature fusion", making it difficult to achieve end-to-end optimization; in summary, constructing a robust and efficient fusion mechanism by aligning semantic and adjusting dynamic energy is still a key problem that needs to be broken through.

[0083] In addition to the fusion strategy, the optimization imbalance problem in multi-modal learning is also increasingly concerned; To alleviate this problem, current research focuses on gradient adjustment strategies, such as the OGM (Optimize Gradient Magnitudes) series of methods, which achieve collaborative training by balancing the norms of the gradients of each modality, while GradNorm controls the task learning progress through multi-task gradient normalization, which is indirectly applicable to multi-modal optimization adjustment; Unlike traditional methods that only adjust the gradient magnitude, Classifier-guided Gradient Modulation (CGGM) aims to solve the optimization imbalance problem caused by the difference in gradient update speed and direction between different modalities, achieving collaborative optimization of different modalities; But its core idea is to use a fixed classifier to guide gradient direction adjustment, which is still a static strategy and does not change dynamically with the modal learning state during the training process, making it difficult to fully adapt to the dynamic imbalance phenomenon in the multi-modal training process.

[0084] To address the inter-modal information asymmetry and imbalance problem in multi-modal learning, the present application proposes a plug-and-play component, DAGMP, which integrates the fusion- adjustment strategy, EEDF, and the gradient dynamic adjustment mechanism, DEA, which can improve the multi-modal representation ability while achieving dynamic regulation of optimization intensity, thereby enhancing the model's ability to adapt to imbalanced modalities and improving overall performance and training stability. Specifically:

[0085] I. Overall research.

[0086] The DAGMP component proposed by the present application includes two core parts, EEDF and DEA, which aims to improve the robustness and optimization balance of multi-modal learning, and its overall structure is as shown in Figure 2 First, after the audio and video features are extracted by the encoder, EEDF is used to complete the fusion, align the semantic space through cross-attention, introduce energy modeling and temperature adjustment dynamic estimation, and balance the modality weights, combined with a gating mechanism to adaptively select the fusion path, highlighting effective information and suppressing interference modalities; To address the inter-modal imbalance and optimization inconsistency problem, the DEA mechanism is introduced, which dynamically adjusts the gradient of each modality based on the fused features, combined with the gradient score ratio and direction similarity, to guide the optimization direction to be consistent and prevent overfitting of the dominant modality, forming a closed-loop regulation mechanism for fusion and optimization, and improving the overall training effect and stability.

[0087] II. Energy-based enhanced dynamic fusion

[0088] To address the problem of modal information asymmetry in multi-modal learning and the introduction of interference caused by indiscriminate fusion, inspired by the uncertain perception mechanism, the present application proposes the EEDF strategy, as shown in Figure 3The method can effectively avoid problems such as strong modal information being submerged by weak modal information and invalid modal misleading decision, and improve the effectiveness and robustness of multi-modal fusion.

[0089] 2.1 Semantic alignment.

[0090] First, the input multi-modal features , , both with a dimension of 512, are mapped into a unified space through two linear transformations and normalization operations, and the formula is as follows:

[0091] .

[0092] Among them: is a LayerNorm layer; is an activation function; W x , W y is a linear transformation parameter; , denotes the mapped modal feature; taking the attention of to as an example, the query key value mapping is defined, and the cross-attention alignment mechanism is constructed, and the formula is as follows:

[0093] .

[0094] Among them: , , ; the mutual information alignment of and is performed in a residual enhancement manner, and the formula is as follows:

[0095] .

[0096] Among them: is used to calculate the attention distribution; similarly, symmetric operations are performed on to obtain the updated modal feature .

[0097] 2.2 Energy score modeling.

[0098] In order to identify the relative contribution of each modal in different samples, the EEDF strategy introduces an energy calculation module to calculate the expression strength or discriminative potential of each modal information, so as to serve as subsequent modal weight guidance information fusion.

[0099] ​A semantic energy network is introduced to estimate the semantic quality of the modal, which takes the interaction features of the two modalities as input and outputs a scalar representing the current semantic energy; the semantic energy reflects the importance and credibility of the modal in the current context; first, the feature splicing result after receiving the modal interaction is obtained, and after feature alignment, the interaction feature formula is:

[0100] .

[0101] Wherein: represents the feature after the visual modal interaction; represents the feature after the audio modal interaction; this residual connection form fusion not only retains the semantic information of the original modal, but also introduces cross-modal interaction semantics, enhancing the discriminability and context alignment ability; after combining, it is input into the energy network to generate energy score, and its formula is:

[0102] .

[0103] The energy score is used to measure the importance or semantic strength of the current modal, which is used as the basis for adjusting the subsequent fusion and gating; wherein is the function of the energy network, which accepts the input feature combination, passes through two layers of nonlinear transformation and outputs the energy score corresponding to each modal; the specific form is represented as:

[0104] .

[0105] Wherein: ; and are bias terms; this form can capture the complex relationship between modalities, reflecting the relative credibility and importance of the two modalities in each other's context.

[0106] In order to use the energy score for modal fusion, a temperature parameter is introduced to control the energy sensitivity; used to adjust the sensitivity of the energy score, thereby affecting the model's response to different feature combinations, the formula is:

[0107] .

[0108] Wherein: represents function; and are weights used to weight the features, and the temperature is the annealing temperature, which is gradually reduced in the training using cosine annealing.

[0109] 2.3 Dynamic gating fusion and enhancement.

[0110] After obtaining the energy score, the EEDF strategy further introduces a dynamic gating mechanism to enhance the adaptability and discriminability of modal fusion and fine control the fusion using weight information. Specifically, the original interaction features and weighted interaction features are input into the gating network, and two fusion weights are output, which are normalized by the Softmax layer. The modal features modulated by the weights are input into the two-dimensional neural gating fusion network to learn adaptive modal fusion weights. The formula of the gating network is:

[0111] .

[0112] The input vector is used to input the original features and weighted features into the gating network, so that the gating network can fully consider different modal features and their weights, and thus more accurately calculate the fusion weights. Then, the gating neural network maps the vector to the modal fusion weight . The formula of the gating fusion weight is:

[0113] .

[0114] wherein: and are the weights calculated by the two-dimensional gating neural network , is a normalization constraint to avoid one modal feature being too dominant after fusion and being ignored, thereby enhancing the adaptability of different modal feature fusion. The preliminary fusion is obtained, and the formula of the preliminary fusion is:

[0115] .

[0116] In order to capture the residual information between modalities and enhance the final fusion representation, an enhancement module is introduced. The module concatenates the fusion features with the modal residual vector, which can preserve part of the residual information. The preliminary fused features are input into a feedforward neural network for nonlinear mapping, which can learn higher-level feature representations. The concatenated 2 d dimensional vector is mapped back to the d dimensional space through the feedforward neural network, which can maintain the consistency of feature dimensions and remove redundant information. To some extent, it can avoid the increase of computational complexity and overfitting problem caused by high feature dimension, and output the final enhanced fusion features , whose formula is:

[0117] .

[0118] wherein: indicates concatenation to form a 2d intermediate vector of the dimension; is a feed-forward neural network that projects the concatenated vector back d into the d-dimensional space, outputting the final augmented feature .

[0119] The mechanism effectively improves the expression ability and discrimination performance of multi-modal representation by dynamically adjusting the fusion weight of different modalities and fusing the inter-modal residual information.

[0120] III. Gradient energy with angle.

[0121] To improve the fusion effect, the application designs a dynamic mechanism for fusing confidence and gradient consistency adjustment, dynamically regulates the gradient intensity of each modality to enhance the attention to weak modalities, and stabilizes the training process; the structure is as shown in Figure 4 .

[0122] 3.1 Modality confidence ratio adjustment.

[0123] Inspired by OGM, the relative importance of each modality to the current sample is calculated according to the prediction confidence of each modality during the training phase, and the gradient is dynamically scaled accordingly; specifically, let the audio modality prediction of the model output be , the visual modality prediction be , and the fusion prediction be , and the corresponding true label be ; the confidence score formula of each modality is expressed as:

[0124] .

[0125] Among them: denotes the number of samples in the mini-batch; the Softmax function is used to convert the vector to a probability distribution; is the probability value corresponding to the true label selected from the Softmax output; the confidence score of the fusion prediction is s y , and the confidence ratio is defined as p v , p a , and further introduce the adjustment factor based on the compression of the hyperbolic function to dynamically adjust the gradient; and The formula is expressed as:

[0126] .

[0127] .

[0128] in: This is a temperature control hyperparameter used to control the sensitivity of gradient adjustment.

[0129] When the confidence level of a certain mode is significantly higher than that of other modes, the gradient adjustment factor may excessively shrink the gradient of that mode. Adding Gaussian noise can increase the randomness of the gradient and prevent the gradient from being too small to effectively update the model. The specific formula is expressed as follows:

[0130] .

[0131] in: Represents the loss function Regarding the gradient of the parameters, It is a coefficient. This represents a vector with a mean of 0 and a variance of . Normally distributed noise, Gradient The standard deviation.

[0132] 3.2 Modal gradient direction consistency adjustment.

[0133] Besides confidence information, training consistency between modalities also affects the final fusion performance. This invention proposes an adjustment method based on gradient direction consistency, which calculates the cosine similarity of the gradients of two modalities in the current batch. To measure the consistency of its optimization direction, the formula is expressed as:

[0134] .

[0135] in: This represents the dot product of two gradients. Let represent the norms of the two gradients respectively.

[0136] 3.3 Joint gradient adjustment rules.

[0137] By combining the two adjustment mechanisms mentioned above, this invention proposes a unified gradient update strategy, the formula of which is as follows:

[0138] .

[0139] in: Representation Model The gradient of the loss function; It is a coefficient used to adjust the intensity; This is an optional Gaussian perturbation term; It is an angle adjustment factor. Further defining the angle adjustment factor, its formula is expressed as:

[0140] g .

[0141] The joint gradient adjustment rule balances the confidence strength and training consistency between different modalities, effectively improving the robustness and expressiveness of inter-modal fusion. This design enables the model to learn the information of different modalities more evenly during training, improving the generalization ability and adaptability of the model.

[0142] Specific experiments:

[0143] 1. Dataset and experimental equipment

[0144] CREMA-D (Crowd-sourced Emotional Multimodal Actors Dataset) dataset: recorded by 91 actors of different genders, ages, and races, covering 6 emotions (anger, disgust, fear, happiness, neutral, and sadness), audio and video data collected through 12 English sentences, a total of 7442 samples.

[0145] MEAD (Multimodal Emotional Actor Dataset) dataset: a structured large-scale multimodal facial expression dataset, 60 actors recorded 7 emotions under uniform conditions, 3 intensity levels (weak, medium, and strong), containing 8 perspectives, this paper only uses audio and video information from the front perspective.

[0146] AVE (Audio-Visual Event Dataset) dataset: a multimodal dataset for audio-visual event recognition, containing 4,143 video clips, covering 28 common events, and accurately labeling audio and video content and event time range.

[0147] 2. Experimental setup

[0148] In the experiment, the visual encoder uses ResNet-18 as the encoder; the video frame is first randomly cropped and resized to 224x224, then randomly flipped horizontally with a probability of 50%, and then pixel normalized and channel standardized; the preprocessed image is input into a two-dimensional convolutional neural network for feature extraction; the audio modality is sampled at 22050Hz, and Librosa is used to extract 128-dimensional Mel spectrograms, and 3 channels are expanded to adapt to the image network; audio longer than 3 seconds is randomly truncated, and audio shorter than 3 seconds is zero-padded to a fixed length; the optimizer uses the stochastic gradient descent (SGD) algorithm with momentum, the initial learning rate is set to 0.005, the momentum parameter is 0.9, and the weight decay coefficient is 1e-4; for the AVE dataset, the video clips are divided according to the event timestamp, treated as independent samples and processed according to the above process.

[0149] 3. Ablation experiment

[0150] 3.1 Ablation of each component of DAGMP.

[0151] To verify the effectiveness of each part of the proposed components, ablation experiments were designed, removing the EEDF strategy and DEA mechanism respectively, and comparing them with the single modality. Accuracy (Acc) and precision (mAP) were used for evaluation. The experimental results are shown in Table 1. The experiments using only video or audio modalities show that the performance difference between the two modalities is significant. For example, in the MEAD dataset, the accuracy of the visual modality is significantly higher than that of the audio modality, indicating that the performance of the modalities in the dataset is unbalanced. The model using only the EEDF strategy has an accuracy of 63.8% on the CREMA-D dataset, which improves to 68.1% after adding the DEA mechanism, both of which are better than the single modality. This shows that the two modules in DAGMP have a significant effect on performance improvement and enhance the information interaction and fusion ability between modalities. Finally, the complete DAGMP model achieves the best performance on all three datasets, with an accuracy of 71.1% on the CREMA-D dataset and 89.5% on the MEAD dataset, demonstrating good robustness and generalization ability.

[0152] Table 1 Ablation Experiment

[0153]

[0154] To further verify the effect of multimodal fusion on improving feature discrimination capability, this invention uses t-SNE to perform dimensionality reduction and visualization of the features extracted by the model under different modalities. The results are as follows: Figure 5 As shown, comparing the initial and final feature distributions, it can be seen that the fused features form clear clusters with obvious inter-class boundaries, which is significantly better than the chaotic and overlapping distribution effect of the initial modality. This result shows that the DAGMP proposed in this invention can effectively integrate audio and video information, improve the separability of emotion categories in the feature space, and thus provide more discriminative feature representations for multimodal learning tasks.

[0155] 4. Comparative experiment.

[0156] 4.1 Comparison of fusion strategies.

[0157] The EEDF strategy is compared with traditional fusion methods, such as baseline, concatenation, summation, and specially designed fusion methods Film and Gated; it is also compared with dynamic fusion methods such as Sfusion, DynMM, and PDF. The baseline method is a conventional fusion method that uses only activation functions to distinguish the relevant feature components of one modality from another.

[0158] To maintain fairness, all experiments used backbone networks with the same connections; comparisons of the fusion methods are shown in Table 2.

[0159] Table 2 Comparison of Fusion Methods

[0160]

[0161] On the CREMA-D and MEAD datasets, the performance of various mainstream fusion strategies was compared. Overall, the EEDF method achieved the highest classification accuracy on both datasets (69.8% and 84.3%, respectively), while also maintaining a leading or near-optimal mAP, demonstrating its strong capabilities in feature enhancement and modality synergy. The PDF method achieved the highest mAP (90.5%) on the MEAD dataset, indicating that its structure is more suitable for processing data with relatively consistent modal distributions and stable representations, and has good confidence modeling capabilities. In contrast, traditional fusion methods such as Concatenation and Sum only achieved moderate performance on both datasets, failing to effectively capture deep associations between multiple modalities. FiLM and Multi-Head Attention showed significant degradation on datasets with strong modal heterogeneity, such as CREMA-D, especially with lower accuracy, indicating certain limitations in complex modality alignment and feature difference modeling.

[0162] Furthermore, methods such as DynMM, Sfuison, and Gated showed stable performance on both datasets, with improvements in both accuracy and mAP, indicating that dynamic modeling and gating mechanisms help improve multimodal fusion results. However, they still lag behind EEDF and PDF. Overall, the experimental results fully validate the effectiveness and advantages of the EEDF method in multimodal learning.

[0163] 4.2 Incorporate DEA mechanism.

[0164] Building upon section 3.1, the DEA mechanism was incorporated, and the results are shown in Table 3. Significant improvements were achieved in both accuracy and mAP on the CREMA-D and MEAD datasets, validating the effectiveness and versatility of DEA in dynamic alignment and multimodal information enhancement. Taking EEDF (DAGMP) as an example, its accuracy and mAP on CREMA-D increased to 71.1% and 75.1%, respectively, and on MEAD, it reached 89.5% and 90.1%, achieving the best performance among all methods, indicating a highly synergistic gain effect between the DEA mechanism and the EEDF strategy. Traditional methods such as Concatenation and Sum also benefited significantly, achieving performance improvements of over 2% on both datasets. Particularly on MEAD, Sum achieved accuracy and mAP of 87.2% and 91.5%, respectively, surpassing some advanced fusion strategies. Multi-Head, which previously performed relatively poorly on CREMA-D, also showed improvement. The Attention and Baseline methods have also achieved significant improvements after the introduction of DEA; in addition, fusion methods with dynamic modeling capabilities, such as PDF, Sfuison, and DynMM, have also seen stable performance improvements after the introduction of DEA; thus, the effectiveness of the DEA mechanism in gradient adjustment is demonstrated.

[0165] Table 3. Integration of DEA Mechanism

[0166]

[0167] 4.3 Comparison with other gradient adjustments.

[0168] To ensure experimental fairness, three gradient adjustment methods—Ogm-ge, AGM, and PMR—were introduced for comparison based on the EEDF strategy. The results are as follows: Figure 6 As shown, DAGMP (Ours) achieves state-of-the-art performance on both the CREMA-D and AVE datasets. (a) shows the results on the CREMA-D dataset, and (b) shows the results on the AVE dataset. In particular, DAGMP leads in both accuracy (71.1%) and mAP (75.1%) on CREMA-D, demonstrating a stronger gradient guidance effect. Although its overall performance drops slightly on the more challenging AVE dataset, DAGMP still maintains the highest mAP (64.9%), demonstrating good robustness and generalization ability. In comparison, AGM outperforms OGM-GE and PMR, but is still inferior to DAGMP overall, indicating that the latter has better modeling ability and practical value in multimodal learning tasks.

[0169] This invention proposes Enhanced Energy-based Dynamic Fusion (EEDF): by dynamically adjusting modal weights based on energy fractions, it effectively suppresses noisy modes and highlights key information, thereby improving the discriminative ability of the fusion representation; it also proposes Gradient Energy with Angle (DEA): by adjusting modal gradients from two dimensions, gradient fraction perception and gradient direction alignment, it makes the optimization process of each mode more coordinated and alleviates the problems of over-optimization of the dominant mode and gradient inconsistency.

[0170] This invention addresses the common problems of modality data imbalance and information redundancy in multimodal learning by proposing a plug-and-play component, DAGMP, with a unified fusion strategy and optimization adjustment mechanism. This component consists of an EEDF strategy and a dual-perception-guided gradient modulation DEA mechanism, enabling dynamic and interconnected adjustment of modality representation quality and optimization intensity. In terms of feature fusion, the EEDF strategy effectively strengthens the representation of key information and suppresses redundant noisy modalities. At the optimization level, the DEA mechanism, guided by energy perception and direction consistency, alleviates the problem of uncoordinated optimization between modalities, improving the stability of the training process and the final performance. Experimental results demonstrate that the proposed method performs well on multiple benchmark datasets, validating the effectiveness and universality of DAGMP in improving model generalization ability and robustness.

[0171] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A multimodal learning method under different sensory sources, characterized in that, Includes the following steps: Acquire perceptual data from different perceptual sources and extract modal features from the perceptual data to obtain different modal features; among which, different modal features include audio modal features and visual modal features; Semantic space alignment of features from different modalities is performed to capture complementary information between modalities, resulting in interaction features of different modalities. The information expression intensity of interaction features of different modalities is calculated to obtain semantic energy scores of different modalities. The semantic energy scores of different modalities are adjusted using a learnable temperature parameter to control energy sensitivity, resulting in semantic energy weight scores of different modalities. The interaction features are then weighted using the semantic energy weight scores to obtain weighted interaction features of different modalities. Based on the weighted interaction features of different modalities, the weights of the interaction features of different modalities are obtained, and the interaction features of different modalities are weighted based on these weights to obtain fused features. Among them, the learnable temperature parameter is the annealing temperature, which corresponds to the critical point of the system energy function in the simulated annealing algorithm, and is gradually reduced in the learning process using cosine annealing. The confidence scores of the perception results predicted by individual modal features and the perception results predicted by fused features are calculated separately, and the ratio of the confidence score of individual predictions to the confidence score of fused predictions is used as the confidence ratio. Based on the confidence ratio, a gradient adjustment factor is calculated using a hyperbolic tangent function and a temperature parameter. The cosine similarity between gradient vectors of different modalities is obtained to generate a direction consistency adjustment factor. Based on the gradient adjustment factor and the direction consistency adjustment factor, the modal gradients in the fused features are adjusted to obtain the final features, thus completing multimodal learning.

2. The multimodal learning method under different sensory sources according to claim 1, characterized in that, The obtained interaction features of different modalities include: Multimodal features are , All dimensions are 512; Through two layers of linear transformation and normalization operations, the features of the two modalities are mapped to a unified space, as expressed by the formula: ； in: This represents the LayerNorm layer; Indicates the activation function; W x , W y Represents the parameters of a linear transformation; , Represents the mapped modal features; by right Taking attention as an example, we define a query key-value mapping and construct a cross-attention alignment mechanism, expressed by the formula: ； in: , , ; Using residual enhancement method to and Mutual information alignment is performed using the following formula: ； in: Used to calculate attention distribution; similarly, for Perform symmetric operations to obtain the modal after interactive updates. Features .

3. The multimodal learning method under different sensory sources according to claim 2, characterized in that, The obtained weighted interaction features of different modalities include: The interaction features of different modalities are represented as follows: ； in: Represented as features after visual modal interaction; Represented as features after audio modal interaction; and After combination, the energy network is input to generate an energy score, the formula of which is: ； in: The function is a function of the energy network, which performs two nonlinear transformations and outputs the energy score corresponding to each mode. Its formula is expressed as: ； in: ; and Represented as a bias term; Utilizing learnable temperature parameters Adjusting the semantic energy scores of different modalities to control energy sensitivity is expressed by the following formula: ； in: express function; and Indicates weight; temperature parameter This refers to the annealing temperature. Using semantic energy weight scores and The interaction features are weighted to obtain the weighted interaction features for different modalities.

4. The multimodal learning method under different sensory sources according to claim 3, characterized in that, The acquisition of the fusion features includes: The interaction features of different modalities and the weighted interaction features are input into a two-dimensional neural gated fusion network. The formula for the input vector of the two-dimensional neural gated fusion network is expressed as follows: ； Two-dimensional neural gated fusion network Map the input vector to modality fusion weights The formula for calculating the gating fusion weight is expressed as: ； in: and Through a two-dimensional gated neural network The calculated weights; For normalization constraints; based on weights and The interaction features of different modalities are initially weighted and fused, and the initial fusion formula is expressed as follows: ； Fusion features Concatenating it with the modal residual vector yields 2 d 2-dimensional vectors, and concatenate the resulting 2 d 1D vector is mapped back through a feedforward neural network d 3D space, outputting the final enhanced fusion features. Its formula is expressed as: ； in: This indicates that the two will be joined together; This represents the learning function.

5. The multimodal learning method under different sensory sources according to claim 1, characterized in that, The acquisition of the gradient adjustment factor includes: Let the audio modal prediction output by the model be... Visual modality prediction is fusion prediction The corresponding real label is The confidence score for each modality is defined by the following formula: ； in: This represents the number of samples in the mini-batch; the Softmax function is used to convert the vector into a probability distribution. This indicates selecting the label from the Softmax output that matches the true label. The corresponding probability value; The confidence score of the fusion prediction is obtained as follows s y And the confidence ratio is defined as ρ v , ρ a The gradient adjustment factor is obtained based on hyperbolic function compression, and the gradient adjustment factor of the audio modality is obtained. Gradient adjustment factor for visual modality They are represented as follows: ； ； in: This indicates that the temperature control parameters are exceeded.

6. The multimodal learning method under different sensory sources according to claim 5, characterized in that, The generation of the direction consistency adjustment factor includes: Calculate the cosine similarity of the gradients of two modes. Its formula is expressed as: ； in: This represents the dot product of two gradients. Let these represent the norms of the two gradients respectively; The calculated cosine similarity Set as the directional consistency adjustment factor.

7. The multimodal learning method under different sensory sources according to claim 6, characterized in that, The modal gradient in the adjusted fusion feature is expressed as: Modal gradient adjustment is performed by combining the gradient adjustment factor and the orientation consistency adjustment factor, and the formula is expressed as follows: ； in: Representation Model The gradient of the loss function; This indicates the strength of a coefficient used for adjustment; This represents an optional Gaussian perturbation term; The adjustment factor representing an angle is expressed by the following formula: γ 。 8. A multimodal learning device for different sensory sources, characterized in that, include: The feature extraction module is used to acquire perceptual data from different perceptual sources and extract modal features from the perceptual data to obtain different modal features; among which, different modal features include audio modal features and visual modal features; The fusion module is used to semantically align features from different modalities, capture complementary information between different modalities, and obtain interaction features of different modalities; calculate the information expression intensity of interaction features of different modalities to obtain semantic energy scores of different modalities; adjust the semantic energy scores of different modalities using a learnable temperature parameter to control energy sensitivity, obtain semantic energy weight scores of different modalities, and use the semantic energy weight scores to weight the interaction features to obtain weighted interaction features of different modalities; obtain the weights of interaction features of different modalities based on the weighted interaction features of different modalities, and weight the interaction features of different modalities based on these weights to obtain fused features; Among them, the learnable temperature parameter is the annealing temperature, which corresponds to the critical point of the system energy function in the simulated annealing algorithm, and is gradually reduced in the learning process using cosine annealing. The optimization learning module is used to calculate the confidence scores of the perception results predicted by individual modal features and the perception results predicted by fused features, and to use the ratio of the confidence scores of individual predictions to the confidence scores of fused predictions as the confidence ratio. Based on the confidence ratio, a gradient adjustment factor is calculated using a hyperbolic tangent function and a temperature parameter. The cosine similarity between gradient vectors of different modalities is obtained to generate a direction consistency adjustment factor. Based on the gradient adjustment factor and the direction consistency adjustment factor, the modal gradients in the fused features are adjusted to obtain the final features, thus completing multimodal learning.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the steps of the multimodal learning method under different sensory sources as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the steps of a multimodal learning method under different sensory sources as described in any one of claims 1 to 7.

Images