Model training method, system and equipment based on multi-modal data and medium

Abstract

The invention provides a model training method, system and device based on multi-modal data and a medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining an original feature vector of the multi-modal data in a training set; the multi-mode data at least comprises original data of three modes of video, text and audio; inputting the original feature vector into a multi-modal fusion model to perform forward propagation processing to obtain a preliminary classification result; reversely calculating a correlation value of multi-modal fusion features of at least one level layer by layer from the preliminary classification result; the multi-modal fusion feature comprises an initial fusion feature; performing weighting processing on the multi-modal fusion features of the corresponding levels based on the association values to obtain weighted fusion features; and re-inputting the weighted fusion features of each hierarchy into the corresponding subsequent hierarchy in the multi-modal fusion model, and carrying out iterative training to obtain a final multi-modal fusion model. According to the method, the accuracy and generalization ability of the model are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a model training method, system, device, and medium based on multimodal data. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal fusion classification technology has shown great potential in fields such as emotion recognition and intent recognition due to its ability to comprehensively utilize the complementarity of information from multiple modalities, including speech, text, and video. Theoretically, by fusing features from different modalities, the model can obtain richer and more robust semantic representations than a single modality. However, as the number of fused modalities increases and the complexity of features rises, while maintaining information diversity and complementarity, the model inevitably introduces a large amount of redundant information, leading to parameter inflation, increased computational complexity, and even overfitting problems.

[0003] To address the feature redundancy problem in multimodal fusion, existing technologies have increasingly focused on improving model interpretability, attempting to identify and filter key features by understanding the model's decision-making mechanisms. For example, some methods preserve task-relevant features by maximizing the mutual information between single-modal and fused features; other studies employ variational autoencoders (VAEs) to compress task-irrelevant feature dimensions by optimizing the evidence lower bound (ELBO); furthermore, graph neural networks have been used to compute causal relationships between features across modalities to demonstrate interpretability. The core idea behind these interpretability methods is to discover and enhance feature dimensions crucial to downstream classification tasks while ignoring insensitive features, aiming to reduce redundancy and improve model performance.

[0004] However, the aforementioned existing technologies share a common and significant drawback: their interpretability analysis is typically limited to calculating the correlation between two adjacent variables with a direct nonlinear mapping relationship (such as adjacent layers in a neural network). This approach cannot directly and quantitatively measure the end-to-end causal relationship between the objective function of the final downstream classification task and the deep features of the entire multimodal fusion model (especially features within complex modules such as the second encoding layer). Therefore, the interpretability of existing technologies is weak, and their ability to identify redundancy in deep features is limited, resulting in insufficient redundancy removal in deep networks. This is particularly problematic in small-sample scenarios where they struggle to effectively guide model optimization to alleviate underfitting and class imbalance issues. Summary of the Invention

[0005] In view of this, it is necessary to provide a model training method, system, device and medium based on multimodal data to solve the technical problems of incomplete feature redundancy elimination and insufficient model optimization in the existing technology.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a model training method based on multimodal data, comprising: Obtain the original feature vectors of multimodal data in the training set; the multimodal data includes at least the original data of three modalities: video, text, and audio. The original feature vector is input into a multimodal fusion model for forward propagation to obtain a preliminary classification result. The multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence. The preliminary classification result is obtained by processing the original feature vector after decomposing and concatenating shared and private features to obtain initial fusion features. Starting from the preliminary classification result, the association values ​​of at least one level of multimodal fusion features are calculated in reverse, layer by layer; the multimodal fusion features include the initial fusion features; The multimodal fusion features at the corresponding levels are weighted based on the association values ​​to obtain weighted fusion features; The weighted fusion features of each level are re-inputted into the corresponding subsequent levels of the multimodal fusion model, and iterative training is performed to obtain the final multimodal fusion model. The multimodal fusion model is evaluated and adjusted based on the test set.

[0007] In one possible implementation, inputting the original feature vector into a multimodal fusion model for forward propagation to obtain a preliminary classification result includes: The original features of each modality are input into the first coding layer to obtain the coded features of each modality; The encoded features of each modality are input into the feature decomposition fully connected layer, which decomposes them into shared features and private features of each modality. The shared and private features of each modality are concatenated to form the initial fused features; Based on the initial fusion features, the data are sequentially processed through the second encoding layer, the first fully connected fusion layer, and the second fully connected fusion layer to generate the preliminary classification result.

[0008] Secondly, the present invention also provides a model training system based on multimodal data, comprising: The acquisition module is used to acquire the original feature vectors of multimodal data in the training set; the multimodal data includes at least the original data of three modalities: video, text, and audio. The preliminary classification module is used to input the original feature vector into the multimodal fusion model for forward propagation processing to obtain a preliminary classification result. The multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence. The preliminary classification result is obtained by processing the original feature vector after decomposing and concatenating shared and private features to obtain initial fusion features. The association calculation module is used to calculate the association values ​​of at least one level of multimodal fusion features in reverse, layer by layer, starting from the preliminary classification results; the multimodal fusion features include the initial fusion features; The weighted fusion module is used to perform weighted processing on the multimodal fusion features at the corresponding level based on the association value to obtain weighted fusion features; The training and adjustment module is used to re-input the weighted fusion features of each level into the corresponding subsequent levels of the multimodal fusion model, perform iterative training to obtain the final multimodal fusion model, and evaluate and adjust the multimodal fusion model based on the test set.

[0009] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the model training method based on multimodal data as described in any of the above implementations.

[0010] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the model training method based on multimodal data described in any of the above implementations.

[0011] The beneficial effects of this invention are as follows: The model training method based on multimodal data provided by this invention first systematically applies the LRP algorithm to the deep structure of the multimodal fusion model (including the second encoding layer and the multi-head attention mechanism), breaking the limitation of traditional interpretability methods that can only analyze shallow or adjacent network layers. Through the calculated correlation values, the specific contribution of each dimension in the fusion features to the classification decision can be clearly and quantitatively revealed, making the model's "black box" decision-making process transparent. Furthermore, by using correlation values ​​to weight the fusion features, key feature dimensions beneficial to classification can be automatically enhanced, while irrelevant or redundant dimensions are suppressed, thereby reducing feature redundancy and model computational complexity at the source. This enables dynamic and accurate feature deredundancy removal, improving model efficiency and performance. Furthermore, through iterative LRP deredundancy training, the model can focus more on learning discriminative feature patterns on small sample datasets, avoiding interference from a large number of noisy features, and effectively preventing underfitting caused by too many parameters and too little data. Meanwhile, enhancing key features helps improve the model's ability to identify categories with fewer samples, effectively alleviating underfitting and class imbalance issues under low-resource data, and improving overall classification performance. Furthermore, a complete, end-to-end automated process has been developed, encompassing feature extraction, fusion, interpretability analysis, and optimized training. This process can be widely applied to various multimodal classification tasks, such as emotion recognition, intent recognition, and sentiment classification, demonstrating good versatility and scalability. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic flowchart of an embodiment of the model training method based on multimodal data provided by the present invention; Figure 2 For the present invention Figure 1 A schematic diagram of an embodiment of S100; Figure 3 This is a schematic diagram of the overall framework of the MVVAE-LRP model of the present invention; Figure 4 This is a schematic diagram of the forward propagation path of the MVVAE-LRP model of this invention; Figure 5 This is the LRP backpropagation path of the MVVAE-LRP model of this invention; Figure 6 For the present invention Figure 1A schematic diagram of an embodiment of S104; Figure 7 This is a schematic diagram of the R-value distribution of the MVVAE-LRP model of the present invention on the CH-SIMS test set based on classification-based latent space fusionEmbedding; Figure 8 This is a schematic diagram of an embodiment of the model training system based on multimodal data provided by the present invention; Figure 9 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0015] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0016] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0017] The use of the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the described embodiments can be combined with other embodiments.

[0018] This invention provides a model training method, system, device, and medium based on multimodal data, which are described below.

[0019] Figure 1 This is a schematic diagram of an embodiment of the model training method based on multimodal data provided by the present invention, as shown below. Figure 1 As shown, model training methods based on multimodal data include: S101. Obtain the original feature vectors of multimodal data in the training set; the multimodal data includes at least the original data of three modalities: video, text, and audio.

[0020] It should be noted that the raw multimodal data for each sample is obtained from the training set (such as CH-SIMS, MOSI, or MOSEI datasets). Subsequently, mature pre-trained models are used to extract features from each multimodal data point, resulting in high-dimensional raw feature vectors. Specifically, visual features are extracted from video frames to obtain visual feature vectors corresponding to the video modalities. Input text is encoded to obtain context-dependent text feature vectors, and acoustic features are extracted from audio waveforms to obtain acoustic feature vectors. The raw feature vectors must include at least the visual feature vectors. Text feature vectors Harmony acoustic eigenvectors .

[0021] S102. The original feature vector is input into the multimodal fusion model for forward propagation processing to obtain a preliminary classification result. The multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence. The preliminary classification result is obtained by processing the original feature vector after decomposing and concatenating shared and private features to obtain initial fusion features.

[0022] It should be noted that: the visual feature vector obtained in step S101 Text feature vectors Harmony acoustic eigenvectors The input is fed into a multimodal fusion model based on MISA (Multimodal Invariant and Specific Representations). The three original features are respectively processed by a shared encoder E. s (2-layer 256-d FC+ReLU) and proprietary encoder E p (2-layer 256-d FC+ReLU) to obtain shared and private features. The shared and private features are concatenated by channel to obtain the initial fused features. After processing based on the initial fused features, a preliminary classification result is obtained. .

[0023] S103. Starting from the preliminary classification result, calculate the association value of at least one level of multimodal fusion feature layer by layer in reverse; the multimodal fusion feature includes the initial fusion feature.

[0024] It should be noted that the preliminary classification results obtained from step S102 using the LRP algorithm are as follows: To begin, backpropagation calculations are performed, working backwards and calculating the associated values ​​for each level.

[0025] S104. Based on the association value, the multimodal fusion features at the corresponding level are weighted to obtain weighted fusion features.

[0026] It should be noted that the correlation value calculated in step S103 is used as a weight matrix, and weighted with the original multimodal fusion features of the corresponding level to obtain weighted fusion features.

[0027] S105. The weighted fusion features of each level are re-input into the corresponding subsequent levels of the multimodal fusion model, and iterative training is performed to obtain the final multimodal fusion model.

[0028] It should be noted that the weighted fusion features obtained in step S104 are re-inputted into the subsequent parts of the model corresponding to that level. For example, the initial weighted fusion features are input back into the second encoder in the following embodiment, and forward propagation continues to the output to obtain a new, optimized classification result. Through this closed loop of "forward propagation > LRP analysis > feature weighting > retraining", the model continuously performs self-diagnosis and optimization during training, thereby learning more effectively in small sample scenarios and alleviating overfitting and underfitting problems.

[0029] S106. Evaluate and adjust the multimodal fusion model based on the test set.

[0030] It should be noted that the performance of the finally trained model is evaluated using the reserved test set. Key metrics such as accuracy, F1 score, and correlation coefficient (Corr) are recorded to verify the model's effectiveness. The LRP algorithm is also used to calculate the correlation values ​​of the fused features on the test set. Through statistical analysis (such as plotting the R-value distribution), a threshold is set (e.g., 99th percentile, threshold=1.42) to select the feature dimensions most critical to the classification task. Based on the evaluation results, hyperparameters (such as loss function weights α, β, γ, learning rate, etc.) can be adjusted, and steps S101-S105 are repeated for training until the model performance reaches its optimal level.

[0031] In summary, the model training method based on multimodal data provided in this invention first systematically applies the LRP algorithm to the deep structure of the multimodal fusion model (including the second encoding layer and multi-head attention mechanism), breaking the limitation of traditional interpretability methods that can only analyze shallow or adjacent network layers. Through the calculated correlation values, the specific contribution of each dimension in the fusion features to the classification decision can be clearly and quantitatively revealed, making the model's "black box" decision-making process transparent. Furthermore, by using correlation values ​​to weight the fusion features, key feature dimensions beneficial to classification can be automatically enhanced, while irrelevant or redundant dimensions are suppressed, thereby reducing feature redundancy and model computational complexity at the source. This enables dynamic and accurate feature deduplication, improving model efficiency and performance. Furthermore, through iterative LRP deduplication training, the model can focus more on learning discriminative feature patterns on small sample datasets, avoiding interference from a large number of noisy features and effectively preventing underfitting caused by too many parameters and too little data. Meanwhile, enhancing key features helps improve the model's ability to identify categories with fewer samples, effectively alleviating underfitting and class imbalance issues under low-resource data, and improving overall classification performance. Furthermore, a complete, end-to-end automated process has been developed, encompassing feature extraction, fusion, interpretability analysis, and optimized training. This process can be widely applied to various multimodal classification tasks, such as emotion recognition, intent recognition, and sentiment classification, demonstrating good versatility and scalability.

[0032] In some embodiments of the present invention, such as Figure 2 As shown, the multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence; step S102 inputs the original feature vector into the multimodal fusion model for forward propagation processing to obtain a preliminary classification result, including: S201. Input the original features of each modality into the first coding layer to obtain the coding features of each modality.

[0033] It should be noted that the structure of MVVAE-LRP (MAX-VAR constraint Variational Autoencoder with Layer-wise Relevance Propagation, hereinafter referred to as the multimodal fusion model) is as follows: Figure 3 and Figure 4As shown, the system mainly consists of a first encoding layer, a fully connected eigenvalue decomposition layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer, connected sequentially. The first encoding layer performs deep encoding and alignment of the original feature vectors extracted from the pre-trained model, which may suffer from information sparsity or dimensionality inconsistencies. The original features of each modality are input into the first encoding layer, which includes three independent encoders (e.g., all of which can be Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs)). Each LSTM encoder learns the temporal dependencies and contextual information of its corresponding modality, outputting more refined, higher-order modality-specific encoded features, thus transforming the visual feature vectors... Text feature vectors Harmony acoustic eigenvectors The inputs are fed into three independent encoders, each of which learns the temporal dependencies and contextual information of its corresponding mode, and outputs more refined, higher-order mode-specific encoded features. , , These coding features , , As input for the next stage of feature decomposition.

[0034] S202. Input the encoded features of each modality into the feature decomposition fully connected layer to decompose them into shared features and private features of each modality.

[0035] It should be noted that feature decomposition is performed in a fully connected layer. This layer typically consists of one or more fully connected (DNN) layers and is driven by a specific loss function (such as subsequent consistency and difference constraints). The visual feature vector obtained in step S201 is then... Text feature vectors Harmony acoustic eigenvectors Their respective coding features , , The inputs are fed into their respective feature decomposition DNNs. The encoded features of each modality are non-linearly mapped to two independent subspaces to obtain shared features. and private features , Shared features Used to capture semantic information common to other modalities within this modality that is relevant to the target task. Private features This method preserves unique details and representations specific to a particular modality that cannot be shared with other modalities. By decomposing features into shared and private components, the model is forced to learn modality-invariant general representations and modality-specific supplementary information. This decomposition effectively addresses the semantic inconsistency between modalities and fundamentally reduces feature redundancy resulting from simply concatenating all features.

[0036] S203. Combine the shared and private features of each modality to form the initial fused features.

[0037] It should be noted that after feature decomposition, the shared and private features of all modalities are concatenated. These six feature vectors are concatenated along the feature dimension to form a comprehensive and information-rich initial fusion feature. ,Right now Initial fusion features after splicing It incorporates common knowledge across modalities (shared features) while retaining unique information for each modality (private features), forming a comprehensive and hierarchical multimodal representation that serves as input for subsequent deep fusion networks.

[0038] S204. Based on the initial fusion features, the data are sequentially processed through the second encoding layer, the first fully connected fusion layer, and the second fully connected fusion layer to generate the preliminary classification result.

[0039] It should be noted that: the initial fusion features The input is fed into the second coding layer (i.e., the second coding layer). The Transformer utilizes its multi-head self-attention mechanism (…). This layer can automatically calculate the global dependencies between different dimensions in the fused features, capturing long-distance cross-modal interaction information. After encoding by this layer, a deep interaction-encoded hidden feature is output. Hidden features of the Transformer output. The input is fed into a fully connected neural network (i.e., the first fully connected fusion layer, DNN FC1 Fusion), where nonlinear transformations and feature dimensionality reduction are performed, and intermediate fused features are output. . Integrate intermediate features The information is then fed into another fully connected neural network (the second fully connected fusion layer, DNNFC2 Fusion) to further refine the data and map it to a dimension equal to the number of classification categories. The output of this layer is the preliminary classification result. Also known as logits value, it represents the initial classification result. This represents the model's raw prediction scores for each category before final Softmax normalization. Through a deep stacked structure of "Transformer + DNN," the model can progressively refine features from shallow interactions to high-level semantic abstraction. The Transformer is responsible for capturing complex cross-modal dependencies, while the DNN layer handles task-specific feature integration and decision-making, ultimately generating high-quality preliminary classification results, providing a starting point for subsequent LRP interpretability analysis.

[0040] In this embodiment, a progressive structure consisting of a first encoding layer, a feature decomposition layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer achieves hierarchical processing from modality-specific encoding to feature-semantic decoupling, and then to deep cross-modal fusion. This ensures the orderly and effective flow of information and avoids confusion between information at different levels. Furthermore, the feature decomposition fully connected layer forces the model to separate information into shared and private subspaces to obtain private and shared features, selectively organizing information and significantly reducing input noise and redundancy in subsequent fusion stages, thus improving the model's computational efficiency and generalization ability. Furthermore, the initial fusion feature concatenation operation integrates general semantics and specific information to form a complete fusion representation. Subsequently, leveraging the powerful global modeling capabilities of the second encoding layer, deep and comprehensive interaction is performed on the initial fusion features, uncovering complex cross-modal associations that shallow models cannot capture, thereby generating hidden features with stronger discriminative power and richer semantics. This lays a solid foundation for high-precision classification. Furthermore, the clear hierarchical division of the model (such as explicit Transformer layers, DNN FC1 layers, and DNN FC2 layers) provides an ideal, structured path for the backpropagation of the LRP algorithm, enabling the importance score (R value) to be accurately calculated and traced back to specific feature dimensions at specific levels (such as initial fused features). Transformer output hidden features (etc.), which greatly enhances the interpretability and optimizability of the model.

[0041] In some embodiments of the present invention, the at least one layer includes at least one of the second coding layer, the first fully connected fusion layer, and the second fully connected fusion layer.

[0042] It should be noted that: such as Figure 5 As shown, the backpropagation path of LRP can cover and operate on these three core layers. An implementation that operates on a single layer means selecting only one of the three layers for LRP redundancy removal optimization. In specific implementations, depending on specific task requirements, data characteristics, or computing resources, any of the following strategies can be selected: Option 1: Optimize only the second coding layer (second coding layer), and calculate the initial fused features in the LRP backpropagation chain. The associated value Then, further backpropagation stops. Subsequently, only the associated values ​​are used. For initial fusion features Weighting is performed to obtain the initial fusion features. and will The input is then re-inputted into the second encoding layer to begin subsequent forward propagation and iterative training. When cross-modal interaction is considered crucial to model performance, and redundancy is desired at the source fusion feature level, this approach has relatively low computational cost and can directly impact the input quality of all subsequent layers.

[0043] Option 2: Optimize only the first fully connected fusion layer (DNN FC1 Fusion), and let the LRP backpropagation chain pass through the second fully connected fusion layer (DNN FC2) to calculate the intermediate fused features output from the first fully connected fusion layer. The associated value Subsequently, only using For intermediate fusion features We perform weighted summation to obtain and will The input is then fed back into the second fully connected fusion layer (DNN FC2) to begin subsequent iterative training. This approach can directly optimize intermediate features leading to the final classifier when it is believed that the features encoded by the Transformer still need to be refined in the high-level semantic space.

[0044] Option 3: Optimize only the second fully connected fusion layer (DNN FC2 Fusion), and directly use the correlation values ​​of the output layer (logits). Calculate its correlation value with the input of the second fully connected fusion layer. and to After weighting, it is directly used to calculate the optimized classification loss. As a lightweight optimization technique, it is used to quickly fine-tune the feature representation of the last layer of the model, and is suitable for fine-tuning when computational resources are extremely limited or the model has basically converged.

[0045] The implementation method for multi-level joint optimization refers to simultaneously selecting two or all three of the three levels for joint optimization.

[0046] Perform a complete LRP backpropagation, as shown in the attached document. Figure 3 As shown, the initial fusion correlation values ​​related to the second coding layer (Transformer) are calculated sequentially. The first fully connected fusion layer outputs correlation values. The second fully connected fusion layer input correlation value (This value is used in calculation) (As obtained during the process). In one iteration of model training, using Weighted get And input it into the second encoding layer (Transformer).

[0047] use Output of the weighted second coding layer (Transformer) get And input it into the first fully connected fusion layer (DNN FC1 Fusion). Using Output of the weighted first fully connected fusion layer (DNN FC1Fusion) get The weighted features are then fed into a second fully connected fusion layer (DNN FC2). All weighted features are re-inputted into their corresponding layers for forward propagation, and the total loss is calculated based on the final optimized classification result, thus updating the model parameters uniformly. This multi-level joint optimization achieves end-to-end, refined redundancy removal from shallow fused features to deep abstract features, ensuring that each layer receives high-quality features filtered and enhanced by the LRP algorithm throughout the entire information extraction path, thereby maximizing redundancy removal and model performance improvement.

[0048] In this embodiment, by limiting LRP redundancy removal optimization to at least one of the second coding layer, the first fully connected fusion layer, and the second fully connected fusion layer, users can flexibly choose the depth and breadth of optimization according to actual conditions. For example, when computational resources are scarce, only the most critical Transformer layer can be optimized; when pursuing ultimate performance, full-link optimization can be performed. This "configurable" optimization strategy greatly enhances the practicality and applicability of the method. Furthermore, optimizing the second encoding layer (Transformer) removes redundancy at the source of cross-modal interactions, with a wide impact, focusing on improving the overall quality and diversity of features. Optimizing the first fully connected fusion layer (DNN FC1) refines the features in the high-level semantic feature space, focusing on enhancing the discriminative power of the features. Optimizing the second fully connected fusion layer (DNN FC2) fine-tunes the layer closest to the decision layer, directly affecting the classification results. This allows users to perform precise, targeted optimization based on the specific location of the model's performance bottleneck. When multiple layers are implemented jointly, the redundancy removal effects of each layer are not simply additive, but rather mutually reinforcing and synergistic. Shallow redundancy removal provides cleaner input to deeper layers, while feedback from deeper layers guides more effective filtering in shallower layers through LRP, significantly improving the model's representation learning ability and generalization performance overall, especially in complex small-sample multimodal tasks. Furthermore, forcing drastic weight adjustments to all layers simultaneously may lead to training instability; this application allows for a phased, gradual optimization strategy. For example, other layers can be fixed first, and only the Transformer layer can be optimized. After stabilization, optimization of other layers can be introduced. This flexibility effectively ensures the training robustness of complex models in the process of introducing interpretability-driven optimization.

[0049] In some embodiments of the present invention, step S103 calculates the association value of at least one level of multimodal fusion features in reverse, layer by layer, starting from the preliminary classification result, including: The preliminary classification result is set as the initial association value for backpropagation, where L represents the output layer; Starting from layer L to layer 1, the correlation value of each layer is calculated sequentially using backpropagation according to the following preset rule formula: ; in, For the first l +1 floor to the l The first layer i The first neuron and the second j The correlation values ​​between neurons For the first l +1 floor j The correlation value of each neuron. For the firstl The first layer i The activation value of each neuron. For the first l +1 floor j The net input of each neuron before it passes through the activation function. For the first l The first layer i The activation value of each neuron. For symbolic functions, For connection weights, It is a positive smoothing coefficient.

[0050] It should be noted that in the multimodal fusion model, the layer that generates the preliminary classification result is defined as the Lth layer, which is the starting point of backpropagation. The output of this layer is a preliminary classification result. Each component corresponds to the original score of a category. The association value of the Lth layer is directly set as the preliminary classification result itself or a transformation thereof. Specifically, each component of the logits vector can be directly used as the association value of the output neuron, i.e. This method preserves the original evidence for all categories. Alternatively, the correlation values ​​from the output layer (logits) can also be used. Initialization is performed. Specifically, the logits vector (i.e., the preliminary classification result) is selected. The class with the largest component value (i.e., the prediction the model is most confident in) retains its value, while the values ​​of the remaining classes are set to 0. By focusing backpropagation on the model's most likely prediction, the interpretation becomes more targeted.

[0051] After initialization, iterative calculations are performed starting from layer L, proceeding sequentially to layers L-1, L-2, ..., until layer 1 (usually the input layer for feature fusion). The entire process follows a core principle: l The sum of the association values ​​of all neurons in the layer should be approximately equal to that of the first layer. l The sum of the correlation values ​​of all neurons in layer +1 maintains correlation conservation. For two adjacent layers... l +1 and l Calculate using formula from l +1 layer per neuron j arrive l Each neuron in the layer i Correlation contribution: ; For the l The total association value of each neuron i in the layer It is all from the next layer l +1 propagates to the sum of its associated values: .

[0052] Using the ε-rule, the algorithm fairly and reasonably distributes the importance of higher layers back to each neuron in the lower layers based on the activation strength and connection weights during forward propagation. Ultimately, each dimension of the fused features at the lowest level of the model receives an association value, which clearly indicates the contribution of that dimension to the final classification decision. This enables deep interpretability analysis and provides precise weighting criteria for subsequent feature weighting and redundancy removal.

[0053] In this embodiment, by directly using the preliminary classification results (the model's final output) as the starting point for analysis and establishing a backpropagation chain that leads directly to the underlying fusion features, this invention successfully establishes a quantitative causal bridge between the model's "input (features)" and "output (decision)." This allows users to clearly answer the core question: "Based on which features did the model make this prediction?" The ε-rule formula (i.e., the preset rule formula) is a mathematically grounded, precise allocation algorithm. It calculates a continuous real value for the association value, which not only distinguishes between the positive and negative contributions (promotion or inhibition) of features but also quantifies their relative strength. This quantitative measurement is a fundamental prerequisite for effective feature selection and redundancy removal optimization. Through layer-by-layer recursion, it can penetrate the entire deep network, including complex second encoding layers and multi-layer DNNs, tracing back to the initial fusion features. This global, deep interpretive capability is a significant advantage and breakthrough of this invention compared to existing technologies.

[0054] In some embodiments of the present invention, such as Figure 6 As shown, step S104 weights the multimodal fusion features at the corresponding level based on the association value to obtain weighted fusion features, including: S601. The first weighted fusion feature is obtained by multiplying the correlation value of the first coding layer with the output feature of the first coding layer.

[0055] It should be noted that during the model's forward propagation, the output features of the first encoding layer are recorded, which are the hidden features after deep cross-modal interaction. Simultaneously, the correlation value of the features at this layer is calculated through the LRP backpropagation chain. Associated values With hidden features Perform element-wise multiplication. The specific formula is as follows: ; Where ⊙ denotes element-wise multiplication. The calculated... This is the first weighted fusion feature. The first weighted fusion feature... As input, it is fed back into the subsequent first fully connected fusion layer (DNN FC1) to continue forward propagation and participate in model training.

[0056] S602. The second weighted fusion feature is obtained by multiplying the correlation value of the first fully connected fusion layer with the output feature of the first fully connected fusion layer.

[0057] It should be noted that the output features of the first fully connected fusion layer (DNN FC1 Fusion) are recorded. The correlation value of the features in this layer is calculated through the LRP backpropagation chain. Associated values and Perform element-wise multiplication. The specific formula is as follows: Calculated This is the second weighted fusion feature. The second weighted fusion feature... As input, it is fed back into the subsequent second fully connected fusion layer (DNN FC2) to continue forward propagation and participate in model training.

[0058] S603. The third weighted fusion feature is obtained by multiplying the correlation value of the second fully connected fusion layer with the output feature of the second fully connected fusion layer.

[0059] It should be noted that the output features of the second fully connected fusion layer (DNN FC2 Fusion) are recorded, which represents the preliminary classification result. The correlation value of the features in this layer is calculated through the LRP backpropagation chain. (Right now Associated values With features Perform element-wise multiplication. The specific formula is as follows: Calculated This is the third weighted fusion feature. It's important to note that the weighting here is primarily used to calculate the weighted classification loss. A common approach is to use association values. (i.e., only retaining the associated value vector with the largest logits value) and Multiplying these features allows for a more focused calculation of the primary category predicted by the model when calculating the loss. This involves incorporating a third weighted fusion feature. Used to calculate the optimized classification loss It also participates in the calculation of the model's total loss and parameter updates.

[0060] In this embodiment, weighted optimization is performed at different stages of feature evolution (after deep interaction, after high-level semantic extraction, and during final decision-making). This allows the model to perform customized redundancy removal based on the different characteristics of features at each level (such as global relevance of the Transformer layer, abstract semantics of the DNN layer, and category evidence of the output layer), achieving unprecedented fine-grained control. Furthermore, the interpretable results (association value R) calculated by LRP are directly transformed into optimization actions (feature weighting), and the optimized features are re-input into training, forming a powerful closed loop. This enables the model to continuously self-diagnose and self-improve during training, using its own decision-making logic to guide parameter updates, greatly improving learning efficiency. Furthermore, weighting through element-wise multiplication smoothly adjusts the intensity of each dimension in the feature space, amplifying key signals while not completely eliminating potential useful information. This significantly enhances the discriminative power of the features ultimately learned by the model. Simultaneously, due to reduced interference from noise and redundant features, the model exhibits stronger robustness when facing unseen data. Furthermore, the hierarchical independent weighting allows for the use of progressive or selective optimization strategies. In the early stages of training, a certain level can be optimized first, and other levels can be optimized after the model stabilizes. This flexibility effectively avoids the training oscillations or instability that may be caused by drastic adjustments to all parameters at the same time, ensuring a smoother and more controllable optimization process, which helps the model converge to a better performance point more quickly.

[0061] In some embodiments of the present invention, it further includes: The training process of the multimodal fusion model is optimized using a multi-objective loss function. Training stops when the multi-objective loss function converges on the validation set or reaches a preset number of training iterations.

[0062] It should be noted that: multi-objective loss function It is not the loss of a single task, but a composite loss function composed of a weighted combination of multiple sub-loss functions with explicit physical meanings. Its general form is the following multi-objective loss function formula: ; in , , It's a hyperparameter. For classifying losses, For consistency constraint loss, For difference constraint loss, To reconstruct the constraint loss.

[0063] ; Where D is the training set. It is the true classification category of sample j. It is the predicted classification category of sample j derived after redundancy removal and optimization of multimodal fusion features. It is the predicted classification category of sample j derived from the forward propagation of the model before optimization.

[0064] The consistency constraint is measured using the central moment difference (CMD) method, as shown in the following formula: ; in, It is a shared feature among multiple modal features. It is a private feature in multimodal features.

[0065] Differentiation constraints are used to measure and drive the degree of independence between mode-invariant features and mode-specific features of each mode. The differentiation constraint algorithm takes the square of the F-norm and calculates the objective function value of the constraint. During model training, the distance between the private feature components and the shared feature components of each modality is maximized.

[0066] ;in, It is the square of the F-norm.

[0067] Reconstruction constraints are used by the decoder (including video decoding module, text decoding module and semantic decoding module) to generate original modality feature representations, reconstruct the features of each modality, and enable the model to learn the hidden layer fusion features.

[0068] ,in, It is the Reconstruction embedding layer (i.e. Figure 3 Reconstruction features of each modality in the mid-reconstruction feature layer. It is the encoder embedding layer (i.e. Figure 3 The encoding features of each modality in the encoder feature coding layer.

[0069] In each training iteration (Batch), forward propagation and LRP redundancy removal optimization are performed first, and then the total loss mentioned above is calculated. Finally, all weight parameters of the model are updated through backpropagation and gradient descent algorithms (such as Adam) to minimize At the end of each training epoch, the performance of the current model is evaluated using a validation set that was not used in training, and the results are calculated on that set. Set a patience value, for example, 10 epochs. If within a consecutive patience value of epochs... The decrease is less than a preset threshold (Δ), or If the price starts to rise continuously, it is considered to have converged, and training should be stopped immediately.

[0070] Typically, the model from the epoch with the lowest validation set loss is saved as the final model. Alternatively, a maximum number of training iterations, Nmax, can be set before training begins. Regardless of whether the validation set loss converges, the program automatically stops training once the training epoch reaches the maximum number of training iterations, Nmax.

[0071] In this embodiment, the multi-objective loss function unifies the model's primary objective (classification) with multiple auxiliary regularization objectives (consistency, difference, reconstruction) through weighted fusion. This design ensures that the model, while pursuing high classification accuracy, must also learn semantic alignment, structural decoupling, and information-complete feature representations. These constraints mutually promote and restrict each other, guiding the model towards more robust and interpretable optimization, avoiding feature overfitting or modality bias that might occur with a single classification loss. Furthermore, early stopping based on validation set convergence automatically and promptly captures the peak point of the model's generalization ability, preventing performance degradation due to overtraining and ensuring optimal quality of the final deployed model. A preset safety limit on the number of iterations guarantees the controllability and engineering feasibility of the training process, avoiding the risk of indefinite training. These two conditions complement each other, forming a robust training termination scheme. Furthermore, strong regularization through multi-objective loss and early stopping through validation set monitoring effectively alleviate the core pain points in low-resource scenarios. This enables the device proposed in this invention to stably train high-performance, highly reliable multimodal classification models in real-world industrial environments, even without massive amounts of labeled data.

[0072] In some embodiments of the present invention, prior to obtaining the original feature vectors of the multimodal data in the training set, the following steps are included: The original dataset is divided into a training set and a test set; the original dataset includes the original data for all modalities. Based on the object detection model, language model, and speech model, feature preprocessing is performed on the original data of the video, text, and speech modalities in the training set and the test set, respectively, to obtain their respective original feature vectors.

[0073] It should be noted that publicly available or self-built multimodal datasets, such as CH-SIMS, MOSI, and MOSEI, are used. Each sample in these datasets is strictly aligned and contains raw data and labeled information for the same event in three modalities: video (V), audio (A), and text (T). A stratified sampling strategy is used to randomly divide the raw dataset into training and test sets. During the partitioning, it is ensured that the proportion of samples of each category in the two sets is substantially consistent with the overall distribution of the original dataset.

[0074] The training set is used for learning and optimizing model parameters; it is the source of the model's "learned knowledge." The test set is used to evaluate the model's generalization performance once and for the last time after the model training is complete. Common practices in machine learning typically follow this approach: for example, 70% to 80% of the dataset is allocated to the training set, and the remaining 20% ​​to 30% to the test set. The specific proportions can be adjusted based on the total amount of data in the dataset to ensure that the training set has sufficient data while the test set provides reliable statistical evaluation. A pre-trained object detection model (e.g., Faster R-CNN) is used to extract visual feature vectors from video keyframes. This involves sampling keyframes from the video stream and inputting each frame into the Faster R-CNN model. The deep convolutional feature maps from its region proposal network are extracted, or the feature vector set of salient objects detected by the model is directly used. These are then aggregated into fixed-dimensional visual feature vectors through pooling operations (such as average pooling) to transform visual information from pixel space to a high-level semantic feature space, capturing key visual information such as scenes, objects, and people. Language models (e.g., BERT-based uncased models) are used to extract text feature vectors. This involves segmenting and encoding the input text, then feeding it into the BERT model. The output vector corresponding to the [CLS] token of the last hidden layer is taken, or all output token vectors are averaged and pooled to obtain the text feature vector. This leverages BERT's powerful contextual understanding capabilities to generate deep text representations containing rich semantic and emotional information. Speech models (e.g., Wav2Vec2 models) are used to extract speech acoustic feature vectors. This involves inputting the original audio waveform into the wav2vec 2.0 model to extract its contextual representation. Typically, features from all time steps are averaged or features from specific locations are taken to form fixed-dimensional acoustic feature vectors. This extracts robust acoustic features from the audio signal containing paralinguistic information such as intonation, pitch, and rhythm, which are crucial for emotion and intent recognition. In summary, all parameters and models in the feature preprocessing (e.g., tokenizers, pooling strategies, etc.) are learned or determined only from the training set data and then directly applied to the test set. No information from the test set is used to guide the preprocessing process to prevent data leakage and ensure the fairness of the evaluation. Ultimately, each sample is transformed into a feature representation in the form of a triplet. , , This serves as the direct input for subsequent multimodal fusion models.

[0075] In this embodiment, by dividing the dataset into training and test sets and strictly adhering to the test set isolation principle, an unbiased and reliable generalization performance estimate can be obtained, avoiding performance overestimation due to data leakage. Furthermore, feature extraction is performed using state-of-the-art (SOTA) models (Faster R-CNN, BERT, wav2vec2) pre-trained on large-scale data. This allows the original feature vectors to carry a large amount of general knowledge, exhibiting stronger representational power and robustness compared to hand-designed features or model features trained from scratch on small datasets. This provides a high-starting point and extremely high-information-density input for subsequent multimodal fusion and classification tasks, laying the foundation for high-performance models. Furthermore, the three heterogeneous modalities of data (images, text, and audio) are uniformly transformed into fixed-dimensional numerical vectors, enabling deep neural networks to perform computation and learning efficiently.

[0076] For example, the multimodal classification model dataset (including multimodal intent classification and multimodal sentiment classification) is divided into training and testing sets. Feature preprocessing is performed on the multimodal classification dataset (including multimodal intent classification and multimodal sentiment classification), including extracting the original feature vectors (original embeddings) of each modality using three pre-trained models (video, text, and speech): Faster R-CNN (pre-trained model for video), Bert-based-uncase (pre-trained model for text), and wav2vec2 (pre-trained model for speech). (i) represents video, t represents text, and a represents semantics. A deep neural network model MVVAE-LRP is constructed, including the original multimodal fusion model based on the separation of shared and private features from MISA, the LRP backpropagation chain, and a redundancy-removing iterative training module. The multimodal fusion model based on the separation of shared and private features from MISA is the backbone model, which integrates the original feature vectors of the sample corpora from the three modalities. After being input into the backbone model, the features of each modality are decomposed into shared features and private features after passing through the encoder layer and the DNN layer (fully connected layer). These features are then processed through the second encoder layer and forward propagation to generate classification values ​​(logits). The LRP backpropagation chain associates these logits with the features of the second encoder layer, the DNN FC1 fusion layer, and the DNN FC2 fusion layer, calculating the association value R of each layer's features based on the logits. The features R of each layer are then applied to the features of the corresponding layer, resulting in a redundancy-free iterative training mode for the second encoder layer, the DNN FC1 fusion layer, and the DNN FC2 fusion layer. First, the backbone model of the MVVAE-LRP model is trained using the multimodal classification model training set to obtain classification logits. Then, the LRP backpropagation chain is used to calculate the association value R of the second encoder layer, the DNN FC1 fusion layer, and the DNN FC2 fusion layer. The association values ​​R of the second encoding layer, the DNN FC1 fusion layer, and the DNN FC2 fusion layer are used as weights to multiply the feature vectors of the backbone model of the above three layers, generating weighted deredundancy fusion features at three levels. Each level of weighted deredundancy fusion feature is then forward-propagated to iteratively train the model. The performance of the MVVAE-LRP model is evaluated using a multimodal classification model test set, generating various evaluation metrics. The association values ​​R of the second encoding layer, the DNN FC1 fusion layer, and the DNN FC2 fusion layer are used as importance coefficients for each dimension of the above three-layer fusion features to achieve model interpretability.

[0077] Among them, the LRP algorithm (layer-wise relevance propagation) calculates the correlation value between the objective function of the classification task and the features of each layer of the model. express The association value R of the i-th neuron in the layer. yes Layer j node to The association value of node i in layer i is inferred as shown in formula (2) and is used to calculate the association value of each neuron in each layer of the model.

[0078] No. The association value of layer neuron i is calculated as follows: DNN forward propagation, through the input of the first... l The feature vectors of each neuron in the layer are used to perform forward propagation of the neural network to calculate the first... l +1 floor j Feature vector of each neuron The calculation formula is as follows: (1); in, Indicates the DNN's first... l layer to the firstl The weight matrix of layer +1. l Activation values ​​of a layer of neural network Enter the number l +1 layer, the linear net activation value of this layer is calculated using formula (1). Then through the first l Activation function of layer Get the first l +1 layer feature vectors .

[0079] (2); Assuming the DNN network has L layers, the classification prediction value logits is based on the probability distribution of the classification, and its value represents the R-value of the output layer L. The R-value of the k-th component of logits is... k refers to the k-th component value among all components of logits. For example, if logits is a 5-class classification (0.05, 0.1, 0.2, 0.6, 0.05), when k=3, the component value is 0.6.

[0080] Model interpretability research focuses on how the R-value of logits in a classification task propagates back to the upstream model nodes. Two main methods for calculating LRP are based on the ε-rule and the α-β rule. The upstream model refers to... Figure 3 The code consists of modules from logits up to the fusion embedding layer (i.e., the fusion feature encoding layer). The ε-rule is a rule with a stability term. Its core idea is to proportionally allocate the importance score Rl+1 of the previous layer's neurons to the Rl of the next layer's neurons. The allocation ratio is determined by the connection weights. and neuron activation values Decide.

[0081] To prevent numerical instability caused by a zero denominator, a small positive stabilizing term ε is introduced in the rule. The α-β rule is a rule for separating positive and negative contributions; it explicitly distinguishes between positive facilitation (excitation) and negative inhibition (inhibition) connections between neurons and processes them separately. The rule uses two parameters, α and β, to control the distribution ratio of positive and negative contributions, and typically satisfies α... β=1 is used to ensure conservation. In a neural network, the activation value of a neuron is obtained by weighted summation of the inputs of all neurons in the previous layer, followed by an activation function. Among these inputs, some have positive products of connection weights (w_ij) and activation values ​​(x_i), which promote and excite the activation of the current neuron; while others have negative products, which inhibit and hinder the activation of the current neuron.

[0082] This application adopts the ε-rule, which has a simpler formula, as shown in formula (3).

[0083] (3); The ε-rule is used to calculate the correlation value between two adjacent layers in a neural network. l Given that the R value of the +1 level node is known, from l +1 floor to l The layer's association value ε is a relatively small positive integer. The purpose is to ensure that the denominator is not zero. The final layer's R value is R( logits The components of ) = logits, so we deduce from the end to the beginning. l Given the R value of layer +1, it can be calculated that l Layer R-value. The ε-rule is calculated from the perspective of linear addition in the neural network. l Layer node pairs l The contribution of node +1 is used to backpropagate and calculate the R value. The result is obtained according to formula (2). l The relationship values ​​of each node in the layer This process is repeated until the relational values ​​of each neuron node in each layer are calculated. A MAX-VAR-based decoupling algorithm is used to optimize the consistency constraints, resulting in the target value feature vector. Used for consistency constraint objective function The calculation involves using the LRP algorithm to calculate the association value R of the fusion features at each layer from the fusion feature encoding layer to the output, based on the classification logits value. This value is then multiplied with the corresponding multimodal fusion features using weighted multiplication to obtain the deredundant fusion feature we (weight embedding), which is used for optimization during classification task training.

[0084] The correlation value R of each layer (second coding layer, first and second fully connected fusion layer) in the logits derivation process is calculated through backpropagation using the LRP algorithm. No LRP route represents the residual network in the logits derivation process; the correlation value of nodes on this path cannot be calculated, and it does not participate in the backpropagation calculation of the LRP chain. The principle of the MVVAE-LRP model is as follows: The input layer of the MVVAE-LRP model receives raw data from three modalities: text (t), video (i), and speech (a). The raw feature vectors are extracted from the pre-trained models Bert-based-uncase, Faster R-CNN, and wav2vec2 respectively. The LSTM coding layer (i.e., the first coding layer, which includes a video coding module, a text coding module, and a speech coding module) generates the coding features for each modality. , , After passing through a DNN layer (i.e., a fully connected layer for feature decomposition), the features are decomposed into shared features through a nonlinear mapping. ) and private characteristics ( Two component characteristics, The fusion feature is obtained by splicing together the features of each component. . The input features of the Transformer encoding layer (i.e., the second encoding layer) are used to generate hidden features. The initial classification results are then generated by passing the FC1 fusion layer (the first fully connected fusion layer) and the FC2 fusion layer (the second fully connected fusion layer) of the DNN. .like Figure 3 As shown.

[0085] As an important component of the second coding layer, the multi-head attention module For extracting key dimensions of multimodal features, existing research has not addressed the computational complexity of this method. The LRP backpropagation calculation limits the LRP algorithm to the fully connected (FC) layers of deep neural networks, preventing it from deduplicating features at deeper levels. This impacts the model's optimization performance and leads to insufficient interpretability. Therefore, it is necessary to derive the LRP backpropagation function... The association values ​​R of the multimodal fusion features of the transformer encoder layer of the module form the classification logits (i.e., the preliminary classification result). The LRP link between the fusion feature coding layer (fusion embedding layer) and the fusion feature encoding layer (fusion embedding layer).

[0086] (4); (5); This is the smoothing coefficient, with a default value of 0.01. This is the output value of the classification task. Forward propagation neural network with FC2 layers in DNN The weights are obtained according to the LRP formula (3) of the ε-rule. Classification-based association values . Is it like this? Figure 3 The fused feature vector of the DNN_FC1 layer of the MVVAE-LRP model is input into the DNN_FC2 layer. yes The feature dimensions. This is the classification category dimension. R(cls) is the value of the component with the largest value in logits, with the rest set to 0, and is used as the association value for the target classification.

[0087] (6); yes The correlation value R of the feature vector based on classification logitis is calculated using the following formula: (7); For a DNN with FC1 layer forward propagation neural network The weight matrix. It is the j-th dimension component of the fusion feature.

[0088] (8); like Figure 4 As shown, It is a feedforward neural network module The multimodal fusion features are defined as follows: bs (batch size) is the batch size, and hs is the vector dimension of the multimodal fusion features at each level of the MVVAE-LRP model. It is set to 256. It is a multi-head attention mechanism module The generated features are added to the relevant residual network to generate the feature vector. By concatenating feature variables along the dimension of feature components, the feature vector of the second coding layer is obtained. m is The number of characteristic components. It is the input feature vector of the Transformer encoder layer.

[0089] (9); (10); yes The intermediate feature representation of the neural network is shown in Equation (10).

[0090] (11); In formula (11) LRP : R ( output → R ( input The correlation value R output during the forward propagation of the neural network is mapped to the input correlation value R using the LRP formula of the ε-rule. The calculation process is similar to that of formulas (5) and (7).

[0091] (12); like Figure 4 As shown, V is the module related to the multi-head attention mechanism in the second coding layer. The feature vectors of the parallel residual network branches, i.e., the features input from the second encoding layer. to feature vector There was no multi-head attention mechanism module between them The branch. It is a multi-head attention mechanism module The output feature vector.

[0092] (13); When calculating the residual network using LRP backpropagation, only the main branch is considered. Formulas (12) and (13) will... Mapped to . yes The output of the multi-head attention module. Based on the LRP calculation formula and the ScaledDot-Product Attention calculation principle, from... Reverse derivation .

[0093] It is the fusion feature of the fusion feature coding layer. One of its component characteristics. Q , K , V After linear mapping Further decomposition ∈R bs×hn×dk . Q , K , V All are input variables for multi-head attention mechanisms. They are Q , K , V After their respective weights , , The calculated weighted input variables for the multi-head attention mechanism are: hn is the number of multi-heads, and dk is the dimension of each multi-head. , , Attention weights , The scaled multi-head weighted attention features are obtained through Scaled Dot-Product Attention. bs is the batch size. The intermediate variable feature representation is obtained after merging and scaling the multi-head weighted attention features. X liner =Concat( X att ), Xliner ∈R bs×hs hs is the vector dimension of the multimodal fusion features at each level of the MVVAE-LRP model, set to 256. Then, after linear mapping, the multi-head attention is obtained. Output features V att = Line ( X liner ), V att ∈R bs×hs The corresponding LRP backpropagation formula is as follows: (14); yes Vatt arrive X line The weight matrix of the linear mapping function between them. R(X) line ) is X liner The feature vector is based on the correlation value R of logits. Decompose multiple heads to obtain .

[0094] (15); Through / hn obtains the standard correlation value R of the multi-head intermediate variable characteristics. Formula (15) shows that for each , Xatt The derivation formula for calculating LRP based on multi-head features, will Mapping Multi-head correlation value Merging multiple positions yields new .

[0095] (16); The final V-based association value is obtained. .

[0096] (17); (18); (19); (20); (twenty one); It is a component shared across multiple modal features. It is a private component in multimodal features.

[0097] Thus, after loading the multimodal fusion model into memory, and inputting all samples from the CH-SIMS test set, the above process can be used to obtain the results for each batch on the test set. The average of the association values ​​based on classification is used. The fused features on the test set are obtained by iterating through all batches of the CH-SIMS test set. The average of the association values ​​based on the classification. See below. Figure 7 As shown, the 99th percentile of the R value is selected as the threshold value, threshold=1.42. The value of is selected as the most critical feature dimension. A multimodal sample (containing VAT trimodal features) from the CH-SIMS test set is input into the multimodal fusion model for deep learning forward propagation calculation, yielding the sentiment classification result NEG, which is consistent with the true classification label. In summary, after inputting text, speech, and video corpora through the above process, multimodal classification (including intent recognition and sentiment classification) is performed on the VAE (Variational Autoencoder) model. Based on this model, the LRP algorithm is used to calculate the R-value of each layer's features based on logits, and then the R-value is applied to the features at the corresponding level to remove redundancy and optimize the multimodal classification model.

[0098] To better implement the model training method based on multimodal data in the embodiments of the present invention, based on the model training method based on multimodal data, correspondingly, as follows: Figure 8 As shown, this embodiment of the invention also provides a model training system 800 based on multimodal data, the model training system 800 based on multimodal data comprising: The acquisition module 801 is used to acquire the original feature vectors of multimodal data in the training set; the multimodal data includes at least the original data of three modalities: video, text, and audio. The preliminary classification module 802 is used to input the original feature vector into the multimodal fusion model for forward propagation processing to obtain a preliminary classification result. The multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence. The preliminary classification result is obtained by processing the original feature vector after decomposing and concatenating shared and private features to obtain initial fusion features. The association calculation module 803 is used to calculate the association value of at least one level of multimodal fusion features in reverse, layer by layer, starting from the preliminary classification result; the multimodal fusion features include the initial fusion features; The weighted fusion module 804 is used to perform weighted processing on the multimodal fusion features of the corresponding level based on the association value to obtain weighted fusion features; The training and adjustment module 805 is used to re-input the weighted fusion features of each level into the corresponding subsequent levels of the multimodal fusion model, perform iterative training to obtain the final multimodal fusion model, and evaluate and adjust the multimodal fusion model based on the test set.

[0099] The model training system 800 based on multimodal data provided in the above embodiments can implement the technical solutions described in the above embodiments of the model training method based on multimodal data. The specific implementation principles of each module or unit can be found in the corresponding content in the above embodiments of the model training method based on multimodal data, and will not be repeated here.

[0100] like Figure 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902, and a display 903. Figure 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0101] In some embodiments, processor 901 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 902 or process data, such as the model training method based on multimodal data in this invention.

[0102] In some embodiments, processor 901 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 901 may be local or remote. In some embodiments, processor 901 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0103] In some embodiments, memory 902 may be an internal storage unit of electronic device 900, such as a hard disk or memory of electronic device 900. In other embodiments, memory 902 may also be an external storage device of electronic device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 900.

[0104] Furthermore, the memory 902 may include both internal storage units of the electronic device 900 and external storage devices. The memory 902 is used to store application software and various types of data installed on the electronic device 900.

[0105] In some embodiments, display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 903 is used to display information from electronic device 900 and to display a visual user interface. Components 901-903 of electronic device 900 communicate with each other via a system bus.

[0106] In one embodiment, when processor 901 executes the multimodal classification program in memory 902, the following steps can be implemented: Obtain the original feature vectors of multimodal data in the training set; the multimodal data includes at least the original data of three modalities: video, text, and audio. The original feature vector is input into a multimodal fusion model for forward propagation to obtain a preliminary classification result. The multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence. The preliminary classification result is obtained by processing the original feature vector after decomposing and concatenating shared and private features to obtain initial fusion features. Starting from the preliminary classification result, the association values ​​of at least one level of multimodal fusion features are calculated in reverse, layer by layer; the multimodal fusion features include the initial fusion features; The multimodal fusion features at the corresponding levels are weighted based on the association values ​​to obtain weighted fusion features; The weighted fusion features of each level are re-inputted into the corresponding subsequent levels of the multimodal fusion model, and iterative training is performed to obtain the final multimodal fusion model.

[0107] It should be understood that when the processor 901 executes the multimodal classification program in the memory 902, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0108] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 900 mentioned. Electronic device 900 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 900 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0109] Accordingly, embodiments of this application also provide a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the model training methods based on multimodal data provided in the above-described method embodiments.

[0110] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0111] The above provides a detailed description of the model training method, apparatus, device, and medium based on multimodal data provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A model training method based on multimodal data, characterized in that, include: Obtain the original feature vectors of multimodal data in the training set; the multimodal data includes at least the original data of three modalities: video, text, and audio. The original feature vector is input into a multimodal fusion model for forward propagation to obtain a preliminary classification result. The multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence. The preliminary classification result is obtained by processing the original feature vector after decomposing and concatenating shared and private features to obtain initial fusion features. Starting from the preliminary classification results, the correlation values ​​of at least one level of multimodal fusion features are calculated in reverse, layer by layer. The multimodal fusion feature includes the initial fusion feature; The multimodal fusion features at the corresponding levels are weighted based on the association values ​​to obtain weighted fusion features; The weighted fusion features of each level are re-inputted into the corresponding subsequent levels of the multimodal fusion model, and iterative training is performed to obtain the final multimodal fusion model. The multimodal fusion model is evaluated and adjusted based on the test set.

2. The model training method based on multimodal data according to claim 1, characterized in that, The step of inputting the original feature vector into a multimodal fusion model for forward propagation to obtain preliminary classification results includes: The original features of each modality are input into the first coding layer to obtain the coded features of each modality; The encoded features of each modality are input into the feature decomposition fully connected layer, which decomposes them into shared features and private features of each modality. The shared and private features of each modality are concatenated to form the initial fused features; Based on the initial fusion features, the data are sequentially processed through the second encoding layer, the first fully connected fusion layer, and the second fully connected fusion layer to generate the preliminary classification result.

3. The model training method based on multimodal data according to claim 2, characterized in that, The at least one layer includes at least one of the second coding layer, the first fully connected fusion layer, and the second fully connected fusion layer.

4. The model training method based on multimodal data according to claim 3, characterized in that, The step of calculating the association value of at least one level of multimodal fusion features in reverse, layer by layer, starting from the preliminary classification result includes: The preliminary classification result is set as the initial association value for backpropagation, where L represents the output layer; Starting from layer L to layer 1, the correlation value of each layer is calculated sequentially using backpropagation according to the following preset rule formula: ； in, This represents the association value between the i-th neuron in layer (l+1) and the j-th neuron in layer l. This represents the association value of the j-th neuron in the (l+1)-th layer. Let be the activation value of the i-th neuron in the l-th layer. This represents the net input of the j-th neuron in layer (l+1) before it passes through the activation function. Let be the activation value of the i-th neuron in the l-th layer. For symbolic functions, For connection weights, It is a positive smoothing coefficient.

5. The model training method based on multimodal data according to claim 3, characterized in that, The weighted fusion features obtained by weighting the multimodal fusion features at the corresponding levels based on the association values ​​include: The first weighted fusion feature is obtained by multiplying the correlation value of the first coding layer with the output feature of the first coding layer. The second weighted fusion feature is obtained by multiplying the correlation value of the first fully connected fusion layer with the output feature of the first fully connected fusion layer. The third weighted fusion feature is obtained by multiplying the correlation value of the second fully connected fusion layer with the output feature of the second fully connected fusion layer.

6. The model training method based on multimodal data according to claim 1, characterized in that, Also includes: The training process of the multimodal fusion model is optimized using a multi-objective loss function. Training stops when the multi-objective loss function converges on the validation set or reaches a preset number of training iterations.

7. The model training method based on multimodal data according to claim 1, characterized in that, Before obtaining the original feature vectors of the multimodal data in the training set, the following steps are included: The original dataset is divided into a training set and a test set; the original dataset includes the original data for all modalities. Based on the object detection model, language model, and speech model, feature preprocessing is performed on the original data of the video, text, and speech modalities in the training set and the test set, respectively, to obtain their respective original feature vectors.

8. A model training system based on multimodal data, characterized in that, include: The acquisition module is used to acquire the original feature vectors of multimodal data in the training set; the multimodal data includes at least the original data of three modalities: video, text, and audio. The preliminary classification module is used to input the original feature vector into the multimodal fusion model for forward propagation processing to obtain a preliminary classification result. The multimodal fusion model includes a first encoding layer, a feature decomposition fully connected layer, a second encoding layer, a first fully connected fusion layer, and a second fully connected fusion layer connected in sequence. The preliminary classification result is obtained by processing the original feature vector after decomposing and concatenating shared and private features to obtain initial fusion features. The association calculation module is used to calculate the association value of at least one level of multimodal fusion features in reverse, layer by layer, starting from the preliminary classification result; The multimodal fusion feature includes the initial fusion feature; The weighted fusion module is used to perform weighted processing on the multimodal fusion features at the corresponding level based on the association value to obtain weighted fusion features; The training and adjustment module is used to re-input the weighted fusion features of each level into the corresponding subsequent levels of the multimodal fusion model, perform iterative training to obtain the final multimodal fusion model, and evaluate and adjust the multimodal fusion model based on the test set.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the model training method based on multimodal data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the model training method based on multimodal data as described in any one of claims 1 to 7.

Images