Fusion classification method, system, storage medium and device for multimodal data

Abstract

The present invention relates to a fusion classification method, system, storage medium and device for multimodal data, which obtains data of a complete modality A and a missing modality B and extracts corresponding shallow features, and further extracts shared features and proprietary features; distills the inter-class relationship of the complete modality A into the missing modality B, corrects the distribution parameters of the missing modality B by using a scaling factor and a bias factor, makes the distribution parameters of the missing modality B consistent with the inter-class relationship of the complete modality A after correction, and determines the inter-class relationship of the missing modality B; reconstructs the sample of the missing modality B by using the sample features of the complete modality A and the relationship between the sample and the category center; regards the shared features of the two modalities as equally important, fuses the shared features and proprietary features of the two modalities by weighted summation, and performs weighted summation on the channel attention weight and the spatial attention weight in the fusion weight on the corresponding channel dimension and the spatial dimension respectively, and classifies the obtained fusion features by using a classifier.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and specifically to a fusion classification method, system, storage medium and device for multimodal data. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Multimodal Fusion refers to a technical method that combines data from different modalities (such as vision, hearing, text, etc.) to improve information processing and understanding capabilities. It can be applied in a variety of scenarios, such as:

[0004] Image captioning: Generate natural language descriptions based on image content.

[0005] Text-to-image generation: Generate corresponding images based on text descriptions.

[0006] Autonomous driving: Integrate data from multiple sensors such as cameras, radars, and lidar to improve the safety and reliability of autonomous driving.

[0007] Human-computer interaction: Combine voice, gestures and facial expressions to achieve natural human-computer interaction.

[0008] Recommendation system: Combines multiple modal data such as user browsing history, comments, images and videos to provide personalized and accurate recommendation services.

[0009] However, in practical applications, multimodal fusion often faces the problem of modality missing, that is, the dataset lacks data of a specific modality.

[0010] Traditional multimodal fusion methods usually use simple filling or ignoring to deal with the problem of missing modal data, which cannot effectively reconstruct the missing modal data, especially under small sample conditions, and it is difficult to achieve satisfactory performance. In addition, when performing modal feature extraction and fusion, traditional methods often need to establish multiple independent models, resulting in a complex model structure that is difficult to optimize and improve.

[0011] In terms of utilizing modal structure knowledge, traditional multimodal fusion methods mainly rely on statistical models and semantic models. These methods learn and fuse modal features through statistical analysis and semantic representation. However, traditional methods usually rely on manual feature extraction, and the model's expression ability is limited. It is difficult to fully explore and utilize the data structure relationship within the modality and the inter-class relationship between modalities, resulting in poor model generalization ability. These defects limit the generalization ability and robustness of traditional methods in practical applications. Summary of the invention

[0012] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a fusion classification method, system, storage medium and device for multimodal data, which combines the key technologies of modal structure knowledge distillation and feature decoupling, deeply mines and utilizes generalized knowledge such as the inter-class relationship between modalities and the neighbor relationship between class centers and samples within the modality, to assist in the reconstruction of missing modal samples under small sample conditions.

[0013] In order to achieve the above object, the present invention adopts the following technical solution:

[0014] A first aspect of the present invention provides a fusion classification method for multimodal data, comprising the following steps:

[0015] Obtain the data of complete modality A and missing modality B and extract the corresponding shallow features, and extract shared features and proprietary features based on the obtained shallow features;

[0016] The inter-class relationship of the complete modality A is distilled into the missing modality B, and the distribution parameters of the missing modality B are corrected using the scaling factor and the bias factor, so that the distribution parameters of the missing modality B are consistent with the inter-class relationship of the complete modality A after correction, and the inter-class relationship of the missing modality B is determined;

[0017] According to the obtained inter-class relationship of the missing modality B, the sample features of the complete modality A and the relationship between the sample and the category center are used to reconstruct the sample of the missing modality B;

[0018] With the constraint loss minimized, the shared features of the two modalities are regarded as equally important, and the shared features and proprietary features of the two modalities are fused through weighted summation. The channel attention weight and spatial attention weight in the fusion weight are weighted summed on the corresponding channel dimension and spatial dimension respectively. The obtained fused features are used for multimodal fusion classification using a classifier.

[0019] Furthermore, the data of the complete modality A and the missing modality B are obtained and the corresponding shallow features are extracted. The shared features and the proprietary features are extracted based on the obtained shallow features. Specifically, the shallow features R of the complete modality A and the missing modality B are extracted using the Transformer-CNN double-branch network structure. A and R B , and further obtain the shared feature H A and H B , and the unique feature P A and P B .

[0020] Furthermore, we get the shared feature H A and H B There is a shared attention mechanism during this period, specifically:

[0021] The shallow feature R is transformed intoA and R B Mapped to a low-dimensional vector space, we get the embedded vector E A and E B ;

[0022] Perform a linear transformation on the embedding vector to generate the query vector Q A and Q B , key vector K A and K B and the value vector V A and V B ;

[0023] Calculate the dot product of the query vector and the key vectors of all modalities to get the attention score, and divide all the attention scores by Perform softmax operation to obtain attention weight;

[0024] Use attention weights to calculate the value vector V of each modality A and V B Perform weighted summation to obtain the final multimodal attention output H A and H B ;

[0025] H A and H B Input it into the shared attention module of the next layer of the Transformer encoder and repeat the above steps for iterative update.

[0026] Furthermore, the inter-class relationship of the complete modality A is distilled into the missing modality B, and the distribution parameters of the missing modality B are corrected using the scaling factor and the bias factor, so that the distribution parameters of the missing modality B are consistent with the inter-class relationship of the complete modality A after correction, and the inter-class relationship of the missing modality B is determined; specifically:

[0027] Construct a graph using the centers of all classes of the complete modality A And mean square error construction diagram Represents the relationship between different classes in the complete modality A; in the same way, construct the graph of modality B and Distill the inter-class relationship of the complete modality A into the model of the missing modality B, so that the inter-class relationship of the missing modality B is consistent with the inter-class relationship of the complete modality A;

[0028] Compute graphs through graph matching methods and picture and The inverse of the similarity is used as the class structure loss between the modalities to measure the difference between the graphs constructed by the class centers of the two modalities and the difference between the graphs constructed by the covariance.

[0029] By minimizing the class structure loss, the scaling factor γ and bias factor δ of the missing mode B are determined so that the distribution parameters of the missing mode B are consistent with the class structure of mode A after correction, and the inter-class relationship of the missing mode B is determined.

[0030] Furthermore, according to the obtained inter-class relationship of the missing modality B, the sample features of the complete modality A and the relationship between the sample and the class center are used to reconstruct the sample of the missing modality B; specifically: a fully connected network with the same number of output layer nodes as the number of class centers is used as a learner, with the sample features of the complete modality A as input, and the output is the weight corresponding to each class center. The learner is trained by minimizing the reconstruction loss and the bidirectional knowledge distillation loss, so that the weights it outputs are used to reconstruct the samples of the missing modality B.

[0031] Furthermore, reconstruction loss is used to ensure that the weighted sum of all class centers in the same modality is consistent with the current sample; bidirectional knowledge distillation loss is used to ensure that the inter-class relationship of the same sample between modalities remains consistent.

[0032] Furthermore, the channel attention weight and spatial attention weight in the fusion weight are weighted and summed in the corresponding channel dimension and spatial dimension respectively to obtain the fusion feature, which is specifically:

[0033] Channel attention performs weighted summation of the features of each modality in the channel dimension and calculates the channel attention weight through the attention mechanism;

[0034] Spatial attention performs a weighted summation of the features of each modality in the spatial dimension and calculates the spatial attention weight through the attention mechanism.

[0035] A second aspect of the present invention provides a fusion classification system for multimodal data, including.

[0036] The feature extraction module is configured to: obtain data of the complete modality A and the missing modality B and extract corresponding shallow features, and extract shared features and proprietary features based on the obtained shallow features;

[0037] The knowledge distillation module is configured to: distill the inter-class relationship of the complete modality A into the missing modality B, correct the distribution parameters of the missing modality B using the scaling factor and the bias factor, so that the distribution parameters of the missing modality B are consistent with the inter-class relationship of the complete modality A after correction, and determine the inter-class relationship of the missing modality B;

[0038] The sample reconstruction module is configured to: reconstruct the sample of the missing modality B according to the obtained inter-class relationship of the missing modality B, using the sample features of the complete modality A and the relationship between the sample and the class center;

[0039] The feature fusion module is configured as follows: to minimize the constraint loss, the shared features of the two modalities are regarded as equally important, the shared features and proprietary features of the two modalities are fused by weighted summation, the channel attention weight and spatial attention weight in the fusion weight are weighted summed in the corresponding channel dimension and spatial dimension respectively, and the obtained fusion features are used for multimodal fusion classification by classifier.

[0040] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned multimodal data fusion classification method.

[0041] A fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned multimodal data fusion classification method when executing the program.

[0042] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0043] It combines the key technologies of modal structure knowledge distillation and feature decoupling, deeply explores and utilizes generalized knowledge such as the inter-class relationship between modalities and the neighbor relationship between class centers and samples within a modality. It can reconstruct missing modal samples under small sample conditions, and in the process of multimodal feature fusion, it breaks through the limitations of the traditional attention mechanism and uses the dual attention mechanism based on feature decoupling to fully consider the inherent characteristics of shared features and proprietary features. By introducing prior information of feature decoupling, it ensures that the fusion weights of the shared features of the two modalities are consistent, and uses orthogonal loss to constrain proprietary features, which effectively improves the multimodal fusion classification performance under small sample conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 is a schematic diagram of a fusion classification process of multimodal data provided by one or more embodiments of the present invention;

[0046] Figure 2 It is a schematic diagram of class structure knowledge organization during fusion classification of multimodal data provided by one or more embodiments of the present invention;

[0047] Figure 3 It is a schematic diagram of a system for implementing fusion classification of multimodal data using images to be tested provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0049] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] The following embodiments take retail-oriented electronic cigarettes as an example to provide a fusion classification method, system, storage medium and device for multimodal data, and adopt a Transformer-CNN dual-branch network structure to extract and fuse shared features and proprietary features. This method combines the key technologies of modal structure knowledge distillation and feature decoupling, deeply explores and utilizes generalized knowledge such as inter-class relationships between modalities and the neighbor relationship between class centers and samples within modalities, thereby assisting in the reconstruction of missing modal samples under small sample conditions. In addition, in the process of multimodal feature fusion, the limitations of the traditional attention mechanism are broken through, and a dual attention mechanism based on feature decoupling is proposed, which fully considers the inherent characteristics of shared features and proprietary features, and ensures that the fusion weights of the shared features of the two modalities are consistent by introducing prior information of feature decoupling, and adopts orthogonal loss to constrain proprietary features, which effectively improves the multimodal fusion classification performance under small sample conditions.

[0051] Embodiment 1:

[0052] The fusion classification method of multimodal data includes the following steps:

[0053] Obtain the data of complete modality A and missing modality B and extract the corresponding shallow features, and extract shared features and proprietary features based on the obtained shallow features;

[0054] The inter-class relationship of the complete modality A is distilled into the missing modality B, and the distribution parameters of the missing modality B are corrected using the scaling factor and the bias factor, so that the distribution parameters of the missing modality B are consistent with the inter-class relationship of the complete modality A after correction, and the inter-class relationship of the missing modality B is determined;

[0055] According to the obtained inter-class relationship of the missing modality B, the sample features of the complete modality A and the relationship between the sample and the category center are used to reconstruct the sample of the missing modality B;

[0056] With the constraint loss minimized, the shared features of the two modalities are regarded as equally important, and the shared features and proprietary features of the two modalities are fused through weighted summation. The channel attention weight and spatial attention weight in the fusion weight are weighted summed on the corresponding channel dimension and spatial dimension respectively. The obtained fused features are used for multimodal fusion classification using a classifier.

[0057] Network structure. This embodiment uses Restormer to extract shallow features of modality A and modality B. In the shared feature extraction stage, these features are input into the Transformer, and multimodal attention output is generated through the shared attention mechanism, and multiple iterative updates are performed to fully learn the shared information between modalities. In the proprietary feature extraction stage, these features are input into CNN respectively to obtain proprietary features. To avoid redundant information, orthogonal loss is used to constrain proprietary features so that they are mutually orthogonal, thereby effectively extracting personalized features of different modalities. This method enhances the generalization ability and robustness of the model.

[0058] Modal structure knowledge distillation

[0059] The first step is to learn the missing modal class distribution based on the relationship between classes. Assuming that each type of data of modality A and missing modality B obeys the standard Gaussian distribution, calculate the class center and mean square error of modality A. Since modality B has less data, it is difficult to obtain complete class distribution knowledge by directly using the data of each type of sample to calculate the average value and mean square error. Therefore, the present invention introduces scaling factors and bias factors to correct the distribution parameters of modality B. A graph is constructed based on the class center and mean square error of modality A to represent the relationship between different classes in modality A, and a graph of modality B is also constructed. The graph similarity of modality A and modality B is calculated by the graph matching method, and the inverse of the similarity is used as the class structure loss between modalities. By minimizing the loss, the scaling factor and bias factor of modality B are solved so that the distribution parameters of modality B are consistent with the class structure of modality A after correction.

[0060] The second step is to use a sample reconstruction learner based on the consistency of the inter-modal category relationship. Based on the idea that the relationship between the category centers of the data samples of the two modalities of the same entity should be similar, the relationship between the sample and the category center is introduced to reconstruct the sample of the missing modality B. A fully connected network with two hidden layers is constructed as a learner, with sample features as input and weights corresponding to each class center as output for sample reconstruction. During the training process, a reconstruction loss is introduced to ensure that the weighted sum of all class centers in the same modality is consistent with the current sample. In order to enhance the interaction of the category structure knowledge between the two modalities and ensure that the inter-class relationship of the same sample between the modalities is consistent, a bidirectional knowledge distillation loss is also introduced. By minimizing the reconstruction loss and the bidirectional knowledge distillation loss, the learner is trained so that the weights it outputs are used to reconstruct the missing modality samples, and finally the missing modality B samples are reconstructed based on the sample features of the available modality A.

[0061] Dual attention multi-feature fusion based on feature decoupling

[0062] In the dual attention multi-feature fusion process based on feature decoupling, the shared features of the two modalities are regarded as equally important, and the fusion weights of the shared features are ensured to be consistent through the prior information of feature decoupling. Specifically, a constraint term is introduced to make the fusion weights of the shared features of the two modalities as close as possible. Next, the shared features and proprietary features of the two modalities are fused by weighted summation, where the fusion weights are determined by channel attention and spatial attention. Channel attention performs weighted summation in the channel dimension, and spatial attention performs weighted summation in the spatial dimension. Through this dual attention mechanism, shared features and proprietary features can be more fully fused to ensure that no important information is lost during the fusion process. Finally, the fused features are input into the classifier for multimodal fusion classification. After comprehensively considering the loss functions of each step, the total loss of network training is defined to optimize the performance of the entire model.

[0063] The network training process mainly includes the following loss functions:

[0064] (1) Orthogonal loss for constraining the learning process of unique features

[0065] (2) Class structure loss for inter-modal knowledge distillation

[0066] (3) Reconstruction loss

[0067] (4) Bidirectional knowledge distillation between modal structures

[0068] (5) Shared feature fusion weight constraint loss L c .

[0069] The method of this embodiment can improve the efficiency of feature extraction: it adopts the Transformer-CNN dual-branch network structure, in which one branch focuses on extracting the data structure relationship within the modality, and the other branch focuses on capturing the inter-class relationship between modalities, fully mining and utilizing structural knowledge, and improving the model's generalization ability for small sample data.

[0070] The method of this embodiment can enhance knowledge transfer between modalities: through the modal structure knowledge distillation technology, the inter-class relationship between modalities and the relationship between class centers and samples within a modality are fully utilized, thereby improving the small sample reconstruction accuracy in the case of missing modalities.

[0071] The method of this embodiment can improve the training effect: by introducing prior information of feature decoupling, the fusion weights of shared features are made consistent, ensuring the effective fusion of shared features between two modalities, thereby improving the overall effect during training.

[0072] The method of this embodiment can improve the feature fusion method: by introducing a dual attention mechanism of feature decoupling, it breaks through the limitations of the traditional attention mechanism, more fully integrates shared features and proprietary features, and ensures that important information is retained during the fusion process.

[0073] About network structure

[0074] like Figure 1-Figure 2 As shown in Figure 1, this embodiment uses the Transformer-CNN dual-branch network structure to extract shared features and proprietary features. Since Restormer can provide an effective initial representation and has a low computational complexity, Restormer is used to extract shallow features of the complete modality A and the missing modality B, which are denoted as R and B, respectively. A and R B .

[0075] In the shared feature extraction stage, R A and R B Input into the shared feature extractor Transformer to obtain the shared feature H A and H B This process is used to learn the shared information between the two modalities and provide a basis for subsequent knowledge distillation.

[0076] In order to better extract the shared information between modalities, a shared attention mechanism is designed. Specifically, R A and R B Mapped to a low-dimensional vector space, we get the embedded vector E A and E B .

[0077] Perform a linear transformation on the embedding vector to generate the query vector Q A and Q B , key vector K a and K B and the value vector V A and V B . Calculate the dot product of the query vector and the key vectors of all modalities to get the attention score, and then divide all the attention scores by Perform softmax operation to obtain attention weights. Use attention weights to calculate the value vector V of each modality. A and V B Perform weighted summation to obtain the final multimodal attention output H A and H B . A and H B Input it into the shared attention module of the next layer of the Transformer encoder and repeat the above steps for iterative update.

[0078] In the stage of proprietary feature extraction, R A and R B Input the proprietary feature extractor CNN to obtain the proprietary features P A and P B , which is used to extract personalized features of each modality. In order to avoid extracting redundant information, the orthogonal loss is introduced to constrain the extraction of exclusive features of modality A and modality B. The orthogonal loss expression is:

[0079]

[0080] Among them, ||.|| F represents the Frobenius norm. By minimizing this loss, the unique features extracted from different modalities can be guaranteed to be orthogonal to each other.

[0081] About Modal Structure Knowledge Distillation

[0082] The first step is to learn the missing modality class distribution based on the inter-class relationship. First, assuming that each class of data of complete modality A and missing modality B follows a standard Gaussian distribution, calculate the class center μ of modality A A and the mean square error ε A Since there is less data in mode B, it is difficult to obtain complete category distribution knowledge by directly using the data of each type of sample to calculate the mean and mean square error. Therefore, the scaling factor γ and the bias factor δ are introduced to correct the distribution parameters of mode B. The corrected distribution parameters are:

[0083] μ′ B =γ0μ B +δ0 and ε′ B =γ1ε B +δ1.

[0084] Since the data of both modalities are generated based on the same entity, the inter-class structure relationship between the modalities should be consistent. Based on this prior, the center of all classes of modality A is used to construct the graph And mean square error construction diagram Represents the relationship between different classes in modality A. Using the same idea, construct the graph of modality B and The inter-class relationship of modality A is distilled into the model of modality B, so that the inter-class relationship of modality B is consistent with the inter-class relationship of modality A. and picture and The reciprocal of the similarity is used as the class structure loss between modalities. The formula is as follows:

[0085]

[0086] Where sim(G1,G2) is the similarity between graphs G1 and G2, and λ1,λ2 are the weight coefficients of the two parts in the class structure loss. This formula consists of two parts. The first part is used to measure the difference between the graphs constructed by the class centers of mode A and mode B, and the second part is used to measure the difference between the graphs constructed by the covariance of mode A and mode B. By minimizing this loss, the scaling factor γ and the bias factor δ of mode B are solved, so that the distribution parameters of mode B are corrected to be consistent with the class structure of mode A.

[0087] The second step is to reconstruct the learner based on the consistency of the category relationship between modalities. Based on the data samples of the two modalities of the same entity, its category center μ A and μ B The relationship between should be similar. This embodiment introduces the relationship between samples and class centers to reconstruct the samples of missing mode B. A fully connected network is constructed as a learner, including two hidden layers, and the number of nodes in the output layer is consistent with the number of class centers. The input of the learner is the sample features, and the output is the weight corresponding to each class center, which is used for sample reconstruction.

[0088] During the training process, reconstruction loss is introduced to ensure that the weighted sum of all class centers in the same modality is consistent with the current sample. The specific formula is:

[0089]

[0090] in, represents the characteristics of the i-th sample in mode m, represents the true label of the i-th sample in modality m, represents the weight of the c-th class center in mode m, and C represents the number of categories.

[0091] In order to enhance the interaction of the category structure knowledge between the two modalities and ensure that the inter-class relationship of the same sample between the modalities remains consistent, a bidirectional knowledge distillation loss is introduced, and the formula is as follows:

[0092]

[0093] in, represents the characteristics of the i-th sample in mode A, represents the characteristics of the i-th sample in mode B, represents the weight of the c-th class center in mode A, represents the weight of the c-th class center in modality B. The learner is trained by minimizing the reconstruction loss and the bidirectional knowledge distillation loss so that the weights it outputs are used to reconstruct the missing modality samples. Finally, for the samples of missing modality B, the following formula can be used for reconstruction:

[0094]

[0095] in, represents the sample features of the available modality A corresponding to the missing modality B, Represents the weight of the c-th class center output by the learner in mode B.

[0096] About multi-feature fusion

[0097] Different from the traditional attention mechanism that ignores the characteristics of shared features and proprietary features, this embodiment introduces the prior information of feature decoupling and regards the shared features of the two modalities as equally important in the fusion process. In order to ensure that the fusion weights of the shared features of the two modalities remain consistent, the following constraints are introduced:

[0098]

[0099] in, and Respectively represent the shared feature fusion weights of modality A and modality B, and ||.||2 represents the L2 norm. By minimizing this loss, the fusion weights of the shared features of the two modalities can be made closer, thereby ensuring the importance of shared features in the fusion process. Next, the common features and exclusive features of the two modalities are fused using a weighted summation method:

[0100]

[0101] Among them, H A and H B Represent the shared features of modality A and modality B, P A and P B represent the unique features of mode A and mode B respectively, and Represent the shared feature fusion weights of modality A and modality B respectively, and They represent the fusion weights of the exclusive features of modality A and modality B respectively.

[0102] In order to more fully integrate shared features and proprietary features, the fusion weight is determined by channel attention and spatial attention. Channel attention performs weighted summation of the features of each modality in the channel dimension, and calculates the channel attention weight through the attention mechanism. Spatial attention performs weighted summation of the features of each modality in the spatial dimension, and calculates the spatial attention weight through the attention mechanism. The dual attention mechanism can more fully integrate shared features and proprietary features, ensuring that important information is not lost during the fusion process.

[0103] The fused feature f is input into the classifier for multimodal fusion classification. This step uses the classifier to finally classify the fusion features of different modalities to achieve fusion classification of multimodal data. Considering the loss functions of each step, the total loss of network training is defined as:

[0104] L=λ1*L ortho +λ2*L struct +λ3*L rec +λ4*L dis +λ c *L c

[0105] Among them, λ1, λ2, λ3, λ4 and λ c are the weight coefficients of the loss function of each part.

[0106] Through the above steps, while fully integrating shared features and proprietary features, the sample reconstruction of the missing modality and feature fusion are coupled into one process, which is expected to further strengthen the correlation between the missing modality sample reconstruction and multimodal fusion classification, thereby significantly improving the classification accuracy.

[0107] Take the image to be tested as an example, Figure 3 As shown, classification is finally achieved through the steps of feature extraction, knowledge organization and sample reconstruction, and multi-feature fusion in this embodiment.

[0108] The above method uses the Transformer-CNN dual-branch network structure to extract and fuse shared features and proprietary features. This method combines the key technologies of modal structure knowledge distillation and feature decoupling, deeply explores and utilizes generalized knowledge such as inter-class relationships between modalities and the neighbor relationships between class centers and samples within a modality, and then assists in the reconstruction of missing modal samples under small sample conditions.

[0109] In addition, in the process of multimodal feature fusion, the limitations of the traditional attention mechanism are broken through, and a dual attention mechanism based on feature decoupling is proposed. The inherent characteristics of shared features and proprietary features are fully considered, and the prior information of feature decoupling is introduced to ensure that the fusion weights of the shared features of the two modalities are consistent. The orthogonal loss is used to constrain the proprietary features, which effectively improves the multimodal fusion classification performance under small sample conditions.

[0110] Embodiment 2:

[0111] The fusion classification system of multimodal data includes:

[0112] The feature extraction module is configured to: obtain data of the complete modality A and the missing modality B and extract corresponding shallow features, and extract shared features and proprietary features based on the obtained shallow features;

[0113] The knowledge distillation module is configured to: distill the inter-class relationship of the complete modality A into the missing modality B, correct the distribution parameters of the missing modality B using the scaling factor and the bias factor, so that the distribution parameters of the missing modality B are consistent with the inter-class relationship of the complete modality A after correction, and determine the inter-class relationship of the missing modality B;

[0114] The sample reconstruction module is configured to: reconstruct the sample of the missing modality B according to the obtained inter-class relationship of the missing modality B, using the sample features of the complete modality A and the relationship between the sample and the class center;

[0115] The feature fusion module is configured as follows: to minimize the constraint loss, the shared features of the two modalities are regarded as equally important, the shared features and proprietary features of the two modalities are fused by weighted summation, the channel attention weight and spatial attention weight in the fusion weight are weighted summed in the corresponding channel dimension and spatial dimension respectively, and the obtained fusion features are used for multimodal fusion classification by classifier.

[0116] It combines the key technologies of modal structure knowledge distillation and feature decoupling, deeply explores and utilizes generalized knowledge such as the inter-class relationship between modalities and the neighbor relationship between class centers and samples within a modality. It can reconstruct missing modal samples under small sample conditions, and in the process of multimodal feature fusion, it breaks through the limitations of the traditional attention mechanism and uses the dual attention mechanism based on feature decoupling to fully consider the inherent characteristics of shared features and proprietary features. By introducing prior information of feature decoupling, it ensures that the fusion weights of the shared features of the two modalities are consistent, and uses orthogonal loss to constrain proprietary features, which effectively improves the multimodal fusion classification performance under small sample conditions.

[0117] Embodiment three:

[0118] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the multimodal data fusion classification method described in the above-mentioned embodiment 2 are implemented.

[0119] Embodiment 4:

[0120] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the multimodal data fusion classification method described in the above-mentioned embodiment 2 are implemented.

[0121] The steps involved in the above embodiments 2 to 4 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fusion classification method for multimodal data, characterized in that: The following steps are involved: Obtain the data of complete modality A and missing modality B and extract the corresponding shallow features, and extract shared features and proprietary features based on the obtained shallow features; The inter-class relationship of the complete modality A is distilled into the missing modality B, and the distribution parameters of the missing modality B are corrected using the scaling factor and the bias factor, so that the distribution parameters of the missing modality B are consistent with the inter-class relationship of the complete modality A after correction, and the inter-class relationship of the missing modality B is determined; According to the obtained inter-class relationship of the missing modality B, the sample features of the complete modality A and the relationship between the sample and the class center are used to reconstruct the sample of the missing modality B. Specifically, a fully connected network with the same number of nodes in the output layer as the number of class centers is used as a learner, and the sample features of the complete modality A are used as input. The output is the weight corresponding to each class center. The learner is trained by minimizing the reconstruction loss and the bidirectional knowledge distillation loss, so that the output weight is used to reconstruct the sample of the missing modality B. With the constraint loss minimized, the shared features of the two modalities are considered equally important, and the shared features and exclusive features of the two modalities are fused through weighted summation. The channel attention weight and spatial attention weight in the fusion weight are weighted summed in the corresponding channel dimension and spatial dimension respectively. The obtained fusion features are used for multimodal fusion classification by classifier. Among them, multimodal fusion refers to combining data from different modalities, including visual, auditory and textual data; Among them, the inter-class relationship of the missing mode B is determined; specifically: Construct a graph using the centers of all classes of the complete modality A And mean square error construction diagram , which represents the relationship between different classes in the complete modality A; in the same way, the graph of modality B is constructed and , distill the inter-class relationship of the complete modality A into the model of the missing modality B, so that the inter-class relationship of the missing modality B is consistent with the inter-class relationship of the complete modality A; Compute graphs through graph matching methods and ,picture and The inverse of the similarity is used as the class structure loss between the modalities to measure the difference between the graphs constructed by the class centers of the two modalities and the difference between the graphs constructed by the covariance. Determine the scaling factor for the missing mode B by minimizing the class structure loss and bias factor , so that the distribution parameters of the missing mode B are consistent with the class structure of mode A after correction, and the inter-class relationship of the missing mode B is determined.

2. The fusion classification method of multimodal data according to claim 1, characterized in that: Obtain the data of complete modality A and missing modality B and extract the corresponding shallow features. Based on the obtained shallow features, extract shared features and proprietary features. Specifically, use the Transformer-CNN dual-branch network structure to extract the shallow features of complete modality A and missing modality B. and , and further obtain the shared features and , and unique features and .

3. The fusion classification method of multimodal data according to claim 2, characterized in that: Get shared features and There is a shared attention mechanism during this period, specifically: Through linear transformation, shallow features and Mapped to a low-dimensional vector space, we get an embedded vector and ; Perform a linear transformation on the embedding vector to generate the query vector and , key vector and and the value vector and ; Calculate the dot product of the query vector and the key vectors of all modalities to get the attention score, and divide all the attention scores by Perform softmax operation to obtain attention weight; Use attention weights to calculate the value vector of each modality and Perform weighted summation to obtain the final multimodal attention output ; Will Input it into the shared attention module of the next layer of the Transformer encoder and repeat the above steps for iterative update.

4. The fusion classification method of multimodal data according to claim 1, characterized in that: The reconstruction loss is used to ensure that the weighted sum of all class centers in the same modality is consistent with the current sample; the bidirectional knowledge distillation loss is used to ensure that the inter-class relationship of the same sample between modalities remains consistent.

5. The fusion classification method of multimodal data according to claim 1, characterized in that: The channel attention weight and spatial attention weight in the fusion weight are weighted and summed in the corresponding channel dimension and spatial dimension respectively to obtain the fusion feature, which is specifically: Channel attention performs weighted summation of the features of each modality in the channel dimension and calculates the channel attention weight through the attention mechanism; Spatial attention performs a weighted summation of the features of each modality in the spatial dimension and calculates the spatial attention weight through the attention mechanism.

6. A fusion classification system for multimodal data, characterized in that: include: The feature extraction module is configured to: obtain data of the complete modality A and the missing modality B and extract corresponding shallow features, and extract shared features and proprietary features based on the obtained shallow features; The knowledge distillation module is configured to: distill the inter-class relationship of the complete modality A into the missing modality B, correct the distribution parameters of the missing modality B using the scaling factor and the bias factor, so that the distribution parameters of the missing modality B are consistent with the inter-class relationship of the complete modality A after correction, and determine the inter-class relationship of the missing modality B; The sample reconstruction module is configured to: reconstruct the samples of the missing modality B according to the obtained inter-class relationship of the missing modality B, using the sample features of the complete modality A and the relationship between the sample and the class center; specifically: use a fully connected network with the same number of output layer nodes as the number of class centers as a learner, take the sample features of the complete modality A as input, and output the weight corresponding to each class center. The learner is trained by minimizing the reconstruction loss and the bidirectional knowledge distillation loss, so that the output weight is used to reconstruct the samples of the missing modality B; The feature fusion module is configured to: minimize the constraint loss, regard the shared features of the two modalities as equally important, fuse the shared features and exclusive features of the two modalities through weighted summation, and perform weighted summation of the channel attention weight and spatial attention weight in the fusion weight on the corresponding channel dimension and spatial dimension respectively. The obtained fusion features are used for multi-modal fusion classification by classifier; Among them, multimodal fusion refers to combining data from different modalities, including visual, auditory and textual data; Among them, the inter-class relationship of the missing mode B is determined; specifically: Construct a graph using the centers of all classes of the complete modality A And mean square error construction diagram , which represents the relationship between different classes in the complete modality A; in the same way, the graph of modality B is constructed and , distill the inter-class relationship of the complete modality A into the model of the missing modality B, so that the inter-class relationship of the missing modality B is consistent with the inter-class relationship of the complete modality A; Compute graphs through graph matching methods and ,picture and The inverse of the similarity is used as the class structure loss between the modalities to measure the difference between the graphs constructed by the class centers of the two modalities and the difference between the graphs constructed by the covariance. Determine the scaling factor for the missing mode B by minimizing the class structure loss and bias factor , so that the distribution parameters of the missing mode B are consistent with the class structure of mode A after correction, and the inter-class relationship of the missing mode B is determined.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps in the fusion classification method for multimodal data as described in any one of claims 1 to 5 are implemented.

8. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the fusion classification method for multimodal data as described in any one of claims 1 to 5 are implemented.

Images