A multi-modal learning method based on a self-supervised knowledge distillation strategy

Abstract

The present application relates to the technical field of machine learning, in particular to a multi-modal learning method based on a self-supervised knowledge distillation strategy, comprising: dividing image data in a data set into a training set and a test set; constructing a multi-modal learning model; inputting the training set and the test set into an RTN network and a FIN network respectively for processing; performing a training process between the RTN network and the FIN network to calculate a cross-entropy loss of task learning; extracting fusion feature information in the RTN and the FIN network; calculating a prediction result based on the fusion feature information in the RTN network; calculating a loss of an ITN network based on the prediction result, the fusion feature information in the RTN and the FIN network; calculating a loss of a WMEN network based on the fusion feature information in the FIN network; calculating a total loss of the model, and optimizing parameters according to the total loss of the model. The present application processes the training set data and the test set data through the RTN network, the FIN network and the MNB network, and combines the WMEN network to deal with the situation of missing some modalities in multi-modal data, thereby greatly improving the performance of the multi-modal learning model.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and specifically to a multimodal learning method based on a self-supervised knowledge distillation strategy. Background Technology

[0002] Self-supervised learning is a traditional machine learning paradigm that uses cleverly designed pre-tasks to predict a subset of information using information beyond the part to be predicted. This uncovers latent patterns in large-scale unsupervised data, learning data representations and inductive biases that improve the performance of downstream tasks. These data representations need to encompass good semantic or structural meaning. Then, through pre-training and optimization, the learned general data representations are applied to specific tasks.

[0003] Knowledge distillation is a common method for model compression. It involves constructing a lightweight, smaller model and training it using supervision information from a larger, more powerful model to achieve better performance and accuracy. It is widely used in supervised learning. Because knowledge distillation can extract essential features of the teacher network in an unsupervised manner and improve the performance of the student network based on these features, more and more researchers are attempting to combine knowledge distillation with self-supervised learning.

[0004] Multimodal learning has achieved great success in many visual tasks such as classification, object detection, and segmentation. It is a complex learning paradigm. Due to the increasing amount of multimodal data, multimodal learning has received more and more attention. Many studies are dedicated to using information from multiple modes to improve the performance of models on specific tasks. However, due to the influence of its training strategy, multimodal learning methods cannot fully utilize its multimodal information and produce underoptimized unimodal representations. This makes the performance of multimodal models fall short of expectations, so better methods are needed to improve its performance.

[0005] Currently, researchers commonly use joint training strategies and gradient mixing methods to improve the performance of multimodal models. However, these detection methods have the following problems: different modes have different convergence speeds, which makes it impossible for the jointly trained multimodal model to match or outperform its single-modal counterpart model. This requires the introduction of additional neural network modules, making the training process more complicated.

[0006] To address the problems existing in the prior art, those skilled in the art have made numerous efforts. For example, Chinese patent application CN115169593A discloses a deep learning-based multimodal learning method, which adaptively controls the optimization of each modality through a dynamic gradient modulation-based balanced multimodal learning method to reduce training errors and thus improve model performance. However, the performance improvement is poor when dealing with incomplete modal data. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a multimodal learning method based on a self-supervised knowledge distillation strategy. This method processes training and test data using RTN, FIN, and MNB networks, and incorporates a WMEN network to handle situations where certain modalities are missing in the multimodal data, thereby significantly improving the performance of the multimodal learning model.

[0008] To achieve the above objectives, the technical solution adopted in this invention is a multimodal learning method based on a self-supervised knowledge distillation strategy, which includes the following steps:

[0009] Step S1: Obtain the dataset and divide the image dataset into training and test sets;

[0010] Step S2: Construct a multimodal learning model, which includes a final prediction network FIN, a coarse training network RTN, and a modality balancing network MBN; the modality balancing network includes a spatial location refinement module SPR and a spatial feature refinement module SIR;

[0011] Step S3: Input the training set images from the dataset into the corresponding modality encoder in the RTN network. Meanwhile, the test set images from the dataset are input into the corresponding modality encoders pre-built in the FIN network. middle;

[0012] Step S4: Multiple auxiliary classifiers are added to the intermediate features in the RTN network to generate different self-supervised information, which is then passed one-to-one to the FIN network. Simultaneously, the FIN network obtains the self-supervised reinforcement distribution of the corresponding layer in the RTN network in a one-to-one manner, acquiring knowledge. This knowledge is then trained by the MBN network to obtain the cross-entropy loss for task learning. ;

[0013] Step S5: Extract the fusion module from the RTN and FIN networks respectively. and and extract and Fusion feature information and ;

[0014] Step S6: [The sentence is incomplete and requires more context to be translated accurately.] The input is processed in the spatial location refinement module, and the output is then fed into the task predictor in the RTN network. The prediction matrix is ​​used to make predictions, and the prediction results are obtained. ;

[0015] Step S7: Calculate the prediction results Fusion of feature information and Inputting the knowledge transfer network ITN, the loss of the ITN network is calculated. ;

[0016] Step S8: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The input spatial feature refinement module outputs the results into the MBN network. The resulting data, along with the Δ values ​​from the samples, are then input into the WMEN network to calculate the loss of the WMEN network. ;

[0017] Step S9: Combine cross-entropy loss Losses of the ITN network And the loss of WMEN network The total loss of the multimodal learning model was calculated. The calculation formula is: , where α and β are hyperparameters.

[0018] Furthermore, in step S4, the intermediate features in the RTN network are appended with multiple auxiliary classifiers to generate different self-supervised information, which is then passed one-to-one to the FIN network. The specific method for this is as follows:

[0019] Step S4-1: Input a sample Mapped to the original class space The resulting probability distribution for the ordinary class is as follows: ,in For the original classification network, For temperature hyperparameters, The softmax activation function is used. It is a probability distribution;

[0020] Step S4-2: Add an auxiliary classifier after each of the L stages in the RTN network to obtain L auxiliary classifiers, and then assign the Lth auxiliary classifier to the appropriate auxiliary classifier. The feature map extracted after the stage is represented as ,pass The self-supervised enhancement probability distribution is obtained as follows: ; where the L auxiliary classifiers are represented as: ;

[0021] Step S4-3: Given the rotated augmented data, train L auxiliary classifiers. Self-supervised reinforcement labels are learned through cross functions, and through the first... The features extracted in each stage are mapped and transferred to the FIN network in a one-to-one manner.

[0022] Furthermore, in step S7, the loss of the ITN network is obtained. The specific method is as follows:

[0023] Step S7-1: ... , As input, via the ITN network Remodeling and and through the same relational function Calculate the relation matrix and The first representation of the RTN network model is obtained. and the Relationship between samples The expression is: ; and represent the first digit of the FIN network model. and the Relationship between samples The expression is: ;in It is a relation matrix. ; and To integrate the feature information into the output after reshaping by the ITN network, , ;

[0024] Step S7-2: ITN Network Calculation and The difference matrix between them is obtained, and the sum of the multiple differences is used to obtain the relational difference vector. The calculation formula is: ;in ;

[0025] Step S7-3: Based on the prediction results of the RTN network model ,from Start calculation Furthermore, the information entropy of each sample is used as the uncertainty of its classification, resulting in the... Classification uncertainty probability of a sample The calculation formula is: ;in It is the softmax function used for normalization; It is the entropy value;

[0026] Step S7-4: The ITN network uses As the component corresponding to e The total loss is calculated using the weights. The calculation formula is: .

[0027] Furthermore, in step S7-1, to eliminate the inhibition caused by dimensionality, cosine distance is chosen as the relation function, and the expression of the relation function is: .

[0028] Furthermore, in step S6, the SPR module... The processing steps are as follows: using classification information to guide the extraction, and adding spatial information for refinement; using a neighborhood sampler and statistically guided adaptive sampling to extract features from the image; and through a self-distillation mechanism, learning semantic space correspondence knowledge from reliable samples in the dataset, and passing this knowledge to the FIN network to generate high-quality predictions.

[0029] Furthermore, in step S8, the SIR module... The processing procedure is as follows: by comprehensively evaluating the accuracy metric IoU and classification confidence, the top-k prediction results are obtained, and the same target features of different modalities are distinguished. The IoU between the missing modality and the existing modality is predicted, and the estimated IoU is multiplied by the corresponding classification score to predict the missing modality.

[0030] Furthermore, in step S8, the loss of the WMEN network is calculated. The specific method is as follows: Iterative training is performed using the WMEN network, gradually increasing the number of iterations; when the number of iterations E is less than the number of warm-up iterations N for the multimodal learning model, a comparative ranking strategy is used to select the next model from the previous one. Mining weak mode combination sets The prediction loss of the weak mode combination is calculated; when the number of iterations E in the iterative training is greater than the number of warm-up times N of the multimodal learning model, the prediction results are combined. and are vectors representing the difference Δ. weak mode combination set Calculate the loss of the WMEN network .

[0031] Furthermore, in step S8, a comparison sorting strategy is used to sort from... Mining weak mode combination sets The specific method for calculating the prediction loss of weak mode combinations is as follows:

[0032] Step S8-1: Calculate the training set after each training iteration Predicted output The calculation formula is: ,in, ∆i means that the i-th component of ∆ is 0, and ∆0 means that no component of ∆ is 0;

[0033] Step S8-2: Combining The predicted distribution is optimized using the following formula: ;in To predict the distribution; The optimized predicted distribution; ;

[0034] Step S8-3: Calculate the prediction loss based on the optimized prediction distribution. The calculation formula is: ;in, ; Represents the KL divergence; .

[0035] Step S8-4: A memory library was introduced. To preserve the preheating period Then, an average filtering operation is performed to obtain the final prediction loss, calculated using the following formula: ;in, It is the i-th iteration epoch. , .

[0036] Furthermore, combining the prediction results The vector of difference Δ weak mode combination set Calculate the loss of the WMEN network The specific method is as follows: First, calculate the origin from and Weak combination mask The calculation formula is: ;in, ; It is the ∆ of the i-th sample in this minimum batch, and then according to calculate The calculation formula is: ;in, yes The basic true vector, [.] represents the indexing operator.

[0037] The beneficial effects of the technical solution described in this invention are as follows:

[0038] 1. This invention introduces modal encoders through RTN and FIN network models to process training and test data. Combined with a self-supervised knowledge distillation strategy, it guides the network to learn the joint distribution of the original recognition task and the self-supervised auxiliary task, comprehensively transferring multimodal knowledge and achieving better performance in multimodal learning tasks. Furthermore, the introduction of a WMEN network, through a comparative ranking strategy, mines weak modal signals, effectively addressing the problem of missing modalities in multimodal data and significantly improving the performance of the multimodal learning model.

[0039] 2. This invention processes the fused feature information of the training and test datasets through a spatial location refinement module and a spatial feature refinement module, and trains the processing results through an ITN network and an MBN network. It introduces edge-aware distillation and modality-aware regularization methods, which helps to solve the problem of weak modality combination. Furthermore, edge-aware distillation achieves knowledge transfer through relation consistency, effectively avoiding overfitting and further improving the performance of the multimodal learning model.

[0040] 3. This invention introduces a perceptual regularization method through the WMEN network modality and guides the network to pay more attention to the loss of weak modality combinations through a comparative ranking strategy, thereby further improving the generalization ability of the model and enabling the multimodal learning model to have excellent task performance. It has excellent performance in practical applications such as multimodal classification and segmentation tasks. Attached Figure Description

[0041] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0042] Figure 1 This is a flowchart illustrating the overall implementation of a multimodal learning method based on a self-supervised knowledge distillation strategy.

[0043] Figure 2 This is a training flowchart between the RTN network and the FIN network;

[0044] Figure 3 This is a flowchart of the data processing process of the SPR module;

[0045] Figure 4 This is a flowchart of the data processing process of the SIR module;

[0046] Figure 5 This is a flowchart illustrating the process of calculating the loss of the ITN network.

[0047] Figure 6 This is a flowchart illustrating the process of calculating the loss of the WMEN network.

[0048] Figure 7 This is a flowchart of the MBN network's data processing procedure;

[0049] Figure 8 This invention provides a comparative experiment to compare the average classification error rate (ACER) of five traditional multimodal learning methods with the method of this invention for classifying images with different missing combinations on the CASIA-SURF dataset.

[0050] Figure 9This invention provides a comparative experiment to compare the average IOU (mIOU) values ​​of five traditional multimodal learning methods with the method of this invention for image segmentation results of different missing combinations in the NYUv2 dataset. Detailed Implementation

[0051] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0052] like Figure 1 As shown, this invention proposes a multimodal learning method based on a self-supervised knowledge distillation strategy, which includes the following steps:

[0053] Step S1: Obtain the CASIA-SURF open-source liveness detection dataset and divide the image dataset into training and test sets.

[0054] Step S2: Construct a multimodal learning model, which includes a final prediction network FIN, a coarse training network RTN, and a modality balancing network MBN; the modality balancing network includes a spatial location refinement module SPR and a spatial feature refinement module SIR.

[0055] Step S3: Extract the training set images from the dataset. Input the corresponding mode encoder in the RTN network Meanwhile, the test set images from the dataset are input into the corresponding modality encoders pre-built in the FIN network. middle;

[0056] Step S4: Multiple auxiliary classifiers are added to the intermediate features in the RTN network to generate different self-supervised information, which is then passed one-to-one to the FIN network. Simultaneously, the FIN network obtains the self-supervised reinforcement distribution of the corresponding layer in the RTN network in a one-to-one manner, acquiring knowledge. This knowledge is then trained by the MBN network to obtain the cross-entropy loss for task learning. The specific content is as follows:

[0057] Step S4-1: Input a sample Mapped to the original class space The resulting probability distribution for the ordinary class is as follows: ,in For the original classification network, For temperature hyperparameters, The softmax activation function is used. It represents a probability distribution.

[0058] Step S4-2: Add an auxiliary classifier after each of the L stages in the RTN network to obtain L auxiliary classifiers, and then assign the Lth auxiliary classifier to the appropriate auxiliary classifier. The feature map extracted after the stage is represented as ,pass The self-supervised enhancement probability distribution is obtained as follows: ;in, Composed of progressive convolutional blocks, global average pooling layers, and fully connected layers, the L auxiliary classifiers are represented as follows: .

[0059] Step S4-3: Given the rotated augmented data, train L auxiliary classifiers. Self-supervised reinforcement labels are learned through cross functions, and through the first... The features extracted in each stage are mapped and transferred to the FIN network in a one-to-one manner.

[0060] Step S5: Extract the fusion module from the RTN and FIN networks respectively. and and extract and Fusion feature information and .

[0061] Step S6: [The sentence is incomplete and requires more context to be translated accurately.] The input is processed in the spatial location refinement module, and the output is then fed into the task predictor in the RTN network. The prediction matrix is ​​used to make predictions, and the prediction results are obtained. The SPR module, in particular, is used for... The processing steps are as follows: using classification information to guide the extraction, and adding spatial information for refinement; using a neighborhood sampler and statistically guided adaptive sampling to extract features from the image; and through a self-distillation mechanism, learning semantic space correspondence knowledge from reliable samples in the dataset, and propagating this knowledge to the FIN network to generate high-quality predictions.

[0062] Step S7: Calculate the prediction results Fusion of feature information and Inputting the knowledge transfer network ITN, the loss of the ITN network is calculated. The specific method is as follows:

[0063] Step S7-1: ... , As input, via the ITN network Remodeling and Furthermore, to eliminate the inhibition caused by dimensionality, cosine distance is chosen as the same relational function. The relational function expression is: .pass and And the relation function calculates the relation matrix and The first representation of the RTN network model is obtained. and the Relationship between samples The expression is: ; and represent the first digit of the FIN network model. and the Relationship between samples The expression is: ;in It is a relation matrix. ; and To integrate the feature information into the output after reshaping by the ITN network, , .

[0064] Step S7-2: ITN Network Calculation and The difference matrix between them is obtained, and the sum of the multiple differences is used to obtain the relationship difference vector. This represents the difference between the coarse training network and the FIN network from the first sample in the same mini-batch to other samples, calculated using the following formula: ;in .

[0065] Step S7-3: Based on the prediction results of the RTN network model ,from Start calculation Furthermore, the information entropy of each sample is used as the uncertainty of its classification, resulting in the... Classification uncertainty probability of a sample The calculation formula is: ;in It is the softmax function used for normalization; It is the entropy value.

[0066] Step S7-4: The ITN network uses As the component corresponding to e The total loss is calculated using the weights. The calculation formula is: .

[0067] This prompts the FIN network to focus on samples near the decision boundary and preserve the inter-class margins refined by comprehensive multimodal information from the RTN network model.

[0068] Step S8: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The input spatial feature refinement module outputs the results into the MBN network. The resulting data, along with the Δ values ​​from the samples, are then input into the WMEN network to calculate the loss of the WMEN network. The specific content is as follows: with , and As input, calculate the loss of the WMEN network. To improve the FIN network's ability to identify weak modal combinations, the WMEN network is iteratively trained, with the number of iterations gradually increasing. When the number of iterations E is less than the warm-up number N of the multimodal learning model, a comparative ranking strategy is used to select the next network from the previous one. Mining weak mode combination sets The prediction loss of the weak mode combination is calculated; when the number of iterations E in the iterative training is greater than the number of warm-up times N of the multimodal learning model, the prediction results are combined. and are vectors representing the difference Δ. weak mode combination set Calculate the loss of the WMEN network .

[0069] Among them, the SIR module is for The processing procedure is as follows: by comprehensively evaluating IoU and classification confidence, the top-k best prediction results are obtained, and the target relevance enhancement module ORE distinguishes the same target features of different modalities, predicts the IoU between the missing modality and the existing modality, and multiplies the estimated IoU by the corresponding classification score to predict the missing modality.

[0070] By comparing sorting strategies from Mining weak mode combination sets The specific method for calculating the prediction loss of weak mode combinations is as follows:

[0071] Step S8-1: Calculate the training set after each training iteration Predicted output The calculation formula is: ,in, ∆i means that the i-th component of ∆ is 0, and ∆0 means that no component of ∆ is 0.

[0072] Step S8-2: Combining The predicted distribution is optimized using the following formula: ;in To predict the distribution; The optimized predicted distribution; .

[0073] Step S8-3: Calculate the prediction loss based on the optimized prediction distribution. The calculation formula is: ;in, ; Represents the KL divergence; .

[0074] Step S8-4: A memory library was introduced. To preserve the preheating period Then, an average filtering operation is performed to obtain the final prediction loss, calculated using the following formula: ;in, It is the i-th iteration epoch. , .

[0075] Based on the prediction results The vector of difference Δ weak mode combination set Calculate the loss of the WMEN network The specific method is as follows: First, calculate the origin from and Weak combination mask The calculation formula is: ;in, ; It is the ∆ of the i-th sample in this minimum batch, and then according to calculate The calculation formula is: ;in, yes The basic true vector, [.] represents the indexing operator.

[0076] Step S9: Combine cross-entropy loss Losses of the ITN network And the loss of WMEN network The total loss of the multimodal learning model was calculated. The calculation formula is: α and β are hyperparameters, with α = 0.25 and β = 0.25. Based on the calculated loss of the multimodal learning model, parameter optimization is performed to improve the performance of the multimodal learning model.

[0077] Working principle:

[0078] In steps S3 and S4, the RTN network model is pre-trained with complete multimodal data, possessing comprehensive multimodal knowledge. Simultaneously, multiple auxiliary classifiers are added to the hierarchical intermediate feature maps to generate different self-supervised knowledge, and a one-to-one transfer is performed to thoroughly teach the FIN network to introduce a self-supervised knowledge distillation strategy. This strategy includes the transfer of relational consistency, enabling the FIN network to simulate the inter-sample relationships of the RTN network model, thereby refining the inter-class boundaries. Furthermore, by reweighting the classification uncertainty of samples, this strategy ensures that the FIN network focuses more on samples near the decision boundary. The FIN network is configured to perform final inference to meet the requirements of the multimodal learning task. It has a modality encoder, including a modality indicator for simulating modality exit. By randomly setting the indicator for some modalities, robustness to predictions of missing modalities is achieved.

[0079] The MBN network model generates additional predictions for all inputs, which helps the network pay more attention to edge information while maintaining robustness to incomplete modal inputs, forcing multimodal learning models to improve their ability to represent weak combinations.

[0080] In step S6, the main idea of ​​the SPR module is to use classification information to guide the extraction process, and to add spatial information to improve the extraction. It adopts a basic neighborhood sampler and statistically guided adaptive sampling to extract the features of the image, avoiding the problems of setting hyperparameters and discrete sampling space. This allows the module to better mine spatial information. Through a self-distillation mechanism, the SPR module learns semantic space correspondence knowledge from reliable samples in the dataset, and then propagates this knowledge to the FIN network to generate higher quality predictions.

[0081] In step S7, the ITN network assists in training the FIN network, improving its ability to represent weak modal combinations. It introduces a modality-aware multimodal regularization algorithm, which can mine and guide the FIN network to learn the features of weak modal combinations. This algorithm uses a contrastive ranking strategy, combined with a repository and average filtering, to ensure effective training of weak modal combinations. Furthermore, the ITN network proposes measuring class boundaries and guides the unified model to pay more attention to samples near the edges, encouraging the FIN network to inherit refined inter-class gaps from the RTN network model. To address the difficulty in boundary detection due to boundary irregularities, the ITN network introduces the classification uncertainty of each sample to reweight its contribution to the total loss. This allows the FIN network to focus more on samples near the decision boundary and maintain the refined inter-class boundary information derived from the comprehensive multimodal information from the RTN network model.

[0082] In step S8, the main idea of ​​the SIR model is to obtain the top-k best prediction results through a comprehensive evaluation of IoU and classification confidence. An ORE module for enhancing target relevance is designed to distinguish the features of the same target in different modalities, predict the IoU value between the missing modality and the existing modality, and multiply the estimated IoU by the corresponding classification score to better predict the missing modality.

[0083] To evaluate the feasibility and effectiveness of the proposed method, the proposed multimodal learning method based on a self-supervised knowledge distillation strategy was experimentally compared with several commonly used multimodal learning methods. These mainstream multimodal learning methods include those focusing on extracting modality-invariant features, such as HeMIS and LCR, as well as enhancement methods that further consider improving the recognition ability of single-modality combinations, such as RFNet and MMFormer. Additionally, a baseline method was included, where all modality combinations are trained on a single model, such as ShuffleNet. The various methods and the proposed method were tested on the CASIA-SURF dataset, and the results were compared.

[0084] This experiment includes two parts: multimodal classification and segmentation. The specific content is as follows:

[0085] In the multimodal classification experiments, the experimental platform used a 64-bit Ubuntu system, version 20.04.4, and a GeForce RTX 2080 Ti GPU. Python 3.8 was used as the programming language, and PyCharm was used as the software development platform. The model was implemented using the deep learning framework PyTorch 1.4. The input images were from the CASIA-SURF dataset, consisting of RGB images, depth images, and infrared images, divided into training, validation, and test sets containing 29k, 1k, and 57k samples respectively. During training, the batch size was 24. With the acceleration of three GeForce RTX 2080 Ti GPUs, the training time for each network was approximately one hour. Random flipping, rotation, and cropping were used to augment the data. All models were optimized and trained using a single SGD epoch for 100 iterations, with a batch size of 24. The learning rate was initialized to 0.001, corresponding to 5 linear warm-up phases, and divided by 10 at iterations 16, 33, and 50. Weight decay and momentum were set to 0.0005 and 0.9, respectively. For the CeFA dataset, we split it into training, validation, and test sets containing 35k, 18k, and 54k samples, respectively, and used the average classification error rate as a metric to represent the results on the test set.

[0086] In the multimodal segmentation experiments, the experimental platform used a 64-bit Ubuntu system, version 20.04.4, a GeForce RTX 2080 Ti GPU, Python 3.8 as the programming language, and PyCharm as the software development platform. The models were implemented using the deep learning framework PyTorch 1.4. Experiments were conducted on the NYUv2 and Cityscapes datasets, both consisting of RGB and infrared images. NYUv2 contained 1449 indoor RGB-D images, of which 795 were used for training and 654 for testing. The Cityscapes dataset contained 5000 samples with a resolution of 2048×1024, of which 2975 were used for training, 500 for validation, and 1525 for testing. All models were optimized and trained using Adam for 300 epochs, with a batch size of 8, a learning rate of 1e-2, and 20 warm-up iterations.

[0087] Result comparison:

[0088] In multimodal classification and multimodal segmentation experiments, the average classification error rate of image classification results and the average IOU value of image segmentation results are usually used as indicators to evaluate the performance. The smaller the average classification error rate of image classification results, the better the performance; the larger the average IOU value of image segmentation results, the better the performance.

[0089] like Figure 8 As shown, this is a comparative experimental data showing the average classification error rate (ACER) of five traditional multimodal learning methods and the method of this invention for image classification results with different missing combinations on the CASIA-SURF dataset in a comparative experiment. According to the data, the multimodal learning method based on the self-supervised knowledge distillation strategy proposed in this invention has the lowest average classification error rate for image classification results with different missing combinations compared to other multimodal learning methods.

[0090] like Figure 9 As shown, this is a comparative experimental data showing the average IOU values ​​of five traditional multimodal learning methods and the method of this invention for image segmentation results with different missing combinations in the NYUv2 dataset. The data shows that the multimodal learning method based on a self-supervised knowledge distillation strategy proposed in this invention achieves the highest average IOU value for image segmentation results under different missing combinations compared to other multimodal learning methods.

[0091] Therefore, in summary, the multimodal learning method based on the self-supervised knowledge distillation strategy proposed in this invention has superior performance.

[0092] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal learning method based on a self-supervised knowledge distillation strategy, characterized in that, The method includes the following steps: Step S1: Obtain the dataset and divide the image dataset into training and test sets; Step S2: Construct a multimodal learning model, which includes a coarse training network RTN, a final prediction network FIN, a knowledge transfer network ITN, a modal balancing network MBN, a spatial location refinement module SPR, and a spatial feature refinement module SIR; The RTN network is used for pre-training on complete multimodal data; the FIN network is used for final inference; the ITN network is used to assist in training the FIN network; the MBN network is used to generate additional predictions for all inputs; the SPR module is used to extract image features and mine spatial information; and the SIR module is used to optimize predictions for missing modalities. Step S3: Input the training set images from the dataset into the corresponding modality encoder in the RTN network. Meanwhile, the test set images from the dataset are input into the corresponding modality encoders pre-built in the FIN network. middle; Step S4: Multiple auxiliary classifiers are added to the intermediate features in the RTN network to generate different self-supervised information, which is then passed one-to-one to the FIN network. Simultaneously, the FIN network obtains the self-supervised reinforcement distribution of the corresponding layer in the RTN network in a one-to-one manner, acquiring knowledge. This knowledge is then trained by the MBN network to obtain the cross-entropy loss for task learning. ; Step S5: Extract the fusion module from the RTN and FIN networks respectively. and and extract and Fusion feature information and ; Step S6: [The sentence is incomplete and requires more context to be translated accurately.] The input is processed into the Spatial Position Refinement (SPR) module, and the output is fed into the task predictor in the RTN network. The prediction matrix is ​​used to make predictions, and the prediction results are obtained. ; Step S7: Calculate the prediction results Fusion of feature information and Inputting the knowledge transfer network ITN, the loss of the ITN network is calculated. ; Step S8: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The input spatial feature refinement module (SIR) is used to refine the input spatial features, and the output is then fed into the MBN network. The resulting input, along with the Δ values ​​from the samples, is then fed into the WMEN network to calculate the loss of the WMEN network. ; Step S9: Combine cross-entropy loss Losses of the ITN network And the loss of WMEN network The total loss of the multimodal learning model was calculated. The calculation formula is: , where α and β are hyperparameters; In step S4, the intermediate features in the RTN network are appended with multiple auxiliary classifiers to generate different self-supervised information, which is then passed one-to-one to the FIN network. The specific method is as follows: Step S4-1: Input a sample Mapped to the original class space The resulting probability distribution for the ordinary class is as follows: ,in For the original classification network, For temperature hyperparameters, The softmax activation function is used. It is a probability distribution; Step S4-2: Add an auxiliary classifier after each of the L stages in the RTN network to obtain L auxiliary classifiers, and then assign the Lth auxiliary classifier to the appropriate auxiliary classifier. The feature map extracted after the stage is represented as ,pass The self-supervised enhancement probability distribution is obtained as follows: ; where the L auxiliary classifiers are represented as: ; Step S4-3: Given the rotated augmented data, train L auxiliary classifiers. Self-supervised reinforcement labels are learned through cross functions, and through the first... The features extracted in each stage are mapped and transferred to the FIN network in a one-to-one manner; In step S7, the loss of the ITN network is obtained. The specific method is as follows: Step S7-1: ... , As input, via the ITN network Remodeling and and through the same relational function Calculate the relation matrix and The first representation of the RTN network model is obtained. and the Relationship between samples The expression is: ; and represent the first digit of the FIN network model. and the Relationship between samples The expression is: ;in It is a relation matrix. ; and To integrate the feature information into the output after reshaping by the ITN network, , ; Step S7-2: ITN Network Calculation and The difference matrix between them is obtained, and the sum of the multiple differences is used to obtain the relational difference vector. The calculation formula is: ;in ; Step S7-3: Based on the prediction results of the RTN network model ,from Start calculation Furthermore, the information entropy of each sample is used as the uncertainty of its classification, resulting in the... Classification uncertainty probability of a sample The calculation formula is: ;in It is the softmax function used for normalization; It is the entropy value; Step S7-4: The ITN network uses As the component corresponding to e The total loss is calculated using the weights. The calculation formula is: .

2. The multimodal learning method based on a self-supervised knowledge distillation strategy according to claim 1, characterized in that, In step S7-1, to eliminate the inhibition caused by dimensionality, cosine distance is chosen as the relation function, and the expression of the relation function is: .

3. The multimodal learning method based on a self-supervised knowledge distillation strategy according to claim 1, characterized in that, In step S6, the SPR module... The processing steps are as follows: using classification information to guide the extraction, and adding spatial information for refinement; using a neighborhood sampler and statistically guided adaptive sampling to extract features from the image; and through a self-distillation mechanism, learning semantic space correspondence knowledge from reliable samples in the dataset, and passing this knowledge to the FIN network to generate high-quality predictions.

4. The multimodal learning method based on a self-supervised knowledge distillation strategy according to claim 1, characterized in that, In step S8, the SIR module... The processing procedure is as follows: by comprehensively evaluating the accuracy metric IoU and classification confidence, the top-k prediction results are obtained, and the same target features of different modalities are distinguished. The IoU between the missing modality and the existing modality is predicted, and the estimated IoU is multiplied by the corresponding classification score to predict the missing modality.

5. The multimodal learning method based on a self-supervised knowledge distillation strategy according to claim 1, characterized in that, In step S8, the loss of the WMEN network is calculated. The specific method is as follows: Iterative training is performed using the WMEN network, gradually increasing the number of iterations; when the number of iterations E is less than the number of warm-up iterations N for the multimodal learning model, a comparative ranking strategy is used to select the next model from the previous one. Mining weak mode combination sets The prediction loss of the weak mode combination is calculated; when the number of iterations E in the iterative training is greater than the number of warm-up times N of the multimodal learning model, the prediction results are combined with the prediction results. and are vectors representing the difference Δ. weak mode combination set Calculate the loss of the WMEN network .

6. The multimodal learning method based on a self-supervised knowledge distillation strategy according to claim 5, characterized in that, By comparing sorting strategies from Mining weak mode combination sets The specific method for calculating the prediction loss of weak mode combinations is as follows: Step S8-1: Calculate the training set after each training iteration Predicted output The calculation formula is: Where i represents the i-th component in the training set. ∆i means that the i-th component of ∆ is 0, and ∆0 means that no component of ∆ is 0; Step S8-2: Combining The predicted distribution is optimized using the following formula: ;in To predict the distribution; This represents the optimized prediction distribution; j is the j-th component in the training set. ; Step S8-3: Calculate the prediction loss based on the optimized prediction distribution. The calculation formula is: ;in, ; Represents the KL divergence; ; Step S8-4: A memory library was introduced. To preserve the preheating period Then, an average filtering operation is performed to obtain the final prediction loss, calculated using the following formula: ;in, It is the i-th iteration epoch. , N represents the number of iterations.

7. The multimodal learning method based on a self-supervised knowledge distillation strategy according to claim 6, characterized in that, Based on the prediction results The vector of difference Δ weak mode combination set Calculate the loss of the WMEN network The specific method is as follows: First, calculate the origin from and Weak combination mask The calculation formula is: ;in, ; It is the ∆ of the i-th sample in this minimum batch, and then according to calculate The calculation formula is: ;in, yes The basic true vector, [.] represents the indexing operator.

Images