A multimodal data classification method and device based on cross supervision

Abstract

The present invention discloses a multimodal data classification method based on cross-supervision, comprising: step 1, obtaining multimodal data, and constructing a sample set including labeled data and unlabeled data; step 2, constructing a first classification model and a second classification model based on the same network structure; step 3, using the sample set, training and adjusting parameters of the first classification model and the second classification model; step 4, using labeled data, respectively training the first classification model and the second classification model for testing, and selecting the model with the best test result as the final multimodal data classification model; step 5, inputting the multimodal data to be classified into the multimodal data classification model, and outputting the classification result corresponding to the multimodal data. The present invention also provides a multimodal data classification device. The method provided by the present invention can ensure the robustness, generalization ability and prediction accuracy of the classification model under the condition of small sample multimodal data.

Description

Technical Field

[0001] The present invention relates to the technical field of data classification of deep learning, and in particular to a multimodal data classification method and device based on cross-supervision. Background Art

[0002] The development of the Internet and 5G technology has provided a large amount of multimodal data (referring to data containing text, video, video, and images) for deep learning research. When using multimodal data for research, we can make full use of the characteristics of each modal data, thereby avoiding the limited expressiveness of a single modal data. For example, when using deep learning for short video classification and prediction tasks, only the short video title data is used for classification. The data is relatively one-sided, which affects the accuracy of classification. At the same time, combining video, audio and other modal data can better obtain prediction results. In multimodal data scenarios, there is little data that has been manually labeled and a lot of unlabeled data. How to combine labeled data and unlabeled data during model training to improve the accuracy of model prediction has become an important issue.

[0003] Patent document CN114443864A discloses a cross-modal data matching method, device and computer program product, the method: obtaining a training sample set; wherein the training sample set includes first modal data, second modal data and a label representing whether the multimodal data matches; extracting the first-layer feature information and the second-layer feature information of the first modal data and the second modal data in the training sample respectively; using a matching loss to constrain the matching results between the first modal data and the second modal data based on the first-layer feature information, and the matching results based on the second-layer feature information, and using a classification loss function to constrain the classification results based on the first modal data and the first-layer feature information, and the second-layer feature information, and the classification results based on the first-layer feature information and the second-layer feature information of the second modal data, respectively, to train a cross-modal matching model. This method can make good use of unlabeled multimodal data and improve the model's relevance to data of different modalities through cross-modal comparative learning. However, if it is to be applied in multimodal data classification scenarios, it is necessary to retrain a model based on the labeled data combined with the model. In addition, the network structure of the model cannot be shared in many cases, and it is more troublesome and costly to update the model parameters according to new data at a later stage.

[0004] Patent document CN110363239A discloses a small sample machine learning method, system and medium for multimodal data. The method includes training and testing using three functional modules: multimodal data representation, hierarchical pooling and relational network. First, the multimodal data features are vectorized by an encoder, and then hierarchical pooling with maximum pooling and average pooling is used to reduce the dimension of the time / space continuous vector sequence into a category feature vector. Finally, the learning classification under the condition of small sample is carried out based on the relational network. This method only uses a small amount of existing, partially labeled data, which not only makes the model have poor generalization ability, but also cannot use the information in the unlabeled sample data, thereby affecting the robustness, generalization ability and prediction accuracy of the model. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a multimodal data classification method based on cross-supervision, which can complete the training of the classification model under the condition of small sample multimodal data, while ensuring the robustness, generalization ability and prediction accuracy of the classification model.

[0006] A multimodal data classification method based on cross supervision, comprising:

[0007] Step 1: Obtain multimodal data, and label some of the multimodal data in three dimensions: text, audio, and video, to construct a sample set containing labeled data and unlabeled data;

[0008] Step 2: Based on the same network structure, a first classification model and a second classification model are constructed, and the parameter initialization methods of the first classification model and the second classification model are different;

[0009] Step 3: Using the sample set constructed in step 1, the first classification model and the second classification model constructed in step 2 are trained and parameter adjusted, wherein the training includes supervised training and cross-supervised training;

[0010] Step 4: Use the labeled data to test the first classification model and the second classification model trained in step 3 respectively, and select the model with a higher corresponding F1 value according to the test results as the final multimodal data classification model;

[0011] Step 5: Input the multimodal data to be classified into the multimodal data classification model, and output the classification results corresponding to the multimodal data.

[0012] The present invention introduces a cross-supervised training method into the training process of a multimodal data classification model. Two classification models with the same network structure and different model parameter initialization methods supervise each other, fully utilize limited sample data to train the models, and finally select the model with the highest F1 value as the final multimodal data classification model by judging the F1 value of each model. The above training strategy ensures the generalization ability and prediction accuracy of the multimodal data classification model.

[0013] Preferably, the sample set construction in step 1 further includes preprocessing of multimodal data:

[0014] For the text, the corresponding digital combination is generated through word segmentation conversion method;

[0015] For video and audio, use the pre-trained model to extract features and generate corresponding tensors based on the extracted features;

[0016] It is convenient to unify the expression of multimodal features so that the model can learn better.

[0017] Preferably, the ratio of annotated data to unlabeled data in step 1 is 1:10, which not only saves the time of manual labeling, but also increases the proportion of unlabeled data so that the model finally trained has better generalization ability.

[0018] Specifically, the network structure in step 2 includes a feature extraction module, a processing module, a fusion module and a classification module. The feature extraction module includes a multimodal feature extractor, and the multimodal feature extractor is used to extract multiple modal feature vectors of input data. The processing module is used to convert the extracted multiple modal feature quantities into a dense vector of a unified dimension and input it into the fusion module. The fusion module is used to concatenate the input multiple dense vectors to obtain a fusion vector and input it into the classification module. The classification module performs an implicit operation according to the input fusion vector to obtain a classification result.

[0019] Specifically, the unified dimension of the processing module is realized by normalization processing, and its expression is as follows:

[0020]

[0021] Where E(x) represents the mean of the input x, Var(x) represents the variance of x, ∈ represents noise, and in the initialization stage, γ = 1, β = 0. During the training process of the neural network, γ and β are adjusted accordingly according to the gradient of back propagation.

[0022] Specifically, the splicing operation of the fusion module is expressed as follows:

[0023] E i=concat({P t (x i ),P m (v i ),P v (y i )})

[0024] Where P t (x i ) represents the embedding function of text modality data, P m (v i ) represents the embedding function of video modality data, P v (y i ) represents the embedding function of audio modality data.

[0025] Specifically, the supervised training in step 3 is to train the first classification model and the second classification model respectively with the same labeled data set to minimize the gap between the model prediction results and the actual labeled labels, and comprehensively consider the results of the loss function to back-propagate and update the gradient to learn the model prediction target. The expression of the loss function is as follows:

[0026]

[0027] In the formula, bs represents the number of batch_size in the labeled data, j represents the model number, i represents the i-th data in the batch, CE represents the cross entropy loss function, and y i Indicates the label of the i-th data, p ji Represents the prediction result of model j for the i-th multimodal data.

[0028] Specifically, the cross-supervised training in step 3 includes prediction result cross-supervision and cross-modal cross-supervision.

[0029] Preferably, the cross-supervision of the prediction results is to use the first classification model and the second classification model to predict the unlabeled data respectively, and exchange their respective prediction results as training data labels for supervised training to minimize the gap between the prediction results of the two models, comprehensively consider the results of their respective cross-loss functions to back-propagate and update the gradient to update the model parameters, and use pseudo-labels to expand the unlabeled data set, so that the model can learn a more compact feature encoding representation during the training process.

[0030] Preferably, the cross-modal cross-supervision is to separate and reconstruct the modalities of the data to obtain partial modal data, and use the first classification model and the second classification model to respectively predict the partial modal data to minimize the gap between the prediction results of the two models, and comprehensively consider the results of each cross-loss function to back-propagate and update the gradient to update the model parameters, so that the model can better learn the cross-modal connections in multimodal data, and at the same time avoid contamination of the model due to weak modal correlation in some data when learning cross-modal connections.

[0031] Specifically, the expression of the cross loss function is as follows:

[0032]

[0033] In the formula, un_bs represents the number of labeled data of batch_size, i represents the i-th data in the batch, and CE is the cross entropy loss function.

[0034] Specifically, after completing the cross-supervision and trans-membrane cross-supervision training, the Loss generated by the cross-supervision and trans-membrane cross-supervision training is calculated, and its expression is as follows:

[0035] Loss unlabeled =Loss csc *β+Loss cs *γ

[0036] In the initialization stage, γ=1, β=1. During the training process of the neural network, the two parameters γ and β are learnable and can be adjusted accordingly according to the gradient of back propagation.

[0037] Specifically, the calculation formula of the model F1 value in step 4 is as follows:

[0038]

[0039] In the formula, l represents the number of categories, Precision i Represents the prediction accuracy of the i-th category, Recall i represents the predicted recall of the i-th category.

[0040] The present invention also provides a multimodal data classification device, comprising a computer, a computer processor, and a computer program stored in the computer memory and executed on the computer processor, wherein the multimodal data classification model described above is adopted in the computer memory; when the computer processor executes the computer program, the following steps are implemented: the multimodal data to be classified is input into the multimodal data classification model, and after calculation and analysis, the classification corresponding to the multimodal data is output.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) Based on traditional supervised training, cross-supervision of prediction results and cross-modal cross-supervision are introduced, which can not only improve the utilization rate of sample data, but also ensure the robustness, generalization ability and prediction accuracy of the model.

[0043] (2) Using two models with the same network structure but different model parameter initialization methods for supervised training can not only ensure the correlation of different modal data, but also save the cost of adjusting model parameters in the preparation stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention provides a flow chart of a multimodal data classification method based on cross-supervision;

[0045] Figure 2 A schematic diagram of a network structure provided for this embodiment;

[0046] Figure 3 A schematic diagram of the process of cross-supervision of prediction results provided in this embodiment;

[0047] Figure 4 A schematic diagram of the cross-modal supervision process provided for this embodiment. DETAILED DESCRIPTION

[0048] There is a large amount of multimodal English learning data on the Internet, such as English learning short videos, English audio, English examples, English essays, etc., and each English learning content provider adopts different chapter classification standards and classification granularity for the English learning content they produce. We collect and organize the multimodal content on the Internet, and some data are classified by chapters with reference to the corresponding textbook data (labeled data). There is still a large amount of data that is not classified according to the corresponding chapters of the textbook (unlabeled data).

[0049] like Figure 1 As shown, in order to complete the chapter classification of the textbook, this embodiment provides a multimodal data classification method based on cross-supervision, including:

[0050] Step 1: Obtain multimodal data including text data, video data and audio data, and process the multimodal data:

[0051] To process the text data, a fixed value y is selected to represent the audio length used when processing video and audio data. For example, when y=256, it means that the first 256 characters of the text are used as data for model training. The part exceeding 256 characters is truncated, and the part less than 256 characters is completed with placeholders. Then, the text content is segmented and converted into corresponding ID representations according to the vocabulary. When the placeholder is identified, it is replaced with "-1".

[0052] The video data is processed and a fixed value k is selected to represent the number of frames used when processing the video modality data. For example, when k = 32, it means that the first 32 frames of the video are used as data for model training. The part exceeding 32 frames is truncated, and the part less than 32 frames is completed with placeholders. Then, the video frame features extracted by the pre-trained model (Swin-Transformer) are converted into corresponding tensors. When the placeholder is identified, it is replaced with a tensor of all zeros.

[0053] The audio data is processed, and the audio modal data is processed. A fixed value x is selected to represent the audio length used when processing the audio data. For example, when x=32, it means that the first 32 seconds of the audio are used as the data for model training. The part exceeding 32 seconds is truncated, and the part less than 32 seconds is completed with placeholders. Then, the pre-trained model (TERA) is used to extract the features of the audio, and the audio features are converted into corresponding tensors. When the placeholder is identified, it is replaced with a tensor of all zeros.

[0054] The multimodal data are labeled in three dimensions: text, audio, and video at a labeling ratio of 1:10, and a sample set containing labeled data and unlabeled data is constructed.

[0055] Step 2: Based on the same network structure, construct the first classification model and the second classification model.

[0056] like Figure 2 As shown in the figure, the network structure includes three identical embedding layers, which are used to perform embedding operations on data of three modes respectively. For the embedding layer, the pre-trained model of Bert is used for corresponding initialization.

[0057] For text data, after the data preparation stage, the text modality has been converted into the corresponding ID representation. By using the embedding layer of the text modality, three dense vectors with different functions representing the text are obtained: word_embeddings, token_type_embeddings and position_embeddings. The three vectors are added together to convert the text modality into a dense vector of fixed dimension. Then, the embedding representation corresponding to the text data is obtained through normalization operation, which is recorded as P t (x i ).

[0058] For the video modality, after the data preparation stage, the video modality has been converted into the corresponding tensor representation. Through a linear layer, the tensor dimension of the video modality is mapped to the same dimension as the embedding representation of the text modality as the vector representation of the video modality. According to the size of the video modality and whether there is placeholder content, the token_type_embeddings and position_embeddings corresponding to the video modality are obtained. The three vectors are added and then normalized to obtain the embedding representation corresponding to the video modality, which is recorded as P m (v i ).

[0059] For the audio modality, after the data preparation stage, the video modality has been converted into the corresponding tensor representation. Through a linear layer, the tensor dimension of the audio modality is mapped to the same dimension as the embedding representation of the text modality as the vector representation of the audio modality. According to the size of the video modality and whether there is placeholder content, the token_type_embeddings and position_embeddings corresponding to the audio modality are obtained. The three vectors are added and then normalized to obtain the embedding representation corresponding to the audio modality, which is recorded as P v (y i ).

[0060] If the data of one of the modalities is missing, a tensor with all zeros and the same dimension is used to represent the embedding representation of the missing modality.

[0061] Among them, the specific expression of the normalization operation is as follows:

[0062]

[0063] Where E(x) represents the mean of the input x, Var(x) represents the variance of x, ∈ represents noise, and in the initialization stage, γ = 1, β = 0. During the training process of the neural network, γ and β are adjusted accordingly according to the gradient of back propagation.

[0064] After completing the embedding representation conversion of multimodal data, the embedding representations of different modalities are concatenated. The concatenated part can be expressed as:

[0065] E i =concat({P t (x i ),P m (v i ),P v (y i )})

[0066] Then, the embedding representation E i Pass the Encoder layer of Bert to obtain the output last_hidden_state, perform mean pooling on last_hidden_state, pass the result through a linear layer corresponding to the number of categories, and obtain the subscript of the maximum element of the linear layer to obtain the final corresponding classification result.

[0067] Step 3: Use the sample set constructed in step 1 to train and adjust the parameters of the first classification model and the second classification model constructed in step 2:

[0068] 1. Supervised data training phase

[0069] In this stage, the first classification model and the second classification model use two different sets of labeled data for model training, and learn by minimizing the gap between the predicted results of the labeled data and the actual labeled labels. In this stage, the cross entropy loss function is used as a method to measure the gap between the predicted results and the actual labels. The loss function can be expressed as:

[0070]

[0071] In the formula, bs represents the number of batch_size in the labeled data, j represents the model number, i represents the i-th data in the batch, CE represents the cross entropy loss function, and y i Indicates the label of the i-th data, p ji Represents the prediction result of model j for the i-th multimodal data.

[0072] After completing the calculation of the loss function, the result of minimizing the loss function is used as the training goal, and back propagation is performed according to the results of the loss function to update the gradients of the first classification model and the second classification model to learn the model prediction target.

[0073] 2. Cross-supervision stage of prediction results

[0074] like Figure 3As shown in the figure, after completing the training of the model with a batch of supervised data, the cross-supervised training stage is entered. In this stage, unlabeled data is used for training, and the first classification model and the second classification model are used to predict the same data respectively. After passing through a linear classification layer of the corresponding category, the prediction result of the first classification model is recorded as Pa, and the prediction result of the second classification model is recorded as Pb. The lower mark corresponding to the maximum value in Pa is Ya, and the lower mark corresponding to the maximum value in Pb is Yb. The cross entropy loss of Yb and Pa, and the cross entropy loss of Ya and Pb are calculated to calculate the loss in the cross-supervised training stage. The calculation method can be expressed as:

[0075]

[0076] Among them, un_bs is the number of labeled data in a batch_size, i is the i-th data in the batch, and CE is the cross entropy loss function. In this stage, the gap between the prediction results of the second classification model and the first classification model is minimized, and the unlabeled data set is expanded using the pseudo-label method. In the cross-supervised training process, the model can learn a more compact feature encoding representation.

[0077] 3. Cross-modal supervision stage

[0078] like Figure 4 As shown in the figure, both labeled data and unlabeled data are used for training. By cross-modal combination of different parts of a piece of data, two data forms are generated. The first part of the data includes a part of the modality of the data, such as X1 in the figure, including the text and video modalities in the data X; the second part of the data includes another part of the modality of the data, such as X2 in the figure, including the video and audio modalities in the data; the first classification model is used to predict X1, and the second classification model is used to predict X2. After passing through a linear classification layer of the corresponding category, the prediction result of the first classification model is recorded as Pa, and the prediction result of the second classification model is recorded as Pb. The lower mark corresponding to the maximum value in Pa is Ya, and the lower mark corresponding to the maximum value in Pb is Yb. The cross entropy loss of Yb and Pa, and the cross entropy loss of Ya and Pb are calculated to calculate the loss in the cross-modal cross-supervised training stage. The calculation method can be expressed as:

[0079]

[0080] Where un_bs represents the number of labeled data in a batch_size, i represents the i-th data in the batch, and CE is the cross-entropy loss function, which can enable the model to better learn the cross-modal connections in multimodal data, while avoiding the pollution of the model caused by weak modal correlation when learning cross-modal connections.

[0081] After completing the above training, the generated Loss is calculated as follows:

[0082] Loss unlabeled =Loss csc *β+Loss cs *γ

[0083] In the initialization stage, γ=1, β=1. During the training process of the neural network, the two parameters γ and β are learnable and can be adjusted accordingly according to the gradient of back propagation.

[0084] After calculating the Loss unlabeled After that, back propagation is performed according to the loss to update the gradients of the first classification model and the second classification model, and the above steps are iterated until the model converges and the model training is completed. After the model training is completed, the corresponding model parameters are frozen to realize the prediction of specific feature data.

[0085] Step 4: Use the labeled data to test the first classification model and the second classification model trained in step 3 respectively:

[0086]

[0087] In the formula, l represents the number of categories, Precision i Represents the prediction accuracy of the i-th category, Recall i represents the predicted recall of the i-th category.

[0088] The model with the highest F1 value among the two models is selected as the final multimodal data classification model.

[0089] Step 5: Input the unclassified textbook content into the multimodal data classification model and output the chapter classification results corresponding to the textbook content.

[0090] The present invention also provides a multimodal data classification device, comprising a computer, a computer processor, and a computer program stored in the computer memory and executed on the computer processor, wherein the multimodal data classification model described above is adopted in the computer memory;

[0091] When the computer processor executes the computer program, the following steps are implemented: the multimodal data to be classified is input into the multimodal data classification model, and after calculation and analysis, the classification corresponding to the multimodal data is output.

[0092] The method provided by the present invention can make full use of labeled data and unlabeled data, train a model with strong generalization ability and high prediction accuracy, and realize efficient classification of massive multimodal English data.

Claims

1. A multimodal data classification method based on cross supervision, characterized in that: include: Step 1: Obtain multimodal data, and label some of the multimodal data in three dimensions: text, audio, and video, to construct a sample set containing labeled data and unlabeled data; Step 2: Based on the same network structure, a first classification model and a second classification model are constructed, and the parameter initialization methods of the first classification model and the second classification model are different; Step 3: Using the sample set constructed in step 1, the first classification model and the second classification model constructed in step 2 are trained and parameter adjusted, wherein the training includes supervised training and cross-supervised training, wherein the cross-supervised training includes prediction result cross-supervision and cross-modal cross-supervision, wherein the prediction result cross-supervision is to use the first classification model and the second classification model to predict the unlabeled data respectively, and exchange their respective prediction results as training data labels for supervised training to minimize the gap between the prediction results of the two models, and comprehensively consider the results of the respective cross-loss functions to back-propagate and update the gradients to update the model parameters, wherein the cross-modal cross-supervision is to separate and reconstruct the modality of the data to obtain partial modal data, and use the first classification model and the second classification model to predict the partial modal data respectively to minimize the gap between the prediction results of the two models, and comprehensively consider the results of the respective cross-loss functions to back-propagate and update the gradients to update the model parameters; Step 4: Use the labeled data to test the first classification model and the second classification model trained in step 3 respectively, and select the model with a higher corresponding F1 value according to the test results as the final multimodal data classification model; Step 5: Input the multimodal data to be classified into the multimodal data classification model, and output the classification results corresponding to the multimodal data.

2. The cross-supervision-based multimodal data classification method according to claim 1, characterized in that: The sample set construction in step 1 also includes preprocessing of multimodal data: For the text, the corresponding digital combination is generated through word segmentation conversion method; For video and audio, use the pre-trained model to extract features, and generate corresponding tensors based on the extracted features.

3. The cross-supervision-based multimodal data classification method according to claim 1, characterized in that: The ratio of annotated data to unannotated data in step 1 is 1:

10.

4. The cross-supervision-based multimodal data classification method according to claim 1, characterized in that: The network structure in step 2 includes a feature extraction module, a processing module, a fusion module and a classification module. The feature extraction module includes a multimodal feature extractor, and the multimodal feature extractor is used to extract multiple modal feature vectors of input data. The processing module is used to convert the extracted multiple modal feature quantities into a dense vector of uniform dimension and input it into the fusion module. The fusion module is used to concatenate the input multiple dense vectors to obtain a fusion vector and input it into the classification module. The classification module performs an implicit operation according to the input fusion vector to obtain a classification result.

5. The cross-supervision-based multimodal data classification method according to claim 1, characterized in that: The supervised training in step 3 is to train the first classification model and the second classification model respectively using the same labeled data set to minimize the gap between the model prediction results and the actual labeled labels, and comprehensively consider the results of the loss function to back-propagate and update the gradient to learn the model prediction target.

6. The cross-supervision-based multimodal data classification method according to claim 1, characterized in that: The calculation formula of the model F1 value in step 4 is as follows: In the formula, l represents the number of categories, Precision i Represents the prediction accuracy of the i-th category, Recall i represents the predicted recall of the i-th category.

7. A multimodal data classification device, comprising a computer, a computer processor, and a computer program stored in a memory of the computer and executed on the computer processor, characterized in that: The multimodal data classification model of claim 1 is used in the memory of the computer; the computer processor implements the following steps when executing the computer program: inputting the multimodal data to be classified into the multimodal data classification model, and outputting the classification corresponding to the multimodal data after calculation and analysis.

Images