A Video Classification Method Based on Knowledge Distillation and Multimodal Fusion
By combining knowledge distillation and multimodal fusion methods in video classification, the problem of modal failure and external knowledge in multimodal learning is solved, and the generalization ability of students' network and the accuracy of video classification are improved.
Patent Information
- Application Number
- CN202210609575.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-31
- Publication Date
- 2025-07-25
- Estimated Expiration
- 2042-05-31
AI Technical Summary
There are negative effects of modal failure and joint training objectives in multimodal learning. The self-distillation method lacks external knowledge guidance, resulting in limited video classification performance.
Using the method of knowledge distillation and multimodal fusion, by adding knowledge distillation modules within the multimodal learning framework, the knowledge of the teacher network is transferred to the student network, and feature fusion is performed in the middle and late stages, and self-distillation strategies with different initializations of the same model are combined with cross entropy loss and KL divergence loss for training.
It improves the generalization ability and accuracy of the student network, alleviates the problem of modal failure, enhances the video classification performance, and realizes more robust single-modal feature learning.
Smart Images
Figure CN115147641B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a video classification method based on knowledge distillation and multimodal fusion, belonging to the fields of digital image processing and deep learning. Background Art
[0002] Video classification is an important task in the field of computer vision. The main goal of the video classification task is to understand the content contained in the video and determine the key theme corresponding to the video. Video classification algorithms based on deep learning can automatically analyze the semantic information contained in the video, automatically classify, annotate, and describe the video, and can effectively operate on large-scale videos.
[0003] Videos contain rich multimodal information, such as visual objects, motion, sound, and text (captions or text obtained through speech recognition) information. These different modal information can provide similar but different and transferable semantic information (such as color, texture, event, and location, etc.) for feature learning in multimodal fusion. Therefore, the performance of multimodal learning is usually better than that of unimodal learning. However, according to the research by Du et al. in 2021, there is a performance degradation phenomenon called modal failure in multimodal learning, and at the same time, the multimodal joint training objective has a negative impact on the performance of unimodal networks.
[0004] The knowledge distillation method is a model compression strategy proposed by Hinton in 2015, which extracts knowledge from a large-scale teacher network to a small-scale student network. Since 2018, researchers have found that the knowledge distillation method can not only be used for model compression, but also be used to further improve the model performance through distillation strategies such as mutual learning and self-distillation of neural networks. By performing self-distillation on different initializations of the same model, the student network can learn features from different perspectives learned by the same network, achieving an effect similar to ensemble learning. However, compared with the model compression strategy, the self-distillation strategy lacks rich external knowledge.
[0005] In summary, constructing a video classification method based on knowledge distillation and multimodal fusion can not only alleviate a series of problems in multimodal learning through knowledge distillation, but also use other modal features as external knowledge to guide the student network through multimodal fusion, further promoting the development of video classification. Summary of the Invention
[0006] Aiming at the existing problems and deficiencies in the current multimodal fusion and self-distillation methods, the present invention proposes a video classification method combining knowledge distillation and multimodal fusion, breaking through some limitations existing in current multimodal learning, adding a knowledge distillation module within the multimodal learning framework and applying it to the video classification task, and further improving the video processing ability of convolutional neural networks.
[0007] To achieve the above object, the technical solution of the present invention is as follows: The present invention adopts the following technical solutions to solve the above technical problems:
[0008] The present invention provides a video classification method based on knowledge distillation and multimodal fusion, and the specific steps are as follows:
[0009] Step 1, extract the audio and visual datasets and preprocess them
[0010] 1.1, construct the audio dataset, use FFmpeg to extract the audio in the video dataset, the single-channel sampling rate is 16,000, and the format is wav;
[0011] 1.2, extract the features of the audio, first perform Fourier transform on the audio data to obtain the spectrogram, then perform logarithmic transformation, and finally normalize and convert it into a tensor;
[0012] 1.3, construct the visual dataset, use FFmpeg to extract the image frames in the video dataset, uniformly sample 32 frames of images from them to represent the video modality, and at the same time use the middle image frame of the video to represent the image modality, a total of 33 frames;
[0013] 1.4, extract the features of the vision, first randomly scale the visual data by 0.8 to 1.2 times, then center crop it to a size of 224×224, and finally normalize and convert it into a tensor;
[0014] The advantages of Step 1 are as follows: Through the audio data with a fixed sampling rate and the same spectrogram transformation process, tensors of the same size can be obtained as the input of the audio network; the visual dataset contains 33 frames of images, which can be used as the input of the 3D convolutional network using the video modality, and can also be used as the input of the 2D convolutional network using the image modality; by using the normalization and data augmentation methods, the dataset is diversified and easier to converge to the optimal solution.
[0015] Step 2, build a knowledge distillation architecture
[0016] 2.1, according to the two-modal data extracted by the methods in 1.1 to 1.4, select the same convolutional neural network, adjust the parameters and train multiple times, and select the best model as the single-modal teacher network;
[0017] 2.2, use the model output as the transmitted knowledge, that is, use the KL divergence loss as the distillation loss to measure the difference between the teacher network and the student network outputs;
[0018] The advantages of Step 2 are as follows: By using the knowledge distillation method, the single-modal student network in the multimodal fusion network can obtain the view features of the teacher network under different parameters, making the model more generalized and accelerating the model convergence.
[0019] Step 3, multimodal fusion student network
[0020] 3.1, Use the same convolutional neural network as the teacher network as the unimodal student network, and adopt a different random initialization from the teacher network;
[0021] 3.2, After obtaining their respective unimodal features for knowledge distillation, perform mid-term fusion on the features of each layer of the visual and audio modality student networks;
[0022] 3.3, For the features of the last layer of the visual and audio modality student networks, use the method of adding the SE module after averaging for late fusion.
[0023] 3.4, Add their respective classification losses to the visual and audio modality student networks and the output after late fusion. The classification loss uses the cross-entropy function.
[0024] The advantage of step 3 is that: on the basis of the late fusion paradigm, a mid-term fusion module is added. Feature fusion starting from the middle layer of the model is more conducive to the learning of the network and the interaction of features between different modalities.
[0025] As a further optimization scheme of the present invention, the software environment of this method is FFmpeg and Python3.7. Use scipy to extract spectrograms with a window length of 512 and an overlapping part of 274, and use PIL and torchvision as the visual data processing methods.
[0026] As a further optimization scheme of the present invention, knowledge distillation uses the self-distillation strategy of the same model with different initializations of the teacher and the student to transfer the knowledge output by the model, and at the same time distills the two modalities of vision and audio.
[0027] As a further optimization scheme of the present invention, the multimodal fusion method uses mid-term fusion and late fusion at the same time. After obtaining the unimodal features, the total loss is the direct addition of three classification losses and two distillation losses.
[0028] Compared with the prior art by adopting the above technical solutions, the present invention has the following technical effects: The present invention alleviates the modality failure problem in multimodal learning and the negative impact of the joint training objective introduced in the background technology. At the same time, multimodal fusion can effectively solve the problem of insufficient external knowledge in self-distillation. The knowledge distillation and multiple feature fusion schemes used in this method can effectively improve the generalization ability and accuracy of the student network and learn more robust unimodal features without changing the basic network structure by calculating the classification losses of the two student networks and the fusion output, and the audio-visual teacher-student distillation losses, further improving the final video classification performance. Description of the Drawings
[0029] Figure 1Schematic diagram of the network structure of a video classification method based on knowledge distillation and multimodal fusion;
[0030] Figure 2 Example of an image frame in the training set;
[0031] Figure 3 Example of an image frame in the test set;
[0032] Figure 4 Schematic diagram of the feature fusion in the middle layer of the student network;
[0033] Figure 5 Schematic diagram of the late fusion structure of the student network output
[0034] Figure 6 Result of video classification described in the specific implementation manner: the test accuracy rate (1 - test_acc) of the dataset VGGSound on the test set. Specific implementation manner
[0035] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and make the above - mentioned objects, features, and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0036] Example 1: A video classification method based on knowledge distillation and multimodal fusion, the specific steps are as follows:
[0037] Step 1, extract the audio and visual datasets and pre - process them
[0038] 1.1, construct the audio dataset, use FFmpeg to extract the audio in the video dataset, the single - channel sampling rate is 16000, and the format is wav;
[0039] 1.2, extract the features of the audio, first perform Fourier transform on the audio data to obtain the spectrogram, then perform logarithmic transformation, and finally normalize and convert it into a tensor;
[0040] 1.3, construct the visual dataset, use FFmpeg to extract the image frames in the video dataset, uniformly sample 32 frames of images from them to represent the video modality, and at the same time use the middle image frame of the video to represent the image modality, a total of 33 frames;
[0041] 1.4, extract the features of the visual data, first randomly scale the visual data by 0.8 to 1.2 times, then center - crop it to a size of 224×224, and finally normalize and convert it into a tensor;
[0042] Step 2, build the knowledge distillation architecture
[0043] 2.1. Select the same convolutional neural network for the two modal data extracted by the methods in 1.1 to 1.4, adjust the parameters and train multiple times, and select the best model as the single-modal teacher network;
[0044] 2.2. Use the model output as the knowledge to be transferred, that is, use the KL divergence loss as the distillation loss to measure the difference between the outputs of the teacher network and the student network;
[0045] Step 3. Multimodal fusion student network
[0046] 3.1. Use the same convolutional neural network as the teacher network as the single-modal student network, and adopt a different random initialization from the teacher network;
[0047] 3.2. After obtaining the respective single-modal features for knowledge distillation, use the MMTM module to perform mid-term fusion on the features of each layer of the visual and audio modal student networks;
[0048] 3.3. For the features of the last layer of the visual and audio modal student networks, use the method of adding the SE module after averaging for late fusion.
[0049] 3.4. Add the respective classification losses to the outputs of the visual and audio modal student networks and the output after late fusion. The classification loss uses the cross-entropy function.
[0050] The software environment of this method is FFmpeg and Python 3.7. Use scipy to extract spectrograms with a window length of 512 and an overlap of 274, and use PIL and torchvision as the visual data processing methods.
[0051] Knowledge distillation uses the self-distillation strategy of the same model with different initializations for the teacher and the student to transfer the knowledge of the model output, and at the same time performs distillation on the two modalities of vision and audio.
[0052] The multimodal fusion method uses both mid-term fusion and late fusion. After obtaining the single-modal features, the total loss is directly added by three classification losses and two distillation losses.
[0053] The schematic diagram of the network structure of a video classification method based on knowledge distillation and multimodal fusion proposed by the present invention is as Figure 1 shown. This figure is mainly composed of four parts: a single-modal network, a knowledge distillation module, a multimodal fusion module, and an overall loss function. The overall process mainly includes two stages: the training stage and the verification stage.
[0054] The training stage is the same as in the technical solution. It is carried out on the training set. After training the teacher network, perform knowledge distillation and multimodal fusion on the student network of the same model, and train this method through five loss functions.
[0055] In the verification stage, the knowledge distillation module is removed. This stage is carried out on the validation set. The student network after completion of the training stage is used, and the final classification result is obtained through the multi-modal fusion module. Parameter tuning is performed according to the results of the validation set.
[0056] Finally, the tuned network parameters are adopted. After the training is completed, the classification results are statistically analyzed on the test set, the accuracy is calculated, and the model is evaluated. Thus, the video classification network based on knowledge distillation and multi-modal fusion is constructed and the parameter tuning is completed.
[0057] Example 2: The steps of the present invention will be described in detail below taking the VGGSound dataset as an example.
[0058] Experimental environment: The computer configuration is an Intel(R) processor (3.2 GHz) and 16 GB of random access memory (RAM), Ubuntu 20.04.1 64-bit operating system, NVIDIA RTX 3090 (24 GB) graphics card; the software environment is the deep learning framework PyTorch 1.8.1, py-opencv3.4.2.
[0059] Experimental object: The VGGSound dataset contains nearly 200,000 video clips with a length of 10 seconds, annotated with 309 sound categories. The number of samples in each category is controlled between 300 and 1000, and the categories that cannot guarantee the audio-visual consistency are removed. The defined training set and test set of the dataset are used. The ratio of the training set (such as Figure 2 ) and the test set (such as Figure 3 ) is 9:1, with 180,000 in the training set and 15,000 in the test set. The dataset contains a large number of challenging video classification tasks, such as human behavior recognition (such as singing, applauding, skateboarding, etc.), complex event detection (such as crowd gathering, alarm beeping, people screaming, etc.), musical instrument classification (such as playing the harmonica, playing the piano, playing the zheng, etc.), and scene classification (such as waterfall, swimming pool, rainy day, etc.). The VGGSound dataset provides a CSV file for dataset download. Each row of the CSV file has columns defined as follows:
[0060] YouTube ID: Used for video download.
[0061] Start second: The 10-second video segment corresponding to the category starts at the nth second of the current video.
[0062] Label: Category name (in English).
[0063] Training / test set: Distinguish between the training set and the test set.
[0064] Taking a single video as an example, it is necessary to download the corresponding video according to the ID in the CSV file using sox, and then use FFmpeg to intercept the corresponding video segment according to the start time. For the video segment, use FFmpeg to extract audio data with a single-channel sampling rate of 16,000 and a format of wav, and visual data with 33 evenly sampled frames of images. Finally, construct the corresponding json annotation files (dictionaries containing file names and one-hot annotations) for the training set and test set according to the CSV file to complete the data preparation work.
[0065] Experimental steps:
[0066] In the experiment, ResNet-18 is used as the basic network, and the optimization algorithm Adam is adopted with parameters b1 = 0.9, b2 = 0.999, and the learning rate is 1e-3, which is reduced to 0.1 every 5 epochs. The accuracy is used as the evaluation index for the video classification results.
[0067] (1) Training phase:
[0068] Step 1: Training set data processing. For ResNet-18, the 33rd frame image of the visual dataset (i.e., the middle image frame of the video) is used as the input. For visual data feature extraction, random scaling, center cropping, and normalization of torchvision.Transforms are adopted, and finally a tensor of size 3×224×224 is obtained. For audio data feature extraction, soundfile is used to obtain the audio data and the sampling rate. For audio that does not meet the 10-fold sampling rate, np.tile is used to copy it to the specified length. The audio spectrogram is obtained using scipy.signal.spectrogram with parameters nperseg = 512 and noverlap = 274, and finally a tensor of size 1×274×671 is obtained after normalization.
[0069] Step 2: Teacher network training. ResNet-18 is trained multiple times according to the above configuration, noting that the random number seeds are different each time and different random initializations are used. For the visual modality teacher network, it is trained on the pre-trained model of the imagnet dataset, and finally the best accuracy of the visual teacher network is 0.292. For the audio modality teacher network, the number of input channels of the first convolution needs to be modified to 1, and it is trained without using the pre-trained model. Finally, the best accuracy of the audio teacher network is 0.485.
[0070] Step 3: Multimodal fusion student network. The feature learning part of ResNet-18 includes a basic convolutional head, four basic blocks BasicBlock, and the final fully connected layer. As Figure 4As shown in the figure, the output features of the last fully connected layer are used for knowledge distillation, and the output features of the last two BasicBloc blocks are mid-term fused using the MMTM module. The specific process is as follows: First, the features of the two modalities are compressed through global pooling and then concatenated. Then, the excitation vectors of their respective modalities are obtained through their respective fully connected layers. Finally, the excitation vectors are multiplied by the original modality features, and the formula is
[0071]
[0072] where represents the features of the l-th layer of the two modality models, represents global average pooling, represents the fully connected layer of the visual modality, represents the visual features after mid-term fusion. The features obtained from mid-term fusion are continued to pass through the BasicBloc block, and the output of the second MMTM fusion is output. Late fusion uses a squeeze-and-excitation (SE) module as shown in Figure 5 to fuse the output features of the model.
[0073] Step 4: Model training. For each input, the output prediction is calculated using the teacher network, and the output prediction and mid-term fusion output are obtained using the student network. The late fusion is used to fuse the mid-term fusion outputs of the two modality student networks to obtain the final multi-modal joint prediction. The total loss is the classification loss of the output predictions of the two student networks, the classification loss of the multi-modal joint prediction, and the KL divergence knowledge distillation loss between the output predictions of the two modality student networks and the teacher network. The model is trained for a total of 50 epochs.
[0074] (2) Testing phase:
[0075] Step 1: Remove the knowledge distillation module. For the testing phase, only the prediction results and classification loss of the multi-modal fusion of the student network are considered.
[0076] Step 2: Visualization. The test results are as shown in the appendix Figure 6 Using all training samples to train the vision for 50 epochs, and using accuracy as the measurement standard. For multi-classification problems, calculate the average correct rate over all prediction values. The higher the accuracy, the higher the classification correct rate of the model, and the better the model performance. As can be seen from Figure 6 After using this method, the accuracy of the student network has been improved, and the multi-modal fusion accuracy far exceeds that of the teacher network. At the same time, the multi-modal fusion accuracy obtained by using this method is the best among various methods using the ResNet-18 network on the current VGGSound dataset.
[0077] The above is the preferred embodiment of this experiment. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements should also be regarded as the protection scope of the present invention.
Claims
1. A video classification method based on knowledge distillation and multimodal fusion, characterized in that knowledge distillation is used for unimodal networks and multimodal fusion is performed on unimodal student networks. The specific steps are as follows: Step 1: Extract audio and visual data sets and preprocess them. Step 2: Build a knowledge distillation architecture. Step 3: Multimodal fusion student network. Among them, Step 1 is specifically as follows: 1.
1. Construction of audio dataset. FFmpeg is used to extract the audio from the video dataset. The single-channel sampling rate is 16,000 and the format is wav. 1.
2. Feature extraction of audio. First, the Fourier transform is performed on the audio data to obtain the spectrogram, then the logarithmic transform is carried out, and finally, it is normalized and converted into a tensor. 1.
3. Construction of visual dataset. FFmpeg is used to extract the image frames from the video dataset. 32 frames of images are uniformly sampled from them to represent the video modality, and at the same time, the middle image frame of the video is used to represent the image modality, a total of 33 frames. 1.
4. Feature extraction of vision. First, the visual data is randomly scaled by 0.8 to 1.2 times, then centrally cropped to a size of 224×224, and finally normalized and converted into a tensor. Step 2 is as follows: 2.
1. For the two-modal data extracted by the methods in 1.1 to 1.4, select the same convolutional neural network, adjust the parameters and train multiple times, and select the best model as the single-modal teacher network. 2.
2. Use the model output as the transferred knowledge, that is, use the KL divergence loss as the distillation loss to measure the difference between the teacher network and the student network outputs. Step 3 is as follows: 3.
1. Use the same convolutional neural network as the teacher network as the single-modal student network, and adopt a different random initialization from the teacher network. 3.
2. After obtaining the respective single-modal features for knowledge distillation, the features of each layer of the visual and audio modality student networks are fused by the mid-term fusion method. 3.
3. For the features of the last layer of the visual and audio modality student networks, the late fusion is carried out by the method of adding the SE module after averaging. 3.
4. Add the respective classification losses to the visual and audio modality student networks and the output after late fusion. The classification loss uses the cross-entropy function.
2. A video classification method based on knowledge distillation and multi-modal fusion according to claim 1, characterized in that the software environment of this method is FFmpeg and Python 3.
7. SciPy is used to extract the spectrogram with a window length of 512 and an overlap of 274, and PIL and torchvision are used as the visual data processing methods.
3. A video classification method based on knowledge distillation and multi-modal fusion according to claim 2, characterized in that Use the self-distillation method with the same model but different initializations of the teacher and the student to transfer the knowledge of the model output, and at the same time perform distillation on the two modalities of vision and audio.
4. A video classification method based on knowledge distillation and multimodal fusion according to claim 3, characterized in that In the mid-term fusion, after obtaining the single-modal features, the total loss is the direct addition of three classification losses and two distillation losses.
Citation Information
Patent Citations
Multi-modal knowledge distillation method and system
CN113361396A