Short video multi-label classification method and device based on multi-modal knowledge distillation

Abstract

The application discloses a kind of short video multi-label classification method and device based on multimodal knowledge distillation, method includes: the loss function of teacher neural network and student network includes the loss between each modality in the feature layer in the first knowledge transfer and the reconstruction loss of the feature representation output between teacher network and student network in the second knowledge transfer, and student network final classification loss, and this objective function guides student network to find the optimal solution of student network model;Using back propagation to calculate the deviation between output layer objective function result and true value, adjust parameters layer by layer, finally obtain the local optimal solution of student network model;Based on local optimal solution, the best student network model distilled by teacher network training is obtained, and the best student network is applied to short video multi-label classification, and the classification result of short video multi-label is output.The device includes: processor and memory.The application improves the accuracy of short video multi-label classification task, and has important practical application value.

Description

Technical Field

[0001] This invention relates to the field of multi-label classification of short videos, and in particular to a method and apparatus for multi-label classification of short videos based on multimodal knowledge distillation. Background Technology

[0002] With the rise of short video platforms, multi-tag classification technology for short videos has become increasingly important. Multi-tag classification refers to assigning multiple tags to a short video, commonly used in short video recommendation and management. Multi-tag classification is a crucial task in computer vision and machine learning, its challenge lying in mapping different video content to multiple tags, involving knowledge from multiple domains. As a new type of multimedia communication medium, short videos typically derive their content information from multiple modalities, such as visual, audio, and trajectory modalities. Unlike images or text, strengthening the information fusion between multiple modalities in short videos is crucial for obtaining a complete representation of the short video content. This plays a vital role not only in the classification control of short videos but also in personalized recommendations.

[0003] Currently, deep neural networks excel in various tasks within the field of computer vision. However, the complex structure of these networks and the millions or even billions of parameters make their training extremely costly in terms of computing power and time. Furthermore, the excessive parameterization of these models makes their deployment on mobile devices prohibitively expensive and unsuitable for real-time tasks. Traditional multi-label classification methods typically require large amounts of training data and highly optimized models, limiting their effectiveness in practical applications.

[0004] In short video multi-label classification tasks, the importance of different modal information varies, with visual information often being the most accurate and stable. Therefore, constructing video content representations primarily based on visual modal information is reasonable. Improving the accuracy and robustness of short video multi-label classification has become a current research hotspot and challenge. Summary of the Invention

[0005] This invention provides a method and apparatus for multi-label classification of short videos based on multimodal knowledge distillation. This invention offers a novel approach to solving the modality fusion problem, reducing model parameters and computational complexity while maintaining model accuracy, thereby improving the accuracy of short video multi-label classification tasks. It has significant practical application value, as detailed below:

[0006] A multi-label classification method for short videos based on multimodal knowledge distillation, the method comprising:

[0007] A multi-head attention mechanism is constructed for classification, which primarily utilizes visual features from videos and secondarily uses audio and trajectory features.

[0008] Text label encoding information is introduced into the teacher neural network. Visual features, audio features, and trajectory features are fused through cross-modal attention layers and shared attention layers. The fused results are then used to guide the output of the student network.

[0009] Knowledge transfer is performed on the fused results of the teacher's neural network output to guide the final video feature representation output of the student network;

[0010] The loss functions of the teacher neural network and the student network include the loss between each modality in the feature layer during the first knowledge transfer, the reconstruction loss of the feature representation output between the teacher network and the student network during the second knowledge transfer, and the final classification loss of the student network. This objective function guides the student network to find the optimal solution of the student network model.

[0011] Backpropagation is used to calculate the deviation between the output layer objective function result and the true value. Parameters are adjusted layer by layer to finally find the local optimum of the student network model.

[0012] The optimal student network model, distilled from the teacher network training, is obtained based on the local optimal solution. This optimal student network is then applied to short video multi-label classification to output the classification results of short video multi-labels.

[0013] The student network model is as follows: visual features, audio features, and trajectory features are first encoded by an MLP, and the output of the intermediate feature layer in the teacher neural network is used to guide the parameter configuration of the MLP, thereby narrowing the gap between the two networks.

[0014] A short video multi-label classification device based on multimodal knowledge distillation, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to execute any of the steps of the method described above.

[0015] The beneficial effects of the technical solution provided by this invention are:

[0016] 1. This invention explores the multimodal representation learning problem in short video classification. From the perspective of knowledge distillation in the "teacher-student" network, this invention can utilize a complex and efficient teacher model to train a simple and effective student model through knowledge transfer. This approach can reduce model parameters and computational complexity while ensuring model accuracy, and has significant practical application value.

[0017] 2. This invention considers that self-supervised learning can mine its own supervisory information from large-scale unsupervised data. By training the network with this constructed supervisory information, it can learn representations that are valuable to student tasks, which provides a new approach to solving the modality fusion problem.

[0018] 3. From the perspective of modal consistency, since visual information is often the most accurate and stable, it is reasonable to construct a video content representation based on visual modal information. Therefore, it is meaningful for this invention to construct a multi-head attention mechanism (MHA) based on visual features and supplemented by audio features and trajectory features.

[0019] This invention fully utilizes the multimodal information of short videos to perform multi-label classification of short videos, and also provides a novel modality fusion method and an innovative network architecture, thereby improving the accuracy of short video multi-label classification tasks. Attached Figure Description

[0020] Figure 1 A schematic diagram of the overall knowledge distillation process;

[0021] Figure 2 A diagram of the neural network framework for students. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0023] Example 1

[0024] This invention provides a multi-label classification method for short videos based on multimodal knowledge distillation, which fully utilizes the multimodal information of short videos and aims to solve the short video classification problem. See details below. Figure 1 The method includes the following steps:

[0025] 101: Extract visual, audio, and trajectory features from the video and construct a multi-head attention mechanism (MHA) with visual features as the main component and audio and trajectory features as auxiliary components for classification.

[0026] Short video content contains multiple modal information. Here, we take three modalities as an example and extract the visual, audio, and trajectory features of the video respectively. Considering that visual information is the most accurate, reliable, and least likely to be lost in a short video, it is reasonable to construct a video content representation based on visual modal information. Therefore, this embodiment of the invention constructs a multi-head attention mechanism (MHA) based on visual features as the main feature and audio and trajectory features as auxiliary features for classification.

[0027] 102: Text label encoding information is introduced into the teacher neural network, and multiple features are fused through cross-modal attention layers and shared attention layers to extract better video features, which are then used to guide the output of the student network;

[0028] In practical implementation, within the teacher network, considering that self-supervised learning can extract its own supervisory information from large-scale unsupervised data, this constructed supervisory information can be used to train the network. Text label encoding information is introduced into the teacher neural network, and multiple features are fused through cross-modal attention layers and shared attention layers to extract better video features. These features are then used to guide the output of the student network. This involves knowledge transfer at two feature layers.

[0029] Among them, see Figure 2 The two knowledge transfers were: the first was the knowledge transfer at the teacher network feature processing level, and the second was the knowledge transfer at the final feature representation level.

[0030] 103: Construct a multi-head attention mechanism (MHA) that primarily uses visual features and secondarily uses audio and trajectory features to extract video features. Through knowledge transfer from the teacher network's output, guide the final video feature representation output of the student network.

[0031] In practical applications, when designing the student network, it is necessary to ensure that the knowledge of the teacher network and the student network is interconnected. A reasonable student model construction involves first encoding the three modal features through an MLP, and then using the output of the intermediate feature layer in the teacher network to guide the parameter configuration of the MLP, thereby narrowing the gap between the two networks. To ensure modal consistency, a multi-head attention mechanism (MHA) is constructed, primarily based on visual features and supplemented by audio and trajectory features, to extract video features. Then, knowledge transfer from the teacher network's output guides the final video feature representation output of the student network.

[0032] 104: The loss function of the entire network includes the loss between each modality in the feature layer of the first knowledge transfer, the reconstruction loss between the two network outputs in the second knowledge transfer, and the final classification loss. This objective function is used to guide the student network to find the optimal solution of the model.

[0033] In this embodiment of the invention, the performance of the scheme is evaluated using five evaluation metrics: coverage, ranking loss, average accuracy, Hamming loss, and first label error, to ensure the objectivity and accuracy of the experimental results.

[0034] 105: The deviation between the output layer objective function result and the true value is calculated by backpropagation, and the parameters are adjusted layer by layer to finally find the local optimum of this model;

[0035] 106: Based on the local optimal solution, the best student network model distilled from the teacher network is obtained, and the best student network is applied to the multi-label classification of short videos to output the classification results of the multi-label classification of short videos.

[0036] In summary, the embodiments of the present invention provide a new approach to solving the modality fusion problem through the above steps 101-106, which can reduce model parameters and computational complexity while ensuring model accuracy, thereby improving the accuracy of short video multi-label classification tasks.

[0037] Example 2

[0038] The scheme in Example 1 will be further described below with reference to calculation formulas and examples. See the description below for details:

[0039] 201: This model takes a complete multimodal short video as input and extracts three modalities from it, including: visual, audio and trajectory features;

[0040] For the visual modality, this model extracts keyframes from the video and uses a classic visual feature extraction residual network (ResNet) to extract features. Subsequently, the extracted features are subjected to average pooling to obtain the visual modality features X. v Overall characteristics f v :

[0041] f v =Average Pooling(ResNet(X v ;β v (1)

[0042] Among them, X v β represents the original visual features of the short video. v Let d be the network parameters to be learned. v For visual modal features f v In the dimension of the feature map, Average Pooling is the average pooling method, which sums all the pixel values ​​of the feature map and takes the average to obtain a single value, which is used to represent the corresponding feature map.

[0043] For audio modalities, a sound spectrogram is plotted, and sound features f are extracted from the spectrogram using CNN (Convolutional Neural Network) and LSTM (Long Short-Term Memory Network). a :

[0044] f a =LSTM(X) a ;β a (2)

[0045] Among them, X a β represents the original audio features of the short video. aLet d be the network parameters to be learned. a For audio modal features f a Dimensions.

[0046] For trajectory modes, the TDD (Track Pooling Depth Convolution Descriptor) method is used to jointly extract trajectory mode features f from the time and spatial domains. t :

[0047] f t =TDD(X) t ;β t (3)

[0048] Among them, X t For the original trajectory information of the short video, β t Let d be the network parameters to be learned. t For trajectory mode features f t Dimensions.

[0049] 202: Constructing a teacher neural network;

[0050] In this embodiment of the invention, the tag features f encoded by the tag set are first added to the teacher network. l For visual modal features f v Audio modal features f a Trajectory mode features f t (All obtained in step 201) Construct cross-modal fusion modules (cross Attention, i.e., A&L attention, V&L attention, T&L attention) with these modules respectively, and add label information to the three modalities of the short video to better guide the student network, based on:

[0051]

[0052] The output of A&L attention is:

[0053]

[0054] Among them, using f a Generate the key matrix and value matrix, using f l Generate a query matrix. To generate the weights of the query matrix, To generate the weights of the key matrix, To generate the weights of the value matrix and prevent gradient vanishing, a scaling factor d is introduced. k .

[0055] Similarly, the output CA of V&L attention and T&L attention is obtained. v CAt .

[0056] Simultaneously, in this embodiment of the invention, the three modal features are also fed into the parameter-shared self-attention encoder sharedAttention to obtain the visual modal features f. v Audio modal features f a Trajectory mode features f t SA's unique representation v (f v SA a (f a SA t (f t ).

[0057] Then, the three outputs from these two parts are concatenated together and used as the output of the teacher's neural network feature layer to guide the student network's feature learning. That is:

[0058] X m =concat(CA) m SA m (f m (6)

[0059] Where m∈{a, v, t}, then X a X v X t The video is fed into the decoder to obtain the final short video representation Z of the teacher network.

[0060] Z = LN(MHA(X) a X v X t (7)

[0061] Where MHA represents a multi-head attention mechanism network, LN represents its normalization, and X a X v X t This represents the output for three modes.

[0062] 202: Constructing a student neural network;

[0063] In a student network, to design a student model and ensure knowledge sharing between the two networks, this embodiment of the invention constructs the following: First, three modal features are encoded, and a multilayer perceptron (MLP) is used for feature extraction. Then, the output of the intermediate feature layer containing label feature information learned by the teacher network is used to guide the parameter configuration of the MLP, thereby narrowing the gap between the two networks and achieving optimal learning performance. The three features f a f v f tThe output after MLP is x a x v x t :

[0064]

[0065] Where m∈{a, v, t}, This is the output of the student network feature layer.

[0066] From the perspective of modality consistency, since visual information is often the most accurate and stable, it is reasonable to construct video content representations primarily based on visual modality information. This invention proposes a multi-head attention mechanism (MHA), where visual features are primary, and audio and trajectory features are secondary. Knowledge transfer from the teacher network's output guides the student network's final video representation output. This method enhances modality consistency and improves learning effectiveness. Here, three modalities are used as examples, constructing feature representations for visual and audio, visual and trajectory, and the three modalities respectively. (Where, visual modality features f...) v Audio modal features f a Trajectory mode features f t (All are obtained in step 201).

[0067] The encoding rules of the Multi-Head Attention (MAT) network φ are as follows:

[0068]

[0069] The inputs are Q, K, and V, representing the query matrix, value matrix, and key matrix, respectively. To prevent gradient vanishing, a scaling factor d is introduced. k .

[0070] 1) Visual and audio modal fusion

[0071] Utilizing the extracted visual modality features f v and audio features f a Learn the complementary features z1 of the two:

[0072]

[0073] Among them, using f v Generate the key matrix and value matrix, using f a Generate a query matrix. To generate the weights of the query matrix, To generate the weights of the key matrix, To generate the weights of the value matrix and prevent gradient vanishing, a scaling factor d is introduced.k .

[0074] 2) Visual modality and trajectory modality fusion

[0075] Utilizing the extracted visual modality features f v and audio features f t Learn the complementary features z2 of the two:

[0076]

[0077] Among them, using f v Generate the key matrix and value matrix, using f t Generate a query matrix. To generate the weights of the query matrix, To generate the weights of the key matrix, To generate the weights of the value matrix and prevent gradient vanishing, a scaling factor d is introduced. k .

[0078] ① Fusion of visual modality, audio modality and trajectory modality

[0079] A modal complementarity encoder Φ(·) based on a multi-head attention mechanism is constructed to encode visual modal features f. v Based on the modal complementarity features, the content representation of short videos is obtained as a whole:

[0080] Attention(f v f a f t )=z1+z2 (12)

[0081] Where, respectively f v and f a f t By combining the two fusion results, a complete modal representation is constructed.

[0082] The output of Φ is as follows:

[0083] Z * =Φ(f v f a f t )=LN(f a +f t +Attention(f v f a f t (13)

[0084] By concatenating the outputs of all attention heads using the `concat` operation, the final representation of each modality group is obtained:

[0085]

[0086] in, Describe a linear projection matrix. This represents the output of the j-th head of the i-th modal group, where h is the number of heads.

[0087] 203: To better explore knowledge transfer, three components of loss are introduced here;

[0088] 1) The loss between the feature layer outputs of the teacher network and the student network;

[0089] The loss between the feature layer outputs of the teacher network and the student network includes: L v For visual feature layer loss; L t For trajectory feature layer loss; L a The loss is output for the audio feature layer. The calculation method is as follows:

[0090]

[0091] in, Describing the F-norm,

[0092] 2) Introduce the reconstruction loss L based on the feature representations of each teacher-student network. recon ;

[0093] Traditionally expressed as:

[0094]

[0095] Among them, Z * For student networks f v and f a f t The two fusion results are added together to construct a complete modal representation. Z represents the complete modal representation in the teacher network. This represents the square of the F-norm.

[0096] Introducing the cross-entropy loss L between modal representations and labels in the teacher network p :

[0097]

[0098] in, For Z and f l Generate a baseline label vector for the i-th entity, and L is the predicted label vector corresponding to the i-th entity. p For multi-class cross-entropy loss, σ(·) is the sigmoid activation function.

[0099] 204: Reducing model parameters and computational complexity while ensuring model accuracy has significant practical application value. Therefore, taking three modalities as an example, the overall loss function L of the model is:

[0100] L=λ1L a +λ2L v +λ3L t +λ4L recon +μL cls-z +ωL P (18)

[0101] Where λ1, λ2, λ3, λ4, μ, and ω are the hyperparameters of the corresponding loss terms.

[0102] Throughout the training and testing process, the model's performance was evaluated using five metrics: Coverage, Ranking Loss, Average Precision (mAP), Hamming Loss, and One-error. Specifically: (1) Coverage is used to calculate the average number of labels needed to cover all correct labels of an instance. It is loosely related to the precision at the best level of recall. The smaller the value, the better the performance. (2) Ranking Loss is used to calculate the average score of the reversed label pairs of an instance. The smaller the value, the better the performance. (3) mAP represents the average accuracy of m categories. The larger the value, the better the performance. (4) Hamming Loss measures the number of times a label is misclassified. The smaller the value, the better the performance. (5) One-error is used to calculate the number of times the label with the highest predicted probability is not in the true label set. The smaller the value, the better the performance.

[0103] Example 3

[0104] The dataset used in this embodiment of the invention is a large-scale multi-label short video real-time classification dataset released in the 2018 AI Challenge, containing 200,000 short videos covering 63 popular element tags such as dance, fitness, and singing. Most of these videos are 5-15 seconds long and are assigned 1-3 tags. Since this method is supervised learning, only the labeled short video dataset was used. The dataset was divided into training and testing sets in a 4:1 ratio, and three modal features—visual, audio, and trajectory—were extracted for validation. These three modal features were used as examples to verify the effectiveness in short video classification. In all experiments, the model was built using the PyTorch deep learning framework, and the hardware environment deployed during training and testing was a GPU GeForce GTX 2080Ti, a CPU Intel i7 6700, and an operating system environment of Ubuntu 18.04. The parameters are initialized using Xavier, and the optimization method is to use the SGD optimizer (Stochastic Gradient Descent, SGD). The momentum value is set to 0.999, the weight decay is set to 1e-3, the base learning rate is set to 1e-3, the batch size is set to 64, the default number of MHA heads is 4, and the default number of generative adversarial networks is 3.

[0105] The experimental results are shown in Table 1. Table 1 shows that the classification effect of the present invention is superior to that of other existing classification methods. The conclusion is that the short video multi-label classification method based on multimodal knowledge distillation proposed in this invention makes full use of knowledge distillation to train an effective student network model, and learns the consistency and complementarity between multimodalities through the most reliable visual features.

[0106] In summary, this invention provides a multi-modal knowledge distillation-based multi-label classification method for short videos, aiming to solve the short video classification problem. For videos containing multiple modalities, this invention takes three modalities as examples, extracting visual, audio, and trajectory features from the video respectively. Considering that self-supervised learning can extract its own supervisory information from large-scale unsupervised data, this constructed supervisory information can be used to train the network. To introduce text label encoding information, a cross-modal attention layer and a shared attention layer are added to the teacher neural network to fuse multiple features, extract better video features, and use these features to guide the student network output. In this process, two knowledge transfer operations are performed at the feature layers. When fusing the three modalities in the student network, considering that the visual modality has the highest accuracy and strongest reliability, a multi-head attention coding network is used to construct a modal complementary feature representation dominated by the visual modality and a fused feature representation under the three modalities, respectively, from the perspective of modal complementarity. Finally, this invention is applied to the short video classification problem.

[0107] Table 1

[0108]

[0109] Example 4

[0110] A multi-modal knowledge distillation-based short video multi-label classification device includes a processor and a memory. The memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to execute any one of the following method steps:

[0111] A multi-head attention mechanism is constructed for classification, which primarily utilizes visual features from videos and secondarily uses audio and trajectory features.

[0112] Text label encoding information is introduced into the teacher neural network. Visual features, audio features, and trajectory features are fused through cross-modal attention layers and shared attention layers. The fused results are then used to guide the output of the student network.

[0113] Knowledge transfer is performed on the fused results of the teacher's neural network output to guide the final video feature representation output of the student network;

[0114] The loss functions of the teacher neural network and the student network include the loss between each modality in the feature layer during the first knowledge transfer, the reconstruction loss of the feature representation output between the teacher network and the student network during the second knowledge transfer, and the final classification loss of the student network. This objective function guides the student network to find the optimal solution of the student network model.

[0115] Backpropagation is used to calculate the deviation between the output layer objective function result and the true value. Parameters are adjusted layer by layer to finally find the local optimum of the student network model.

[0116] The optimal student network model, distilled from the teacher network training, is obtained based on the local optimal solution. This optimal student network is then applied to short video multi-label classification to output the classification results of short video multi-labels.

[0117] The student network model consists of visual features, audio features, and trajectory features first encoded by an MLP, and then using the output of the intermediate feature layer in the teacher neural network to guide the parameter configuration of the MLP, thereby narrowing the gap between the two networks.

[0118] Furthermore, the teacher neural network is as follows:

[0119] The three modal features are fed into a parameter-shared self-attention encoder to obtain visual modal features f. v Audio modal features f a Trajectory mode features f t SA's unique representation v(f v SA a (f a SA t (f t );

[0120] The three outputs are concatenated together and used as the output of the teacher's neural network feature layer to guide the feature learning of the student network.

[0121] X m =concat(CA) m SA m (f m ))

[0122] Where m∈{a, v, t}, X a X v X t The video is fed into the decoder to obtain the final short video representation Z of the teacher network:

[0123] Z = LN(MHA(X) a X v X t ))

[0124] Where MHA represents Multi-Head Attention Network, LN represents Normalization, and X a X v X t This represents the output for three modes.

[0125] The content of the short video is represented as follows:

[0126] Attention(f v f a f t )=z1+z2

[0127] Among them, f is used respectively v and f a f t By combining the two fusion results, a complete modal representation is constructed.

[0128] The output of Φ is as follows:

[0129] Z * =Φ(f v f a f t )=LN(f a +f t +Attention(f v f a f t ))

[0130] The outputs of all attention heads are concatenated using the `concat` operation to obtain the final representation of each modality group:

[0131]

[0132] in, Describe a linear projection matrix. This represents the output of the j-th head of the i-th modal group, where h is the number of heads.

[0133] Furthermore, the losses between modalities in the feature layer during the first knowledge transfer, the reconstruction loss of feature representation outputs between the teacher network and the student network during the second knowledge transfer, and the final classification loss of the student network are specifically as follows:

[0134] 1) The loss between the feature layer outputs of the teacher network and the student network;

[0135] The loss between the feature layer outputs of the teacher network and the student network includes: L v For visual feature layer loss; L t For trajectory feature layer loss; L a The loss for the audio feature layer output is calculated as follows:

[0136]

[0137] in, Denotes the F-norm;

[0138] 2) Introduce the reconstruction loss L based on the feature representations of each teacher-student network. recon ;

[0139]

[0140] Among them, Z * For student networks f v and f a f t The two fusion results are added together to construct a complete modal representation, where Z is the complete modal representation in the teacher network. Denotes the square of the F-norm;

[0141] Introducing the cross-entropy loss L between modal representations and labels in the teacher network p :

[0142]

[0143] in, For Z and f l Generate a baseline label vector for the i-th entity. L is the predicted label vector corresponding to the i-th entity. p For multi-class cross-entropy loss, σ(·) is the sigmoid activation function;

[0144] The overall loss function L is:

[0145] L=λ1L a +λ2L v +λ3I t +λ4L recon +μL cls-z +ωL P

[0146] Where λ1, λ2, λ3, λ4, μ, and ω are the hyperparameters of the corresponding loss terms.

[0147] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0148] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0149] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0150] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0151] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-label classification method for short videos based on multimodal knowledge distillation, characterized in that, The method includes: A multi-head attention mechanism is constructed for classification, which primarily utilizes visual features from videos and secondarily uses audio and trajectory features. Text label encoding information is introduced into the teacher neural network. Visual features, audio features, and trajectory features are fused through cross-modal attention layers and shared attention layers. The fused results are then used to guide the output of the student network. Knowledge transfer is performed on the fused results of the teacher's neural network output to guide the final video feature representation output of the student network; The loss functions of the teacher neural network and the student network include the loss between each modality in the feature layer during the first knowledge transfer, the reconstruction loss of the feature representation output between the teacher network and the student network during the second knowledge transfer, and the final classification loss of the student network. This objective function guides the student network to find the optimal solution of the student network model. Backpropagation is used to calculate the deviation between the output layer objective function result and the true value. Parameters are adjusted layer by layer to finally find the local optimum of the student network model. The optimal student network model, distilled from the teacher network training, is obtained based on the local optimal solution. This optimal student network is then applied to short video multi-label classification to output the classification results of short video multi-labels. The student network model is as follows: visual features, audio features, and trajectory features are first encoded by an MLP, and the output of the intermediate feature layer in the teacher's neural network is used to guide the parameter configuration of the MLP, thereby narrowing the gap between the two networks. The teacher neural network is as follows: The three modal features are fed into a parameter-shared self-attention encoder to obtain visual modal features. Audio modal features Trajectory modal features Unique representation ( ), ( ), ( ); The three outputs are concatenated together and used as the output of the teacher's neural network feature layer to guide the feature learning of the student network. in, ,Will The data is fed into the decoder to obtain the final short video representation of the teacher network. : Where MHA stands for Multi-Head Attention Network, and LN stands for Normalization. This represents the output for three modes.

2. The short video multi-label classification method based on multimodal knowledge distillation according to claim 1, characterized in that, The content of the short video is represented as follows: in, These are complementary features; As complementary features; respectively using and , By combining the two fusion results, a complete modal representation is constructed. The output is as follows: The outputs of all attention heads are concatenated using the `concat` operation to obtain the final representation of each modality group: （ ） in, Describe a linear projection matrix. Indicates the first The modality group of the first The output of each head, It refers to the number of heads.

3. The short video multi-label classification method based on multimodal knowledge distillation according to claim 1, characterized in that, The losses between modalities in the feature layer during the first knowledge transfer, the reconstruction loss of feature representation outputs between the teacher network and the student network during the second knowledge transfer, and the final classification loss of the student network are specifically as follows: 1) The loss between the feature layer outputs of the teacher network and the student network; The loss between the feature layer outputs of the teacher network and the student network includes: For visual feature layer loss; For trajectory feature layer loss; The loss for the audio feature layer output is calculated as follows: in, express Norm; 2) Introduce reconstruction loss based on the feature representations of each teacher-student network. ; in, For student networks and , The two fusion results are added together to construct a complete modal representation. To provide a complete modal representation in the teacher network, Denotes the square of the F-norm; Introducing cross-entropy loss between modal representations and labels in teacher networks : in, , For Z and Generate a baseline label vector for the i-th entity. , It is the predicted label vector corresponding to the i-th entity. For multi-class cross-entropy loss, It is the sigmoid activation function; Overall loss function for: ； in, , , , , , These are the hyperparameters of the corresponding loss terms.

4. A short video multi-label classification device based on multimodal knowledge distillation, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the steps of the method according to any one of claims 1-3.

Images