An automatic short video annotation method based on modality diversity

Abstract

The present invention provides a short video automatic annotation method based on modality diversity, which belongs to the field of automatic annotation, and includes: obtaining three modalities to describe the overall representation of the short video; combining the three modalities in pairs to form three modal groups, and inputting each modal combination into a multi-head attention mechanism to obtain enhanced visual features f v , enhanced trajectory features f t and enhanced audio features f a , and obtain the consistency loss function to realize the restriction of modal relationship; respectively obtain the visual specific representation #imgabs0#, the trajectory specific representation #imgabs1# and the audio specific representation #imgabs2#, and then obtain the disentanglement loss function; use the formula #imgabs3# to realize the modulation of each modality and obtain the modulation amount #imgabs4#; according to the consistency loss function, the disentanglement loss function and the automatic retrieval loss function, the total loss function is obtained, and the training is performed with the total loss function. The present invention enhances the representation ability of the modality and improves the accuracy of the annotation.

Description

Technical Field

[0001] The present invention belongs to the field of automatic annotation, and in particular relates to a short video automatic annotation method based on modality diversity. Background Art

[0002] In recent years, with the rapid progress of multimedia technology, short videos have become a popular form of communication, attracting a large user group. However, how to provide users with more accurate and personalized services has become a major challenge facing the short video industry. At present, the research directions in the field of short videos are mainly personalized recommendations, popularity prediction, and memorability modeling. The automatic annotation function of short videos is of great significance for the efficient use of video resources, convenient retrieval, classification and management. However, since short videos contain multimodal information, this poses a challenge to multi-label classification. The key lies in how to effectively integrate these multimodal information to form an accurate multimodal representation. Past methods often ignore the consideration of modal diversity when pursuing multimodal consistent representation, which may lead to modal bias. Modal bias not only causes the representation to deviate from the video content, but also may hinder the fusion of information. Therefore, how to overcome these problems and complete automatic annotation with higher accuracy has become the focus of current research.

[0003] In order to obtain more accurate multimodal representations, researchers have conducted research from two perspectives: joint representation learning and coordinated representation learning. Joint representation learning aims to project information from different modalities into a shared space and then integrate the modal information; coordinated representation learning focuses on exploring the potential relationship between them while maintaining the independence of each modality representation. These studies provide many effective solutions for exploring the consistency and complementarity of multi-label classification.

[0004] In multi-label classification, category correlation is a major issue. Early methods such as binary association and calibrated label ranking explored the correlation between two labels. However, these shallow networks have difficulty modeling complex label relationships. In recent years, more and more multi-label classification methods based on graph networks have been developed. However, in multimodal scenarios, multi-label classification faces a more complex challenge - modeling the potential relationship between modalities and multiple labels. Summary of the invention

[0005] The purpose of the present invention is to propose a short video automatic annotation method based on modality diversity, which enhances the representation ability of the modality and improves the accuracy of annotation.

[0006] The present invention is achieved through the following technical solutions:

[0007] A short video automatic annotation method based on modality diversity includes the following steps:

[0008] Step S1, obtaining three modalities from the short video dataset to describe the overall representation of the short video, the three modalities being visual features, trajectory features, and audio features;

[0009] Step S2: Combine the three modalities into three modal groups, and input each modal combination into the multi-head attention mechanism to obtain the trajectory enhanced visual feature f v,t , audio enhanced visual featuresf v,a , visually enhanced trajectory features f t,v , audio enhancement trajectory features f t,a , visually enhanced audio features f a,v and the track-enhanced audio features f a,t , and then get the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a , and obtain the consistency loss function to realize the restriction of modal relationship;

[0010] Step S3: Based on the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a Get the consistent representation of the short video f o , and obtain visual specific representations based on the consistent representation of the video , trajectory specific representation and audio specific representation , and then get the disentanglement loss function;

[0011] Step S4: Based on the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a Get the mode-oriented representation F m , and according to the modal-oriented representation F m Construct a graph to get the graph convolutional network H m , using the formula Realize the modulation of each mode and obtain the modulation amount ,in, m ∈{ v, t, a}, γ m and β m All are modulation parameters;

[0012] Step S5: Fuse the modulation values ​​of the three modes and obtain the category embedding matrix H final , and obtain the automatic retrieval loss function, which is related to the predicted label representation, which is based on the category embedding matrix H finalObtain, based on the consistency loss function, the disentanglement loss function and the automatic retrieval loss function, the total loss function is obtained, and training is performed with the total loss function.

[0013] Furthermore, in step S1, a sample set is obtained from the short video data set at a set interval, and a sample is selected from each sample in the sample set. L The selected images are input into the pre-trained ImageNet-VGG-2048 network to obtain the visual features of the sample set; the trajectory features of the sample set are extracted using the trajectory pooling deep convolutional descriptor; and the audio features of the sample set are extracted using the 2048D-Mel frequency cepstral coefficients.

[0014] Further, in step S2, according to the formula , , Get enhanced visual features f v , enhanced trajectory features f t and enhanced audio features f a .

[0015] Furthermore, in step S2, a fully connected layer is set to enhance the visual feature f v , enhanced trajectory features f t and enhanced audio features f a Input the fully connected layer to get the consistency loss function ,in, N represents the number of samples in the sample set, ( v, t )、( v, a ) ( t, a ) represents three modal groups, FC(.) represents the fully connected layer, represents the norm, Indicates the sample set i The enhanced visual features f of samples v , enhanced trajectory features f t and enhanced audio features f a .

[0016] Further, in step S3, according to the formula Get consistent representation of the video , respectively according to the formula Get visual specific representation , trajectory specific representation and audio specific representation , and then get the disentanglement loss function , where Avgpool represents average pooling and Ω represents the real domain.

[0017] Furthermore, in step S4, the enhanced visual features f v , enhanced trajectory features f t and enhanced audio features f a Input the corresponding encoder to obtain the modality-oriented representation F m =E m (f m ), where E m (.) denotes an encoder consisting of two layers of multilayer perceptrons.

[0018] Furthermore, in step S4, the modality-oriented representation F m Expand by row to get the category associated Z m , Z m Each behavior category is represented by Z m Build a graph structure for the nodes to obtain a graph convolutional network ,in, represents the activation function, A m Represents Z m The adjacency matrix, W m is the weight matrix.

[0019] Further, in step S4, according to the formula and Get modulation parameters γ m and β m ,in, and is the parameter generator, Contains a linear layer, Contains a multi-layer perceptron, f fuse =Concat(f v ;f t ;f a ) is the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a fusion result.

[0020] Furthermore, in step S5, the self-attention mechanism is used to obtain the embedding matrix ,in, It represents the fusion of the modulation quantities of the three modes, and SA(.) represents the self-attention function.

[0021] Furthermore, in step S5, the total loss function is expressed as ,in, To automatically retrieve the loss function, C is the number of categories, yj is the true label representation, s j is a prediction representation, which s j Youjiang C Category representation h j enter C A binary classifier is obtained, and the category representation h j Embedded matrix by category H final It is obtained that is the Sigmoid activation function, λ1 and λ2 are the weights of the consistency loss function and the disentanglement loss function.

[0022] The present invention has the following beneficial effects:

[0023] 1. The present invention obtains three modalities from a short video data set to describe short videos, and then combines the three modalities in pairs to form three modal groups to obtain three enhanced features and a consistency loss function. Secondly, three specific representations are obtained respectively to obtain a disentanglement loss function, and then the modulation of each modality is realized through a graph convolutional network. Finally, the category embedding matrix is ​​obtained and the automatic retrieval loss function is obtained, and then the total loss function is obtained. In this way, the deviations of multiple modalities are adjusted in a progressive manner, and the representation ability of a single modality is enhanced. The representation ability of the modality is further improved through feature domain modulation and category domain modulation, which better reflects the potential relationship between multi-modality and multi-label, and effectively improves the accuracy of annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention is further described in detail below with reference to the accompanying drawings.

[0025] Figure 1 It is a flow chart of the present invention.

[0026] Figure 2 It is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0027] like Figure 1 and Figure 2 As shown in FIG. 1 , the automatic short video annotation method based on modality diversity includes the following steps:

[0028] Step S1, obtaining three modalities from the short video dataset to describe the overall representation of the short video, the three modalities being visual features, trajectory features, and audio features;

[0029] Specifically, we select the large-scale short video multi-label classification dataset MTSVRC as the short video dataset. For visual features, we obtain a sample set from the short video dataset at a set interval and select aL =15 key frame images, the selected images are input into the pre-trained ImageNet-VGG-2048 network to obtain the visual features of the sample set; for the trajectory features, the trajectory pooling deep convolutional descriptor (TDD) is used to extract the trajectory features of the sample set; the 2048D-Mel frequency cepstral coefficient (MFCC) is used to extract the audio features of the sample set. Each short video in the sample set is assigned 1-3 labels. In each category of labels, 1 indicates the presence of the label and 0 indicates the absence.

[0030] Step S2: Combine the three modalities into three modal groups, and input each modal combination into the multi-head attention mechanism to obtain the trajectory enhanced visual feature f v,t , audio enhanced visual featuresf v,a , visually enhanced trajectory features f t,v , audio enhancement trajectory features f t,a , visually enhanced audio features f a,v and the track-enhanced audio features f a,t , and then get the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a , and obtain the consistency loss function to realize the restriction of modal relationship;

[0031] Specifically, a single-modal aggregation strategy is designed to integrate the complementary advantages of each modality. Taking vision as an example, in order to effectively capture the dependency between trajectory and vision, the trajectory is input into the multi-head attention to obtain the enhanced feature trajectory enhanced visual feature f v,t ,Right now , MHA() represents the multi-head attention mechanism, in this way, the audio enhanced visual feature f can be obtained v,a , visually enhanced trajectory features f t,v , audio enhancement trajectory features f t,a , visually enhanced audio features f a,v and the track-enhanced audio features f a,t ;

[0032] According to the formula Get enhanced visual features f v , enhanced trajectory features f t and enhanced audio features f a .

[0033] In order to regulate the relationship between modalities, a single modality is mapped to the latent space and a consistent loss is added. Specifically, the modal relationship is constrained by minimizing the norm in each modality. Specifically, a fully connected layer is set to enhance the visual feature f v, enhanced trajectory features f t and enhanced audio features f a Input the fully connected layer to get the consistency loss function ,in, N represents the number of samples in the sample set, ( v, t )、( v, a ) ( t, a ) represents three modal groups, FC(.) represents the fully connected layer, represents the norm, Indicates the sample set i The enhanced visual features f of samples v , enhanced trajectory features f t and enhanced audio features f a .

[0034] Step S3: Based on the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a Get the consistent representation of the short video f o , and obtain visual specific representations based on the consistent representation of the video , trajectory specific representation and audio specific representation , and then get the disentanglement loss function;

[0035] Specifically, according to the formula Get the consistent representation of the video f o , respectively according to the formula Get visual specific representation , trajectory specific representation and audio specific representation , and then get the disentanglement loss function , where Avgpool represents average pooling and Ω represents the real domain.

[0036] The feature domain modulation is realized through step S2 and step S3.

[0037] Step S4: Based on the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a Get the mode-oriented representation F m , and according to the modal-oriented representation F m Construct a graph to get the graph convolutional network H m , using the formula Realize the modulation of each mode and obtain the modulation amount ,in, m ∈{v, t, a}, γ m and β m All are modulation parameters;

[0038] Specifically, the enhanced visual features f v , enhanced trajectory features f t and enhanced audio features f a Input the corresponding encoder to obtain the modality-oriented representation F m =E m (f m ), m ∈{ v, t, a}, where E m (.) denotes an encoder consisting of two layers of multilayer perceptrons.

[0039] The graph structure is introduced to model the modality-category relationship, and the modality-oriented representation F m Expand by row to get category association , Z k Each behavior category is represented by Build a graph structure for the nodes to obtain a graph convolutional network ,in, Relu function (activation function), d m represents the graph convolution dimension, Represents Z m The adjacency matrix is ​​used to represent the relationship between labels. is the weight matrix;

[0040] The instance-level modulation module can further modulate the category representation. First, the global representation is used to generate guidance features. The parameter generator and Useful representations can be extracted and projected into the modulation weights. Specifically, the fused features are passed through the parameter module to generate parameters γ k and parameters β k , to adjust the category distribution, that is , the generated parameters can explore the instance-level characteristics and assign modulation weights, use the fusion structure to guide the modulation of a single mode, and further reflect the transformation to achieve modulation: ,in, and is the parameter generator, Contains a linear layer, Contains a multi-layer perceptron, f fuse =Concat(f v ;f t;f a ) is the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a fusion result.

[0041] Class domain modulation is implemented by step S4.

[0042] Step S5: Fuse the modulation values ​​of the three modes and obtain the category embedding matrix H final , and obtain the automatic retrieval loss function, which is related to the predicted label representation, which is based on the category embedding matrix H final Obtain, based on the consistency loss function, the disentanglement loss function and the automatic retrieval loss function, the total loss function is obtained, and training is performed with the total loss function.

[0043] Specifically, the self-attention mechanism is used to obtain the embedding matrix ,in, It represents the fusion of the modulation quantities of the three modes, and SA(.) represents the self-attention function.

[0044] Embed the categories into the matrix H final As a representation of a short video, it is obtained C Category representation h j , then C Category representation h j enter C A binary classifier obtains the category score (i.e., prediction representation) s=[ s 1 ,s 2 ,…,s C ]. The automatic retrieval task can be transformed into a binary classification task and predict the score of the category. Taking the binary cross entropy (BCE) loss as the loss function, the automatic retrieval loss function is defined as: , then the total loss function is expressed as ,in, C is the number of categories, y j is the true label representation, s j is a prediction representation, which s j Youjiang C Category representation h j enter C A binary classifier is obtained, and the category representation h j Embedded matrix by categoryH final We get, where is the Sigmoid activation function, λ1 and λ2 are the weights of the consistency loss function and the disentanglement loss function, j =1,2,…, C .

[0045] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A short video automatic annotation method based on modality diversity, characterized by: The steps include: Step S1, obtaining three modalities from the short video dataset to describe the overall representation of the short video, the three modalities being visual features, trajectory features, and audio features; Step S2: Combine the three modalities into three modal groups, and input each modal combination into the multi-head attention mechanism to obtain the trajectory enhanced visual feature f v,t , audio enhanced visual featuresf v,a , visually enhanced trajectory features f t,v , audio enhancement trajectory features f t,a , visually enhanced audio features f a,v and the track-enhanced audio features f a,t , and then get the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a , and obtain the consistency loss function to realize the restriction of modal relationship; Step S3: Based on the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a Get the consistent representation of the short video f o , and obtain visual specific representations based on the consistent representation of the video , trajectory specific representation and audio specific representation , and then get the disentanglement loss function; Step S4: Based on the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a Get the mode-oriented representation F m , and according to the modal-oriented representation F m Construct a graph to get the graph convolutional network H m , using the formula Realize the modulation of each mode and obtain the modulation amount ,in, m ∈{ v,t,a }, γ m and β m All are modulation parameters; Step S5: Fuse the modulation values ​​of the three modes and obtain the category embedding matrix H final , and obtain the automatic retrieval loss function, which is related to the predicted label representation, which is based on the category embedding matrix H final Obtain, based on the consistency loss function, the disentanglement loss function and the automatic retrieval loss function, a total loss function is obtained, and training is performed using the total loss function; In step S2, a fully connected layer is set to enhance the visual feature f v , enhanced trajectory features f t and enhanced audio features f a Input the fully connected layer to get the consistency loss function ,in, N represents the number of samples in the sample set, ( v,t )、( v,a ) ( t,a ) represents three modal groups, FC(.) represents the fully connected layer, represents the norm, Indicates the sample set i The enhanced visual features f of samples v , enhanced trajectory features f t and enhanced audio features f a ; In step S3, according to the formula Get consistent representation of the video , respectively according to the formula Get visual specific representation , trajectory specific representation and audio specific representation , and then get the disentanglement loss function , where Avgpool represents average pooling and Ω represents the real domain.

2. The method for automatically labeling short videos based on modality diversity according to claim 1, characterized in that: In step S1, a sample set is obtained from the short video data set at a set interval, and a sample is selected from each sample in the sample set. L The selected images are input into the pre-trained ImageNet-VGG-2048 network to obtain the visual features of the sample set; the trajectory features of the sample set are extracted using the trajectory pooling deep convolutional descriptor; and the audio features of the sample set are extracted using the 2048D-Mel frequency cepstral coefficients.

3. The method for automatically labeling short videos based on modality diversity according to claim 2, characterized in that: In step S2, respectively, according to the formula , , Get enhanced visual features f v , enhanced trajectory features f t and enhanced audio features f a .

4. The method for automatically labeling short videos based on modality diversity according to claim 3, characterized in that: In step S4, the enhanced visual features f v , enhanced trajectory features f t and enhanced audio features f a Input the corresponding encoder to obtain the modality-oriented representation F m =E m (f m ), where E m (.) represents an encoder consisting of two layers of multilayer perceptrons.

5. The method for automatically labeling short videos based on modality diversity according to claim 4, characterized in that: The step S4 is to transform the modality-oriented representation F m Expand by row to get the category associated Z m , Z m Each behavior category is represented by Z m Build a graph structure for the nodes to obtain a graph convolutional network ,in, represents the activation function, A m Represents Z m The adjacency matrix, W m is the weight matrix.

6. The method for automatically labeling short videos based on modality diversity according to claim 5, characterized in that: In step S4, respectively, according to the formula and Get modulation parameters γ m and β m ,in, and is the parameter generator, Contains a linear layer, Contains a multi-layer perceptron, f fuse =Concat(f v ;f t ;f a ) is the enhanced visual feature f v , enhanced trajectory features f t and enhanced audio features f a fusion result.

7. The method for automatically labeling short videos based on modality diversity according to claim 6, characterized in that: In step S5, the self-attention mechanism is used to obtain the embedding matrix ,in, It represents the fusion of the modulation quantities of the three modes, and SA(.) represents the self-attention function.

8. The method for automatically labeling short videos based on modality diversity according to claim 7, characterized in that: In step S5, the total loss function is expressed as ,in, To automatically retrieve the loss function, C is the number of categories, y j is the true label representation, s j is a prediction representation, which s j Youjiang C Category representation h j enter C A binary classifier is obtained, and the category representation h j Embedded matrix by category H final It is obtained that is the Sigmoid activation function, λ1 and λ2 are the weights of the consistency loss function and the disentanglement loss function.

Images