A Short Video Classification Method Based on Multimodal Information Aggregation

Abstract

The present invention relates to a short video classification method based on multi-modal information aggregation, comprising the following steps: extracting visual, trajectory, and sound modal features of the short video; introducing a group of parallel graph convolutional networks, taking the category embedding matrices obtained after processing the three modal features as the initial node features input for graph convolution, where the correlation matrices of different graph convolutional networks respectively represent two label association modes of modal sharing and modality-specific among each node, using the correlation matrices of graph convolution to update the nodes and obtaining the updated node features; multi-modal information aggregation: introducing a sequential self-attention mechanism with an aggregation bottleneck term to aggregate and concentrate important information from each modality in a specific direction while discarding redundant information to obtain a fused category representation; inputting the obtained final category representation into a classifier to obtain the label prediction score of the short video, and using a multi-label classification loss function to guide the network to find the optimal solution of the model.

Description

Technical Field

[0001] The present invention relates to the field of short video classification, and particularly to a short video classification method based on multi-modal information aggregation. Background Art

[0002] In recent years, major short video platforms have developed vigorously, indicating that short videos, as a new way of user-generated content, are attracting a large number of users with their characteristics of easy operation, instant sharing, and content close to real life, and have become one of the mainstream trends in the social media era. At the same time, this also means that a large amount of multimedia data is generated every moment, and its generation speed is still growing at a high speed, making the information that users really need be covered by a huge amount of irrelevant information and difficult to obtain. Therefore, it has become an urgent need to classify and manage these explosively growing video data. Good video classification can help short video platforms and websites achieve more accurate content search, personalized recommendation, and advertising placement, improve the user experience, and also bring higher economic benefits to the platform, which has important application value and practical significance.

[0003] On the other hand, short videos naturally contain multiple modal information, such as vision, audio, trajectory, and text, etc. These information describe the content of short videos from different angles and play a complementary role. Therefore, it is reasonable to consider solving the short video classification task from a multi-modal perspective, which is also conducive to promoting the development of related fields and is a research topic with research value and application value.

[0004] After retrieval, some relevant patent documents found are as follows:

[0005] Patent 110334689 discloses a video classification method based on multi-modal fusion. First, parallel self-attention mechanisms are used to obtain the category information of each modality respectively, and then the category information of each modality is input into a gated recurrent unit in turn to implement a post-fusion strategy for multi-modal fusion. The attention module used is used to capture the feature relationships within each modality and does not involve the utilization of the correlation between labels.

[0006] Patent CN111246256 discloses a video recommendation method based on multi-modal video content and multi-task learning. According to the user's preference for each modality (i.e., user characteristics), an attention mechanism is used to learn the weights of the features of each modality of the video, and the features of each modality are weighted and summed to obtain a video feature representation, realizing the fusion of the multi-modal features of the video.

[0007] Patent CN115129934 discloses a multi-modal video understanding method, which adopts a late fusion strategy for multi-modal information, that is, the category scores obtained from the features of each modality are weighted and summed as the category score of the entire video.

[0008] Patent CN111382309 provides a short video recommendation method based on a graph model, which uses a user-short video graph structure. The nodes of the graph represent users and short videos, and the edges represent the interactions between users and short videos. It is proposed that instead of treating multi-modal information as a whole, each modality is modeled separately; the multi-modal fusion strategy used in this patent is to directly splice the feature vectors obtained from each modality as the output of the fusion layer.

[0009] Patent CN115329127 discloses a multi-modal short video tag recommendation method that fuses emotional information. First, the content features and emotional features of the video are respectively extracted from the multi-modal information, and then weighted and added as the final representation, where the weights are learned by an attention network. In the learning of content features and emotional features, different modality information interacts through a cross-attention mechanism or a multi-head co-attention mechanism, that is, different modality features directly interact with each other. Summary of the Invention

[0010] The present invention provides a short video classification method based on multi-modal information aggregation, which is beneficial to improving the accuracy of the multi-label short video classification task in a multi-modal scenario. The technical solution is as follows:

[0011] A short video classification method based on multi-modal information aggregation, the method comprising the following steps:

[0012] Step 1: Pre-extract the visual, trajectory, and sound modality features of the short video from the dataset used. Let the three modality features be visual feature x v , trajectory feature x t , and sound feature x a ;

[0013] Step 2: Introduce a group of parallel graph convolutional networks, and use the class embedding matrices obtained after processing the three modality features as the initial node features input for graph convolution. The correlation matrices of different graph convolutional networks respectively represent two label association modes of modality sharing and modality-specific among each node. Use the correlation matrix of graph convolution to update the nodes and obtain the updated node features. The method is as follows:

[0014] (1) Obtain the initial node features of the modality-specific graph convolutional networks: Input the three modality features x v , x t , x a into 3 encoders with the same structure respectively to obtain the encoded modality representations u v , u t , u a , and then expand u m , m ∈ {v, t, a} by rows to obtain the class embedding matrices Z v , Z t,Z a As the initial node features of the graph convolutional network specific to each modality;

[0015] (2) Obtain the initial node features of the modality-shared graph convolutional network: By means of a convolutional operation, fuse the class embedding matrices Z of the three modalities v ,Z t ,Z a to obtain, and the formula is:

[0016]

[0017] where δ() is the Sigmoid activation function, and are the weight matrices of the convolutional layer, C represents the number of graph nodes, which is equal to the number of short video tags here, and d m is the dimensionality of the class embedding;

[0018] (3) Take Z v ,Z t ,Z a and Z s respectively as the initial node features V of the corresponding graph convolutional networks, substitute them into the following formula to obtain the respective updated features of the corresponding nodes:

[0019]

[0020] where V u is the updated node feature, A ∈ R C×C is the correlation matrix, which describes the correlation between the tags, V is the initial node feature, is the state weight update matrix, D is the feature dimensionality of V, and D u is the feature dimensionality of V u , and LeakyReLU() is the non-linear activation function;

[0021] Denote the updated node features obtained from the graph convolutional networks specific to each modality as the enhanced class embeddings H m , m ∈ {v, t, a}, where v, t, and a represent the visual, trajectory, and sound modalities respectively; Denote the updated node features obtained from the modality-shared graph convolutional network as the modality-shared class embedding H s , that is, the updated node feature V u contains H m , m ∈ {v, t, a}, H s ;

[0022] Step 3: Multimodal information aggregation: Introduce a sequential self-attention mechanism with an aggregation bottleneck term to aggregate and concentrate the important information from each modality in a specific direction, while discarding the redundant information to obtain the fused class representation;

[0023] Step 4: Input the obtained final class representation into the classifier to obtain the label prediction score of the short video, and use the multi-label classification loss function to guide the network to find the optimal solution of the model.

[0024] Furthermore, the extraction methods of the three modal features of vision, trajectory, and sound described in Step 1 are as follows:

[0025] Visual feature x v is obtained by extracting multiple image frames of the short video at equal time intervals, using a residual network to extract the depth features of these image frames, and then performing average pooling operation; Trajectory feature x t is obtained by jointly using the trajectory pooling deep convolutional descriptor and Fisher vector for the short video; Sound feature x a is obtained by extracting the Mel cepstral coefficients of the short video audio.

[0026] Furthermore, the multi-modal information aggregation process in Step 3 is as follows:

[0027] (1) Use the enhanced visual class embedding H v to initialize and generate a low-dimensional aggregation bottleneck term H agg through convolution operation;

[0028] (2) The aggregation bottleneck term H agg interacts with each modal class embedding H v ,H t ,H a in a specific order through the sequential self-attention mechanism module, and at the same time introduce the modality-shared class embedding H s to guide the entire aggregation process, the method is as follows:

[0029] Concatenate H agg with H s and the modal class embedding H m ,m∈{v,t,a} respectively, and then input them into the self-attention mechanism. The obtained output result matrix is the updated H s by the modal class embedding and H agg . Split the output result matrix to obtain the updated H agg and H s . Repeat the above process with another modal class, and perform modal feature fusion through the self-attention mechanism until H agg completes the interaction with all modalities; In this process, the important information from each modal class embedding is collected and compressed into H agg . Use the aggregation bottleneck term H agg that has been interacted and updated with all modal class embeddings as the final class representation.

[0030] Further, the loss function is as follows:

[0031]

[0032] where log() is the logarithmic function, Sigmoid() is the non-linear activation function, C is the number of label categories, y i is the true label of the short video, and s i is the label prediction score.

[0033] The beneficial effects of the technical solution provided by the present invention are as follows:

[0034] (1) For the multi-label classification task in the multi-modal scenario, considering that different modalities have different contributions to the labels, the label correlation is decomposed into label association patterns unique to each modality and label association patterns shared by modalities, and a graph convolutional network is introduced to adaptively mine the label correlation relationship, effectively utilizing the label information;

[0035] (2) For the problem that the inherent correlation between modalities in short video multi-modal information is masked due to embedded noise information, a sequential self-attention mechanism is introduced to mine the inherent dependency relationship in the modality feature space, thereby generating a more complete class representation;

[0036] (3) The problem of multi-modal data fusion in short videos is explored, and an aggregation bottleneck term is introduced to prompt the model to better collect and concentrate information from each modality, realizing the aggregation of multi-modal information and providing a multi-modal data fusion solution;

[0037] The present invention makes full use of the natural multi-modal characteristics of short videos, and at the same time considers the dependencies between labels and between modalities and labels, which is beneficial to improving the accuracy of the multi-label classification task in the multi-modal scenario. Description of the Drawings

[0038] Figure 1 is the short video classification method network based on multi-modal information aggregation;

[0039] Figure 2 is the feature fusion process based on the self-attention mechanism;

[0040] Figure 3 is the performance comparison using different modality combinations and the performance comparison with advanced methods. Detailed Embodiments

[0041] The present invention provides a short video classification method based on multi-modal information aggregation. For the specific method framework, please refer to Figure 1 , and the following is a detailed description of the implementation manner:

[0042] The first step is to construct a data set and perform modal feature extraction:

[0043] (1) The present invention uses the large-scale multi-label short video classification dataset released in the 2018 AI Challenge. This dataset contains a total of 200,000 short videos, and 63 types of labels describe the short videos from multiple dimensions such as the video subject, scene, and action. The duration of most videos is 5 to 15 seconds. Randomly select 80% of the short videos in the dataset as the training set, and the remaining 20% as the test set.

[0044] (2) Extract the visual, trajectory, and sound modality features of the short videos respectively:

[0045] For the visual modality, extract 15 image frames of the short video at equal time intervals, use the pre-trained Residual Network (ResNet) model to extract the deep feature representation of the image frames, and then perform average pooling (AveragePooling) operation on them to obtain the overall visual feature x of the short video v :

[0046]

[0047] Among them, ResNet() is the residual network, AvePooling() is the average pooling operation, f v represents the deep visual representation of the image frame, β v is the network parameter to be learned, D v is the dimension of the visual feature.

[0048] For the trajectory modality, jointly use the Trajectory-Pooled Deep-Convolutional Descriptors (TDD) and Fisher vectors to extract the trajectory feature x from the short video t :

[0049]

[0050] Among them, TDD() represents the trajectory-pooled deep convolutional descriptor network, f t is the original trajectory information of the short video, β t is the network parameter to be learned, D t is the dimension of the trajectory feature.

[0051] For the sound modality, extract the Mel-scale Frequency Cepstral Coefficients (MFCC) of the short video audio to obtain the sound feature x a :

[0052]

[0053] Among them, MFCC() represents Mel-frequency cepstral coefficients, and f a is the original audio feature of the short video, and β a is the network parameter to be learned, and D a is the dimension of the sound feature.

[0054] Step 2: Build a short video classification network based on multi-modal information aggregation through the deep learning framework Pytorch. The short video classification network based on multi-modal information aggregation includes three parts: a label dependence information embedding module, a multi-modal information aggregation module, and a multi-label classification module. The following is a detailed introduction to these three parts:

[0055] (1) Label dependence is one of the important influencing factors in multi-label learning. In addition, in a multi-modal scenario, due to the different contributions of different modalities to the label, the correlation between the modality and the label also needs to be considered. Therefore, a set of parallel graph convolutional networks are constructed to separately mine the label correlation patterns shared by modalities and specific to each modality.

[0056] First, introduce the concepts of graph structure and graph convolution. Given a graph G with C label nodes, a matrix V = [v1, v2,..., v C ∈ R C×D can be used to describe the node features, where D is the dimension of the node features. The edges of the graph G, that is, the correlation matrix A ∈ R C×C then reflect the relationship between the labels. Graph convolution can use this correlation to update the value of V, and the formula is defined as follows:

[0057]

[0058] Among them, V u represents the updated node features, is the state weight update matrix, and LeakyReLU() is the non-linear activation function.

[0059] ① Establish the initial graph structure

[0060] Input the modality features x v , x t , x a obtained in the first step into 3 encoders with the same structure to get the encoded modality representations u v , u t , u a :

[0061]

[0062] Among them, m ∈ {v, t, a} is the modality indicator, v, t, and a respectively represent the visual, trajectory, and sound modalities, σ() represents the non-linear activation function, FC() is the fully connected layer operation, and dm is the number of dimensions of the encoded representation. Then, expand u m by rows to obtain the category embedding matrix As the initial node features of the graph, each row of this matrix represents the representation of the corresponding category.

[0063] ② Mine the modality-shared and modality-specific label correlation patterns

[0064] Introduce a set of parallel graph convolutional networks to mine the above two correlation patterns respectively. Let Z v , Z t , Z a be input into the modality-specific graph convolutional networks respectively to obtain the updated node H m :

[0065]

[0066] where A m ∈R C×C , represent the correlation matrix and the state weight update matrix of the modality-specific graph convolutional network respectively, and H m , m = {v, t, a} represents the category embedding enhanced by the modality-specific correlation pattern.

[0067] Meanwhile, fuse the embedding matrices Z v , Z t , Z a as the input of the modality-shared graph convolutional network to obtain the modality-shared category embedding H s :

[0068]

[0069] where δ() represents the Sigmoid activation function, and A s ∈R C×C , represent the correlation matrix and the state weight update matrix of the modality-shared graph convolutional network respectively, and represent the weight matrix of the convolutional layer, and the fusion of Z v , Z t , Z a is achieved through multiple convolutional operations.

[0070] (2) Since the self-attention mechanism can mine the internal connections between features and strengthen the important features among them, the present invention introduces a low-dimensional multi-modal information aggregation bottleneck term to make H agg and H m, m = {v, t, a} interacts through the sequential self-attention mechanism to aggregate the important information of each modality into H agg above.

[0071] ① Initialize the multi-modal information aggregation bottleneck term H agg

[0072] Since the visual modality usually contains richer information than other modalities, it is considered to be mainly based on the visual modality, and other modality information is used as a supplement, so that The initialization is as follows:

[0073]

[0074] where H v is the enhanced visual class embedding, and Conv_1d() represents a one-dimensional convolution operation.

[0075] ② Multi-modal information aggregation

[0076] Introduce the sequential self-attention mechanism module, so that the initialized interacts with each modality H m in a serial order to achieve multi-modal information aggregation. At the same time, H s is introduced to guide the entire aggregation process.

[0077] To simplify the formula expression, define the self-attention operation based on scaled dot product as follows:

[0078]

[0079] where θ is a learnable parameter, Softmax() is a non-linear activation function, W Q , W K , W V are the mapping matrices that transform the input X into the query matrix, key matrix, and value matrix respectively, and d k is the scaling factor used to offset the vanishing gradient effect caused by using the Softmax function.

[0080] During the modality information aggregation process, H agg is concatenated with H s and H m respectively, and then they are input into the self-attention mechanism. The sequential self-attention operation of the l-th layer is defined as follows:

[0081]

[0082] where l = 1, 2, 3 represents the current l-th layer modality feature fusion process, H m is the modality class embedding selected by the current feature fusion layer, represents the matrix concatenation operation, and θ s, θ t are the corresponding learnable parameters. The single-layer modal feature fusion process is as follows Figure 2 shown. Concatenate H agg with H s and the modal category embedding H m respectively, then input them into the self-attention mechanism respectively. Split the obtained output result matrix to obtain the updated H agg and H s .

[0083] Taking the information aggregation order of t→a→v as an example, that is, when l = 1, 2, 3, H m takes H t , H a , H v respectively. Then the whole information aggregation process can be expressed as follows:

[0084]

[0085] In this process, the information from each modality is forced to be compressed and aggregated into the final .

[0086] (3) Use the obtained in (2) as the final class representation matrix to complete the multi-label classification task. Among them, h i (i = 1, 2,..., C) is the representation vector specific to the i-th class and contains the content information from the short video. Input h i into the corresponding classifier to obtain the predicted class scores s = [s1, s2,..., s C . Adopt the traditional multi-label classification loss as the loss function of the model:

[0087]

[0088] where log(·) is the logarithmic function, Sigmoid(·) is the non-linear activation function, C is the number of label categories, y i is the true label of the short video, and s i is the label prediction result.

[0089] The third step, model training and experiment:

[0090] During the training process, the SGD (Stochastic Gradient Descent) optimizer was used, with the momentum set to 0.9 and the weight decay set to 0.001. The initial learning rate was set to 0.01, and the batch_size was set to 64. The epoch was set to 50, meaning the entire network needed to be trained for 50 rounds in total, and the learning rate was decayed to 0.1 times the original value after 45 rounds of training. After the training was completed, the model and parameters of the network were saved. The present invention uses five evaluation metrics, namely mean average precision (mAP), one-error, coverage, ranking loss, and Hamming loss, to evaluate the multi-label classification performance of the model, where:

[0091] (1) Mean average precision represents the average of the accuracies of multiple classes (63 classes in this experiment). The larger its value, the better the classification performance of the model;

[0092] (2) One-error calculates the frequency that the label with the largest predicted probability value does not belong to the set of correct labels. The smaller its value, the better the classification performance of the model;

[0093] (3) Coverage sorts the predicted probability values of the labels from largest to smallest and calculates the average number of values that need to be taken from the front to cover all the correct labels. The smaller its value, the better the classification performance of the model;

[0094] (4) Ranking loss calculates the average score of the reversed label pairs of the instances. The smaller its value, the better the classification performance of the model;

[0095] (5) Hamming loss calculates the difference between the predicted labels and the correct labels. The smaller its value, the better the classification performance of the model.

[0096] The experimental results are as Figure 3 shown, where v, t, and a represent the visual, trajectory, and sound modalities respectively. Figure 3 The classification performance comparisons of different modality combinations and the classification performance comparisons with classical methods are respectively shown, demonstrating that the method proposed in the present invention can effectively aggregate information from multiple modalities and achieve competitive results in comparison with advanced methods.

[0097] In summary, the present invention provides a short video classification method based on multi-modal information aggregation. For the content information of short videos containing multiple modalities, the proposed method aims to aggregate multi-modal information to generate a complete joint representation for multi-label classification tasks. Considering the correlation between labels and the correlation between modalities and labels, the modality dependence relationship is decomposed into modality sharing and label association patterns specific to each modality, and a group of parallel graph convolutional networks are introduced to mine and utilize them. Considering that the correlation between modalities may be masked due to the noise information contained in each modality data, a sequential self-attention mechanism module is introduced to collect and concentrate the important information from each module into the aggregation bottleneck term, while discarding redundant information. The obtained final class representation is used for multi-label classification.

[0098] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0099] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A short video classification method based on multi-modal information aggregation, the method comprising the following steps: Step 1: Pre-extract the visual, trajectory, and audio modality features of the short videos from the dataset used. Let the three modality features be the visual feature x v , the trajectory feature x t , and the audio feature x a ; Step 2: Introduce a set of parallel graph convolutional networks, and use the class embedding matrices obtained after processing the three modal features as the initial node features input for graph convolution. The correlation matrices of different graph convolutional networks respectively represent two label association modes of modal sharing and modality-specific between each node. Use the correlation matrix of graph convolution to update the nodes and obtain the updated node features. The method is as follows: (1) Obtain the initial node features of the graph convolutional networks specific to each modality: Input the three modality features x v , x t , x a into 3 encoders with the same structure respectively to obtain the encoded modality representations u v , u t , u a . Then, expand u m , m ∈ {v, t, a}, by rows to obtain the class embedding matrices Z v , Z t , Z a as the initial node features of the graph convolutional networks specific to each modality; (2) Obtain the initial node features of the modal-shared graph convolutional network: fuse the category embedding matrices \(Z\) of the three modalities through convolutional operations v , \(Z\) t , \(Z\) a to obtain, the formula is: where, δ(·) is the Sigmoid activation function, and is the weight matrix of the convolutional layer, C represents the number of graph nodes, which is equal to the number of short video tags here, and d m is the dimensionality of the class embedding; (3) Take Z v , Z t , Z a and Z s as the initial node features V of the corresponding graph convolutional networks respectively, and substitute them into the following formula to obtain the updated respective features of the nodes: Among them, V u contains each feature corresponding to the updated node, A ∈ R C×C is the correlation matrix, which describes the correlation between labels, is the state weight update matrix, D is the number of feature dimensions of V, D u is for V u is the number of feature dimensions of, LeakyReLU(·) is the non-linear activation function; Denote the updated node features obtained from the graph convolutional network specific to each modality as the enhanced category embeddings \(H_m\) of each modality, where \(m\in\{v, t, a\}\), and \(v\), \(t\), and \(a\) represent the visual, trajectory, and sound modalities respectively; Denote the updated node features obtained from the modality - shared graph convolutional network as the modality - shared category embedding \(H\). m , \(m\in\{v, t, a\}\), where \(v\), \(t\), and \(a\) respectively represent the visual, trajectory, and sound modalities; Denote the updated node features obtained from the modality - shared graph convolutional network as the modality - shared category embedding \(H\). s , that is, the updated node features \(V\). u contains \(H_m\). m , \(m\in\{v, t, a\}\), \(H\). s ; Step 3: Multi-modal information aggregation: Introduce a sequential self-attention mechanism with an aggregation bottleneck term to aggregate and condense important information from each modality in a specific direction, while discarding redundant information, to obtain a fused class representation; Step 4: Input the obtained final class representation into a classifier to obtain the label prediction score of the short video, and use a multi-label classification loss function to guide the network to find the optimal solution of the model.

2. The short video classification method based on multi-modal information aggregation according to claim 1, wherein The extraction methods of the three modal features of vision, trajectory, and sound in Step 1 are respectively: Visual feature x v It is obtained by extracting multiple image frames of a short video at equal time intervals, and performing average pooling operation after using a residual network to extract the depth features of these image frames; Trajectory feature x t It is obtained by jointly using a trajectory pooling deep convolutional descriptor and a Fisher vector for the short video; Voice feature x a Obtained for extracting the Mel cepstral coefficients of the short video audio.

3. The short video classification method based on multi-modal information aggregation according to claim 1, wherein, The multi-modal information aggregation process in Step 3 is: (1) Use the enhanced visual category embedding H v Initialize and generate a low-dimensional aggregated bottleneck term H through a convolution operation agg ; (2) Aggregation bottleneck term H agg Through the sequence self-attention mechanism module, it interacts with each modality category embedding H in a specific order respectively v ,H t ,H a and introduces the modality-shared category embedding H s to guide the entire aggregation process as follows: Combine H agg separately with H s and the modality category embedding H m , m ∈ {v, t, a} in series, and then input them into the self-attention mechanism respectively. The resulting output matrix is the updated H s by the modality category embedding and H agg . Split the output matrix to obtain the updated H agg and H s . Repeat the above process with another modality category again, and perform modality feature fusion through the self-attention mechanism until H agg completes the interaction with all modalities; in this process, the important information from each modality category embedding is collected and compressed into H agg . The aggregated bottleneck term H agg updated by interacting with all modality category embeddings is used as the final class representation.

4. A short video classification method based on multi-modal information aggregation according to claim 1, characterized in that The loss function is: Among them, log() is the logarithmic function, Sigmoid() is the non-linear activation function, C is the number of label categories, and y i is the true label of the short video, and s i is the label prediction score.

Images