A multi-modal micro-video classification method based on time domain information enhancement

Abstract

The application provides a kind of multi-modal micro-video classification method based on time domain information enhancement, it is related to video classification technical field.The method first cuts the micro-video to be classified into multiple video segments;Then separate the visual modal, audio modal and text modal information in the video segment, and extract the visual modal feature, audio modal feature and text modal feature;Then, based on the visual modal feature and audio modal feature of each video segment, select the most expressive video segment;Capture the time domain related information of the visual modal and audio modal of the most expressive video segment, obtain the feature matrix of time domain information enhancement;Finally, the visual modal feature, audio modal feature and original text modal feature of time domain information enhancement are fused to obtain the final classification result.The method fully utilizes the audio and text modal of video, supplements the image information, and further improves the classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of video classification technology, and in particular to a multimodal micro-video classification method based on temporal information enhancement. Background Technology

[0002] With the popularization of the internet and the development of social media, short videos have gradually become a major carrier of information and have been integrated into people's daily lives. In recent years, the number of short video users worldwide has increased dramatically, and the economic growth brought about by the short video industry accounts for a significant proportion of the total economic growth. A challenge that follows is the arduous task of classifying tens of millions of short videos daily on short video platforms. Therefore, using an efficient, accurate, automated, and intelligent micro-video classification model to complete the video classification task is particularly important.

[0003] The paper "Neural Multimodal Cooperative Learning Toward Micro-VideoUnderstanding" proposes a multimodal cooperative learning method for video classification. This method alternately treats three modalities as the primary modality and concatenates the other two guest modalities. Two learnable weight matrices are used to represent the similarity scores between the primary and guest modalities. A threshold is then dynamically learned to distinguish between the consistent and complementary components between modalities. Finally, the consistency and complementarity vectors are concatenated to obtain the final enhanced feature vector.

[0004] The paper "ViViT:AVideoVisionTransformer" proposes a Transformer-based video classification method, and is the first paper to perform video classification solely using Transformer. In this paper, the authors modify the encoder part of the traditional Transformer by adding a model structure that acquires temporal information, thereby learning the similarity relationships between different frames of the video to achieve video classification.

[0005] The paper "Dynamic interaction networks for image-text multimodal learning" designs a meta-network to generate a dynamic parameter W for the main network. This method consists of two parts: a meta-network and a main network. Temporal encoding and self-attention mechanisms are used to generate a dynamic parameter containing cross-modal information. A unified modeling space is constructed for visual and textual modalities. Finally, the generated dynamic parameter is used in the main network to fuse features from both modalities and perform a final video classification.

[0006] The paper "Neural Multimodal Cooperative Learning Toward Micro-Video Understanding" uses learnable similarity scores to distinguish between modal consistency and complementarity. This only helps the model learn the relationships between modalities, ignoring the objective fact that there are also connections between parts of the video within each modality. It lacks modeling of the video's temporal dimension. Furthermore, the model's reliance solely on similarity matrices and learnable thresholds to distinguish between modal consistency and complementarity results in low weighting for intermediate boundary information. This could lead to incorrect penalties for information that is part of the main video content.

[0007] In "ViViT: AVision Transformer," the authors cleverly modeled the temporal dimension of video using a variant of the Transformer. However, this method still suffers from the common problems of Transformers: excessive model weight leading to too many training parameters and excessively long training times. While acceptable for video classification tasks on internet platforms, it's difficult to apply to embedded systems. Furthermore, training Transformer-like models heavily relies on large-scale datasets, making it difficult to train them in situations lacking sufficient data. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a multimodal micro-video classification method based on temporal information enhancement to address the shortcomings of the prior art and realize the classification of multimodal micro-videos.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multimodal micro-video classification method based on temporal information enhancement, comprising the following steps:

[0010] Step 1: Segment the micro-videos to be classified according to scene changes to obtain multiple video segments;

[0011] Step 2: Separate the visual modality, audio modality, and text modality information from the video clip, and extract the visual modality features, audio modality features, and text modality features;

[0012] Step 2.1: Extract 5 RGB frame images at equal intervals from each of the several video segments that have been divided into segments, and use them as the visual modal information of that video segment;

[0013] Step 2.2: Extract the audio information from each video segment separately as the audio modal information of that video segment;

[0014] Step 2.3: Use the pre-trained Vision Transformer model to extract features from the RGB frame images obtained in Step 2.1; for each image, a 1*768 dimensional feature vector will be obtained and used as the visual modality feature;

[0015] Step 2.4: Use the pre-trained VGGish model to extract features from the audio segments obtained in Step 2.2; for each audio segment, a 1*128 dimensional feature vector will be obtained and used as the audio modality feature;

[0016] Step 2.5: Use the pre-trained ALBert model to extract features from the video title text; for each video title, a 1*312 dimensional feature vector is obtained and used as the text modality feature;

[0017] Step 3: Construct the most expressive segment extraction module, which selects the most expressive video segments based on the visual and audio modal features of each video segment;

[0018] Step 3.1: Connect the visual modal features and audio modal features of each video segment obtained in Step 2; feed them into a fully connected layer of depth 2, and optimize the parameters of the fully connected layer based on backpropagation, and obtain the classification probability distribution of the segment for all video categories;

[0019] Step 3.2: For the probability distribution of the classification results of each video segment, take the maximum value of the probability distribution corresponding to each category as the classification confidence of that category; at the same time, for all video segments, compare the classification confidence of each video segment, and take the two video segments with the highest classification confidence as the two video segments with the highest contribution to the classification.

[0020] Step 3.3: For the visual modal features and audio modal features of the two video segments with the highest contribution to classification obtained in Step 3.2, set two learnable weight matrices respectively, perform matrix multiplication with their corresponding feature matrices, and perform matrix addition on the feature matrices of the two modalities corresponding to the two video segments; so that the feature matrices of the two video segments are merged into the feature matrix of one video segment, and the merged video segment is the most expressive video segment.

[0021] Step 4: Construct a temporal information enhancement module to capture the temporal information related to the visual and audio modalities of the most expressive video clips, and obtain the feature matrix for temporal information enhancement;

[0022] Step 4.1: For the most expressive video clip obtained in Step 3, first design three pooling modules: Max pooling module: captures the feature that appears least once in the video clip; Standard deviation pooling module: captures the feature that changes most significantly over time in the video clip; Average pooling module: captures the most frequently occurring feature in the video; The feature vectors of the visual modality and audio modality are respectively fed into the three pooling modules to fix the corresponding temporal information of the video clip;

[0023] Step 4.2: Design three 1*1 dimensional convolutional blocks k, u, and v. The convolution operation learns the parameters on the convolutional kernel while reducing the dimensionality of the input feature vector. The features of the two most expressive video segments are fed into convolutional blocks k and u as input matrices, and the average pooling result from the average pooling module in step 4.1 is also fed into convolutional block v for convolution operation.

[0024] Step 4.3: Perform matrix multiplication on the convolution results of convolutional blocks k and u to obtain the similarity matrix between each frame in the video segment; perform matrix multiplication on the results of convolutional blocks u and v to obtain the similarity matrix between each frame in the video segment and the most frequently occurring feature in the video; perform matrix addition on the two similarity matrices and apply a softmax operation to them; perform matrix multiplication on the result of the softmax operation with the feature matrix of the most expressive video segment, and activate it with ReLU to obtain the feature matrix with enhanced similarity; the above operations are shown in the following formula:

[0025] S ff =K·U (1)

[0026] S fa =U·V (2)

[0027] S = softmax(S ff +S fa (3)

[0028] F a =relu(S·F r (4)

[0029] Among them, S ff S represents the inter-frame similarity, where K, U, and V represent the feature matrices after convolutional blocks k, u, and v, respectively. fa The similarity between a frame and average pooling is represented by S, where S represents the sum of the two similarity matrices, and F represents the similarity between the two frames. a Represents the feature matrix enhanced by similarity S, F r Feature matrix representing the most expressive video clips;

[0030] Step 4.4: Concatenate the similarity-enhanced feature matrix obtained in Step 4.3 with the results of standard deviation pooling and max pooling in Step 4.1, and use this as the overall output of the temporal information enhancement module in Step 4, as shown in the following formula:

[0031] F o =concat(F a ,F m ,F s (5)

[0032] Among them, F o The feature matrix after time-domain information enhancement, F m Represents the feature matrix after max pooling, F s This represents the feature matrix after standard deviation pooling;

[0033] Step 5: Fuse the visual modality features, audio modality features, and original text modality features obtained in Step 4, for a total of three modalities, to obtain the final classification result;

[0034] Step 5.1: Input the visual modality feature matrix, audio modality feature matrix and original text modality feature matrix after temporal information enhancement into two fully connected layers to obtain the probability distribution of all categories for each modality;

[0035] Step 5.2: Set three learnable weight matrices and perform matrix multiplication with the probability distributions of all categories corresponding to each modality in Step 5.1 for late fusion; finally, perform matrix addition on the three probability distributions of the same dimension, and take the probability distribution with the largest confidence value for each category as the final classification category.

[0036] The beneficial effects of adopting the above technical solution are as follows: The multimodal micro-video classification method based on temporal information enhancement provided by the present invention (1) uses video segments based on scene transformation as the basic unit of video classification; in the process of video segmentation, the key information of the video is extracted and the redundant information of the video is discarded. Compared with the traditional video classification method which uses the whole video as the basic processing unit, the method of the present invention has the advantages of being faster and lighter; (2) compared with the traditional video classification model, the method of the present invention makes full use of the inherent characteristic of video temporality, and captures the features that appear least once, change significantly over time, and appear most frequently in the video; and calculates the similarity between frames and the similarity between each frame and the main content, which better highlights the main body of the video while weakening the secondary information and improving the accuracy of model classification. (3) compared with the method of classifying video based solely on images, the method of the present invention makes full use of the audio and text modalities of the video, supplements the image information, and further improves the classification accuracy. Attached Figure Description

[0037] Figure 1 A flowchart illustrating a multimodal micro-video classification method based on temporal information enhancement, provided in an embodiment of the present invention;

[0038] Figure 2 The main structure diagram of a multimodal micro-video classification method based on temporal information enhancement provided in an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the most expressive fragment extraction module provided in an embodiment of the present invention:

[0040] Figure 4 This is a network architecture diagram of the time-domain information enhancement module provided in an embodiment of the present invention.

[0041] Figure 5 The convergence graph of the loss function is provided to demonstrate the convergence of the model results in the embodiments of the present invention. Detailed Implementation

[0042] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0043] In this embodiment, a multimodal micro-video classification method based on temporal information enhancement is described, such as... Figure 1 , 2 As shown, it includes the following steps:

[0044] Step 1: Segment the micro-videos to be classified according to scene changes to obtain multiple video segments;

[0045] For a given micro-video to be classified, the first step is to perform scene segmentation preprocessing. The specific segmentation is based on the inter-frame change rate between every two adjacent frames of the video. This ensures that the segmented video segments have different scene descriptions, and each scene segment itself has the same scene information. In this embodiment, the PySceneDetect tool is used in Python to complete the segmentation of the micro-video to be classified.

[0046] Step 2: Separate the visual modality, audio modality, and text modality information from the video clip, and extract the visual modality features, audio modality features, and text modality features;

[0047] Step 2.1: Use the cv2 library of the OpenCV module in Python to extract 5 RGB frame images at equal intervals from each of the several video segments that have been segmented, as the visual modal information of that video segment;

[0048] Step 2.2: Use the moviepy library in Python to extract the audio information from each video segment individually, as the audio modality information of that video segment;

[0049] Step 2.3: Use the Vision Transformer model pre-trained on the ImageNet image dataset to extract features from the RGB frame images obtained in Step 2.1; for each image, a 1*768 dimensional feature vector will be obtained and used as the visual modality feature;

[0050] Step 2.4: Use the VGGish model pre-trained on the AudioSet audio dataset to extract features from the audio segments obtained in Step 2.2; for each audio segment, a 1*128 dimensional feature vector will be obtained and used as the audio modality feature;

[0051] Step 2.5: Use the ALBert model pre-trained on the text dataset to extract features from the video title text; for each video title, a 1*312 dimension feature vector is obtained and used as the text modality feature;

[0052] Step 3: Construct the module for extracting the most expressive fragments, such as... Figure 3 As shown, based on the visual modal features and audio modal features of each video segment, the most expressive video segment is selected;

[0053] Step 3.1: Connect the visual modal features and audio modal features of each video segment obtained in Step 2; feed them into a fully connected layer of depth 2, and optimize the parameters of the fully connected layer based on backpropagation, and obtain the classification probability distribution of the segment for all video categories;

[0054] Step 3.2: For the probability distribution of the classification results of each video segment, take the maximum value of the probability distribution corresponding to each category as the classification confidence of that category; at the same time, for all video segments, compare the classification confidence of each video segment, and take the two video segments with the highest classification confidence as the two video segments with the highest contribution to the classification.

[0055] Step 3.3: For the visual modal features and audio modal features of the two video segments with the highest contribution to classification obtained in Step 3.2, set two learnable weight matrices respectively, perform matrix multiplication with their corresponding feature matrices, and perform matrix addition on the feature matrices of the two modalities corresponding to the two video segments; merge the feature matrices of the two video segments into the feature matrix of one video segment. The merged video segment is the most expressive video segment, and this is used as the input for Step 4, i.e., input:[visual feature,audio feature].

[0056] Step 4: Construct a time-domain information enhancement module, such as... Figure 4 As shown, temporal domain information related to the visual and audio modalities of the most expressive video clips is captured to obtain the temporal information-enhanced feature matrix F. o ;

[0057] Step 4.1: For the most expressive video clip obtained in Step 3, first design three pooling modules: Max pooling module: captures the feature that appears least once in the video clip; Standard deviation pooling module: captures the feature that changes most significantly over time in the video clip; Average pooling module: captures the most frequently occurring feature in the video; The feature vectors of the visual modality and audio modality are respectively fed into the three pooling modules to fix the corresponding temporal information of the video clip;

[0058] Step 4.2: Design three 1*1 dimensional convolutional blocks k, u, and v. The convolution operation learns the parameters on the convolutional kernel while reducing the dimensionality of the input feature vector. The features of the two most expressive video segments are fed into convolutional blocks k and u as input matrices, and the average pooling result from the average pooling module in step 4.1 is also fed into convolutional block v for convolution operation.

[0059] Step 4.3: Perform matrix multiplication on the convolution results of convolutional blocks k and u to obtain the similarity matrix between each frame in the video segment; perform matrix multiplication on the results of convolutional blocks u and v to obtain the similarity matrix between each frame in the video segment and the most frequently occurring feature in the video; perform matrix addition on the two similarity matrices and apply a softmax operation to them; perform matrix multiplication on the result of the softmax operation with the feature matrix of the most expressive video segment, and activate it with ReLU to obtain the feature matrix with enhanced similarity; the above operations are shown in the following formula:

[0060] S ff =K·U (1)

[0061] S fa =U·V (2)

[0062] S = softmax(S ff +S fa (3)

[0063] F a =relu(S·F r (4)

[0064] Among them, S ff S represents the inter-frame similarity, where K, U, and V represent the feature matrices after convolutional blocks k, u, and v, respectively. fa The similarity between a frame and average pooling is represented by S, where S represents the sum of the two similarity matrices, and F represents the similarity between the two frames. a Represents the feature matrix enhanced by similarity S, F r Feature matrix representing the most expressive video clips;

[0065] Step 4.4: Concatenate the similarity-enhanced feature matrix obtained in Step 4.3 with the results of standard deviation pooling and max pooling in Step 4.1, and use this as the overall output of the temporal information enhancement module in Step 4, as shown in the following formula:

[0066] F o =concat(F a ,F m ,F s (5)

[0067] Among them, F o The feature matrix after time-domain information enhancement, F m Represents the feature matrix after max pooling, F s This represents the feature matrix after standard deviation pooling;

[0068] Step 5: Fuse the visual modality features, audio modality features, and original text modality features obtained in Step 4, for a total of three modalities, to obtain the final classification result;

[0069] Step 5.1: Enhance the visual modality feature matrix and audio modality feature matrix (i.e., the feature matrix F enhanced with temporal information) using temporal information. o The original text modality feature matrix and the original text modality feature matrix are respectively input into two fully connected layers to obtain the probability distribution of all categories for each modality;

[0070] Step 5.2: Set three learnable weight matrices and perform matrix multiplication with the probability distributions of all categories corresponding to each modality in Step 5.1 for late fusion; finally, perform matrix addition on the three probability distributions of the same dimension, and take the probability distribution with the largest confidence value for each category as the final classification category.

[0071] The classification method of this invention, due to its thorough capture of temporal information, achieves classification accuracy exceeding that of many existing multimodal video understanding models. It brings new ideas to micro-video understanding tasks and provides the industry with another effective video classification method, possessing high academic and industrial value. In this embodiment, the model's convergence is as follows... Figure 5 As shown in Table 1, the results were evaluated using the F1-Score as the evaluation metric.

[0072] Table 1 Evaluation Index Values

[0073] Micro-F1 Macro-F1 0.99370 0.99406

[0074] The model structure and basic deep learning operators used in this invention are relatively simple. Compared with Transformer-based classification methods, it is lighter, consumes less GPU computing resources, and further reduces training time, thereby improving the efficiency of model usage.

[0075] Furthermore, this invention also exhibits good scalability; the number of target classification categories can be increased by simply modifying the parameters of the last layer of the fully connected layer. This method integrates preprocessing operations such as video segmentation and feature extraction, automating the model and enabling a more comprehensive completion of video understanding tasks.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multimodal micro-video classification method based on temporal information enhancement, characterized in that: Includes the following steps: Step 1: Segment the micro-videos to be classified according to scene changes to obtain multiple video segments; Step 2: Separate the visual modality, audio modality, and text modality information from the video clip, and extract the visual modality features, audio modality features, and text modality features; Step 3: Construct the most expressive segment extraction module, which selects the most expressive video segments based on the visual and audio modal features of each video segment; Step 4: Construct a temporal information enhancement module to capture the temporal information related to the visual and audio modalities of the most expressive video clips, and obtain the feature matrix for temporal information enhancement; Step 4.1: For the most expressive video clip obtained in Step 3, first design three pooling modules, and send the feature vectors of the visual modality and audio modality into the three pooling modules respectively to fix the corresponding temporal domain information of the video clip; The three pooling modules are max pooling, standard deviation pooling, and average pooling. Max pooling captures the feature that appears least once in the video clip; standard deviation pooling captures the feature that changes most significantly over time; and average pooling captures the most frequently occurring feature in the video. Step 4.2: Design three 1*1 dimensional convolutional blocks k, u, and v. The convolution operation learns the parameters on the convolutional kernel while reducing the dimensionality of the input feature vector. The features of the two most expressive video segments are fed into convolutional blocks k and u as input matrices, and the average pooling result from the average pooling module in step 4.1 is also fed into convolutional block v for convolution operation. Step 4.3: Perform matrix multiplication on the convolution results of convolutional blocks k and u to obtain the similarity matrix between each frame in the video segment; perform matrix multiplication on the results of convolutional blocks u and v to obtain the similarity matrix between each frame in the video segment and the most frequently occurring feature in the video; perform matrix addition on the two similarity matrices and perform a softmax operation on them; perform matrix multiplication on the result of the softmax operation with the feature matrix of the most expressive video segment, and activate it with ReLU to obtain the feature matrix with enhanced similarity. Step 4.4: Connect the similarity-enhanced feature matrix obtained in Step 4.3 with the results of standard deviation pooling and max pooling in Step 4.1 to obtain the temporal information-enhanced feature matrix as the overall output result of the temporal information enhancement module in Step 4; Step 5: Fuse the visual modality features, audio modality features, and original text modality features obtained in Step 4, for a total of three modalities, to obtain the final classification result.

2. The multimodal micro-video classification method based on temporal information enhancement according to claim 1, characterized in that: The specific method for step 2 is as follows: Step 2.1: Extract 5 RGB frame images at equal intervals from each of the several video segments that have been divided into segments, and use them as the visual modal information of that video segment; Step 2.2: Extract the audio information from each video segment separately as the audio modal information of that video segment; Step 2.3: Use the pre-trained Vision Transformer model to extract features from the RGB frame images obtained in Step 2.1; for each image, a 1*768 dimensional feature vector will be obtained and used as the visual modality feature; Step 2.4: Use the pre-trained VGGish model to extract features from the audio segments obtained in Step 2.2; for each audio segment, a 1*128 dimensional feature vector will be obtained and used as the audio modality feature; Step 2.5: Use the pre-trained ALBert model to extract features from the video title text; for each video title, a 1*312 dimensional feature vector is obtained and used as the text modality feature.

3. The multimodal micro-video classification method based on temporal information enhancement according to claim 2, characterized in that: The specific method for step 3 is as follows: Step 3.1: Connect the visual modal features and audio modal features of each video segment obtained in Step 2; feed them into a fully connected layer of depth 2, and optimize the parameters of the fully connected layer based on backpropagation, and obtain the classification probability distribution of the segment for all video categories; Step 3.2: For the probability distribution of the classification results of each video segment, take the maximum value of the probability distribution corresponding to each category as the classification confidence of that category; at the same time, for all video segments, compare the classification confidence of each video segment, and take the two video segments with the highest classification confidence as the two video segments with the highest contribution to the classification. Step 3.3: For the visual modal features and audio modal features of the two video segments with the highest contribution to classification obtained in Step 3.2, set two learnable weight matrices respectively, perform matrix multiplication with their corresponding feature matrices, and perform matrix addition on the feature matrices of the two modalities corresponding to the two video segments; so that the feature matrices of the two video segments are merged into the feature matrix of one video segment, and the merged video segment is the most expressive video segment.

4. The multimodal micro-video classification method based on temporal information enhancement according to claim 3, characterized in that: All operations in step 4.3 are shown in the following formulas: （1）； （2）； （3）； （4）； in, The values ​​represent inter-frame similarity, where K, U, and V represent the feature matrices after convolutional blocks k, u, and v, respectively. The similarity between the frame and the average pooling is represented by S, where S represents the sum of the two similarity matrices. Represents the feature matrix enhanced by similarity S. A feature matrix representing the most expressive video clips.

5. The multimodal micro-video classification method based on temporal information enhancement according to claim 4, characterized in that: The enhanced feature matrix obtained in step 4.4 is shown in the following formula: （5）； in, The feature matrix after time-domain information enhancement Represents the feature matrix after max pooling. This represents the feature matrix after standard deviation pooling.

6. The multimodal micro-video classification method based on temporal information enhancement according to claim 5, characterized in that: The specific method for step 5 is as follows: Step 5.1: Input the visual modality feature matrix, audio modality feature matrix and original text modality feature matrix after temporal information enhancement into two fully connected layers to obtain the probability distribution of all categories for each modality; Step 5.2: Set three learnable weight matrices and perform matrix multiplication with the probability distributions of all categories corresponding to each modality in Step 5.1 for late fusion; finally, perform matrix addition on the three probability distributions of the same dimension, and take the probability distribution with the largest confidence value for each category as the final classification category.

Images