A multi-label classification method for short videos based on multi-scale temporal attention mechanism

Abstract

This invention discloses a multi-label classification method for short videos based on a multi-scale temporal attention mechanism, comprising: selecting a dataset and preprocessing; extracting and fusing features from samples in the training set, generating predicted scores for the model, and comparing them with the true label values; evaluating the model's predictive performance by calculating the objective function; feeding the first few sets of feature maps into a multi-head temporal feature map attention module after layer standardization to obtain processed feature maps; using 3D uniform pooling to directly generate feature vectors from the feature maps, processing them with fully connected layers to obtain feature scores, and then optimizing the model through loss function calculation and gradient descent to achieve multi-label classification of short videos. This invention can perform efficient and accurate multi-label classification on massive amounts of short video data.

Description

Technical Field

[0001] This invention relates to the field of intelligent video classification, and in particular to a multi-label classification method for short videos based on a multi-scale temporal attention mechanism. Background Technology

[0002] Since the rise of the short video industry, major short video companies have shown an extreme thirst for intelligent analysis technology. They have partnered with universities and research institutions to introduce advanced technologies such as big data analysis and artificial intelligence. Short video multi-label classification can be widely applied to scenarios such as short video retrieval and automatic short video annotation, effectively controlling rapidly increasing operating costs and giving companies a competitive edge in the future. Unlike traditional image classification, short video classification possesses not only spatial information but also temporal information. The extraction and utilization of temporal features to achieve high-accuracy short video multi-label classification has always been a key research focus.

[0003] To achieve high-accuracy multi-label classification of short videos, researchers mainly divide the problem into two main directions: feature representation of short videos and feature fusion of short videos. Feature representation of short videos can be divided into three aspects: visual, audio, and trajectory features. Utilizing neural networks to extract deep features is a traditional research direction in computer vision. Visual features are typically extracted in the RGB color space using CNNs (Convolutional Neural Networks), which are further divided into 2D-CNN and 3D-CNN. Audio feature extraction mainly utilizes MFCCs (Mel-Frequency Ceptral Coefficients), and trajectory feature extraction mainly utilizes TDDs (Trajectory-concatenated Deep Descriptors). Feature fusion of short videos mainly involves fusing different types of features extracted from the video, or features at different scales. The main types are front-end fusion, back-end fusion, and mid-stream fusion. The main difference between these three methods is the location where feature fusion occurs, and the choice can be made flexibly according to task requirements.

[0004] One common approach used by existing deep learning models when processing 2D images is multi-scale image feature extraction. This involves sampling image samples at different scales to obtain image data of varying sizes, extracting features from each scale to obtain their respective high-dimensional features, and then fusing these features to create a classification model that focuses on visual features at different scales. However, in short video classification tasks, ordinary spatial domain multi-scale visual features, due to their lack of sufficient utilization of temporal information, are often less effective than temporal action features in representing the visual semantics of actions. Summary of the Invention

[0005] This invention provides a multi-label classification method for short videos based on a multi-scale temporal attention mechanism. The invention aims to perform efficient and accurate multi-label classification on massive amounts of short video data, as detailed below:

[0006] A multi-label classification method for short videos based on a multi-scale temporal attention mechanism, the method comprising:

[0007] Dataset selection and preprocessing;

[0008] After feature extraction and feature fusion of the samples in the training set, the model generates a predicted score, which is compared with the true label value. The predictive performance of the model is evaluated by calculating the value of the objective function.

[0009] After the first few sets of feature maps are processed by layer normalization, they are fed into the multi-head temporal feature map attention module to obtain the processed feature map.

[0010] The feature map is processed using 3D uniform pooling to directly generate feature vectors, and then processed using a fully connected layer to obtain feature scores. Finally, the model is optimized through loss function calculation and gradient descent to enable it to perform multi-label classification of short videos.

[0011] The selection of the dataset and preprocessing are as follows:

[0012] Each video is uniformly extracted into 32 frames, and each image is randomly cropped to a size of 224×224 pixels, then normalized and randomly horizontally flipped.

[0013] The objective function is:

[0014]

[0015] Where M represents the total number of samples, y i,j s represents the true label value of the j-th category for the i-th sample. i,j This represents the model prediction value for the j-th category of the i-th sample.

[0016] The first few sets of feature maps are processed through layer normalization and then fed into the multi-head temporal feature map attention module to obtain the processed feature maps as follows:

[0017] For the pre-processed dataset, the input data to the model is video frames of size 32×224×224. The 32 frames of each sample are divided into 4 non-overlapping groups in chronological order, with 8 frames in each group, to obtain the 1st to 4th groups of video frames. Then, 2 frames are extracted from each of these 1st to 4th groups of video frames at equal intervals and arranged in chronological order to form the 5th group of video frames.

[0018] Five sets of video frames are fed into five pre-trained 3DCNN networks with shared parameters, outputting five sets of feature maps. The first four sets of feature maps are then input and convolved once with a 1×1 kernel. Each set of feature maps is then X... i The number of channels is tripled, and then evenly divided into 3 parts to obtain Q. i ,K i V i Three sets of data, where i represents the i-th feature map;

[0019] Q i K with each set of feature maps j Perform element-wise multiplication to obtain N of the same size. i,j Then V i respectively with N i,1 To N i,4 Perform element-wise multiplication, then add the four sets of feature maps element-wise, and then perform the same operation on each head to reconcat the outputs in channel order to obtain the processed feature maps.

[0020] The beneficial effects of the technical solution provided by this invention are:

[0021] 1. This invention proposes a multi-label classification method for short videos based on a multi-scale temporal attention mechanism. It extracts high frame rate sequences from videos and uniformly segments and samples them to obtain low frame rate sequences, thereby obtaining a more robust temporal multi-scale sequence for video recognition.

[0022] 2. This invention employs a dual-channel extraction method that focuses on low frame rate features emphasizing image information and high frame rate features emphasizing temporal information, enabling the model to comprehensively learn video representations;

[0023] 3. This invention achieves multi-level and efficient extraction and fusion of temporal features of short video data through differentiated sampling, network structure optimization, introduction of self-attention mechanism, and feature fusion strategy. Attached Figure Description

[0024] Figure 1 This is a flowchart of a short video multi-label classification method based on a multi-scale temporal attention mechanism;

[0025] Figure 2 This is an illustration of a short video tagged with "children, dogs". Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0027] This invention proposes a multi-scale temporal attention mechanism for short video multi-label classification. It extracts high-frame-rate sequences from videos and uniformly segments and samples them to obtain low-frame-rate sequences, thus acquiring more robust temporal multi-scale sequences for video recognition. A dual-channel extraction method is employed, focusing on low-frame-rate features emphasizing image information and high-frame-rate features emphasizing temporal information, enabling the model to comprehensively learn video representations. Through differential sampling, network structure optimization, the introduction of a self-attention mechanism, and feature fusion strategies, a system is constructed... Figure 1 The multi-scale fusion network shown.

[0028] Example 1

[0029] A multi-label classification method for short videos based on a multi-scale temporal attention mechanism, see [link to relevant documentation]. Figure 1 The method consists of three parts: dataset selection and preprocessing, objective function selection and construction, and model construction and process.

[0030] I. Dataset Selection and Preprocessing

[0031] For example, the MLSV2018 short video dataset released by Meitu was selected. It contains more than 200,000 short videos, covering 63 popular elements such as dance, fitness, and singing. Most videos are 5-15 seconds long. Furthermore, the videos use a multi-label classification system, with label information including multiple dimensions such as video subject, scene, and actions. The annotation information covers as many elements as possible in the video, and each video has 1-3 labels.

[0032] Before the original video dataset undergoes model training, the video needs to be converted into a series of frame images, which are then fed into the feature extraction part of the 3D-CNN. Therefore, frame extraction is required first. Considering the training cost and time complexity, this embodiment of the invention selects to uniformly extract 32 frames from each video, randomly crop each image to a size of 224×224 pixels, and perform standardization and random horizontal flipping to increase the diversity of the training set.

[0033] II. Constructing the Objective Function and Indicators

[0034] The model uses the common loss function for multi-label classification tasks, the Binary CrossEntropy Loss (BCELoss):

[0035]

[0036] Among them, y j ∈{0,1}, representing the true value of the j-th class label for each sample, s j This represents the model prediction value for this category, and satisfies sj ∈(0,1), δ(·) represents the Sigmoid activation function, and C represents the total number of labels.

[0037] In summary, the overall objective function of this model can be written as follows:

[0038]

[0039] Where M represents the total number of samples, y i,j s represents the true label value of the j-th category for the i-th sample. i,j This represents the model prediction value for the j-th category of the i-th sample.

[0040] In this process, the samples in the training set undergo feature extraction and feature fusion to extract useful information. Then, based on these features, the model generates predicted scores and compares them with the true label values. Next, the model's predictive performance is evaluated by calculating the objective function. To optimize the model's performance, we need to perform backpropagation calculations based on the gradient of the objective function to adjust the model's parameters, making the objective function value decrease as much as possible. This allows the model to better fit the data and improve prediction accuracy.

[0041] III. Model Construction and Process

[0042] For the pre-processed dataset, the input model consists of video frames of size 32×224×224. In this embodiment, the 32 frames of each sample are divided into four non-overlapping groups of eight frames each, resulting in groups 1-4. Then, two frames are extracted from each of these groups at equal intervals, and these are combined to form group 5. These five groups of video frames are then fed into five pre-trained 3DCNN networks with shared parameters. Taking 3DResNet50 pre-trained on the Kinetics-400 dataset as an example, it will output five feature maps of size 7×7 and dimension 2048.

[0043] X = X1, X2, X3, X4, X5

[0044] The first four feature maps X1, X2, X3, and X4 are then processed by layer normalization and fed into the multi-head temporal feature map attention module. Specifically, the channel dimension of each feature map is first evenly divided according to the pre-set number of heads. Taking a 16-head attention module as an example, the feature dimension of each head input is 2048 / 16 = 128.

[0045] In one of the heads, four sets of feature maps are input and each is convolved once with a 1×1 kernel. The result is X... iThe number of channels is tripled, and then evenly divided into 3 parts to obtain Q. i ,K i V i Three sets of data, where i represents the i-th feature map. Next, Q... i K with each set of feature maps j Perform element-wise multiplication to obtain N of the same size. i,j Then V i respectively with N i,1 To N i,4 Perform element-wise multiplication, then add the four sets of feature maps element-wise, and finally concatenate the outputs of each header in channel order to obtain the processed feature map.

[0046]

[0047] Then, concatenate the fifth set of feature maps with the first four sets of feature maps in sequence to obtain:

[0048]

[0049] The feature map is then processed using 3D uniform pooling to directly generate a feature vector with a dimension of 2048. A fully connected layer is then used to process the vector to obtain a 63-dimensional feature score. Finally, the model is optimized through loss function calculation and gradient descent to enable it to perform multi-label classification of short videos.

[0050] Experiments using this model setup yielded an accuracy of 88.76% on the MLSV2018 dataset, demonstrating that the invention can effectively achieve multi-label accurate classification of short videos using only single-modal visual features.

[0051] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-label classification method for short videos based on a multi-scale temporal attention mechanism, characterized in that, The method includes: Dataset selection and preprocessing; After feature extraction and feature fusion of the samples in the training set, the model generates a predicted score, which is compared with the true label value. The predictive performance of the model is evaluated by calculating the value of the objective function. After the first few sets of feature maps are processed by layer normalization, they are fed into the multi-head temporal feature map attention module to obtain the processed feature map. The feature map is directly generated by 3D uniform pooling, and the feature scores are obtained by using a fully connected layer. Then, the model is optimized by calculating the loss function and gradient descent to achieve multi-label classification of short videos. The first few sets of feature maps are processed through layer normalization and then fed into the multi-head temporal feature map attention module to obtain the processed feature maps as follows: For the pre-processed dataset, the input data to the model is video frames of size 32×224×224. The 32 frames of each sample are divided into 4 non-overlapping groups in chronological order, with 8 frames in each group, to obtain the 1st to 4th groups of video frames. Then, 2 frames are extracted from each of these 1st to 4th groups of video frames at equal intervals and arranged in chronological order to form the 5th group of video frames. Five sets of video frames are fed into five pre-trained 3DCNN networks with shared parameters, outputting five sets of feature maps. The first four sets of feature maps are then input and convolved once with a 1×1 kernel. Each set of feature maps is then X... i The number of channels is tripled, and then evenly divided into 3 parts to obtain Q. i , K i V i Three sets of data, where i represents the i-th feature map; Q i K with each set of feature maps j Perform element-wise multiplication to obtain N of the same size. i, j Then V i respectively with N i,1 To N i,4 Perform element-wise multiplication, then add the four sets of feature maps element-wise, and then perform the same operation on each head to reconcat the outputs in channel order to obtain the processed feature maps.

2. The short video multi-label classification method based on a multi-scale temporal attention mechanism according to claim 1, characterized in that, The selection of the dataset and preprocessing are as follows: Each video is uniformly extracted into 32 frames, and each image is randomly cropped to a size of 224×224 pixels, then normalized and randomly horizontally flipped.

3. The short video multi-label classification method based on a multi-scale temporal attention mechanism according to claim 1, characterized in that, The objective function is: ； Where M represents the total number of samples. This represents the true label value of the j-th category for the i-th sample. This represents the model prediction value for the j-th class of the i-th sample. This represents the Sigmoid activation function, and C represents the total number of labels.

Images