A short video classification method based on scene fragments and multimodal feature enhancement

Through the multimodal feature enhancement method based on scene clips, the problem of different modal quality of different scene clips in short videos is solved, more accurate short video classification is achieved, and classification performance is improved.

CN114741556BInactive Publication Date: 2024-11-29NORTHEASTERN UNIV CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210196211.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-01
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

There are challenges in the extraction and fusion of multimodal features of short videos, especially when the modal quality varies greatly in different scene segments, it is difficult to effectively classify short videos.

Method used

Using a multimodal feature enhancement method based on scene segments, crawl short video data, segment scene segments, extract visual, sound and subtitle modal features, dynamically select information-intensive and sparse features, and use pre-trained deep learning models and The neural network performs feature enhancement and fusion, and finally performs short video classification through multimodal features.

Benefits of technology

It improves the performance of short video classification, can effectively supplement modal information, enhance feature expression ability, and achieve more accurate short video classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure QLYQS_1
    Figure QLYQS_1
  • Figure QLYQS_5
    Figure QLYQS_5
  • Figure QLYQS_6
    Figure QLYQS_6
Patent Text Reader

Abstract

The present invention provides a short video classification method based on scene fragments and multimodal feature enhancement, which relates to the technical field of short video classification. The method comprises the following steps: capturing short videos and additional information of a short video platform, annotating short video data to construct a short video data set, dividing the video into multiple scene fragments according to the scene, and extracting multimodal information in each scene fragment, including key frames, audio and subtitles; extracting each modal feature by using a pre-trained deep learning model; dynamically selecting information-intensive features and information-sparse features in the same type of modality of multiple scene fragments of the short video, and using the former to enhance the semantics of the latter, and obtaining the enhanced modal features at the video granularity by connecting with the original features; taking the visual modality as the dominant modality and other modalities as auxiliary modalities, connecting the specific features in the auxiliary modality with the dominant modality, and obtaining the short video classification result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of short video classification, and in particular to a short video classification method based on scene fragments and multimodal feature enhancement. Background Art

[0002] With the rapid development of mobile Internet and social networks, short videos have spread rapidly on social networks with their advantages of being "short, fast and precise", becoming an important part of the current media structure. Studies have shown that short video data is growing explosively, and people are in urgent need of finding useful information quickly and accurately from these massive short videos. However, humans have limited ability to process these unstructured and complex data, so intelligent and automated technologies are needed to process video information.

[0003] Multimodal fusion refers to the integration of information from multiple modalities to complete classification or regression tasks. Short videos are a combination of multiple modalities, including visual, audio, and textual modalities. It is generally believed that introducing multiple modalities when observing the same phenomenon will enhance the robustness of predictions, and exposure to information from multiple modalities will make it more likely to capture complementary information, that is, information unique to only one modality can complement information that other modalities do not have. In addition, multimodal features can ensure good performance even when one modality is missing.

[0004] Therefore, compared with traditional video understanding, short video understanding has some challenges. Short videos themselves are a unity of multiple modalities such as social attributes, vision, sound, and text. Therefore, it is very important to effectively extract features from multiple modalities of short videos and integrate them to fully explore the potential information in them to complete short video understanding.

[0005] After investigation, it was found that for a short video, the contribution of the same modality in different scene segments of the video is different, that is, each modality may show different forms and levels of noise at different time points. Because users add some additional interference information with other intentions when making short videos, or because of the instantaneous change of shooting conditions, or because of the transition of the camera, the modality quality of a short video in different time periods will be different. Therefore, a technology is needed to analyze the modality of different scene segments of short videos to effectively solve the problem of short video classification. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a short video classification method based on scene fragments and multimodal feature enhancement.

[0007] A short video classification method based on scene fragments and multimodal feature enhancement, specifically comprising the following steps:

[0008] Step 1: Crawl short videos and their additional information from short video platforms to build a short video dataset; additional information includes video description, tags, video duration, and number of views;

[0009] Step 1.1: Select some users with a large number of fans on the short video platform as seed users, and crawl the video-related information of the fan list of the seed users, including video description, tags, video duration, and number of views;

[0010] Step 1.2: Extract features from the labels in the video-related information obtained in step 1.1, and use the K-means clustering algorithm to cluster the most similar labels into a cluster;

[0011] Step 1.3: Based on the clusters obtained in step 1.2, manually define the category to which each cluster belongs, which is called the category keyword, such as food, cute pets, and beauty;

[0012] Step 1.4: Enter the category keywords in step 1.3 in the search field on the short video platform, crawl the searched short videos and their additional information, and build a short video dataset;

[0013] Step 2: Segment each short video in the short video dataset obtained in step 1 into multiple scene segments according to the scene, and extract the key frames, audio and subtitles in each scene segment as the visual modality, sound modality and subtitle modality respectively;

[0014] Step 2.1: Use the scene recognition segmentation tool to segment each short video in the short video dataset into multiple scene segments according to the scene, and save the key frame images generated during the segmentation as the visual modality;

[0015] Step 2.2: Use an open source video segmentation tool to separate the audio file from each scene clip as the sound modality;

[0016] Step 2.3: Use open source OCR tools to extract subtitle information from the key frame image obtained in step 2.1 as the subtitle mode;

[0017] Step 3: Use the pre-trained deep learning model to extract feature vectors for the visual modality, audio modality, and subtitle modality obtained in step 2, and save them into an h5 file; the pre-trained deep learning model includes pre-trained visual, audio, and text modality feature extractors;

[0018] Step 3.1: Use the pre-trained visual modality feature extractor to extract visual features for each keyframe, then use the maximum pooling strategy for all keyframes of each scene clip, and finally generate a visual modality feature vector for each scene clip;

[0019] Step 3.2: Use the pre-trained sound modality feature extractor to extract sound features; then perform maximum pooling on the features of the audio file to generate a sound modality feature vector for each scene segment;

[0020] Step 3.3: Concatenate the subtitles in the key frames of each scene clip into a sentence, and then use the pre-trained text feature extractor to generate a subtitle modality feature vector for the subtitle information from each scene clip;

[0021] Step 3.4: Save the visual, sound, and subtitle modal feature vectors and the categories corresponding to the video into an h5 file;

[0022] Step 4: Dynamically select information-intensive features and information-sparse features for the visual modalities, audio modalities, and subtitle modalities in multiple scene clips, and use the information-intensive features to enhance the semantics of the information-sparse features; connect the visual, audio, and subtitle modal feature vectors extracted in step 3 with the corresponding semantically enhanced information-sparse features and input them into a two-layer neural network to learn more expressive visual, audio, and subtitle modal features, and connect the more expressive features in multiple scene clips to obtain enhanced visual modal, audio modal, and subtitle modal features at the video granularity;

[0023] Step 4.1: According to the feature vectors of the visual modality, audio modality and subtitle modality extracted in step 3, their high-level semantic representations are obtained through nonlinear transformation:

[0024]

[0025] in, and They represent the original feature vector, high-level semantic feature vector, activation function and trainable weight matrix of the modality m of the j-th scene segment, respectively. m∈{v,a,s} represents the visual modality, the audio modality and the subtitle modality, respectively.

[0026] Step 4.2: Dynamically select information-intensive features and information-sparse features from the same type of modalities of multiple scene clips, where the same type of modalities refers to the same visual modality, audio modality, or subtitle modality;

[0027] Step 4.2.1: Initialize the learnable weight matrix C represents the number of categories, D m Representing the feature dimension of the modality m, calculating the score size of the weight matrix corresponding to the same type of modality of multiple scene clips, wherein the same type of modality refers to the visual modality, the sound modality, or the subtitle modality;

[0028]

[0029] Among them, Q[α,β] represents the weight value of the αth row and βth column in the matrix Q. represents the score of the weight matrix of modality m of the jth scene segment;

[0030] Step 4.2.2: Calculate the weight matrix with the largest score of the same type of modality of multiple scene clips, denoted as Z. The feature corresponding to the weight matrix is ​​the information-intensive scene clip modality feature, denoted as The modal features of other scene clips are sparse features, expressed as

[0031]

[0032] Where M is the number of scene segments in a video, and Max represents the maximum value function;

[0033] Step 4.3: Calculate the Euclidean distance between information-intensive features and information-sparse features Calculate the sum of the Euclidean distances of all information-intensive and information-sparse features, loss, and use it as the loss function to make the information-sparse features approach the information-intensive features, thereby enhancing the semantics of the information-sparse features by using the information-intensive features.

[0034]

[0035]

[0036] Step 4.4: Obtain enhanced modal features at the video granularity according to the multiple modal features before semantic enhancement and the multiple modal features after semantic enhancement;

[0037] Step 4.4.1: Concatenate the feature vector after semantic enhancement with the feature vector before semantic enhancement and input it into a two-layer neural network to learn a more expressive feature vector, i.e., the scene fragment granularity;

[0038]

[0039]

[0040] Where conc represents the connection function, represents the feature vector of the mode m of the j-th scene segment after connection, represents the trainable weight matrix, σ represents the activation function, A more expressive feature vector representing the modality m of the j-th scene segment;

[0041] Step 4.4.2: Connect the modal features of the same type in multiple scene clips obtained in step 4.4.1 to obtain multiple modal feature vectors at the video granularity;

[0042]

[0043] in, Represents the feature vector of the i-th video modality m;

[0044] Step 4.4.3: Input the multiple modal feature vectors obtained in step 4.4.2 into a fully connected softmax layer to obtain the probability distribution in the short video classification;

[0045] Calculate the probability distribution of short video categories for multiple modal feature vectors of short videos:

[0046]

[0047] Among them, W m represents a trainable weight matrix, softmax represents a normalized exponential function, Represents the probability distribution of mode m for short video categories;

[0048] Calculate the cross entropy loss between the predicted categories and the true categories of multiple modal feature vectors:

[0049]

[0050] N represents the number of short videos, yi represents the true category of the i-th short video;

[0051] Step 5: Take the visual modality feature vector at the video granularity as the dominant modality, and the sound modality feature vector and subtitle modality feature vector at the video granularity as the auxiliary modality, calculate the correlation between the dominant modality and the auxiliary modality, obtain the unique score vector in the auxiliary modality that is not highly correlated with the dominant modality, and connect the unique score vector in the auxiliary modality with the dominant modality as the complementary information of the dominant modality to form the final multimodal fusion feature vector of the video for short video classification, thereby obtaining the short video classification result;

[0052] Step 5.1: Take the visual modality at the video granularity as the dominant modality, the sound modality and subtitle modality at the video granularity as auxiliary modalities, and connect the auxiliary modality and the dominant modality features as E vt , calculate the characteristic correlation score vector of the auxiliary mode and the dominant mode and record it as R vt , R vt The smaller the score in , the smaller the correlation between the feature in the auxiliary mode and the dominant mode;

[0053] E vt=conc(E v ,E t )

[0054] R vt =softmax(W v ·E vt )

[0055] Among them, E v represents the dominant modality feature at the video granularity, namely the visual modality feature, E t represents the auxiliary modality feature at the video granularity, namely the sound modality feature or the subtitle modality feature, t∈{a,s} represents the sound modality and the subtitle modality respectively, v represents the visual modality, conc represents the connection function, softmax represents the normalized exponential function, and W v represents a trainable weight matrix;

[0056] Step 5.2: Pass a learnable threshold ρ m , obtain the characteristic score vector γ in the auxiliary mode that is not closely related to the dominant mode from the characteristic correlation score vector vt :

[0057]

[0058] Among them, R vt [i] represents the relevance score vector R vt The value of position i in , γ vt [i] represents the uniqueness score vector γ vt The value of position i in ;

[0059] Step 5.3: The uniqueness score vector in the auxiliary modality is used as the complementary information of the dominant modality and connected with the dominant modality feature to form the final multimodal fusion feature vector of the video. The final multimodal fusion feature vector of the short video is:

[0060]

[0061] Among them, γ va and γ vs They represent the uniqueness score vectors of the audio modality and subtitle modality that are not very relevant to the visual modality features, E a and E s Represent the audio modality features and subtitle modality features respectively, It means the cross product is the multiplication of corresponding elements of two vectors;

[0062] Step 5.4: Input the final multimodal fusion feature vector into a fully connected softmax layer to obtain the probability distribution in the short video classification, perform loss compensation on the probability distribution, and obtain the short video classification result;

[0063] Calculate the probability distribution of the feature vector of the final multimodal fusion of the short video for the short video category:

[0064]

[0065] Among them, W p represents a trainable weight matrix, and softmax represents a normalized exponential function;

[0066] Calculate the cross entropy loss between the predicted category and the true category:

[0067]

[0068] Add the cross entropy loss to the loss1 obtained in step 4.3 and the loss2 obtained in step 4.4.2 as the objective function to obtain the scene classification result;

[0069] Loss=loss1+loss2+loss3.

[0070] Beneficial effects of the present invention:

[0071] In view of the fact that short videos are a multimodal fusion, the present invention constructs a short video dataset for short video classification. This method mines the subtitle modality in short videos and can effectively supplement the modal information. In addition, the present invention provides a multimodal feature enhancement and supplementary short video classification method based on scene fragments, which analyzes the quality of different modalities from a finer granularity of scene fragments, dynamically selects information-intensive features and information-sparse features, and uses the former to enhance the semantic information of the latter, so that it has stronger expressive power. In addition, this method uses the visual modality as the dominant modality and other modalities as auxiliary modalities, connects the unique features of the auxiliary modalities with the dominant modality, and jointly realizes short video classification. In summary, the present invention can provide better performance for short video classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is an example diagram of the segmentation of scene segments in the present invention.

[0073] Figure 2 This is an example diagram of mining subtitle mode in the present invention.

[0074] Figure 3 This is a framework diagram of a short video classification method based on multimodal feature enhancement and supplementation of scene fragments in the present invention. DETAILED DESCRIPTION

[0075] A short video classification method based on scene fragments and multimodal feature enhancement is further described below in conjunction with the accompanying drawings and embodiments;

[0076] A short video classification method based on scene fragments and multimodal feature enhancement, specifically comprising the following steps:

[0077] Step 1: Crawl short videos and their additional information from short video platforms to build a short video dataset; additional information includes video description, tags, video duration, and number of views;

[0078] Step 1.1: Select some users with a large number of fans on the short video platform as seed users, and crawl the video-related information of the fan list of the seed users, including video description, tags, video duration, and number of views;

[0079] Step 1.2: Extract features from the labels in the video-related information obtained in step 1.1, and use the K-means clustering algorithm to cluster the most similar labels into a cluster;

[0080] Step 1.3: Based on the clusters obtained in step 1.2, manually define the category to which each cluster belongs, which is called the category keyword, such as food, cute pets, and beauty;

[0081] Step 1.4: Enter the category keywords in step 1.3 in the search field on the short video platform, crawl the searched short videos and their additional information, and build a short video dataset;

[0082] Step 2: Segment each short video in the short video dataset obtained in step 1 into multiple scene segments according to the scene, and extract the key frames, audio and subtitles in each scene segment as the visual modality, sound modality and subtitle modality respectively;

[0083] Step 2.1: Use scene recognition and segmentation tools such as the open source tool PySceneDetect to segment each short video in the short video dataset into multiple scene segments according to the scene, and save the key frame images generated during the segmentation as the visual modality; Figure 1 As shown in the figure, the method for detecting scene segments during segmentation is detect-content, which compares each frame and searches for content changes in turn. In addition, the minimum duration of each scene segment is limited to 3 seconds, and 5 frames of images are taken from each scene segment as key frames.

[0084] The naming format of each scene clip is video_id-Scene-n.mp4, where n represents the nth scene clip of the video, and multiple scene clips of a video are saved in a folder named by video_id. The naming format of each key frame is video_id-Scene-nm.png, where n represents the nth scene clip and m represents the mth key frame, and multiple key frames are saved in a folder named by video_id.

[0085] Step 2.2: Use an open source video segmentation tool such as FFmpeg (Fast Forward Moving Picture Experts Group) to separate the audio file from each scene clip as the sound modality;

[0086] The naming format of each audio file is video_id-Scene-n.m4a, where n represents the nth scene segment. The audio files of multiple scene segments of a video are saved in a folder named with video_id.

[0087] Step 2.3: Use an open source OCR tool such as PaddleOCR to extract the subtitle information from the key frame image obtained in step 2.1 as the subtitle mode;

[0088] The subtitle information of each scene segment is saved in the video_id-Scene-n.csv file, where the first column is the name of the key frame corresponding to the subtitle, and the second column is the extracted subtitle information.

[0089] Step 3: Use the pre-trained deep learning model to extract feature vectors for the visual modality, audio modality, and subtitle modality obtained in step 2, and save them into an h5 file; the pre-trained deep learning model includes pre-trained visual, audio, and text modality feature extractors;

[0090] Step 3.1: Use a pre-trained visual modality feature extractor such as the Transformer model to extract 768-dimensional visual features for each keyframe, then use a maximum pooling strategy for all keyframes of each scene clip, and finally generate a 768-dimensional visual modality feature vector for each scene clip;

[0091] Step 3.2: Use a pre-trained sound modality feature extractor, such as the VGGish model pre-trained on the Audioset audio dataset, to extract sound features; extract nums_frames*128-dimensional features for each audio, where nums_frames = audio duration / 0.96. Then perform maximum pooling on the features of the audio file, and finally generate a 128-dimensional sound modality feature vector for each scene segment;

[0092] Step 3.3: Connect the subtitles in the key frames of each scene clip into a sentence, such as Figure 2 As shown. Then, a pre-trained text feature extractor such as the ALBERT model is used to generate a 312-dimensional subtitle modality feature vector for the subtitle information from each scene segment;

[0093] Step 3.4: Save the visual, audio, and subtitle modality feature vectors and the category corresponding to the video (encoded in one-hot format) into an h5 file;

[0094] Step 4: Dynamically select information-intensive features and information-sparse features (such as visual modality features in n scene segments) for the visual modality, audio modality, and subtitle modality in multiple scene segments, and use information-intensive features to enhance the semantics of information-sparse features; connect the visual, audio, and subtitle modality feature vectors extracted in step 3 with the corresponding semantically enhanced information-sparse features and input them into a two-layer neural network to learn more expressive visual, audio, and subtitle modality features, and connect the more expressive features in multiple scene segments to obtain enhanced visual modality, audio modality, and subtitle modality features at the video granularity;

[0095] Step 4.1: According to the feature vectors of the visual modality, audio modality and subtitle modality extracted in step 3, their high-level semantic representations are obtained through nonlinear transformation:

[0096]

[0097] in and They represent the original feature vector, high-level semantic feature vector, activation function and trainable weight matrix of the modality m of the j-th scene segment, respectively. m∈{v,a,s} represents the visual modality, the audio modality and the subtitle modality, respectively.

[0098] Step 4.2: Dynamically select information-intensive features and information-sparse features from the same type of modalities of multiple scene clips, where the same type of modalities refers to the same visual modality, audio modality, or subtitle modality;

[0099] Step 4.2.1: Initialize the learnable weight matrix in the neural network C represents the number of categories, D m Representing the feature dimension of the modality m, calculating the score size of the weight matrix corresponding to the same type of modality of multiple scene clips, wherein the same type of modality refers to the visual modality, the sound modality, or the subtitle modality;

[0100]

[0101] Where Q[α,β] represents the weight value of the αth row and βth column in the matrix Q. represents the score of the weight matrix of modality m of the jth scene segment;

[0102] Step 4.2.2: Calculate the weight matrix with the largest score of the same type of modality of multiple scene clips, denoted as Z. The feature corresponding to the weight matrix is ​​the information-intensive scene clip modality feature, denoted as The modal features of other scene clips are sparse features, expressed as

[0103]

[0104] Where M is the number of scene segments in a video, and Max represents the maximum value function;

[0105] Step 4.3: Calculate the Euclidean distance between information-intensive features and information-sparse features Calculate the sum of the Euclidean distances of all information-intensive and information-sparse features, loss, and use it as the loss function to make the information-sparse features approach the information-intensive features, thereby enhancing the semantics of the information-sparse features by using the information-intensive features.

[0106]

[0107]

[0108] Step 4.4: Obtain enhanced modal features at the video granularity according to the multiple modal features before semantic enhancement and the multiple modal features after semantic enhancement;

[0109] Step 4.4.1: Concatenate the feature vector after semantic enhancement with the feature vector before semantic enhancement and input it into a two-layer neural network to learn a more expressive feature vector, i.e., the scene fragment granularity;

[0110]

[0111]

[0112] Where conc represents the connection function, represents the feature vector of the mode m of the j-th scene segment after connection, represents the trainable weight matrix, σ represents the activation function, A more expressive feature vector representing the modality m of the j-th scene segment;

[0113] Step 4.4.2: Connect the modal features of the same type in multiple scene clips obtained in step 4.4.1 to obtain multiple modal feature vectors at the video granularity;

[0114]

[0115] in, Represents the feature vector of the i-th video modality m;

[0116] Step 4.4.3: Input the multiple modal feature vectors obtained in step 4.4.2 into a fully connected softmax layer to obtain the probability distribution in the short video classification;

[0117] Calculate the probability distribution of short video categories for multiple modal feature vectors of short videos:

[0118]

[0119] Among them, W m represents a trainable weight matrix, softmax represents a normalized exponential function, Represents the probability distribution of mode m for short video categories;

[0120] Calculate the cross entropy loss between the predicted categories and the true categories of multiple modal feature vectors:

[0121]

[0122] N represents the number of short videos, yi represents the true category of the i-th short video;

[0123] Step 5: Select the modality with better modality quality as the dominant modality. Experiments show that the visual modality generally has the best quality. Therefore, this method takes the visual modality as the dominant modality, the visual modality feature vector at the video granularity as the dominant modality, the sound modality feature vector and the subtitle modality feature vector at the video granularity as the auxiliary modality, calculates the correlation between the dominant modality and the auxiliary modality, obtains the uniqueness score vector in the auxiliary modality that is not highly correlated with the dominant modality, and connects the uniqueness score vector in the auxiliary modality as the complementary information of the dominant modality with the dominant modality to form the final multimodal fusion feature vector of the video for short video classification, thereby obtaining the short video classification result; the algorithm model is as follows: Figure 3 As shown;

[0124] Step 5.1: Take the visual modality at the video granularity as the dominant modality, the sound modality and subtitle modality at the video granularity as auxiliary modalities, and connect the auxiliary modality and the dominant modality features as E vt , calculate the characteristic correlation score vector of the auxiliary mode and the dominant mode and record it as R vt , R vt The smaller the score in , the smaller the correlation between the feature in the auxiliary mode and the dominant mode;

[0125] E vt =conc(E v ,E t )

[0126] Rvt =softmax(W v ·E vt )

[0127] Among them, E v represents the dominant modality feature at the video granularity, namely the visual modality feature, E t represents the auxiliary modality feature at the video granularity, namely the sound modality feature or the subtitle modality feature, t∈{a,s} represents the sound modality and the subtitle modality respectively, v represents the visual modality, conc represents the connection function, softmax represents the normalized exponential function, and W v represents a trainable weight matrix;

[0128] Step 5.2: Pass a learnable threshold ρ m , obtain the characteristic score vector γ in the auxiliary mode that is not closely related to the dominant mode from the characteristic correlation score vector vt :

[0129]

[0130] Among them, R vt [i] represents the relevance score vector R vt The value of position i in , γ vt [i] represents the uniqueness score vector γ vt The value of position i in ;

[0131] Step 5.3: The uniqueness score vector in the auxiliary modality is used as the complementary information of the dominant modality and connected with the dominant modality feature to form the final multimodal fusion feature vector of the video. The final multimodal fusion feature vector of the short video is:

[0132]

[0133] Among them, γ va and γ vs They represent the uniqueness score vectors of the audio modality and subtitle modality that are not very relevant to the visual modality features, E a and E s Represent the audio modality features and subtitle modality features respectively, It means the cross product is the multiplication of corresponding elements of two vectors;

[0134] Step 5.4: Pass the final multimodal fusion feature vector through a fully connected softmax layer to obtain the probability distribution in the short video classification, perform loss compensation on the probability distribution, and obtain the short video classification result;

[0135] Calculate the probability distribution of the feature vector of the final multimodal fusion of the short video for the short video category:

[0136]

[0137] Among them, W p represents a trainable weight matrix, and softmax represents a normalized exponential function;

[0138] Calculate the cross entropy loss between the predicted category and the true category:

[0139]

[0140] Add the cross entropy loss to the loss1 obtained in step 4.3 and the loss2 obtained in step 4.4.2 as the objective function to obtain the scene classification result;

[0141] Loss=loss1+loss2+loss3.

[0142] In addition, the present invention conducts experiments on the constructed data set, and the experiments show that the short video method based on multimodal feature enhancement and supplementation of scene fragments proposed in the present invention has a result of 0.9835 on Macro-F1 and a result of 0.9851 on Micro-F1.

Claims

1. A short video classification method based on scene fragments and multimodal feature enhancement, characterized in that: The specific steps include: Step 1: Crawl short videos and their additional information from short video platforms to build a short video dataset; additional information includes video description, tags, video duration, and number of views; Step 2: Segment each short video in the short video dataset obtained in step 1 into multiple scene segments according to the scene, and extract the key frames, audio and subtitles in each scene segment as the visual modality, sound modality and subtitle modality respectively; Step 3: Use the pre-trained deep learning model to extract feature vectors for the visual modality, audio modality, and subtitle modality obtained in step 2, and save them into an h5 file; the pre-trained deep learning model includes pre-trained visual, audio, and text modality feature extractors; Step 4: Dynamically select information-intensive features and information-sparse features for the visual modality, audio modality, and subtitle modality in multiple scene clips, and use the information-intensive features to enhance the semantics of the information-sparse features; The visual, audio, and subtitle modal feature vectors extracted in step 3 are respectively connected with the corresponding semantically enhanced information sparse features and input into a two-layer neural network to learn more expressive visual, audio, and subtitle modal features, and the more expressive features in multiple scene clips are connected to obtain enhanced visual modality, audio modality, and subtitle modality features at the video granularity; Step 5: Take the visual modality feature vector at the video granularity as the dominant modality, and the sound modality feature vector and subtitle modality feature vector at the video granularity as auxiliary modalities, calculate the correlation between the dominant modality and the auxiliary modality, obtain the uniqueness score vector in the auxiliary modality that is not highly correlated with the dominant modality, and connect the uniqueness score vector in the auxiliary modality with the dominant modality as the complementary information of the dominant modality to form the final multimodal fusion feature vector of the video for short video classification, thereby obtaining the short video classification result.

2. According to claim 1, a short video classification method based on scene fragments and multimodal feature enhancement is characterized in that: The step 1 is specifically as follows: Step 1.1: Select some users with a large number of fans on the short video platform as seed users, and crawl the video-related information of the fan list of the seed users, including video description, tags, video duration, and number of views; Step 1.2: Extract features from the labels in the video-related information obtained in step 1.1, and use the K-means clustering algorithm to cluster the most similar labels into a cluster; Step 1.3: Based on the clusters obtained in step 1.2, manually define the category to which each cluster belongs, which are called category keywords, such as food, cute pets, and beauty; Step 1.4: Enter the category keywords in step 1.3 in the search field on the short video platform, crawl the searched short videos and their additional information, and build a short video dataset.

3. The short video classification method based on scene fragments and multimodal feature enhancement according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1: Use the scene recognition segmentation tool to segment each short video in the short video dataset into multiple scene segments according to the scene, and save the key frame images generated during the segmentation as the visual modality; Step 2.2: Use an open source video segmentation tool to separate the audio file from each scene clip as the sound modality; Step 2.3: Use an open source OCR tool to extract the subtitle information from the key frame image obtained in step 2.1 as the subtitle mode.

4. The short video classification method based on scene fragments and multimodal feature enhancement according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.1: Use the pre-trained visual modality feature extractor to extract visual features for each keyframe, then use the maximum pooling strategy for all keyframes of each scene clip, and finally generate a visual modality feature vector for each scene clip; Step 3.2: Use the pre-trained sound modality feature extractor to extract sound features; then perform maximum pooling on the features of the audio file to generate a sound modality feature vector for each scene segment; Step 3.3: Concatenate the subtitles in the key frames of each scene clip into a sentence, and then use the pre-trained text feature extractor to generate a subtitle modality feature vector for the subtitle information from each scene clip; Step 3.4: Save the visual, sound, and subtitle modality feature vectors and the categories corresponding to the video into an h5 file.

5. The short video classification method based on scene segments and multimodal feature enhancement according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.1: According to the feature vectors of the visual modality, audio modality and subtitle modality extracted in step 3, their high-level semantic representations are obtained through nonlinear transformation: in, and They represent the original feature vector, high-level semantic feature vector, activation function and trainable weight matrix of the modality m of the j-th scene segment, respectively. m∈{v,a,s} represents the visual modality, the audio modality and the subtitle modality, respectively. Step 4.2: Dynamically select information-intensive features and information-sparse features from the same type of modalities of multiple scene clips, where the same type of modalities refers to the same visual modality, audio modality, or subtitle modality; Step 4.3: Calculate the Euclidean distance between information-intensive features and information-sparse features Calculate the sum of the Euclidean distances of all information-intensive and information-sparse features, loss, and use it as the loss function to make the information-sparse features approach the information-intensive features, thereby enhancing the semantics of the information-sparse features by using the information-intensive features. Step 4.4: Obtain enhanced modal features at the video granularity based on the multi-modal features before semantic enhancement and the multi-modal features after semantic enhancement.

6. The short video classification method based on scene segments and multimodal feature enhancement according to claim 5, characterized in that: The step 4.2 is specifically as follows: Step 4.2.1: Initialize the learnable weight matrix C represents the number of categories, D m Representing the feature dimension of the modality m, calculating the score size of the weight matrix corresponding to the same type of modality of multiple scene clips, wherein the same type of modality refers to the visual modality, the sound modality, or the subtitle modality; Among them, Q[α,β] represents the weight value of the αth row and βth column in the matrix Q. represents the score of the weight matrix of modality m of the jth scene segment; Step 4.2.2: Calculate the weight matrix with the largest score of the same type of modality of multiple scene clips, denoted as Z. The feature corresponding to the weight matrix is ​​the information-intensive scene clip modality feature, denoted as The modal features of other scene clips are sparse features, expressed as Where M is the number of scene segments in a video, and Max represents the maximum value function.

7. The short video classification method based on scene segments and multimodal feature enhancement according to claim 5, characterized in that: The step 4.4 is specifically as follows: Step 4.4.1: Concatenate the feature vector after semantic enhancement with the feature vector before semantic enhancement and input it into a two-layer neural network to learn a more expressive feature vector, i.e., the scene fragment granularity; Where conc represents the connection function, represents the feature vector of the mode m of the j-th scene segment after connection, represents the trainable weight matrix, σ represents the activation function, A more expressive feature vector representing the modality m of the j-th scene segment; Step 4.4.2: Connect the modal features of the same type in multiple scene clips obtained in step 4.4.1 to obtain multiple modal feature vectors at the video granularity; in, Represents the feature vector of the i-th video modality m; Step 4.4.3: Input the multiple modal feature vectors obtained in step 4.4.2 into a fully connected softmax layer to obtain the probability distribution in the short video classification; Calculate the probability distribution of short video categories for multiple modal feature vectors of short videos: Among them, W m represents a trainable weight matrix, softmax represents a normalized exponential function, Represents the probability distribution of mode m for short video categories; Calculate the cross entropy loss between the predicted categories and the true categories of multiple modal feature vectors: N represents the number of short videos, and yi represents the true category of the i-th short video.

8. The short video classification method based on scene segments and multimodal feature enhancement according to claim 1, characterized in that: The step 5 is specifically as follows: Step 5.1: Take the visual modality at the video granularity as the dominant modality, the sound modality and subtitle modality at the video granularity as auxiliary modalities, and connect the auxiliary modality and the dominant modality features as E vt , calculate the characteristic correlation score vector of the auxiliary mode and the dominant mode and record it as R vt , R vt The smaller the score in , the smaller the correlation between the feature in the auxiliary mode and the dominant mode; AND vt =conc(E v ,AND t ) R vt =softmax(W v ·E vt ) Among them, E v represents the dominant modality feature at the video granularity, namely the visual modality feature, E t represents the auxiliary modality feature at the video granularity, namely the sound modality feature or the subtitle modality feature, t∈{a,s} represents the sound modality and the subtitle modality respectively, v represents the visual modality, conc represents the connection function, softmax represents the normalized exponential function, and W v represents a trainable weight matrix; Step 5.2: Pass a learnable threshold ρ m , obtain the characteristic score vector γ in the auxiliary mode that is not closely related to the dominant mode from the characteristic correlation score vector vt : Among them, R vt [i] represents the relevance score vector R vt The value of position i in , γ vt [i] represents the uniqueness score vector γ vt The value of position i in ; Step 5.3: The uniqueness score vector in the auxiliary modality is used as the complementary information of the dominant modality and connected with the dominant modality feature to form the final multimodal fusion feature vector of the video. The final multimodal fusion feature vector of the short video is: Among them, γ va and γ vs They represent the uniqueness score vectors of the audio modality and subtitle modality that are not very relevant to the visual modality features, E a and E s Represent the audio modality features and subtitle modality features respectively, It means the cross product is the multiplication of corresponding elements of two vectors; Step 5.4: Input the final multimodal fusion feature vector into a fully connected softmax layer to obtain the probability distribution in the short video classification, perform loss compensation on the probability distribution, and obtain the short video classification result; Calculate the probability distribution of the feature vector of the final multimodal fusion of the short video for the short video category: Among them, W p represents a trainable weight matrix, and softmax represents a normalized exponential function; Calculate the cross entropy loss between the predicted category and the true category: Add the cross entropy loss to the loss1 obtained in step 4.3 and the loss2 obtained in step 4.4.2 as the objective function to obtain the scene classification result; Loss=loss1+loss2+loss3.

Citation Information

Patent Citations

  • Multi-mode collaborative learning method and apparatus based on neural network

    CN108846375A

  • Short video classification method, system and medium based on multimodal dictionary learning

    CN109344887A