Video carousel detection method

Abstract

The invention belongs to the technical field of video carousel, and discloses a video carousel detection method, which comprises the following steps of: 1, preparing training data; 2, network scene differentiation; step 3, removing transition frames in scene switching; 4, splicing a new video; 5, extracting data; step 6, calculating related features and time sequence features; and step 7, classifying the videos. According to the method, fusion of local features and classification of global features are adopted, specifically, time sequence + space features obtained by the TSN serve as local features to be subjected to weighted fusion to obtain global features based on the whole video, the global features obtained through fusion serve as input of a full connection layer, classification of labels is finally obtained, the advantages of different networks are combined, and the classification efficiency is improved. The learning ability of the network is enhanced, the accuracy of carousel video detection is effectively improved, meanwhile, the number of data is input and fused, and priori knowledge of system detection is increased.

Description

Technical Field

[0001] The invention belongs to the technical field of video carousels, and in particular relates to a video carousel detection method. Background Art

[0002] The short video service (about 15 seconds) is becoming more and more popular among users, and its traffic is growing explosively. It appears in a large number of mobile apps to provide information and entertainment to users. The intuitive explanation of the carousel video is that it is a video composed of several pictures by adding special effects and other technologies. From the user's perspective, it contains very little information.

[0003] To solve the above problems, the prior art proposes two different solutions;

[0004] The first one is: before the neural network, the carousel video detection can be based on the traditional feature image feature extraction method. The basic steps include the following two steps: the first step is to extract a certain number of frames from the video, such as extracting one frame every ten frames; the second step is to use the opencv method to calculate the features of the pictures in the video, such as SIFT features, SURF features, ORB features, etc., and calculate the similarity between pictures based on the extracted features, such as calculating the Euclidean distance between pictures; the defects of this method are: poor generalization ability, many features of video RGB information cannot be effectively learned; insufficient analysis of the time domain information of the video; poor effect, and low accuracy for classifying more complex scenes in the video.

[0005] The second is: after the neural network, the technical solution 2 based on deep learning appeared. Technical solution 2 includes the following steps: The basic steps include the following two steps: The first step is to extract a certain number of frames from the video, also every ten frames; the second step is to use a neural network such as resnet to extract the features of the pictures in the video, and calculate the similarity between the pictures based on the extracted features. The defects of this method are: only using the image features of the video, although more image features can be learned, but still no other features are added, such as time series features; compared with the opencv method, the accuracy has been improved, but the effect is still not ideal;

[0006] Based on the defects of the above two methods, the present application proposes a complete technical solution for small video carousel detection. Summary of the invention

[0007] The purpose of the present invention is to provide a video carousel detection method to solve the problems raised in the above background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solution: a video carousel detection method, the video carousel detection method comprises the following steps:

[0009] Step 1: Prepare training data: Use a 10-20s short video dataset from the short video library as the training set;

[0010] Step 2: Network scene differentiation: Use network scene differentiation based on Transnet V2 to cut short videos and find the splicing boundary;

[0011] Step 3: Remove transition frames in scene switching: Use a deep learning method based on TransNet V2:

[0012] Specifically, each short video is segmented by scene, different scenes in the video and transition frames between scenes are detected, and then the intermediate transition frames are removed to extract the video segments containing information as the research object. The deep learning strategy greatly increases the accuracy of scene segmentation.

[0013] Step 4: stitching of new videos: remove the intermediate transition frames of scene switching, stitch the filtered frame list to get a new video, form a new data set, and perform subsequent data feature extraction based on the new data set;

[0014] Step 5, data extraction: Sample each scene separately, extract a certain number of frames of data, and extract a total of 300 frames of data to calculate the similarity between each frame;

[0015] Step 6: Calculation of related features and temporal features: The calculation of related features is to input a video and divide it into K segments (T 1 , T 2 , …, T K ), randomly select a snippet from each fragment; the calculation formula is:

[0016] TSN 1 , T 2 , …, T k )=H(G(F(T 1 , W), F(T 1 , W),…,F(T 1 , W)));

[0017] The temporal features are calculated by learning model parameters from the entire video rather than short clips. At the same time, by fixing the number of clips K for all videos and selecting only a small part of each clip as the input of the model, sparse temporal sampling is achieved;

[0018] Step 7: Classify the video: Use the Attention cluster algorithm model to classify the video; the model is divided into three parts.

[0019] Preferably, in the calculation formula in step 6, the F function is based on the analysis of the input snippet, and the output is the score of the snippet in each category. When it comes to the entire video, F represents the distribution of the socre in the category.

[0020] Preferably, in the calculation formula in step 6, the G function (feature fusion function G) is an aggregation function, and the mean function is used to average the scores of all snippets belonging to the same category;

[0021] The feature fusion function G combines the outputs from multiple short clips to obtain the consistency of class hypotheses between them;

[0022] The specific form of the feature fusion function G is:

[0023] G=G(F(T 1 ;ω), F(T 2 ;ω),…,F(T K ;ω))

[0024] Preferably, a feature fusion function G is used to infer G from the scores of the same class on all segments. i , the feature fusion function G adopts the simplest form, where G i =G(F i (T 1 ), …, F i (T K )).

[0025] Preferably, each fragment T in step six k From its corresponding segment S k The sampled fragments only contain a small fraction of the frames.

[0026] Preferably, in step 6, the concept of scene is added when selecting the snippet, and only a snippet in the same scene is randomly selected.

[0027] Preferably, the model in step seven is specifically divided into:

[0028] Part 1: Local feature extraction: Extract the features of the video through the CNN model. The extracted features are represented by X, and the formula is:

[0029] X=(x 1 , x 2 , …, x L )

[0030] Among them, the dimension of X is L, representing L different features;

[0031] Part 2: Local feature integration; obtaining global features based on Attention; the output of Attention is essentially equivalent to a weighted average of the extracted local features X;

[0032] The formula is: v = aX

[0033] Among them, v is the global feature output by an Attention unit, a is the weight vector, and it consists of two fully connected layers. The formula is: a = softmax(w 2 tanh(W 1 X T +b 1 )+b 2 )

[0034] where α and β are learnable scalars.

[0035] Attention clusters are used to combine the outputs of each attention unit to obtain multiple global features g, the formula is:

[0036] g=[v 1 , v 2 , …, v N ]

[0037] N represents the number of clusters;

[0038] Part 3: Global feature classification: After concatenating multiple global features, the final single-label or multi-label classification is performed through the conventional fully connected layer and Softmax.

[0039] The beneficial effects of the present invention are as follows:

[0040] The present invention adopts the fusion of local features and the classification of global features. Specifically, the time series + space features obtained by TSN are used as local features for weighted fusion to obtain global features based on the entire video. The fused global features are used as the input of the fully connected layer to finally obtain the classification of labels. On the basis of traditional correlation calculation, a calculation method that fuses RGB features + time series features is adopted to fully extract the information in the video and effectively improve the classification accuracy. Before feature extraction, a Transnet V2 network is added for scene differentiation, the purpose is to remove the intermediate transition frames and only study the part with carousel information. At the same time, the advantages of different networks are combined to enhance the learning ability of the network, effectively improving the accuracy of carousel video detection. At the same time, the number of input data and fusion are increased to increase the prior knowledge of system detection, and the accuracy rate is increased from the original 85.30% to 91.50%. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1It is a flow chart of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a video carousel detection method, which includes the following steps:

[0044] Step 1: Prepare training data: Use a 10-20s short video dataset from the short video library as the training set;

[0045] Step 2: Network scene differentiation: Use network scene differentiation based on Transnet V2 to cut short videos and find the splicing boundary;

[0046] Short videos are made up of video clips from multiple scenes; the splicing can be done directly or by a transition. To obtain the information of each small clip, the video needs to be cut, and it is crucial to accurately find the splicing boundary.

[0047] Step 3: Remove transition frames in scene switching: Use a deep learning method based on TransNet V2;

[0048] Specifically, each short video is segmented by scene, different scenes in the video and transition frames between scenes are detected, and then the intermediate transition frames are removed to extract the video segments containing information as the research object. The deep learning strategy greatly increases the accuracy of scene segmentation.

[0049] Step 4: stitching of new videos: remove the intermediate transition frames of scene switching, stitch the filtered frame list to get a new video, form a new data set, and perform subsequent data feature extraction based on the new data set;

[0050] Step 5, data extraction: Sample each scene separately, extract a certain number of frames of data, and extract a total of 300 frames of data to calculate the similarity between each frame;

[0051] Step 6, calculation of relevant features and temporal features: The calculation of relevant features is to input a video and divide it into K segments (K here can be 8) (T 1 , T 2 , …, T K), randomly select a snippet from each fragment; the calculation formula is:

[0052] TSN 1 , T 2 ,…,T k )=H(G(F(T 1 , W), F(T 1 , W),…,F(T 1 , W)));

[0053] The temporal features are calculated by learning model parameters from the entire video rather than short clips. At the same time, by fixing the number of clips K for all videos and selecting only a small part of each clip as the input of the model, sparse temporal sampling is achieved;

[0054] The TSN method is used to extract data features, which takes into account the temporal characteristics of the video;

[0055] The network part consists of two-way CNN, namely spatial stream ConvNets and temporal streamConvNets.

[0056] The inputs of the two parts are different. Spatial stream ConvNets take single RGB images as input; temporal stream ConvNets take a series of continuous optical flows as input.

[0057] In general, temporal stream ConvNets is the key part of the network. Its input is no longer RGB information, but also includes the optical flow information in the video. In fact, in order to avoid data redundancy, not all information will be extracted, but sparse sampling will be used to obtain it.

[0058] Step 7: Classify the video: Use the Attention cluster algorithm model to classify the video; the model is divided into three parts.

[0059] Main advantages:

[0060] a) The output of Attention is essentially a weighted average, which can avoid redundancy caused by some repeated features;

[0061] b) For some local key features, Attention can give them higher weights, so that the classification ability can be improved through these key features.

[0062] c) The input of Attention is an unordered set of arbitrary size. First, the temporal characteristics of the video have been studied in the previous TSN stage, so the classification stage does not need to focus on the temporal characteristics. Second, input of arbitrary size can improve the generalization ability of the model.

[0063] d) The simple Attention algorithm cannot describe the local features in the video well, but the Attention cluster algorithm can capture them well, so the Attention cluster is finally selected for the final classification.

[0064] In the calculation formula in step 6, the F function is based on the analysis of the input snippet, and the output is the score of the snippet in each class. When it comes to the entire video, F represents the distribution of the socre in the category.

[0065] In the calculation formula in step 6, the G function (feature fusion function G) is an aggregation function, which uses the mean function to average the scores of all snippets belonging to the same category;

[0066] The feature fusion function G combines the outputs from multiple short clips to obtain the consistency of class hypotheses between them;

[0067] The specific form of the feature fusion function G is:

[0068] G=G(F(T 1 ;ω), F(T 2 ;ω),…,F(T K ;ω))

[0069] Among them, the form of the feature fusion function G does not have a fixed form at present. In this patent, the feature fusion function G is used to infer G from the scores of the same class on all fragments. i , the feature fusion function G adopts the simplest form, where G i =G(F i (T 1 ), …, F i (T K )).

[0070] We evaluate several different forms of the feature fusion function G, including uniform averaging, maximum and weighted averaging in our experiments.

[0071] In fact, what we need are the features in the model. These features will be extracted during the training process, named and saved in the form of X. The feature dimension is saved as 300*2048, and then input into our subsequent classification model.

[0072] Each segment T k From its corresponding segment S k We randomly sample from frames, where the sampled segments only contain a small fraction of the frames. (Compared to previous work that used densely sampled frames, it significantly reduces the computational cost of evaluating ConvNets on frames.)

[0073] Among them, in step 6, the concept of scene is added when selecting the snippet, and only a snippet in the same scene is randomly selected.

[0074] Since there is a strong correlation between frames in each scene of the carousel video, and a weak correlation between frames in different scenes, the concept of scene is added when selecting snippets, and only a snippet in the same scene is randomly selected.

[0075] Among them, the model in step seven is specifically divided into:

[0076] Part 1: Local feature extraction: Extract the features of the video through the CNN model. The extracted features are represented by X, and the formula is:

[0077] X=(x 1 , x 2 , …, x L )

[0078] Among them, the dimension of X is L, representing L different features;

[0079] Part 2: Local feature integration; obtaining global features based on Attention; the output of Attention is essentially equivalent to a weighted average of the extracted local features X;

[0080] The formula is: v = aX

[0081] Among them, v is the global feature output by an Attention unit, a is the weight vector, which consists of two fully connected layers, and its formula is: a = soffmax(w 2 tanh(W 1 X T +b 1 )+b 2 )

[0082] where α and β are learnable scalars.

[0083] Attention clusters are used to combine the outputs of each attention unit to obtain multiple global features g, the formula is:

[0084] g=[v 1 , v2 , …, v N ]

[0085] N represents the number of clusters;

[0086] Part 3: Global feature classification: After concatenating multiple global features, the final single-label or multi-label classification is performed through the conventional fully connected layer and Softmax.

[0087] Other deep learning frameworks can be used for model training. For example, TSN can be replaced by TSM or I3D; Attentionclusters can be replaced by Attention LSTM or NextVlad, etc., replacing part of the network in the entire network. However, these all require a trade-off between accuracy, speed, and efficiency.

[0088] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0089] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A video carousel detection method, characterized in that: The video carousel detection method comprises the following steps: Step 1: Prepare training data: Use a 10-20s short video dataset from the short video library as the training set; Step 2: Network scene differentiation: Use network scene differentiation based on Transnet V2 to cut short videos and find the splicing boundary; Step 3: Remove transition frames in scene switching: Use a deep learning method based on TransNet V2; Specifically, each short video is segmented by scene, different scenes in the video and transition frames between scenes are detected, and then the intermediate transition frames are removed to extract the video segments containing information as the research object. The deep learning strategy greatly increases the accuracy of scene segmentation. Step 4: stitching of new videos: remove the intermediate transition frames of scene switching, stitch the filtered frame list to get a new video, form a new data set, and perform subsequent data feature extraction based on the new data set; Step 5, data extraction: Sample each scene separately, extract a certain number of frames of data, and extract a total of 300 frames of data to calculate the similarity between each frame; Step 6: Calculation of related features and temporal features: The calculation of related features is to input a video and divide it into K segments (T1, T2, ..., T K ), randomly select a snippet from each fragment; the calculation formula is: TSN(T1,T2,…,T k )=H(G(F(T1,W),F(T1,W),…,F(T1,W))); The temporal features are calculated by learning model parameters from the entire video rather than short clips. At the same time, by fixing the number of clips K for all videos and selecting only a small part of each clip as the input of the model, sparse temporal sampling is achieved; Step 7: Classify the video: Use the Attention cluster algorithm model to classify the video; the model is divided into three parts.

2. A video carousel detection method according to claim 1, characterized in that: In the calculation formula of step 6, the F function is based on the analysis of the input snippet, and the output is the score of the snippet in each class. When applied to the entire video, F represents the distribution of the snippet in the categories.

3. A video carousel detection method according to claim 1, characterized in that: In the calculation formula in step 6, the G function (feature fusion function G) is an aggregation function, which uses a mean function to average the scores of all snippets belonging to the same category; The feature fusion function G combines the outputs from multiple short clips to obtain the consistency of class hypotheses between them; The specific form of the feature fusion function G is: G=G(F(T1;ω),F(T2;ω),…,F(T K ;oh))。 4. A video carousel detection method according to claim 3, characterized in that: Use the feature fusion function G to infer G from the scores of the same class on all segments i , the feature fusion function G adopts the simplest form, where G i =G(F i (T1),…,F i (T K )).

5. A video carousel detection method according to claim 1, characterized in that: In step 6, each fragment T k From its corresponding segment S k The sampled fragments only contain a small fraction of the frames.

6. A video carousel detection method according to claim 1, characterized in that: In step 6, the concept of scene is added when selecting the snippet, and only a snippet in the same scene is randomly selected.

7. A video carousel detection method according to claim 1, characterized in that: The model in step seven is specifically divided into: Part 1: Local feature extraction; extract the features of the video through the CNN model; the extracted features are represented by X, and the formula is: X=(x1,x2,…,x L ) Among them, the dimension of X is L, representing L different features; Part 2: Local feature integration; obtaining global features based on Attention; the output of Attention is essentially equivalent to a weighted average of the extracted local features X; The formula is: v = aX Among them, v is the global feature output by an Attention unit, a is the weight vector, and it consists of two fully connected layers. The formula is: a = softmax(w2tanh(W1X T +b1)+b2) Where α and β are learnable scalars; Attention clusters are used to combine the outputs of each attention unit to obtain multiple global features g, the formula is: g=[v1,v2,…,v N ] N represents the number of clusters; Part 3: Global feature classification: After concatenating multiple global features, the final single-label or multi-label classification is performed through the conventional fully connected layer and Softmax.

Images