Content-based video core word generation system and generation method

Abstract

The application discloses a content-based video core word generation system, characterized by comprising a feature extraction module and a video description module, the feature extraction module adopts a VGG16_bn structure, and the video description module adopts a network structure based on Seq2Seq, and the technical problems of complex management and high management cost caused by a large number of video types in current video management are solved. The application further discloses a content-based video core word generation method, comprising the following steps: step 1, extracting video features through the feature extraction module; step 2, sending the extracted video feature information into the video description module to generate video text description; step 3, sending the generated video text description into a core word extraction module to obtain initial core words; and step 4, scoring and sorting the initial core words to obtain the final core words of the video, thereby solving the technical problem of low review efficiency caused by manual video review in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to a content-based video core word generation system and a content-based video core word generation method. Background Technology

[0002] With the advent of the knowledge-paying era and the continuous popularization of legal knowledge, people's awareness of copyright protection is constantly improving. In today's internet age, the development of the new media industry has driven a large number of video creators to produce high-quality video works, bringing cultural and spiritual wealth to society. At the same time, it has also spawned a large number of criminals who steal others' video clips or materials to seek personal gain. In this context, copyright protection becomes particularly important. Currently, in the management of uploaded videos, the large number and variety of videos lead to complex management and high costs. The detection and investigation of video infringement is generally carried out through manual review. However, manual review suffers from low efficiency, high costs, and difficulty in accurately remembering infringing segments, making it difficult to promptly detect infringement in videos and effectively reduce the economic losses suffered by video creators. Therefore, to enable video reviewers to quickly and accurately detect infringement and promptly protect the rights of video creators, this invention proposes a content-based video core keyword generation method. Summary of the Invention

[0003] The primary objective of this invention is to provide a content-based video keyword generation system, which solves the technical problems of complex and costly management caused by the large number and variety of videos in current video uploading management.

[0004] The second objective of this invention is to provide a content-based video keyword generation method, which solves the technical problem of low review efficiency caused by manual review of videos in the prior art.

[0005] The first technical solution adopted in this invention is a content-based video core word generation system, including a feature extraction module and a video description module. The feature extraction module adopts a VGG16_bn structure, and the video description module adopts a Seq2Seq-based network structure.

[0006] The video description module includes an encoder and a decoder. The encoder adopts an LSTM+self-attention network structure and consists of two layers: an LSTM-based video information encoding layer and a self-attention-based encoding information attention layer. The decoder adopts an LSTM+self-attention network structure and consists of two layers: an LSTM-based video information decoding layer and a self-attention-based semantic information attention layer.

[0007] The second technical solution adopted in this invention is a content-based video keyword generation method, which uses the aforementioned content-based video keyword generation system and specifically includes the following steps:

[0008] Step 1: Extract video features using the feature extraction module;

[0009] Step 2: Input the extracted video feature information into the video description module to generate a video text description;

[0010] Step 3: Send the generated video text description to the core word extraction module to obtain the initial core words;

[0011] Step 4: Score and sort the initial core words to obtain the final core words of the video.

[0012] The second technical solution of the present invention is characterized in that,

[0013] Step 1 is as follows:

[0014] The video is extracted using an equal-interval frame extraction method, resulting in a keyframe sequence S = [s1, s2, s3, ..., s...]. n ];

[0015] The obtained keyframe sequence S is fed into the feature extraction module in chronological order for feature extraction.

[0016] Step 2 is as follows:

[0017] Step 2.1: Encode the video feature information;

[0018] Step 2.1.1, according to equation (1), the feature vector e corresponding to each frame is... i The vector is fed into the LSTM network to obtain the initial encoding vector:

[0019] n i =LSTM(W1e i (i = 1, 2, ...) (1)

[0020] In equation (1), W1 is the linear transformation matrix, and n i For the initial encoding vector, e i For each frame, there is a feature vector.

[0021] Step 2.1.2, calculate the initial encoding vector n according to equation (2). i Relevance α i,j :

[0022] α i,j =W2tanh(W v ni +b) (i=1,2,…) (2)

[0023] In equation (2), W2 is the parameter matrix of the linear transformation, W v Let v be the transformation parameter matrix, and n be the parameter matrix. i Let b be the initial encoding vector, α be the bias term, and α be the initial encoding vector. i,j For n i Relevance;

[0024] Step 2.1.3: According to equation (3), normalization is performed using the softmax function to obtain the relevance weights:

[0025]

[0026] In equation (3), a i,j For relevance weights;

[0027] Step 2.1.4: Calculate the final encoding vector based on the relevance weights, as shown in equation (4):

[0028]

[0029] In equation (4), c i For the encoding vector, a i,j For relevance weights, n j This is the initial encoding vector;

[0030] Repeat steps 2.1.1 to 2.1.4 until all feature vectors have been encoded;

[0031] Step 2.2: Decode the video feature information;

[0032] Step 2.2.1, convert the encoded information c obtained in step 2.1 into... i The word w obtained in the previous moment i-1 Word embedding vector w_em i-1 The data is combined and fed into the LSTM network to obtain preliminary decoded information. i With the latent information h at this time i As shown in equations (5) and (6):

[0033] w_em i-1 =ebbedding(w i-1 (5)

[0034]

[0035] In equation (5), when i = 1, i.e. at the initial time, let w0 correspond to the sentence start flag. <sos>;

[0036] In equation (6), o i For the initial decoded information, i represents the hidden state information at this point, and c... i It is an encoded vector;

[0037] Step 2.2.2: A self-attention mechanism is used to calculate the relevance weights of the initial decoded information and the latent information at this moment. Based on the weights of both, a weighted sum is performed to obtain the probability distribution p of the word at this moment. i Specifically:

[0038] Step 2.2.2.1: The preliminary decoded information obtained in step 2.2.1... i With latent information i Equations (2), (3), and (4) are then calculated to obtain the weighted information vector context. i ;

[0039] Step 2.2.2.2, the obtained weighted information vector context i The data is fed into a fully connected layer to obtain the probability distribution p of the word at this moment. i As shown in equation (7):

[0040] p i =W p context i +b (7)

[0041] In equation (7), W p Let b be the parameter matrix in the transformation, and b be the bias term;

[0042] Step 2.2.3, based on the probability distribution p obtained in step 2.2.2. i Sampling is performed to obtain the word w at the current time. i ;

[0043] Repeat steps 2.2.1-2.2.3 until all encoded information has been decoded, yielding the final video text description: sentence={w1,w2,…,w n };

[0044] In step 2.2.3, sampling is performed using a cluster search algorithm.

[0045] Step 3 specifically involves using the TF-IDF algorithm to calculate the w of each word in the video text description sentence. i The TF-IDF value is used to filter out words with smaller TF-IDF values ​​in the video text description, resulting in the core word word = {word1, word2, ...}.

[0046] Step 4 specifically involves: refining the selection of core keywords obtained in Step 3;

[0047] Step 4.1: Extract the semantic feature information of the initial core words and sentences;

[0048] Using the BERT pre-trained model, the initially selected core words and sentences are fed into BERT to obtain the initial semantic feature vectors of the core words and sentences as {u1,u2,…} and v, respectively;

[0049] Step 4.2: Compare the semantic similarity between the initial core words and the sentences;

[0050] The semantic feature vectors of the initial core words are compared one by one with the semantic feature vectors of the sentences, and the cosine similarity calculation method is used to calculate the similarity, as shown in Equation (8):

[0051] sim = cos i ,v> (8)

[0052] Using similarity as the relevance score, the initial core words are sorted in descending order of similarity. The top five words with the highest similarity are selected as the core words to obtain the final core words of the video.

[0053] The beneficial effects of this invention are:

[0054] Compared with existing technologies, this invention can convert long video sequences into more accurate and concise core words, thereby reducing the cost of video management and classification, and solving the problems of high time consumption and management difficulties in reviewing long video sequences.

[0055] This invention provides concise and refined keywords for videos, enabling video reviewers to quickly grasp the core content and effectively improve their work efficiency, allowing for timely detection of infringing content. Simultaneously, it provides a valid basis for video uploading management and reduces the cost of category management. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the feature extraction network structure in the content-based video core word generation system of the present invention;

[0057] Figure 2 This is a schematic diagram of the Seq2Seq network framework in the content-based video core word generation system of this invention;

[0058] Figure 3 This is a schematic diagram of the encoder structure in the content-based video core word generation system of the present invention;

[0059] Figure 4 ​This is a schematic diagram of the decoder structure in the content-based video core word generation system of the present invention. Detailed Implementation

[0060] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0061] like Figure 1-4 As shown, the content-based video core word generation system disclosed in this invention includes a feature extraction module and a video description module. The feature extraction module adopts a VGG16_bn structure, and the video description module adopts a Seq2Seq-based network structure.

[0062] The video description module includes an encoder and a decoder. The encoder adopts an LSTM+self-attention network structure and consists of two layers: an LSTM-based video information encoding layer and a self-attention-based encoding information attention layer. The decoder adopts an LSTM+self-attention network structure and consists of two layers: an LSTM-based video information decoding layer and a self-attention-based semantic information attention layer.

[0063] Example 1

[0064] This invention discloses a content-based video core word generation method. First, video features are extracted using a feature extraction module. Then, the extracted video feature information is fed into a video description module to generate a video description. Next, the generated video text description is fed into a core word extraction module. Finally, the words in the text description are scored and ranked to obtain the video's core words. The method specifically includes the following steps:

[0065] Step 1: Extract video features using the feature extraction module;

[0066] Step 2: Input the extracted video feature information into the video description module to generate a video text description;

[0067] Step 3: Send the generated video text description to the core word extraction module to obtain the initial core words;

[0068] Step 4: Score and sort the initial core words to obtain the final core words of the video.

[0069] Example 2

[0070] Based on Example 1, step 1 specifically includes:

[0071] Step 1.1: Extract keyframes from the video;

[0072] Step 1.2: Extract keyframe features.

[0073] Furthermore, due to the massive amount of information in the video sequence, keyframes need to be extracted to compress the video information and improve processing speed. To ensure that the overall content of the video remains largely unchanged, step 1.1 specifically involves extracting keyframes from the video using an equally spaced frame extraction method, resulting in a keyframe sequence S = [s1, s2, s3, ..., s...]. n ];

[0074] Step 1.2 specifically involves feeding the keyframe sequence S obtained in Step 1.1 into the feature extraction module in chronological order for feature extraction. The feature extraction module can use image classification networks such as ResNet or VGG16, with modifications made to the network. This method uses a modified VGG16_BN network; specifically, the last classification layer is replaced with an average pooling layer to facilitate the extraction of keyframe feature information. Thus, the feature matrix E = [e1, e2, e3, ..., e] of the keyframe sequence is obtained. n ], where e i This represents the feature information of the i-th frame.

[0075] Furthermore, the feature information obtained from step 1 is sent to the video description module according to the video's chronological order. The video description module uses a Seq2Seq-based network framework, such as... Figure 2 As shown, video feature information needs to undergo the following steps in the video description module to obtain the video text description.

[0076] Example 3

[0077] Based on Example 1, step 2 specifically involves:

[0078] Step 2.1: Encode the video feature information;

[0079] The feature information E obtained in step 1 is fed into the encoder to encode the video feature information. The encoder uses an LSTM+self-attention network framework, such as... Figure 3 As shown. The encoding process is as follows:

[0080] Step 2.1.1, according to equation (1), the feature vector e corresponding to each frame is... i The vector is fed into the LSTM network to obtain the initial encoding vector:

[0081] n i =LSTM(W1e i (i=1,2,…) (1)

[0082] In equation (1), W1 is the linear transformation matrix, and n i For the initial encoding vector, e i For each frame, there is a feature vector.

[0083] Step 2.1.2, calculate the initial encoding vector n according to equation (2). i Relevance α i,j :

[0084] α i,j =W2tanh(W v n i +b) (i=1,2,…) (2)

[0085] In equation (2), W2 is the parameter matrix of the linear transformation, W v Let v be the transformation parameter matrix, and n be the parameter matrix. i Let b be the initial encoding vector, α be the bias term, and α be the initial encoding vector. i,j For n i Relevance;

[0086] Step 2.1.3: According to equation (3), normalization is performed using the softmax function to obtain the relevance weights:

[0087]

[0088] In equation (3), a i,j For relevance weights;

[0089] Step 2.1.4: Calculate the final encoding vector based on the relevance weights, as shown in equation (4):

[0090]

[0091] In equation (4), c i For the encoding vector, a i,j For relevance weights, n j This is the initial encoding vector;

[0092] Repeat steps 2.1.1 to 2.1.4 until all feature vectors have been encoded;

[0093] Step 2.2: Decode the video feature information to obtain the video text description;

[0094] The encoded information c = [c1, c2, c3, ... c1] obtained in (1) is used to encode the information c = [c1, c2, c3, ... c1] n The data is fed into the decoder, which uses a roughly the same network structure: an LSTM + self-attention network structure, such as... Figure 4 As shown.

[0095] Step 2.2.1, convert the encoded information c obtained in step 2.1 into... i The word w obtained in the previous moment i-1 Word embedding vector w_em i-1 The data is combined and fed into the LSTM network to obtain preliminary decoded information. i With the latent information h at this time i As shown in equations (5) and (6):

[0096] w_em i-1 =embedding(w i-1 (5)

[0097]

[0098] In equation (5), when i = 1, i.e. at the initial time, let w0 correspond to the sentence start flag. <sos>In equation (6), o i For the initial decoded information, i represents the hidden state information at this point, and c... i It is an encoded vector;

[0099] Step 2.2.2: A self-attention mechanism is used to calculate the relevance weights of the initial decoded information and the latent information at this moment. Based on the weights of both, a weighted sum is performed to obtain the probability distribution p of the word at this moment. i The specific calculation process is as follows:

[0100] Step 2.2.2.1: The preliminary decoded information obtained in step 2.2.1... i Using the latent information i, perform equations (2), (3), and (4) to calculate the weighted information vector context. i .

[0101] Step 2.2.2.2, the obtained weighted information vector context i The data is fed into a fully connected layer to obtain the probability distribution p of the word at this moment. i As shown in equation (7):

[0102] p i =W p context i +b (7)

[0103] In equation (7), W p Let be the parameter matrix in the transformation, and b be the bias term.

[0104] Step 2.2.3, based on the probability distribution p obtained in step 2.2.2. i Sampling is performed to obtain the word w at the current time. i ;

[0105] Repeat steps 2.2.1 to 2.2.3 until all encoded information has been decoded, obtaining the final video text description sentence = {w1, w2, ..., w...}. n };

[0106] Furthermore, in step 2.2.3, sampling is performed using a beam search algorithm. That is, based on the probability distribution p of word i at each time step. i The algorithm calculates the probability of the currently generated sequence and searches for sequences within a pre-defined search range until all sequences are generated, identifying the sequence with the highest probability. Finally, it retrieves the text description of the video using a dictionary lookup.

[0107] Example 4

[0108] Based on Example 1, step 3 specifically involves: using the TF-IDF algorithm to calculate the w of each word in the video text description sentence. i The TF-IDF value is used to filter out words with smaller TF-IDF values ​​in the video text description, resulting in the core word word = {word1, word2, ...}.

[0109] Example 5

[0110] Based on Example 1, step 4 specifically involves: refining the core words obtained in step 3. Since the core words extracted in step 3 only consider word frequency and fail to effectively combine the semantic information between individual words and the sentence as a whole, the selected core words cannot truly represent the semantic information the sentence intends to convey. Therefore, further refining of the core words selected in step 3 is necessary. This refining process consists of the following steps:

[0111] Step 4.1: Extract the semantic feature information of the initial core words and sentences;

[0112] To effectively extract the semantic information of both, a language model is used for extraction. The BERT pre-trained model is used, and the initially selected core words and sentences are fed into BERT to obtain the initial semantic feature vectors of the core words and sentences as {u1,u2,…} and v, respectively.

[0113] Step 4.2: Compare the semantic similarity between the initial core words and the sentences;

[0114] The semantic feature vectors of the initial core words are compared one by one with the semantic feature vectors of the sentences, and the cosine similarity calculation method is used to calculate the similarity, as shown in Equation (8):

[0115] sim = cos i ,v> (8)

[0116] Using similarity as the relevance score, the initial core words are sorted in descending order of similarity. The top five words with the highest similarity are selected as the core words to obtain the final core words of the video.​< / sos> < / sos>

Claims

1. A content-based video keyword generation method, characterized in that, Specifically, the following steps are included: Step 1: Extract video features using the feature extraction module; Step 2: Input the extracted video feature information into the video description module to generate a video text description; Step 3: Send the generated video text description to the core word extraction module to obtain the initial core words; Step 4: Score and sort the initial core keywords to obtain the final core keywords of the video; Step 2 specifically involves: Step 2.1: Encode the video feature information; Step 2.1.1, according to equation (1), the feature vector corresponding to each frame is... The vector is fed into the LSTM network to obtain the initial encoding vector: （1） In equation (1), It is a linear transformation matrix. This is the initial encoding vector. For each frame, there is a feature vector. Step 2.1.2, calculate the initial encoding vector according to equation (2). Relevance : （2） In equation (2), Let be the parameter matrix of the linear transformation. Let v be the transformation parameter matrix. This is the initial encoding vector. For bias terms, for Relevance; Step 2.1.3: According to equation (3), normalization is performed using the softmax function to obtain the relevance weights: （3） In equation (3), For relevance weights; Step 2.1.4: Calculate the final encoding vector based on the relevance weights, as shown in equation (4): （4） In equation (4), For encoding vectors, For relevance weights, This is the initial encoding vector; Repeat steps 2.1.1 to 2.1.4 until all feature vectors have been encoded; Step 2.2: Decode the video feature information; Step 2.2.1, the encoded information obtained in step 2.1 The words obtained in the previous moment Word embedding vectors The data is combined and fed into an LSTM network to obtain preliminary decoded information. With the latent information at this time As shown in equations (5) and (6): （5） （6） In equation (5), when i=1, i.e. at the initial time, let Corresponding sentence start marker <sos> ；< / sos> In equation (6), For the initial decoding of information, This is the latent information at this time. It is an encoded vector; Step 2.2.2: A self-attention mechanism is used to calculate the relevance weights of the initial decoded information and the latent information at this point. Based on the weights of both, a weighted sum is performed to obtain the probability distribution of the word at this moment. Specifically: Step 2.2.2.1: The preliminary decoded information obtained in step 2.2.1... With latent information Equations (2), (3), and (4) are then calculated to obtain the weighted information vector. ; Step 2.2.2.2, the obtained weighted information vector The data is fed into a fully connected layer to obtain the probability distribution of the word at this moment. As shown in equation (7): （7） In equation (7), The parameter matrix in the transformation, For bias terms; Step 2.2.3, based on the probability distribution obtained in step 2.2.

2. Sampling is performed to obtain the word at the current time. ; Repeat steps 2.2.1-2.2.3 until all encoded information has been decoded to obtain the final video text description. ; Step 4 specifically involves: performing a refined screening of the core words obtained in step 3; Step 4.1: Extract the semantic feature information of the initial core words and sentences; Using the BERT pre-trained model, the initially selected core words and sentences are fed into BERT to obtain the initial semantic feature vectors of the core words and sentences, respectively. and ; Step 4.2: Compare the semantic similarity between the initial core words and the sentences; The semantic feature vectors of the initial core words are compared one by one with the semantic feature vectors of the sentences, and the cosine similarity calculation method is used to calculate the similarity, as shown in Equation (8): （8） Using similarity as the relevance score, the initial core words are sorted in descending order of similarity. The top five words with the highest similarity are selected as the core words to obtain the final core words of the video.

2. The content-based video keyword generation method according to claim 1, characterized in that, Step 1 specifically involves: The video is processed by extracting keyframes using an equal-interval frame extraction method, resulting in a keyframe sequence. ; The obtained keyframe sequence The data are sequentially fed into the feature extraction module for feature extraction according to the time sequence.

3. The content-based video keyword generation method according to claim 1, characterized in that, In step 2.2.3, sampling is performed using a cluster search algorithm.

4. The content-based video keyword generation method according to claim 1, characterized in that, Step 3 specifically involves: calculating the video text description using the TF-IDF algorithm. Each word in Chinese The TF-IDF value is used to filter out words with smaller TF-IDF values ​​in the video text description, resulting in the core word word={word1,word2,…}.

5. A content-based video keyword generation system, used to implement the content-based video keyword generation method according to any one of claims 1 to 4, characterized in that, It includes a feature extraction module and a video description module. The feature extraction module adopts the VGG16_bn structure, and the video description module adopts a Seq2Seq-based network structure. The video description module includes an encoder and a decoder. The encoder adopts an LSTM+self-attention network structure and consists of two layers: a video information encoding layer based on an LSTM network and an encoding information attention layer based on self-attention. The decoder adopts an LSTM+self-attention network structure. The decoder consists of two layers: a video information decoding layer based on the LSTM network and a semantic information attention layer based on self-attention.

Images