Video description automatic generation method based on background knowledge

Abstract

The invention provides a video description automatic generation method based on background knowledge in a short video era. According to the method, more accurate video description is generated mainly by combining bullet screens and comment information related to short videos with video information. A traditional video description method is not ideal in video description quality due to lack of related background knowledge (the background knowledge is usually priori knowledge related to videos, such as character names, events, places and the like), and in the method, bullet screens and comment information are used as related background knowledge sources, and video content is combined for description. The method is characterized in that information such as bullet screens and comments is introduced to assist in short video description generation, noise in background knowledge is reduced based on a CLIP text preprocessing scheme, and the method effectively improves the video description generation quality.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and natural language processing, and in particular relates to a method for automatically generating video descriptions based on background knowledge. Background Art

[0002] With the rapid development of the Internet, video data has become an important information medium, especially on platforms such as Douyin, Video Account, Bilibili, Xiaohongshu and Kuaishou, where video content occupies a dominant position. At the same time, emerging business models such as live streaming have also become popular. In this context, how to effectively mine video data and deeply understand video content is a question worth exploring. The information in the video is not limited to the video itself, but is usually accompanied by a large amount of text information, such as barrage and comments. These text contents often contain information that complements the video and provide rich background knowledge. If these text information can be mined in depth, the quality of video description will be enhanced.

[0003] However, research related to this technology is relatively scarce, and KaVD [7] This background knowledge-based video description task is proposed for the first time. It extracts relevant named entities and event information from the background knowledge text, uses a multi-layer LSTM structure to integrate this information into the video description, and generates a natural language description. [8] In the description generation process, a pointer network is introduced to generate descriptions. [9] The description quality is improved by introducing sentence sorters and template generation. However, these methods do not fully exploit the background knowledge text information, and the denoising process is not reasonable, resulting in unsatisfactory model quality. In addition, there is a lack of correlation between video features and text features, resulting in low accuracy of generated descriptions. Summary of the invention

[0004] Automatic generation of video descriptions based on background knowledge is an important task in the multimodal field involving natural language processing and computer vision. In the era of short videos, video descriptions based on background knowledge can help platforms to promptly detect illegal live broadcast content and illegal video content to avoid causing huge losses.

[0005] In response to the above problems, the present invention aims to generate video descriptions based on background knowledge. First, the bullet screen and comment information are filtered through video information, and the filtered background knowledge is mined for named entities. Then, the video information, filtered text information, and named entity information are used to generate video descriptions, which can effectively improve the accuracy of video descriptions.

[0006] Technical solution of the present invention:

[0007] The present invention provides a method for automatically generating video description based on background knowledge, and the specific steps include:

[0008] Step 1: Input video data and background knowledge text related to the video, use a text preprocessing solution to process the noise information in the background knowledge text, sort the video frame and the background knowledge text according to the similarity, and obtain the processed sorted sentence;

[0009] Specifically, the text preprocessing scheme adopts the CLIP model;

[0010] Step 2: Design the overall architecture of the model, including video encoder, text encoder, cross-module encoder and multimodal decoder;

[0011] Step 3: Fine-tune the text encoder in step 2 using the MNE fine-tuning scheme;

[0012] Step 4: Fine-tune the video encoder in step 2 and the text encoder in step 3 using contrastive learning to obtain a fine-tuned text encoder and video encoder;

[0013] Step 5: Use the text encoder and video encoder in step 4 and the overall model in step 2 to perform model training to obtain the final model;

[0014] Step 6: Use the trained model to generate video description.

[0015] The present invention aims to solve the problem that the existing video description automatic generation method based on background knowledge cannot accurately generate news video descriptions, and uses contrastive learning and multimodal fusion methods to achieve description generation. First, the CLIP model is fine-tuned to perform background knowledge text preprocessing. Secondly, the MNE fine-tuning scheme is designed to fine-tune RoBerta. Then, contrastive learning is used to enhance the association between video features and text features. Finally, the final video description is output for multimodal input, which is of great significance to the field of multimodal semantic understanding.

[0016] Beneficial Effects

[0017] The present invention focuses on combining video with background knowledge to generate better video descriptions. In view of the problems that existing methods lack background knowledge text noise filtering methods, lack of correlation between video features and text features, and insufficient attention to key named entities, a method for automatically generating video descriptions combined with background knowledge is implemented. With the goal of generating video descriptions, the present invention uses a fine-tuned CLIP model to enhance the relevance between background knowledge text and video; designs the MNE fine-tuning of the Roberta model and a dual-granularity text encoding scheme to enhance the model's ability to perceive named entities; and uses contrastive learning and cross-coding to improve the relevance between video features and text features. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to promote a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present invention. In the accompanying drawings:

[0019] Figure 1 A technical roadmap for the present invention based on background knowledge video description;

[0020] Figure 2 It is a schematic diagram of the CLIP-based text preprocessing in step 1 of the present invention;

[0021] Figure 3 This is the MNE fine-tuning principle diagram of Roberta in step 2 of the present invention;

[0022] Figure 4 This is the overall architecture diagram of the model in step 4 of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the specific implementation methods of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0024] like Figure 1 As shown, a method for automatically generating video description based on background knowledge includes the following steps:

[0025] Step 1: Input video data and background knowledge text related to the video, use a text preprocessing solution to process the noise information in the background knowledge text, sort the video frame and the background knowledge text according to the similarity, and obtain the processed sorted sentence;

[0026] Step 2: Design the overall architecture of the model, including video encoder, text encoder, cross-module encoder and multimodal decoder;

[0027] Step 3: Fine-tune the text encoder in step 2 using the MNE fine-tuning scheme;

[0028] Step 4: Fine-tune the video encoder in step 2 and the text encoder in step 3 using contrastive learning to obtain a fine-tuned text encoder and video encoder;

[0029] Step 5: Use the text encoder and video encoder in step 4 and the overall model in step 2 to perform model training to obtain the final model;

[0030] Step 6: Use the trained model to generate video description.

[0031] Among them, in step 1, the text preprocessing scheme adopts the CLIP model, such as Figure 2 As shown; including an image encoder and a text encoder;

[0032] The processing process is as follows:

[0033] 1.1: Extract several picture frames from the video and provide them to 1.2;

[0034] 1.2: Using CLIP

[10] Model training using contrastive learning

[0035] The input of the CLIP model is the real description of the video and the video frame, and the output is the InfoNCE loss of the two.

[11] , the training process minimizes this InfoNCE loss.

[0036] The image encoder inputs the picture frame extracted in 1.1, the text encoder of CLIP inputs the real description T in the training set to obtain the vector Q, and the image encoder of CLIP inputs the video frame V extracted in 1.1 to obtain the vector K. The two are trained using contrastive learning.

[0037]

[0038] Among them, E_V and E_T represent the image encoder and text encoder of the CLIP model respectively; sim represents the similarity calculation, K + represents the positive sample, K - represents negative samples. By minimizing Implement fine-tuning the CLIP model for the training objective.

[0039] 1.3: Use the fine-tuned CLIP model in 1.2 to filter and sort, calculate the CLIP similarity of the video frame and the background knowledge text, and sort them according to the similarity to obtain the sorted background knowledge text sentences (sorted sentences).

[0040] Step 2: Design the overall architecture of the model, including video encoder, text encoder, cross-modal encoder and multimodal decoder; the model structure is as follows Figure 4 shown.

[0041] The video encoder includes an S3D model and a CLIP model; and the text encoder is a Roberta model.

[0042] Step 3: Use the MNE fine-tuning solution to adjust RoBerta

[12] Model fine-tuning; Figure 3 .

[0043] The RoBerta model is a ready-made pre-trained language model based on the Transformer architecture. The processing process is as follows:

[0044] 3.1: Using Spacy

[13] Perform named entity extraction on the sorted sentences obtained in step 1 to obtain named entities NE; Spacy is a deep learning model used to implement named entity recognition of text content.

[0045] 3.2: Fine-tuning the RoBerta model for named entity masking (MNE)

[0046] The named entity information in the background knowledge text is masked with [MASK]. The mask operation here is to replace the named entity in the text with the [MASK] tag, and the training model will restore the [MASK] part. The training adopts a probability increasing masking strategy to gradually mask the named entities. The specific process is divided into three parts:

[0047] 30% Probability Masking: Perform a preliminary masking operation on the named entities, masking the named entities with a probability of 30%. This stage enables the model to begin learning the context of the named entities while retaining some of the information of the named entities.

[0048] 60% probability mask: Based on the initial learning of the model, the mask probability is increased to 60%. This allows the model to gradually strengthen its attention to named entities during the learning process while still maintaining sufficient relevance.

[0049] 80% probability masking: Finally, an 80% masking operation is performed on the named entities. At this stage, the model will face a higher proportion of masked named entity challenges, which will enable it to deepen its understanding of the relationship between named entities and context.

[0050] Furthermore, the processing of step 4 is as follows:

[0051] 4.1: Using S3D for Video

[14] Encode the motion features of the video and use the CLIP model to encode the representation features of the video. The two are concatenated to obtain the video feature e v , use the RoBerta model fine-tuned by MNE in step 3 to extract text features of background knowledge t .

[0052] 4.2: For e v and e t InfoNCE loss is used to calculate the contrast loss.

[0053]

[0054] The text encoder and video encoder are fine-tuned using contrastive loss.

[0055] Step 5: Use the text encoder and video encoder in step 4 and the overall model in step 2 to perform model training to obtain the final model; the details are as follows:

[0056] 5.1 Feature Encoding

[0057] Input video, text, and the named entities extracted in step 3.1, and encode the video features X respectively V′ =Concat(S3D(V),CLIP(V)), background knowledge text feature X t′ =BERT(C) and named entity features X n =Embedding(NE).X V′ and X t′ After the cross encoder, X is obtained V ,X t =CrossModeEncoder(X V′ ,X t′ )

[0058] 5.2 Feature Decoding

[0059] Multimodal decoder based on M-layer transformer

[15] Recursively generate video description. At time t, the input received by the decoder is R t-1 ={w0,w1,w2,...,w t-1}, X V ,X t ,X n ; The output is R t ={w0,w1,w2,...,w t}. The specific process of each decoding layer is

[0060]

[0061] The DecoderLayer is implemented based on the Transformer's attention mechanism.

[0062] Decoder={DecoderLayer1,...DecoderLayer m}

[0063]

[0064] Among them, MHA means multi-head attention mechanism; Concat means concatenating vectors; Add&Norm means batchnormalization after addition; FC means fully connected layer;

[0065] 5.3 It is the output feature of the last layer of decoder. After linear mapping and softmax operation, the vocabulary mapping is performed to obtain w t .

[0066]

[0067] 5.4 Calculation of loss optimization model

[0068] Given a video V, context C, and a true description S = (w0, w1, ..., w t-1 ), using cross entropy loss as the loss function for training:

[0069]

[0070] The overall model composed of encoder-decoder is optimized by gradient descent.

[0071] Step 6: Use the model trained in step 4 to generate video description. Input the video and background knowledge text, and the model generates video description.

[0072] The present invention proposes a method for automatically generating video description based on background knowledge. Compared with the existing generation methods, the video description generated by the present invention is more accurate. The present invention first performs CLIP text preprocessing on the background knowledge text, filters out the sentences related to the video from the background knowledge, then designs the MNE fine-tuning task to fine-tune Roberta, then uses contrastive learning to perform contrastive learning on video encoding and text encoding, and finally uses video, background knowledge text and named entities for model training.

[0073] Using the news video data set, the method of the present invention is compared with the existing typical NVDT, KaVD, S2VT-pointer and other methods. The results are as follows:

[0074] BLEU_4 METEOR ROUGE_L CIDEr The present invention 0.1069 0.1348 0.2268 0.6374 NVDT 0.0347 0.1117 0.1956 0.2461 ikB - 0.102 0.189 - S2VT-pointer - 0.108 0.186 0.257

[0075] The comparison indicators include: BLEU_4, METEOR, ROUGE_L and CIDEr. Among them, BLEU_4 is an indicator for measuring the accuracy of description generation; METEOR is an indicator for measuring semantic matching and language fluency; ROUGE_L is an indicator that comprehensively considers accuracy and recall; CIDEr is a score that calculates weighted n-gram similarity and is an indicator that focuses on description semantic evaluation. The indicators of the present invention are significantly better than the comparison method.

[0076] It can be seen that the method proposed in the present invention improves the quality of video description generation.

[0077] Appendix: References

[0078] [1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C] / / International conference on machine learning. PMLR, 2021: 8748-8763.

[0079] [2] Oord A, Li Y, Vinyals O. Representation learning with contrastive predictive coding[J]. arXiv preprint arXiv:1807.03748, 2018.

[0080] [3] Liu Y. Roberta: A robustly optimized bert pretraining approach[J]. arXiv preprint arXiv:1907.11692, 2019.

[0081] [4] Honnibal M, Montani I. Natural language understanding with Bloom embeddings, convolutional neural networks, and incremental parsing. Unpublished software application. https: / / spacy.Io(2017)

[0082] [5] Xie S, Sun C, Huang J, et al. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification[C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 305-321.

[0083] [6]Vaswani A,Shazeer N,Parmar N,et al.Attention is all you need.In:Advances in neural information processing systems,5998-6008(2017)

[0084] [7]Whitehead S,Ji H,Bansal M,et al.Incorporating background knowledgeinto video description generation.In:2018 Conference on Empirical Methods inNatural Lan-guage Processing,EMNLP 2018.Association for ComputationalLinguistics,3992-4001(2018)

[0085] [8]Rimle P,Dogan- P,Gross M.Enriching video captions withcontextual text.In:2020 25th International Conference on Pattern Recognition(ICPR).IEEE,5474-5481(2021)

[0086] [9]Devlin J,Chang M W,Lee K,et al.Bert:Pre-training of deepbidirectional transformers for language understanding[J].arXiv preprintarXiv:1810.04805,2018.

[0087]

[10] Radford A,Kim J W,Hallacy C,et al.Learning transferable visualmodels from natural language supervision[C] / / International conference onmachine learning.PMLR,2021:8748-8763.

[0088]

[11] Oord A,Li Y,Vinyals O.Representation learning with contrastivepredictive coding[J].arXiv preprint arXiv:1807.03748,2018.

[0089]

[12] Liu Y.Roberta:A robustly optimized bert pretraining approach[J].arXiv preprint arXiv:1907.11692,2019.

[0090]

[13] Honnibal M,Montani I.Natural language understanding with Bloomembeddings,convolutional neural networks,and incremental parsing.Unpublishedsoftware ap-plication.https: / / spacy.Io(2017)

[0091]

[14] Xie S,Sun C,Huang J,et al.Rethinking spatiotemporal featurelearning:Speed-accuracy trade-offs in video classification[C] / / Proceedings ofthe European conference on computer vision(ECCV).2018:305-321.

[0092]

[15] Vaswani A,Shazeer N,Parmar N,et al.Attention is all you need.In:Advances in neural information processing systems,5998-6008(2017)。

Claims

1. A method for automatically generating video description based on background knowledge, characterized in that: The following steps are involved: Step 1: Input video data and background knowledge text related to the video, use a text preprocessing solution to process the noise information in the background knowledge text, sort the video frame and the background knowledge text according to the similarity, and obtain the processed sorted sentence; Step 2: Design the overall architecture of the model, including video encoder, text encoder, cross-module encoder and multimodal decoder; Step 3: Fine-tune the text encoder in step 2 using the MNE fine-tuning scheme; Step 4: Fine-tune the video encoder in step 2 and the text encoder in step 3 using contrastive learning to obtain a fine-tuned text encoder and video encoder; Step 5: Use the text encoder and video encoder in step 4 and the overall model in step 2 to perform model training to obtain the final model; Step 6: Use the trained model to generate video description.

2. The method for automatically generating video description based on background knowledge as claimed in claim 1, characterized in that: The processing of step 1 is as follows: 1.1: Extract several picture frames from the video and provide them to 1.2; 1.2: Use CLIP model for contrastive learning training; The input of the CLIP model is the real description of the video and the video frame, and the output is the InfoNCE loss of the two. The training process minimizes the InfoNCE loss. The image encoder inputs the picture frame extracted in 1.1, the text encoder of CLIP inputs the real description T in the training set to obtain the vector Q, and the image encoder of CLIP inputs the video frame V extracted in 1.1 to obtain the vector K. The two are trained using contrastive learning. Among them, E_V and E_T represent the image encoder and text encoder of the CLIP model respectively; sim represents the similarity calculation, K + represents the positive sample, K - Represents negative samples; by minimizing Fine-tune the CLIP model for the training objective; 1.3: Use the fine-tuned CLIP model in 1.2 to filter and sort, calculate the CLIP similarity of the video frame and the background knowledge text, and sort them according to the similarity to obtain the sorted background knowledge text sentences (sorted sentences).

3. The method for automatically generating video description based on background knowledge as claimed in claim 1, characterized in that: The processing of step 3 is as follows: 3.1: Use Spacy to extract named entity entities from the sorted sentences obtained in step 1 to obtain named entities NE; Spacy is a deep learning model used to implement named entity recognition of text content; 3.2: Fine-tuning the RoBerta model for named entity masking (MNE) The named entity information in the background knowledge text is masked with [MASK], that is, the named entities in the text are replaced with [MASK] tags, and the training model restores the [MASK] part; the training adopts a masking strategy with increasing probability to gradually mask the named entities; The specific process is divided into three parts: 30% probability masking: Perform a preliminary masking operation on the named entities, masking the named entities with a probability of 30%; this stage enables the model to begin learning the context of the named entities while retaining some of the information of the named entities; 60% probability mask: Based on the initial learning of the model, the mask probability is increased to 60%; this allows the model to gradually strengthen its focus on named entities during the learning process while still maintaining sufficient relevance; 80% Probability Masking: Finally, an 80% masking operation is performed on the named entities; at this stage, the model will face a higher proportion of masked named entity challenges, which will enable it to deepen its understanding of the relationship between named entities and context.

4. The method for automatically generating video description based on background knowledge as claimed in claim 1, characterized in that: The processing of step 4 is as follows: 4.1: Use S3D to encode the motion features of the video, and use the CLIP model to encode the representation features of the video. The two are spliced ​​to obtain the video feature e v , use the RoBerta model fine-tuned by MNE in step 3 to extract text features of background knowledge t ; 4.2: For e v and e t Use InfoNCE loss to calculate contrast loss The text encoder and video encoder are fine-tuned using contrastive loss.

5. The method for automatically generating video description based on background knowledge as claimed in claim 1, characterized in that: The processing of step 5 is as follows: 5.1 Feature Encoding Input video, text, and the named entities extracted in step 3.1, and encode the video features X respectively V′ =Concat(S3D(V),CLIP(V)), background knowledge text feature X t′ =BERT(C) and named entity features X n =Embedding(NE);X V′ and X t′ After the cross encoder, X is obtained V ,X t =CrossModeEncoder(X V′ ,X t′ ) 5.2 Feature Decoding The multimodal decoder recursively generates video descriptions based on the M-layer transformer; at time t, the input received by the decoder is R t-1 ={w0,w1,w2,...,w t-1 }, X V ,X t ,X n ; The output is R t ={w0,w1,w2,...,w t }; The specific process of each decoding layer is Among them, DecoderLayer is implemented based on Transformer's attention mechanism; Decoder={DecoderLayer1,...DecoderLayer m } Among them, MHA means multi-head attention mechanism; Concat means concatenating vectors; Add&Norm means batchnormalization after addition; FC means fully connected layer; 5.3 It is the output feature of the last layer of decoder. After linear mapping and softmax operation, the vocabulary mapping is performed to obtain w t ; 5.4 Calculation of loss optimization model Given a video V, context C, and a true description S = (w0, w1, ..., w t-1 ), using cross entropy loss as the loss function for training: The overall model composed of encoder-decoder is optimized by gradient descent.

6. The method for automatically generating video description based on background knowledge as claimed in claim 1, characterized in that: Step 6 is as follows: Use the model trained in step 5 to generate video description, input the video and background knowledge text, and the model generates video description.

Images