A method for automatically generating news video descriptions

Abstract

A method for automatically generating news video descriptions can produce an accurate description containing entity information for a news video with attached text. Current news video description methods are all based on a one-segment generation model and do not filter out redundant contextual sentences, resulting in inaccurate generated descriptions. The innovation of the present invention is that the attached text is first filtered using the video, and then a two-segment generation method based on template generation and entity insertion is used to generate the news video description. The template generation part focuses on the content of the event, while the entity insertion part focuses on entity information such as the participants of the event, effectively improving the accuracy of the news video description.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and natural language processing. Background Art

[0002] With the rapid development of internet technology, video data is being produced and disseminated at a rapid pace, making video a crucial medium for information dissemination. News video description aims to generate a description containing entity information for a news video with attached text. Due to the nature of news videos, the generated description must include the time, location, people, and the cause, process, and outcome of an event. The challenge lies in leveraging video information to mine event and named entity information from the attached text.

[0003] KaVD proposed by Whitehead et al. [7] The model first extracts relevant entity information from the context and uses the pointer network during decoding [8] The entity embedding vector enables the model to output entity class information. [9] Building on KaVD,

[15] proposed an end-to-end video description model that uses all contextual words in the decoding phase. However, neither

[16] filters the complex context and uses a one-paragraph generation structure, which fails to fully exploit relevant contextual information. This results in low accuracy of generated descriptions and low entity recognition rates. Summary of the Invention

[0004] News video description is a critical multimodal task involving natural language processing and computer vision, with applications ranging from assisting the visually impaired to robotic perception. Current methods for automatically generating news video descriptions fail to filter complex accompanying text and fail to fully exploit named entity information during description generation, resulting in low-quality automatically generated descriptions.

[0005] To address these issues, this invention aims to generate news video descriptions. It first filters additional contextual articles from video information, then uses a two-stage generation method, a template generator and an entity inserter, to produce the final news video description. The template generation phase focuses on the content of the event, while the entity insertion phase focuses on entity information such as the event participants, effectively improving the accuracy of news video descriptions.

[0006] In order to achieve the above object, the technical solution provided by the present invention is:

[0007] The present invention provides a two-segment news video description automatic generation method, comprising:

[0008] Step 1: Sort each statement in the attached file using a statement sorter;

[0009] Step 2: Perform named entity recognition on the sorting statement obtained in step 1 to obtain a sorting statement template and a sorting entity list;

[0010] Step 3: Generate a description template of the target video using the video data and the sorting sentence template obtained in step 2;

[0011] Step 4: Use the sorted entity list generated in step 2 to insert entities into the description template generated in step 3 to obtain the final news video description.

[0012] Beneficial effects

[0013] This paper addresses the problem that existing automatic news video description generation methods cannot accurately generate news video descriptions. It implements a two-stage automatic generation method based on template generation and entity insertion. First, a sentence sorter is used to select relevant sentences with high similarity to the video. Second, a template generator based on the attention mechanism and an entity inserter based on sentence similarity are used to generate video descriptions in two stages. This method has significant significance for the field of multimodal semantic understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are provided to further illustrate the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present invention. In the accompanying drawings:

[0015] Figure 1 A technical roadmap for automatic description generation of news videos;

[0016] Figure 2 This is the flowchart of the statement sorter in step 1;

[0017] Figure 3 Shows an example of generating news video descriptions; DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the specific implementation methods of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0019] The specific implementation process of the present invention is as follows Figure 1 As shown, it includes the following four aspects:

[0020] Step 1: Sort each statement in the attached file using a statement sorter;

[0021] Step 2: Perform named entity recognition on the sorting statement obtained in step 1 to obtain a sorting statement template and a sorting entity list;

[0022] Step 3: Generate a description template of the target video using the video data and the sorting sentence template obtained in step 2;

[0023] Step 4: Use the sorted entity list generated in step 2 to insert entities into the description template generated in step 3 to obtain the final news video description.

[0024] The steps are detailed below:

[0025] Step 1: Use the statement sorter to sort each statement in the attached file, such as Figure 2 shown

[0026] 1.1: Using ResNet-50 [1] and I3D [2] The model extracts features from the video to obtain the dynamic and appearance features of the video, and then uses LSTM [3] And splice to get the video feature vector X containing timing information V , provided to step 1.3;

[0027] 1.2: Leveraging BERT [4] Extract features from each sentence in the attached text to obtain the intermediate text features X, which are provided to step 1.3;

[0028] 1.3: Training the Sentence Ranker

[0029] The input of the sentence sorter is video features and text features, and the output is the similarity sim(x,v) between the two.

[0030] Using the attention mechanism [5] The aggregation operation Agg(·; v) is defined as follows: , where α i =softmax(w T x i ), aggregate the intermediate text feature vector to obtain the intermediate text aggregate vector e x =Agg(X; w x ), similarly, the video vector is aggregated to obtain the visual aggregation vector e v =Agg(X V ;w v ). Among them, w x With w v are the aggregation parameters for text and video respectively. The similarity between video and sentence is defined as follows: sim(x,v)=sigmod(W r [e v ,e x ]+b rThe ROUGE index value corresponding to the context sentence and the real description of the video is used as the target value, and the model is trained by minimizing the cross entropy loss between the two.

[0031] 1.4: Use the trained sentence sorter to obtain the similarity between each sentence in the attached text and the video, sort them according to this similarity, obtain the sorted sentences, and provide them to steps 2.1 and 2.2.

[0032] Step 2: Perform named entity recognition on the sorting statement obtained in step 1 to obtain the sorting statement template and sorting entity list

[0033] 2.1: Using Spacy [6] Perform named entities on the sorted statements obtained in step 1 to obtain the entities contained in each statement and the corresponding entity types. Then, sort the entities according to the original similarity to obtain a sorted entity list, which is provided to step 4.

[0034] 2.2: Replace the entities of the original sorting statement with the corresponding entity categories to obtain the sorting statement template, and provide it to step 3.1.

[0035] Step 3: Generate a description template for the target video using the video data and the sorting sentence template obtained in step 2

[0036] 3.1: Concatenate the top 20 ranked sentence templates and vectorize them to obtain the final text feature vector X C ;

[0037] 3.2: Training Template Generator

[0038] The input of the template generator is the video and text feature vector, and the output is the description template of the target news video.

[0039] Template generator based on M-layer transformer [5] , a serialized description of the video is generated. At time t, it accepts the vocabulary S generated at the previous time <t ={s0,s1,s2,…,s t-1} and the eigenvector X V ,X C , generating a hidden state sequence where i∈{0,1,…,M}, For S <t The corresponding embedding amount, the specific process is:

[0040]

[0041] use The last dimension of the element is linearly mapped and softmaxed to get the word output at the current moment. The DecoderLayer is implemented based on the attention mechanism. Given a set of queries Q, keys K, and values V, the attention mechanism performs a weighted sum of the value vectors based on the similarity between the query and key vectors. This can be expressed as follows:

[0042]

[0043] Multi-head attention consists of H attention layers:

[0044] MA(Q,K,V)=Concat(head1,…,head h )W O

[0045]

[0046] For the L-th layer Decoder layer, first use the self-attention mechanism to get

[0047]

[0048] Then perform MA operation on text and video features respectively to obtain and in Concatenate the two and pass them through the linear layer to get Used as the next DecoderLayer l Given a video V, context C, and true description S, the cross entropy loss is used as the objective function for training. The loss function is:

[0049]

[0050] 3.3: Use the trained template generator to generate a description template with entity category placeholders for the target video and provide it to step 4.

[0051] Step 4: Use the sorted entity list generated in step 2 to insert the entity into the description template generated in step 3 to obtain the final news video description

[0052] The description template generated in step 3 above contains different entity categories. According to the entity category, corresponding entities are selected from the sorted entity list generated in step 2 and filled in in sequence to obtain the final video description.

[0053] A specific description generation example is Figure 3As shown in the figure, after steps 1 and 2, the given video data is sorted into entities such as the location (GPE) category, the organization (ORG) category, and the person (PERSON) category. After step 3, a description template is generated, which contains placeholders for different types of entities. These placeholders are then inserted according to the sorted entities to obtain the final news video description.

[0054] References

[0055] [1]He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In: Pro-ceedings of the IEEE conference on computer vision and pattern recognition, 770-778 (2016)

[0056] [2]Carreira J, Zisserman A. Quo Vadis, action recognition? a new model and the kinet-ics dataset.In:Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition,6299-6308(2017)

[0057] [3]Hochreiter S,Schmidhuber J.Long short-term memory.In:Neuralcomputation,9(8):1735-1780(1997)

[0058] [4]Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deepbidirectional trans-formers for language understanding. arXiv preprint arXiv:1810.04805(2018)

[0059] [5]Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Advances in neural information processing systems, 5998 - 6008(2017)

[0060] [6]Honnibal M, Montani I. Natural language understanding with Bloom embeddings, convolutional neural networks, and incremental parsing. Unpublished software application. https: / / spacy.Io(2017)

[0061] [7]Whitehead S, Ji H, Bansal M, et al. Incorporating background knowledge into video description generation. In: 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. Association for Computational Linguistics, 3992 - 4001(2018)

[0062] [8]See A, Liu P J, Manning C D. Get to the point: Summarization with pointer - generator networks. arXiv preprint arXiv:1704.04368(2017)

[0063] [9]Rimle P, Dogan - P, Gross M. Enriching video captions with contextual text. In: 2020 25th International Conference on Pattern Recognition(ICPR). IEEE, 5474 - 5481(2021)

[0064] Innovation points

[0065] This paper proposes a method for automatically generating news video descriptions. Compared with existing methods, the generated video descriptions are more accurate. First, a sentence sorter is designed using a pretrained language model to filter video-related sentences from attached articles. Next, an attention-based template generator and a sentence-similarity-based entity inserter are designed to generate accurate video descriptions in two stages.

[0066] The method proposed in this paper performs well in news video description datasets and improves the accuracy of automatically generated video descriptions.

Claims

1. A method for automatically generating news video descriptions, comprising: Step 1: sort each sentence of the attached text using a sentence sorter; Step 2: Perform named entity recognition on the sorting statement obtained in step 1 to obtain a sorting statement template and a sorting entity list; Step 3: Generate a description template of the target video using the video data and the sorting sentence template obtained in step 2; Step 4: Use the sorted entity list generated in step 2 to perform entity insertion on the description template generated in step 3 to obtain the final news video description; in, Step 1: Sort each sentence of the attached text using the sentence sorter 1.1: Use ResNet-50 and I3D models to extract video features to obtain dynamic and appearance features of the video, and then use LSTM splicing to obtain a video feature vector containing timing information , provided to step 1.3; 1.2: Use BERT to extract features from each sentence of the attached text to obtain intermediate text features , provided to step 1.3; 1.3: Training the Sentence Ranker The input of the sentence sorter is video features and text features, and the output is the similarity between the two ; Defining aggregation operations using attention mechanisms as follows: ,in , aggregate the intermediate text feature vectors to obtain the intermediate text aggregate vector , Similarly, the video vector is aggregated to obtain the visual aggregation vector ;in, and are the aggregation parameters for text and video respectively; the similarity between video and sentence is defined as follows: The ROUGE index value corresponding to the context sentence and the real description of the video is used as the target value, and the model is trained by minimizing the cross entropy loss between the two. 1.4: Use the trained sentence sorter to obtain the similarity between each sentence in the attached text and the video, sort them according to this similarity, and obtain the sorted sentences, which are provided to steps 2.1 and 2.2; Step 3: Generate a description template for the target video using the video data and the sorting sentence template obtained in step 2 3.1: Concatenate the top 20 ranked sentence templates and vectorize them to obtain the final text feature vector ; 3.2: Training Template Generator The input of the template generator is the video and text feature vector, and the output is the description template of the target news video; Template generator based on M The transformer of the layer generates a description of the video in sequence; at time t, it accepts the vocabulary generated at the previous time and the eigenvector , , generating a hidden state sequence ,in , for The corresponding embedding vector, the specific process is: use The last dimension of the element is linearly mapped and softmaxed to get the word output at the current moment. DecoderLayer is implemented based on the attention mechanism. For a set of query Q, key vector K and value vector V, the attention mechanism performs a weighted sum of the value vector V according to the similarity between the query Q and key K vectors. It is expressed as follows: Multi-headed attention MA It consists of H attention layers: For the L-th layer Decoder layer, first use the self-attention mechanism to get ; Then the text and video features are analyzed separately. MA Operation, get and ,in , , concatenate the two and pass them through the linear layer to get , is used as the next Input; given a video V , context C , and a true description S , using cross entropy loss as the objective function of training, the loss function is: 3.3: Use the trained template generator to generate a description template with entity category placeholders for the target video and provide it to step 4.

2. The method according to claim 1, wherein Step 2: Perform named entity recognition on the sorting statement obtained in step 1 to obtain the sorting statement template and sorting entity list 2.1: Use Spacy to name entities for the sorted statements obtained in step 1, obtain the entities contained in each statement and the corresponding entity types, sort them according to the original similarity to obtain a sorted entity list, and provide it to step 4; 2.2: Replace the entities of the original sorting statement with the corresponding entity categories to obtain the sorting statement template, and provide it to step 3.

1. <img src='' class="img-anchor" img-id="QLYQS_35" / >. The method according to claim 1, characterized in that Step 4: Specifically: The description template generated in step 3 contains different entity categories. According to the entity category, the corresponding entities are selected from the sorted entity list generated in step 2 and filled in in sequence to obtain the final video description.

Images