Short video news generation system based on large-scale pre-training model

Abstract

The present invention proposes a short video news generation system based on a large-scale pre-trained model, comprising: a front-end web page module and a back-end short video news generation platform; the front-end web page module is specifically divided into an interactive component, a video parameter visualization component, a data transmission and upload component, and a data receiving component; the back-end short video news generation platform is specifically divided into a summary generation module, a picture-text matching module, and a video synthesis module. The present invention is based on summary generation, information extraction, picture-text matching, and video synthesis technologies, and provides a short video news generation system based on a large-scale pre-trained model, which to a certain extent reduces the work pressure of journalists in finding materials and editing videos. By simply inputting the news text, a smooth short video news can be output, which is convenient for workers to edit later. In actual applications, it can be expanded and flexibly generated according to specific different application scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of computer multimodality, and specifically refers to a short video news generation system based on a large-scale pre-training model. Background Art

[0002] With the development and progress of the times, short videos, as an emerging form of entertainment, have sparked the curiosity and desire of the general public with their short duration, cascading, intuitive presentation, and direct content output. Data shows that in the first quarter of 2020 alone, 1.8 million content creators on Bilibili submitted 4.9 million videos; as of June 2020, nearly 300 million users had published their works on the Kuaishou platform. By the end of 2021, the number of short video users had reached a staggering 934 million, with a usage rate of 90.5% and an average daily usage time of over 2.5 hours per person. To increase public awareness of news, journalists have begun to incorporate short videos into news dissemination. According to statistics at the end of 2019, there were over one million journalists nationwide.

[0003] With the growing prosperity of the news and short video intersection industry, the market is placing higher demands on related personnel. Editors play a crucial role in the short video ecosystem, assembling videos from a vast amount of collected footage. However, with numerous small and medium-sized news events occurring daily, editors face significant workload pressure. Furthermore, some news content often incorporates excessive personal emotions, attitudes, and values ​​from the creators, and even some news stories exhibit distortion in their descriptions. Furthermore, the development and operation of short video news require significant resources and financial resources, making such high costs difficult for typical news media outlets to sustain. Therefore, a low-cost short video news generation system that generates objective news descriptions is needed. However, most existing short video news generation technologies suffer from shortcomings such as an inability to extract key news information, a lack of rich video footage, and clunky video transitions and playback effects.

[0004] Therefore, the present invention proposes a short video news generation system based on a large-scale pre-training model. This technical method can be applied to real life and has a strong universal significance for journalists, short video producers and some people with special needs. Especially for journalists, this technology can reduce the workload of journalists, such as segmenting news texts into paragraph units and generating summaries, matching the summaries with suitable image materials through information extraction technology combined with search engines, and synthesizing short video news using video synthesis technology; and for short video workers, this technology can generate sample videos, saving the time of searching for materials, so that short video workers only need to optimize the videos through their own editing technology, and ultimately achieve the effect of saving time and effort. In addition, this technology also reduces the subjective influence of journalists' emotions to a certain extent, and can generate more objective short video news.

[0005] In the technology of the present invention, the summary generation module uses the text pre-training model GPT2[1] proposed by OpenAI, which is constructed using an autoregressive method on a large-scale text corpus based on the Transformer[2] decoder architecture; the image-text matching module uses the TextRank algorithm[3] to implement keyword extraction, the DRGREE model[4] to implement event extraction, and the image-text pre-training model CLIP[5] to implement image-text matching; and the video synthesis module uses the Whisper model[6] to implement speech recognition.

[0006] [1].adford A, Wu J, Child R, et al. Language models are unsupervised multitask learners [J]. OpenAI blog, 2019, 1(8):9.

[0007] [2].swani A,Shazeer N,Parmar N,et al.Attention is all you need[J].Advances in neural information processing systems,2017,30.

[0008] [3].ihalcea R,Tarau P.Textrank:Bringing order into text[C] / / Proceedings of the 2004conference on empirical methods in natural languageprocessing.2004:404-411.

[0009] [4].Hsu IH,Huang KH,Boschee E,et al.DEGREE:A data-efficientgeneration-based event extraction model[C] / / Proceedings of the 2022Conference of the North American Chapter of the Association for ComputationalLinguistics:Human Language Technologies.2022:1890-1908.

[0010] [5].Radford A,Kim JW,Hallacy C,et al.Learning transferable visualmodels from natural language supervision[C] / / International conference onmachine learning.PMLR,2021:8748-8763.

[0011] [6].Radford A,Kim JW,Xu T,et al.Robust speech recognition via large-scale weak supervision[J].arXiv preprint arXiv:2212.04356,2022. Summary of the Invention

[0012] In order to reduce the workload of journalists in finding materials and editing videos, the present invention provides a short video news generation system based on a large-scale pre-trained model, based on summary generation, information extraction, image-text matching, and video synthesis technology. The system supports workers in selecting rich video template resources and transition effects, has strong interactivity, and users can input press releases to generate a smooth short video news in a relatively short time. The synthesis efficiency is high, which facilitates the subsequent editing work of journalists. In actual applications, the system can be expanded and flexibly generated according to specific application scenarios to generate corresponding short video news.

[0013] The short video news generation system based on a large-scale pre-trained model includes: a front-end web page module and a back-end short video news generation platform.

[0014] The front-end web page module,

[0015] It is specifically divided into interactive components, video parameter visualization components, data sending and uploading components, and data receiving components.

[0016] Interactive components, which enable interaction with users, include buttons, text input boxes, prompt boxes, and page switching;

[0017] The video parameter visualization component visualizes pre-selectable video parameters for users to choose from, including video transition effects, subtitle style, speech speed, and timbre;

[0018] The data upload component uploads the user's uploaded news releases, selected templates, and video parameters to the backend short video news generation platform;

[0019] Data receiving and conversion component, receiving press releases and short video news processed by the back-end short video news generation platform;

[0020] Furthermore, the front end is the web page interface. The present invention uses the widely used and stable Vue framework for development and programming to realize the functions of each component. The Vue framework has the advantages of good interactivity, easy use, simple operation, and wide application.

[0021] The back-end short video news generation platform module,

[0022] Specifically divided into summary generation module, image-text matching module, and video synthesis module;

[0023] Summary generation module

[0024] The input news release is pre-processed through data cleaning and segmentation, and then a summary generation model is used to generate a news summary. When using the website, users can upload the URL, text, or file of the news release. The website backend will automatically extract the news text and transmit it to the backend model, ultimately outputting the news summary.

[0025] Furthermore, the construction of the news summary generation model is specifically implemented as follows:

[0026] We collected news summary datasets from Sina Weibo, Sogou News, and other sources, and cleaned the data by removing URLs, image tags, and topic tags to construct a news Chinese summary dataset M.

[0027] Use the Chinese word segmenter to analyze the original content of each news in the dataset M. i and News Abstract i Perform word segmentation, M={(Content i , Abstract i )|1≤i≤n}; where n is the number of news in the data set, and is a word vector corresponding to the text, and is the number of word vectors;

[0028] The news summary generation model used is the text pre-training model GPT2, and the autoregressive learning model of the GPT2 model used is The GPT2 model loss function is modified to only calculate the summary of the predicted news for training;

[0029] The GPT2 model used is constructed through the Transformer decoder, using beam search as the decoding strategy, retaining num at each time step beams Output the output with the highest current score, output the sequence with the largest product of conditional probabilities at each time step, and use n-grams penalty to reduce sequence or word duplication problems;

[0030] The news text input during the test is segmented before generating the summary; first, the news text is divided into p paragraphs. The specific steps are to traverse the entire news text. If the cumulative number of words at the end of a sentence exceeds num, the segmentation operation is performed. If the number of words at the end of the text is less than num, it is spliced ​​to the previous paragraph.

[0031] Image-text matching module

[0032] That is, based on the input news summary, the image with the greatest relevance is matched to each paragraph of text through the image-text matching model; when using the web page, the user can choose whether to modify the news summary output in the previous step and re-upload it. The front-end web page will transmit it to the back-end model, and finally the news release with images and text will be output.

[0033] Furthermore, the image-text matching function is specifically implemented as follows

[0034] Through information extraction, keywords and event information are extracted from the text segment to form key information. Then, relevant image datasets are searched through search engines. Finally, the pre-trained Chinese CLIP model is used to calculate the similarity between the key information in the text and the image. After performing softmax, the image with the highest probability is selected as the matching image.

[0035] The information extraction adopts the TextRank algorithm to return the k keywords with the largest weights, and performs trigger word recognition and argument recognition on the summary segment based on the DEGREE model to obtain the triples of the corresponding events; among them, the trigger word here represents the core word of the event, which is mostly a verb or noun, and the argument is the participant of the event, mainly composed of entity, value, and time. The layer parameters of the trigger word and argument recognition network are initialized using BERT.

[0036] The CLIP image encoder uses ResNet, and the text encoder uses Tansformer. The image and text encoding results obtained by inputting into the encoder are used to extract their respective features. The corresponding low-dimensional vector representation is obtained through projection mapping and L2 normalization. The cross entropy of these two vectors is calculated to obtain the similarity matrix for classification. After obtaining the predicted similarity matrix, the cross entropy objective function needs to be calculated in combination with the true value, and finally the model is learned.

[0037] Video synthesis module

[0038] That is, short video news is synthesized from the input news text and image information and video parameters; users can choose whether to modify the news summary with text and images output in the previous step and re-upload it. They can upload their own video templates or select the platform's preset video templates, and select the preset video transition effects, subtitle style, speech speed, and timbre. The webpage transmits the video parameters to the back-end model and finally outputs the short video news.

[0039] Furthermore, the video synthesis function is specifically implemented as follows

[0040] Based on the input news summary image and text information, the news text and corresponding images are extracted. The text-to-speech Python library pyttsx3 is used to convert the text to speech based on the preset timbre and speaking speed to generate an audio file. The Whisper model is used to recognize the speech and generate a subtitle file.

[0041] According to the time flow in the subtitle file, the total duration of the video can be obtained as t all =t s +t end , where t end is the ending time of the last sentence in the subtitle file, t s is the duration of the preset opening template video, and the time when each sentence should appear is t i '=t s +t i , 1≤i≤n w , where t i is the starting time of a sentence in the subtitle file, n w is the number of sentences in the subtitle file;

[0042] Repeat after the opening pattern is played s The middle template of the video at the time is added to the end template of the video to synthesize a complete video template;

[0043] The moment the picture appears is the same as the beginning of the corresponding paragraph. Then, using the Python library moviepy for video editing, the picture is dynamically rotated, scaled, and translated in the video, thus achieving different preset transition effects.

[0044] Use ffmpeg, an open source framework for audio and video processing, to add subtitles and audio to the video to generate complete short video news.

[0045] The advantages and beneficial results of the present invention are as follows: 1) The present invention reduces the work pressure of journalists in finding materials and editing videos through summary generation, information extraction, image-text matching and video synthesis technology. Only by inputting the news text, a smooth short video news can be output, which is convenient for workers to edit later. 2) The summary generation module designed by the present invention reconstructs the GPT2 model and trains the news summary model. The generated news summary has strong semantics and conforms to the normal reading order. 3) The image-text matching module designed by the present invention extracts key information of the text through information extraction technology, thereby improving the effect of image-text matching and retrieval, and obtains pictures related to the current text segment through network retrieval, thereby enhancing the real-time performance of the pictures matched by the current news segment. 4) The video synthesis module designed by the present invention supports workers in selecting rich video template resources and transition effects, has strong interactivity, can generate news videos in a relatively short time, and has high synthesis efficiency. 5) The present invention has strong scalability. The GPT2 model of the summary generation module is retrained using summary data sets from different fields, so that short video news in different fields can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Module connection diagram of an embodiment of the present invention;

[0047] Figure 2 The main implementation process of the embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the GPT2 model structure in an embodiment of the present invention;

[0049] Figure 4 GPT2 model test case in the embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the CLIP model structure in an embodiment of the present invention;

[0051] Figure 6 CLIP model test case in the embodiment of the present invention;

[0052] Figure 7 An example of speech recognition using the Whisper model in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the specific implementation methods of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0054] Short video news generation system based on large-scale pre-training model,

[0055] It is specifically divided into the front-end web page module and the back-end short video news generation platform. The connection diagram between the modules is as shown in the attached Figure 1 shown.

[0056] The front-end webpage module is specifically implemented as follows:

[0057] The present invention uses the widely used and stable Vue framework for development, and programming realizes the functions of each component. The Vue framework has the advantages of good interactivity, easy use, simple operation, and wide application.

[0058] The process for web users is as follows:

[0059] When the user logs in to the webpage, he first clicks to enter the summary generation interface and can choose to upload news text, files or URLs. The webpage background converts the uploaded data into news text and transmits it to the backend. The summary generation module first pre-processes the data through data cleaning and segmentation, and then uses the pre-trained news summary model to generate a news summary and output it to the front end.

[0060] Users can view the summary file and modify it, and then enter the image-text matching interface. They can choose to upload the modified summary file to the backend. The image-text matching module first extracts information from each paragraph of the news summary to obtain keywords and event triplets, and then crawls the image set related to the paragraph through the search engine. The pre-trained Chinese CLIP model is used to match the most similar images, and after splicing them into the paragraph, the generated news release with images and text is finally output to the frontend.

[0061] Users can view the generated press releases after matching pictures and texts and make modifications, and finally enter the video synthesis interface. Users can choose the platform's preset video template or upload their own video template, and select the preset video transition effects, subtitle style, speech speed, and timbre. The front end will transmit the video parameters to the back-end video synthesis module for video synthesis, and finally output the synthesized short video news to the user, who can then perform video editing based on this.

[0062] The back-end short video news generation platform module,

[0063] Specifically, it is divided into summary generation module, image-text matching module, and video synthesis module; the present invention adopts GPT2 model to realize summary generation, TextRank algorithm and DEGREE model to realize information extraction, CLIP model to realize image-text matching, and Whisper model to realize speech recognition, as shown in the attached figure. Figure 2 As shown, the specific implementation steps of the three modules are as follows:

[0064] The summary generation module is specifically implemented as follows:

[0065] We collected news summary datasets from Sina Weibo, Sogou News, etc., and cleaned the data by removing URLs, image tags, and topic tags. We then constructed a news Chinese summary dataset M and used a Chinese word segmenter to segment the original content of each news item in the dataset M. i and News Abstract i Perform word segmentation, M={(Content i , Abstract i )|1≤i≤n}; where n is the number of news in the data set, and is a word vector corresponding to the text, and is the number of word vectors;

[0066] The news summary generation model used is the text pre-training model GPT2, and the model structure is as shown in the attached Figure 3 As shown, the autoregressive learning model of the GPT2 model used is modeled as The GPT2 model loss function is modified to only calculate the summary of the predicted news for training;

[0067] The GPT2 model used is constructed through the Transformer decoder, using beam search as the decoding strategy, retaining num at each time step beams Output the output with the highest current score, output the sequence with the largest product of conditional probabilities at each time step, and use n-grams penalty to reduce sequence or word duplication problems;

[0068] The training set and test set were trained with a base learning rate of 0.0001, a validation interval of 4000 times, and a warm-up probability of 0.1. The trained network was then used to test the news summary generation model on the test set, resulting in a highly accurate summary generation model. The input news release text was cleaned, including removing image tags, URLs, and hashtags, and then segmented. Finally, each segment of the news text was summarized and concatenated to produce a news summary.

[0069] The news text input during the test is segmented before generating the summary; first, the news text is divided into p paragraphs. The specific steps are to traverse the entire news text. If the cumulative number of words at the end of a sentence exceeds num, the segmentation operation is performed. If the number of words at the end of the text is less than num, it is spliced ​​to the previous paragraph. Here, num is 100.

[0070] Attachment Figure 4 An example of a news summary generated from a news text in a press release is given. The final news summary is "The whole country has entered the "quick freeze" mode. How to respond and protect during the cold wave? The cold wave hit more than 20 provinces across the country, and the temperature in Beijing dropped to freezing point." It can be used for reference.

[0071] The image-text matching module is specifically implemented as follows:

[0072] Through information extraction, keywords and event information are extracted from the text segment to form key information. Then, relevant image datasets are searched through search engines. Finally, the pre-trained Chinese CLIP model is used to calculate the similarity between the key information in the text and the image. After performing softmax, the image with the highest probability is selected as the matching image.

[0073] Information extraction uses the TextRank algorithm, which returns the k most heavily weighted keywords. The DEGREE model is then used to identify trigger words and arguments in the summary, thereby obtaining triples corresponding to the event. Trigger words are the core words that trigger the event, often verbs or nouns, while arguments are the participants in the event, primarily consisting of entities, values, and time. The layer parameters of the trigger word and argument identification networks are initialized using BERT.

[0074] The CLIP image encoder uses ResNet, and the text encoder uses Tansformer. The image and text encoding results obtained by inputting into the encoder are used to extract their respective features. The corresponding low-dimensional vector representation is obtained through projection mapping and L2 normalization. The cross entropy of these two vectors is calculated to obtain the similarity matrix for classification. After obtaining the predicted similarity matrix, the cross entropy objective function needs to be calculated in combination with the true value, and finally the model is learned.

[0075] Attachment Figure 6 The following is a picture that matches a certain section of the news summary output by the news summary module for reference.

[0076] The video synthesis module is specifically implemented as follows:

[0077] According to the time flow in the subtitle file, the total duration of the video can be obtained as t all =t s +t end , where t endis the ending time of the last sentence in the subtitle file, t s is the duration of the preset opening template video, and the time when each sentence should appear is t i '=t s +t i , 1≤i≤n w , where t i is the starting time of a sentence in the subtitle file, n w is the number of sentences in the subtitle file;

[0078] Repeat after the opening pattern is played s The middle template of the video at the time is added to the end template of the video to synthesize a complete video template;

[0079] The moment the picture appears is the same as the beginning of the corresponding paragraph. Then, using the Python library moviepy for video editing, the picture is dynamically rotated, scaled, and translated in the video, thus achieving different preset transition effects.

[0080] Use ffmpeg, an open source framework for audio and video processing, to add subtitles and audio to the video to generate complete short video news.

[0081] Attachment Figure 7 An example of using the Whisper model to recognize a speech segment is given for reference.

[0082] While the embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments, including components, without departing from the principles and spirit of the present invention are within the scope of protection of the present invention.

Claims

1. A short video news generation system based on a large-scale pre-training model, characterized by: include: Front-end webpage module and back-end short video news generation platform; Front-end webpage module, including: interactive component, video parameter visualization component, data sending and uploading component, and data receiving component; Interactive components, which enable interaction with users, including buttons, text input boxes, and page switching; Video parameter visualization component, which visualizes pre-selectable video parameters for users to choose; The data upload component uploads the user's uploaded news releases, selected templates, and video parameters to the backend short video news generation platform; Data receiving and conversion component, which receives press releases and news videos processed by the back-end short video news generation platform; The back-end short video news generation platform includes: summary generation module, image-text matching module, and video synthesis module; The summary generation module pre-processes the input news release through data cleaning and segmentation, and then generates a news summary through the summary generation model; The image-text matching module extracts keywords and event triplets from each news segment through information extraction based on the input news summary. It then crawls the image collection related to each news segment through a search engine, matches the most relevant images using an image-text matching model, and generates news articles with images and text. The video synthesis module synthesizes news videos based on the input news image and text information; the webpage transmits the video parameters to the backend model, and finally outputs the news video; Summary generation module, including: We collected news summary datasets, cleaned the data, and constructed the news Chinese summary dataset M; Use the Chinese word segmenter to analyze the original content of each news in the dataset M. i and News Abstract i Perform word segmentation, M={(Content i , Abstract i )|1≤i≤n}; where n is the number of news in the data set, and is a word vector corresponding to the text, and is the number of word vectors; The news summary generation model used is the text pre-training model GPT2, and the autoregressive learning model of the GPT2 model used is The GPT2 model loss function is modified to only calculate the summary of the predicted news for training; the GPT2 model used is constructed through the Transformer decoder, using beam search as the decoding strategy, and retaining num at each time step. beams Output the output with the highest current score, output the sequence with the largest product of conditional probabilities at each time step, and use n-grams penalty to reduce sequence or word duplication problems; The news summary input during the test is segmented before generation; the news text is divided into p paragraphs, and the entire news text is traversed. If the cumulative number of words at the end of a sentence exceeds num, the segmentation operation is performed. If the number of words at the end of the text is less than num, it is spliced ​​to the previous paragraph. Video synthesis module, including: Based on the input news summary image and text information, the news text and corresponding images are extracted. The text-to-speech Python library pyttsx3 is used to convert the text to speech based on the preset timbre and speaking speed to generate an audio file. The Whisper model is used to recognize the speech and generate a subtitle file. According to the time flow in the subtitle file, the total duration of the video is obtained as t all =t s +t end , where t end is the ending time of the last sentence in the subtitle file, t s is the duration of the preset opening template video, and the time when each sentence should appear is t i '=t s +t i , 1≤i≤n w , where t i is the starting time of a sentence in the subtitle file, n w is the number of sentences in the subtitle file; Repeat after the opening pattern is played s The middle template of the video at the time is added to the end template of the video to synthesize a complete video template; The moment the picture appears is the same as the beginning of the corresponding paragraph. Then, using the Python library moviepy for video editing, the picture is dynamically rotated, scaled, and translated in the video, thus achieving different preset transition effects. Use ffmpeg, an open source framework for audio and video processing, to add subtitles and audio to the video to generate complete short video news.

2. The short video news generation system based on a large-scale pre-training model according to claim 1 is characterized in that: The image-text matching module, after extracting key information from the text segment through information extraction, and extracting the key information from the event information, and then searching the relevant image dataset through a search engine, further includes: using a pre-trained Chinese CLIP model to calculate the similarity between the key information in the text and the image, and taking the image with the highest probability after performing softmax as the matching image.

3. The short video news generation system based on a large-scale pre-training model according to claim 2 is characterized in that: The information extraction adopts the TextRank algorithm to return the k keywords with the largest weights, and performs trigger word recognition and argument recognition on the summary segment based on the DEGREE model to obtain the triples of the corresponding events; among them, the trigger word here represents the core word of the event, which is mostly a verb or noun, and the argument is the participant of the event, mainly composed of entity, value, and time. The layer parameters of the trigger word and argument recognition network are initialized using BERT.

4. The short video news generation system based on a large-scale pre-training model according to claim 2 is characterized in that: CLIP image encoder uses ResNet, and text encoder uses Tansformer. By inputting the image and text encoding results obtained by the encoder, the respective features are extracted. The corresponding low-dimensional vector representation is obtained through projection mapping and L2 normalization. Use these two vectors to calculate the cross entropy to obtain the similarity matrix for classification. After obtaining the predicted similarity matrix, it is necessary to calculate the cross entropy objective function in combination with the true value. Finally, the model is learned.

Images