Text-based video automatic production method and device, equipment and medium

Abstract

The invention discloses a text-based video automatic production method and device, equipment and a medium, and relates to the technical field of electric data processing. The method comprises the following steps: establishing a video material library and a background music material library; wherein content keywords and corresponding time slice labels are marked on the video materials, and emotion category labels are marked on the background music materials; acquiring an input text, segmenting the input text into a plurality of sentences according to punctuations, respectively performing speech synthesis to generate speech, and recording the audio duration of each sentence; performing keyword matching on each sentence of input text and a video material label to generate a video candidate material set; performing emotion classification on the full text of the input text by using the fine-tuned BERT model, matching emotion types with background music material labels, and generating a background music candidate material set; segmenting the input text according to sentences, and matching audio duration to generate subtitles; and calling the MoviePy library to integrate the video, the voice, the subtitles and the background music, and carrying out accelerated processing by utilizing a GPU to generate the video. Content authenticity and production efficiency are both considered.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic data processing, and in particular to a text-based automatic video production method, device, equipment and medium. Background Art

[0002] With the rapid development of the self-media industry, more and more people are trying to express their opinions, share their experiences, and spread knowledge through their own videos. Traditional video production relies heavily on professional editing software such as Adobe Premiere, iMovie, TikTok, and Jianying. Although the operational threshold of such tools has gradually decreased, users still need to manually complete complex steps such as material selection, editing and splicing, adding subtitles, and dubbing synthesis. In terms of automatic video production, while AI video generation models such as Sora can directly generate video content through text input, the images generated by their synthesis technology often have defects that violate the laws of physics or real-world logic, resulting in a lack of video authenticity and often causing audiences to question the credibility of the content.

[0003] Patent application number CN119402728A proposes a video generation method based on text parsing. This method extracts entity information from text segments, matches it to a preset template, and synthesizes the video. Its limitations are: (1) it relies on a preset video template; (2) it does not address the issue of dealing with the lack of entities or keywords in the text; and (3) the location of text segments must be determined by character position or paragraph number, making it difficult to capture semantic associations.

[0004] Patent application number CN119211458A proposes to generate videos by presetting video groups and splicing them in a certain order. However, its limitation is that the video groups need to be manually preset, and the generation process still requires human intervention, which cannot be automated. Summary of the Invention

[0005] The present invention provides a text-based automatic video production method, device, equipment and medium, which solves the problem of how to automatically generate videos with high authenticity.

[0006] To achieve the above objectives, this application adopts the following technical solutions: In a first aspect, a text-based automatic video production method is provided, comprising: Establish a video material library and a background music material library; among them, the video materials are marked with content keywords and corresponding time segment labels, and the background music materials are marked with emotion category labels; Obtaining input text, dividing the input text into multiple sentences according to punctuation, performing speech synthesis on each sentence to generate speech, and recording the audio duration of each sentence; Perform keyword matching on each sentence of the input text and the video material label to generate a set of candidate video materials; Use the fine-tuned BERT model to classify the sentiment of the full input text, match the sentiment type with the background music material label, and generate a set of background music candidate materials; Split the input text into sentences and generate subtitles based on the audio duration; The MoviePy library is called to integrate video, voice, subtitles, and background music, and GPU-accelerated processing is used to generate video. For sentences with the same video candidate set, the sentences in the same candidate set are merged into video units, and the order of the materials is randomly shuffled and spliced together. The material loop or cutoff is adjusted according to the total length of the voice.

[0007] In a first possible implementation manner of the first aspect, after performing keyword matching between each input text and a video material tag: If the match fails, the semantic similarity is calculated and a set of video candidate materials is generated through threshold screening.

[0008] Based on the first possible implementation of the first aspect, in a second possible implementation of the first aspect, calculating the semantic similarity includes: Suppose the N sentences of input text after segmentation are ,material There are k labels, ; respectively and Encode through BERT to obtain the encoding vector and ; Compute the cosine similarity between two encoded vectors: .

[0009] when hour, Selected candidate material set; in, is the semantic similarity threshold, , set by humans.

[0010] In a third possible implementation manner of the first aspect, after generating the candidate video material set: If the video candidate material set of this sentence is empty, the video candidate material set of the previous sentence will be inherited; If this sentence is the first sentence of the text, assign the interlude material to this sentence; When the number of sentences in the same video candidate material set exceeds a set value, the video candidate material set of subsequent sentences will be changed to a running material until a new video candidate material set appears.

[0011] In a second aspect, a text-based video automatic production device is provided, comprising: The material library construction module is used to build a video material library and a background music material library; the video material is annotated with content keywords and corresponding time segment labels, and the background music material is annotated with emotion category labels; The speech synthesis module is used to obtain input text, segment the input text into multiple sentences according to punctuation, perform speech synthesis on each sentence to generate speech, and record the audio duration of each sentence; A video candidate material set generation module is used to perform keyword matching between each sentence of the input text and the video material tag to generate a video candidate material set; The background music candidate material set generation module is used to use the fine-tuned BERT model to classify the emotions of the full input text, match the emotion type with the background music material label, and generate a background music candidate material set; The subtitle generation module is used to split the input text into sentences and generate subtitles according to the audio duration; The video generation module is used to call the MoviePy library to integrate video, voice, subtitles, and background music, and use GPU acceleration to generate video. Among them, for sentences with the same video candidate set, the sentences with the same candidate set are merged into video units, and the order of the materials is randomly shuffled and spliced together; the material loop or interception is adjusted according to the total length of the voice.

[0012] In a first possible implementation manner of the second aspect, the video candidate material set generation module is further configured to: After performing keyword matching on each input text and the video material tag: If the match fails, the semantic similarity is calculated and a set of video candidate materials is generated through threshold screening.

[0013] Based on the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, calculating the semantic similarity includes: Suppose the N sentences of input text after segmentation are ,material There are k labels, ; respectively and Encode through BERT to obtain the encoding vector and ; Compute the cosine similarity between two encoded vectors: .

[0014] when hour, Selected candidate material set; in, is the semantic similarity threshold, , set by humans.

[0015] In a third possible implementation manner of the second aspect, the video candidate material set generation module is further configured to: After generating the video candidate material set, If the video candidate material set of this sentence is empty, the video candidate material set of the previous sentence will be inherited; If this sentence is the first sentence of the text, assign the interlude material to this sentence; When the number of sentences in the same video candidate material set exceeds a set value, the video candidate material set of subsequent sentences will be changed to a running material until a new video candidate material set appears.

[0016] In a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the text-based automatic video production method as described in the first aspect.

[0017] In a fourth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the text-based automatic video production method as described in the first aspect are implemented.

[0018] The text-based automatic video production method of the present invention has the following advantages: This application realizes the automatic production of videos. Videos can be produced by simply inputting text, without any additional manual operation. The input text is analyzed, a set of candidate materials is specified for each sentence, and the text is divided into several video units based on this. The final generated video achieves a balance between authenticity and automation, taking into account both content authenticity and production efficiency. Compared with the method based on video editing software, it has the advantages of simple operation, no need for professional knowledge, and fully automatic video generation. Compared with the method based on large video models, it has the advantage of producing more realistic and credible videos. Compared with the approximate solution, this method does not require a preset template, can process any text, uses sentences as the smallest processing unit, can use GPU acceleration, and does not require manual intervention.

[0019] The present invention and the corresponding devices, electronic devices and readable storage media of the text-based automatic video production method can achieve the same technical effects. To avoid repetition, they will not be described here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A schematic flow chart of a text-based automatic video production method provided in an embodiment of the present application; Figure 2 A schematic flow chart of another text-based automatic video production method provided in an embodiment of the present application; Figure 3 A schematic flow chart of a speech synthesis process provided in an embodiment of the present application; Figure 4 A schematic flow chart of a text analysis process provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a text-based automatic video production device provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] To further illustrate the technical means and effects of the present invention to achieve its intended purpose, the technical solutions in the embodiments of this application are clearly described. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of this application are within the scope of protection of this application.

[0022] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0023] The description of the method flow in the specification of this application and the steps in the flowcharts in the drawings of the specification of this application do not necessarily need to be strictly executed according to the step numbers. The method steps can be executed in a different order. In addition, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0024] The text-based automatic video production method, device, equipment and medium provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0025] Most videos we commonly see consist of two parts: image and sound. Images are composed of a series of continuous images. Each image is called a frame, and multiple frames switched continuously at a certain speed constitute the image portion of a video. Each video clip (excluding the audio) is called a video clip. The core content of this clip is called the clip's label, such as "tiger," "elephant," or "war." The sound of a video is usually called audio, which can include voice, music, sound effects, and so on. The image and sound components of a video are usually related.

[0026] See Figure 1-2 , the embodiment of the present application provides a text-based video automatic production method, such as Figure 1-2 Shown, including: Step S1, establishing a video material library and a background music material library; wherein, the video material is marked with content keywords and corresponding time segment labels, and the background music material is marked with emotion category labels.

[0027] This method requires two types of materials: video and background music. Both types of materials must be pre-tagged, and the tags are stored in JSON format. Video tags include content and time. Content is a word representing the main content of the video, and time is the start and end time of the content in the video.

[0028] For example, a video of "Animal World" contains a clip of a giraffe. This clip appears from the 600th to the 605th second of the entire video. After this clip is selected as the video material, its label format is as follows: { "file_path": "video_examples / Animal World.mp4", "tags": ["giraffe"], "time": [(600, 605)] } Among them, the "file_path" field indicates the path of the video; the "tags" field indicates the material content, which can have multiple keywords; the "time" field indicates the time period when the content appears, and each tuple corresponds to a keyword in "tags".

[0029] The background music material tag only includes an emotion category, which indicates the video scene that the background music is suitable for. The emotion categories are divided into ten types: happy, excited, calm, narrative, soothing, sad, angry, tense, humorous, and romantic.

[0030] For example, if there is a background music material called True Blue and its label is happy, the label is as follows: { "file_path": "music_examples / True Blue.mp3", "tags": "happiness" } All video materials and background music materials are stored in the material database for system use.

[0031] Step S2: obtain input text, divide the input text into multiple sentences according to punctuation, perform speech synthesis on each sentence to generate speech, and record the audio duration of each sentence.

[0032] The system converts the input text into the audio part of the output video through speech synthesis. The speech synthesis process is as follows Figure 3 As shown in the figure, the entire input text is segmented and divided into N sentences by searching for punctuation marks such as periods, exclamation points, question marks, and semicolons. Then, speech synthesis is performed on each of the N sentences. Numerous speech synthesis algorithms are available, such as the open-source projects CosyVoice, GPT-SoVits, and XTTS. After synthesizing the N sentences, the audio duration of each sentence needs to be calculated and recorded. The speech synthesis results serve as the clue for the entire system. Video material, background music material, and subtitles are all guided by the speech synthesis results and ultimately form the output video.

[0033] Step S3: perform keyword matching on each input text and the video material tag to generate a set of candidate video materials.

[0034] Furthermore, if the matching fails, the semantic similarity is calculated and a set of video candidate materials is generated through threshold screening.

[0035] Specifically, let the N input texts after segmentation be ,right Calculate the semantic similarity between M video materials. Assume that material There are k labels, If any of the tags appear in In and The semantic similarity between , that is, the match is successful, Selected candidate material set.

[0036] if None of the tags in If they appear in and Encode through BERT (Bidirectional Encoder Representations from Transformers) to obtain the encoding vector and . Then calculate the cosine similarity of the two encoding vectors: is the semantic similarity threshold, which is set manually. hour, Selected candidate material set.

[0037] In step S4, the fine-tuned BERT model is used to perform emotion classification on the full input text, and the emotion type is matched with the background music material label to generate a background music candidate material set.

[0038] In this step, BERT is a pre-trained model. It needs to be fine-tuned with training data to enable the model to have the ability to recognize text emotions. The steps of fine-tuning are as follows: (1) Prepare training data The training data should contain text and its corresponding emotion labels. These labels are the 10 emotion labels mentioned above, and the data is a labeled text dataset. The dataset format is as follows: text label Today's sunshine is exceptionally bright, and my mood is as clear as the blue sky! Walking down the street, every corner is filled with laughter and joy, as if the whole world is celebrating this beautiful day with me. Let's enjoy this joy together and spread smiles to everyone! happiness Come on! We're about to embark on an unprecedented adventure! The road ahead is filled with unknowns and challenges, but it's precisely these that make our blood boil! Every step is filled with strength, and every attempt is filled with hope. Let's push our limits together and create our own legend! excitement ··· ··· The dataset is a CSV file containing two columns: one is text (text) and the other is sentiment label (label).

[0039] (2) Data preprocessing When fine-tuning BERT, you need to convert the text data into a format that BERT can accept. This includes tokenization, adding special tags (such as [CLS] and [SEP]), and converting the text into corresponding token IDs.

[0040] (3) Pre-training model training Load the pre-trained BERT model, add a classification layer, and train the model using the prepared training data. The trained model is now capable of recognizing sentiment in text. When new text is input, the model's output data is the sentiment label.

[0041] The above steps S3-S4 are the process of text analysis on the input text, and the results of the analysis are used as the basis for searching video materials and background music materials. Figure 4As shown in the figure, the input text is segmented into sentences, and the semantic similarity between each sentence and all video materials is calculated. The output results are used by the video material search module. On the other hand, emotion recognition is performed on the entire input text, and the output results are used to search for background music materials.

[0042] Through the processing of step S3, the video candidate material set of some sentence texts can be obtained, while the video candidate material set of other sentences is empty. By adjusting the semantic similarity threshold, the number of sentences with non-empty candidate sets can be controlled. In order to specify the video candidate material set for all sentences, this method adopts the following strategy for processing, including: Step S31: If the video candidate material set of the current sentence is empty, the video candidate material set of the previous sentence is inherited.

[0043] Step S32: If the sentence is the first sentence of the text, a link material is assigned to the sentence. The first sentence of the text, that is, there is no previous sentence. The link material is a general video image, which can be similar to the image broadcast by the news anchor.

[0044] In step S33, if the number of sentences in the same candidate video material set exceeds a set value, the candidate video material set for subsequent sentences is changed to interlude material until a new candidate video material set appears. If the number of sentences in the same candidate set exceeds five (or another number), the candidate set for subsequent sentences is changed to interlude material. This is done to avoid excessive reuse of video content.

[0045] Through the processing of step S4, the emotional category of the input text can be obtained, and background music can be selected from the background music library based on the emotional category label. If the duration of the entire video is longer than the duration of the background music, the background music is recycled. Otherwise, the background music is intercepted. In addition, the volume of the background music is adjusted to be lower than the volume of the voice to ensure that the volume of the background music does not interfere with the voice broadcast.

[0046] Step S5: Segment the input text into sentences and generate subtitles by matching the audio duration.

[0047] In this step, the subtitles match the input text and appear sentence by sentence at the designated location on the output video screen, which can be centered at the bottom or to the left. The order and duration of the subtitles match the content and duration of each sentence in the input text. Because the duration of each sentence has been determined in the previous step, the start and end times of the subtitles can also be determined.

[0048] Step S6: Call the MoviePy library to integrate video, voice, subtitles, and background music, and use GPU acceleration to generate video. For sentences with the same video candidate set, the sentences in the same candidate set are merged into video units, and the order of the materials is randomly shuffled and spliced together. The material loop or interception is adjusted according to the total length of the voice.

[0049] In other words, sentences with the same candidate set are merged into a single video unit, the video clips in the candidate set are randomly shuffled, and then the video is spliced in order. If the total duration of the speech corresponding to the video unit is longer than the total duration of the candidate clips, the candidate clips are recycled. If the total duration of the speech corresponding to the video unit is shorter than the total duration of the candidate clips, the first several clips are cut to make it equal to the total duration of the speech, and the last video clip can be cut as needed.

[0050] Related technologies, such as patent application CN119402728A, rely solely on the CPU for computing resources, without utilizing GPU acceleration, resulting in inefficiency. This step, including video capture, splicing, adding subtitles, and adding audio, is implemented using the MoviePy library. This library not only provides a wealth of video processing functions but also supports GPU computing, accelerating the processing by invoking the GPU during video processing.

[0051] Based on the above technical solution, this application realizes automatic video production. Videos can be produced by simply inputting text, and no additional manual operation is required throughout the process. The input text is analyzed, and the semantic similarity between the text sentence and the material is calculated for control. By inheriting the previous sentence, specifying the linking material, and whether the cumulative number of sentences exceeds the limit, a candidate material set is specified for each sentence, and the text is divided into several video units accordingly. The final generated video achieves a balance between authenticity and automation, taking into account both content authenticity and production efficiency. Compared with the method based on video editing software, it has the advantages of simple operation, no need for professional knowledge, and fully automatic video generation. Compared with the method based on large video models, it has the advantage of producing more realistic and credible videos. Compared with the approximate solution, this method has the advantages of not requiring a preset template, can process any text, uses sentences as the smallest processing unit, can use GPU acceleration, and does not require human intervention throughout the process.

[0052] See also Figure 5 Corresponding to the above-mentioned text-based automatic video production method embodiment, the present application embodiment provides a text-based automatic video production device, including: The material library construction module 1001 is used to establish a video material library and a background music material library; wherein the video material is annotated with content keywords and corresponding time segment labels, and the background music material is annotated with emotion category labels; The speech synthesis module 1002 is used to obtain input text, segment the input text into multiple sentences according to punctuation, perform speech synthesis on each sentence to generate speech, and record the audio duration of each sentence; The video candidate material set generation module 1003 is used to perform keyword matching between each sentence of the input text and the video material tag to generate a video candidate material set; Background music candidate material set generation module 1004, configured to use the fine-tuned BERT model to perform emotion classification on the full input text, match the emotion type with the background music material label, and generate a background music candidate material set; The subtitle generation module 1005 is used to segment the input text into sentences and generate subtitles by matching the audio duration; The video generation module 1006 is used to call the MoviePy library to integrate video, voice, subtitles, and background music, and use GPU acceleration to generate video; among them, for sentences with the same video candidate set, the sentences in the same candidate set are merged into video units, the order of the materials is randomly shuffled and spliced; the material loop or interception is adjusted according to the total length of the voice.

[0053] Furthermore, the video candidate material set generation module 1003 is further configured to: After performing keyword matching on each input text and the video material tag: If the match fails, the semantic similarity is calculated and a set of video candidate materials is generated through threshold screening.

[0054] Furthermore, the semantic similarity is calculated, including: Suppose the N sentences of input text after segmentation are ,material There are k labels, ; respectively and Encode through BERT to obtain the encoding vector and ; Compute the cosine similarity between two encoded vectors: .

[0055] when hour, Selected candidate material set; in, is the semantic similarity threshold, , set by humans.

[0056] Furthermore, the video candidate material set generation module 1003 is further configured to: After generating the video candidate material set, If the video candidate material set of this sentence is empty, the video candidate material set of the previous sentence will be inherited; If this sentence is the first sentence of the text, assign the interlude material to this sentence; When the number of sentences in the same video candidate material set exceeds a set value, the video candidate material set of subsequent sentences will be changed to a running material until a new video candidate material set appears.

[0057] The above-mentioned text-based video automatic production device implements the steps and various processes of the above-mentioned text-based video automatic production method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0058] See also Figure 6 Corresponding to the above-mentioned text-based video automatic production method embodiment, the embodiment of the present application provides an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, the steps and various processes of the above-mentioned text-based video automatic production method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0059] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0060] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0061] Corresponding to the above-mentioned text-based video automatic production method embodiment, the embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps and various processes of the above-mentioned text-based video automatic production method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0062] The processor is the processor in the electronic device described in the embodiment of the present application. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0063] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0065] It will be appreciated that the embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative and not restrictive, and those skilled in the art will appreciate that various changes or equivalent replacements may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, those of ordinary skill in the art may modify these features and embodiments to adapt to specific circumstances and materials under the inspiration or guidance of this application without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application fall within the scope protected by the present invention.

Claims

1. A text-based video automatic production method, characterized in that: include: Establish a video material library and a background music material library; among them, the video materials are marked with content keywords and corresponding time segment labels, and the background music materials are marked with emotion category labels; Obtaining input text, dividing the input text into multiple sentences according to punctuation, performing speech synthesis on each sentence to generate speech, and recording the audio duration of each sentence; Perform keyword matching on each sentence of the input text and the video material label to generate a set of candidate video materials; Use the fine-tuned BERT model to classify the sentiment of the full input text, match the sentiment type with the background music material label, and generate a set of background music candidate materials; Split the input text into sentences and generate subtitles based on the audio duration; The MoviePy library is called to integrate video, voice, subtitles, and background music, and GPU-accelerated processing is used to generate video. For sentences with the same video candidate set, the sentences in the same candidate set are merged into video units, and the order of the materials is randomly shuffled and spliced together. The material loop or cutoff is adjusted according to the total length of the voice.

2. The text-based automatic video production method according to claim 1, characterized in that: After performing keyword matching on each input text and the video material tag: If the match fails, the semantic similarity is calculated and a set of video candidate materials is generated through threshold screening.

3. The text-based automatic video production method according to claim 2, characterized in that: Calculate semantic similarity, including: Suppose the N sentences of input text after segmentation are ,material There are k labels, ; Respectively and Encode through BERT to obtain the encoding vector and ; Compute the cosine similarity between two encoded vectors: ,when hour, Selected candidate material set; in, is the semantic similarity threshold, , set by humans.

4. The text-based automatic video production method according to claim 1, characterized in that: After generating the video candidate material set as described above: If the video candidate material set of this sentence is empty, the video candidate material set of the previous sentence will be inherited; If this sentence is the first sentence of the text, assign the interlude material to this sentence; When the number of sentences in the same video candidate material set exceeds a set value, the video candidate material set of subsequent sentences will be changed to a running material until a new video candidate material set appears.

5. A text-based video automatic production device, characterized in that: include: The material library construction module is used to build a video material library and a background music material library; the video material is annotated with content keywords and corresponding time segment labels, and the background music material is annotated with emotion category labels; The speech synthesis module is used to obtain input text, segment the input text into multiple sentences according to punctuation, perform speech synthesis on each sentence to generate speech, and record the audio duration of each sentence; A video candidate material set generation module is used to perform keyword matching between each sentence of the input text and the video material tag to generate a video candidate material set; The background music candidate material set generation module is used to use the fine-tuned BERT model to classify the emotions of the full input text, match the emotion type with the background music material label, and generate a background music candidate material set; The subtitle generation module is used to split the input text into sentences and generate subtitles according to the audio duration; The video generation module is used to call the MoviePy library to integrate video, voice, subtitles, and background music, and use GPU acceleration to generate video. Among them, for sentences with the same video candidate set, the sentences with the same candidate set are merged into video units, and the order of the materials is randomly shuffled and spliced together; the material loop or interception is adjusted according to the total length of the voice.

6. The text-based video automatic production device according to claim 5, characterized in that: The video candidate material set generation module is further used for: After performing keyword matching on each input text and the video material tag: If the match fails, the semantic similarity is calculated and a set of video candidate materials is generated through threshold screening.

7. The text-based video automatic production device according to claim 6, characterized in that: Calculate semantic similarity, including: Suppose the N sentences of input text after segmentation are ,material There are k labels, ; Respectively and Encode through BERT to obtain the encoding vector and ; Compute the cosine similarity between two encoded vectors: ,when hour, Selected candidate material set; in, is the semantic similarity threshold, , set by humans.

8. The text-based automatic video production device according to claim 5, characterized in that: The video candidate material set generation module is further used for: After generating the video candidate material set, If the video candidate material set of this sentence is empty, the video candidate material set of the previous sentence will be inherited; If this sentence is the first sentence of the text, assign the interlude material to this sentence; When the number of sentences in the same video candidate material set exceeds a set value, the video candidate material set of subsequent sentences will be changed to a running material until a new video candidate material set appears.

9. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the text-based automatic video production method according to any one of claims 1 to 4 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the text-based automatic video production method according to any one of claims 1 to 4 are implemented.

Images