Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment and a storage medium, and relates to the technical field of computer application. Segmenting the plot text to obtain segmented plot texts; for each segmented plot text, determining a video segment corresponding to the segmented plot text in the original video, taking the segmented plot text as an explanation text, and converting the explanation text into voice in a text-to-voice TTS mode to obtain explanation voice corresponding to the segmented plot text; extracting a video picture in the video clip as an explanation video picture corresponding to the segmented plot text, and taking an explanation voice corresponding to the segmented plot text as an explanation voice corresponding to the explanation video picture; and according to a segmentation time sequence for segmenting the plot text, synthesizing the explanation video picture and the explanation voice corresponding to each segmented plot text to obtain a target video. The video can be automatically and efficiently generated.

Description

Technical Field

[0001] The present invention relates to the field of computer application technologies, and in particular, to a video generation method, apparatus, electronic device, and storage medium. Background Art

[0002] In the era of short videos, it is an era of competing for dividends. Among them, in the categories of movies and TV dramas, short video content of the "quick look type" based on the plot is more popular. Users can understand the plot or exciting plot segments within just a few minutes, thereby attracting more users for the long video platforms in a short-driven-long way, increasing the viewing duration and consuming long video content. However, its production cost by manual labor is extremely high. How to efficiently generate short video content is a very urgent problem to be solved. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a video generation method, apparatus, electronic device, and storage medium to achieve automatic and efficient video generation. The specific technical solutions are as follows:

[0004] In the first aspect implemented by the present invention, a video generation method is provided, including:

[0005] Obtain a plot text, where the plot text is used to represent the plot summary information of the original video;

[0006] Segment the plot text to obtain segmented plot texts;

[0007] For each segmented plot text, determine the video segment in the original video corresponding to the segmented plot text, and use the text-to-speech (TTS) method to convert the segmented plot text into speech, obtaining the explanatory speech corresponding to the segmented plot text;

[0008] Extract the video frames in the video segment as the explanatory video frames corresponding to the segmented plot text, and use the explanatory speech corresponding to the segmented plot text as the explanatory speech corresponding to the explanatory video frames;

[0009] Synthesize the explanatory video frames and the explanatory speech respectively corresponding to each segmented plot text in the chronological order of segmentation of the plot text to obtain a target video.

[0010] Optionally, determining the video segment in the original video corresponding to the segmented plot text includes:

[0011] Input the segmented plot text and the original video into the plot retrieval model. Obtain the starting time of the video corresponding to the segmented plot text in the original video through the plot retrieval model, and extract the corresponding video segment from the original video using the starting time of the video as the video segment corresponding to the segmented plot text. Among them, the plot retrieval model is pre-trained according to multiple training samples. A training sample pair includes multiple sample plot texts and sample videos, as well as the starting time period labels of the videos in the sample videos corresponding to each sample plot text.

[0012] Optionally, before synthesizing the explanatory video frames and the explanatory speech respectively corresponding to each segmented plot text in the chronological order of splitting the plot text to obtain the target video, the method further includes:

[0013] Determine the key plot point sentences corresponding to the segmented plot text by using the method of plot coherence analysis, or extract the key plot point sentences corresponding to the segmented plot text by using the ChatGPT large model;

[0014] Determine the native content to be inserted based on the key plot point sentences. The native content is the video content extracted from the original video based on the key plot point sentences, including native frames and native audio;

[0015] The step of synthesizing the explanatory video frames and the explanatory speech respectively corresponding to each segmented plot text in the chronological order of splitting the plot text to obtain the target video includes:

[0016] According to the chronological order of splitting, splice the explanatory video frames and the native frames respectively corresponding to each segmented plot text to obtain a video track, and splice the explanatory speech corresponding to the explanatory video frames and the native audio corresponding to the native frames respectively corresponding to each segmented plot text to obtain an audio track;

[0017] Synthesize the video track and the audio track to obtain the target video.

[0018] Optionally, the method of using the plot coherence analysis to determine the key plot point sentences corresponding to the plot text includes:

[0019] Select a sentence in the segmented plot text as a candidate content in turn;

[0020] For each candidate content, delete the candidate content from the segmented plot text, judge whether it affects the smoothness of understanding the semantic content and the degree of influence, and determine the corresponding score based on whether it affects the smoothness of understanding the semantic content and the degree of influence;

[0021] Based on the scores of each candidate content, select the candidate content ranked at the front in the order of scores as the selected key plot point statements.

[0022] Optionally, the original content further includes original text;

[0023] Synthesizing the explanatory video images and the explanatory voice respectively corresponding to each segmented plot text according to the segmentation time sequence of segmenting the plot text to obtain a target video, including:

[0024] According to the segmentation time sequence, splice the explanatory video images and the original images respectively corresponding to each segmented plot text to obtain a video track, splice the explanatory voice corresponding to the explanatory video images and the original audio corresponding to the original images respectively corresponding to each segmented plot text to obtain an audio track, and splice each segmented plot text and the original text to obtain a subtitle track;

[0025] Synthesizing the video track and the audio track to obtain a target video,

[0026] Synthesize the video track, the audio track and the subtitle track to obtain a target video.

[0027] Optionally, after determining the video segment corresponding to the segmented plot text in the original video and using the text-to-speech TTS method to convert the explanatory text into speech to obtain the explanatory voice corresponding to the segmented plot text, the method further includes:

[0028] For each segmented plot text, align the video segment and the explanatory voice.

[0029] Optionally, the alignment of the video segment and the explanatory voice includes:

[0030] If the duration of the video segment is greater than the duration of the explanatory voice, shrink the video segment to the same duration as the explanatory voice;

[0031] If the duration of the video segment is less than the duration of the explanatory voice, expand the video segment to the same duration as the explanatory voice.

[0032] Optionally, the alignment of the video segment and the explanatory voice includes:

[0033] Keep the duration of the video segment unchanged, and adjust the playback speed of the explanatory voice based on the ratio between the duration of the video segment and the duration of the explanatory voice, and the playback speed is used to ensure that the duration of playing the explanatory voice is the same as the duration of the video segment.

[0034] Optionally, before aligning the video segment with the commentary voice for each segmented plot text, the method further includes:

[0035] If the segmented plot text is a key plot point statement, remove the content from the time point corresponding to the key plot point statement to the end of the video segment in the video segment to obtain a post-removal video segment;

[0036] Aligning the video segment with the commentary voice for each segmented plot text includes:

[0037] For each segmented plot text, align the post-removal video segment with the commentary voice.

[0038] In a second aspect of the implementation of the present invention, a video generation device is provided, including:

[0039] An acquisition module for acquiring plot text, where the plot text is used to represent the plot summary information of the original video;

[0040] A splitting module for splitting the plot text to obtain segmented plot texts;

[0041] A first determination module for, for each segmented plot text, determining the video segment in the original video corresponding to the segmented plot text, and using the text-to-speech (TTS) method to convert the segmented plot text into speech as the commentary voice corresponding to the segmented plot text; extracting the video picture in the video segment as the commentary video picture corresponding to the segmented plot text, and using the commentary voice corresponding to the segmented plot text as the commentary voice corresponding to the commentary video picture;

[0042] A synthesis module for synthesizing the commentary video picture and the commentary voice respectively corresponding to each segmented plot text in the chronological order of splitting the plot text to obtain a target video.

[0043] In a third aspect of the implementation of the present invention, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[0044] The memory is used to store a computer program;

[0045] The processor is used to implement the steps of the method described in the first aspect when executing the program stored on the memory.

[0046] In yet another aspect of the implementation of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the video generation method described in any one of the above is implemented.

[0047] In yet another aspect of the implementation of the present invention, there is also provided a computer-readable storage medium, in which a target video is stored, and the target video is obtained by implementing the method steps of any one of the methods described in the first aspect.

[0048] In yet another aspect of the implementation of the present invention, there is also provided a computer program product containing instructions, which when run on a computer, causes the computer to execute the video generation method described in any one of the above.

[0049] The video generation method provided by the embodiments of the present invention includes: obtaining a scenario text for representing the scenario summary information of the original video; splitting the scenario text to obtain segmented scenario texts; for each segmented scenario text, determining a video segment in the original video corresponding to the segmented scenario text, and using the segmented scenario text as an explanatory text, and converting the explanatory text into speech by means of text-to-speech (TTS) to obtain the explanatory speech corresponding to the segmented scenario text; extracting the video frames in the video segment as the explanatory video frames corresponding to the segmented scenario text, and using the explanatory speech corresponding to the segmented scenario text as the explanatory speech corresponding to the explanatory video frames. In this way, according to the segmentation time sequence of the scenario text splitting, the explanatory video frames and explanatory speech respectively corresponding to each segmented scenario text can be synthesized to obtain the target video, realizing the automatic generation of the video, improving the efficiency of video generation, and achieving efficient video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art.

[0051] Figure 1 It is a flowchart of a video generation method in an embodiment of the present invention;

[0052] Figure 2 It is another flowchart of a video generation method in an embodiment of the present invention;

[0053] Figure 3 It is still another flowchart of a video generation method in an embodiment of the present invention;

[0054] Figure 4 It is yet another flowchart of a video generation method in an embodiment of the present invention;

[0055] Figure 5Another flowchart of the video generation method in the embodiments of the present invention;

[0056] Figure 6 Schematic structural diagram of the video generation device in the embodiments of the present invention;

[0057] Figure 7 Schematic structural diagram of the electronic device provided by the embodiments of the present invention. Detailed implementation manners

[0058] Next, the technical solutions in the embodiments of the present invention will be described with reference to the accompanying drawings in the embodiments of the present invention.

[0059] Currently, film and television drama platforms have a large number of high-quality long videos. How to efficiently and automatically complete the work that can only be done by a video team and automatically generate short video material content, such as the short video content form of commentary + partial original sound interspersed, is a very urgent problem to be solved.

[0060] The embodiments of the present invention provide a video generation method, as Figure 1 shown, which may include:

[0061] S101, obtaining a plot text, where the plot text is used to represent the plot summary information of the original video;

[0062] S102, splitting the plot text to obtain segmented plot texts;

[0063] S103, for each segmented plot text, determining the video segment in the original video corresponding to the segmented plot text, and using the segmented plot text as the commentary text, and converting the commentary text into voice by means of TTS to obtain the commentary voice corresponding to the segmented plot text;

[0064] S104, extracting the video pictures in the video segment as the commentary video pictures corresponding to the segmented plot text, and using the commentary voice corresponding to the segmented plot text as the commentary voice corresponding to the commentary video pictures;

[0065] S105, synthesizing the commentary video pictures and commentary voices respectively corresponding to each segmented plot text according to the segmentation time sequence of the plot text splitting to obtain the target video.

[0066] In an embodiment of the present invention, by obtaining a plot text for representing the plot summary information of the original video; splitting the plot text to obtain segmented plot texts; for each segmented plot text, determining a video segment in the original video corresponding to the segmented plot text, and using the segmented plot text as an explanatory text, and converting the explanatory text into speech by means of TTS to obtain the explanatory speech corresponding to the segmented plot text; extracting the video frames in the video segment as the explanatory video frames corresponding to the segmented plot text, and using the explanatory speech corresponding to the segmented plot text as the explanatory speech corresponding to the explanatory video frames. In this way, according to the segmentation time sequence of the plot text, the explanatory video frames and the explanatory speech respectively corresponding to each segmented plot text can be synthesized to obtain the target video, realizing the automatic generation of the video, improving the efficiency of generating the video, and achieving the efficient video generation.

[0067] In S101, according to the name of the TV series, the plot copywriting introduction of the corresponding episode can be crawled from the film and television plot introduction website as the plot text.

[0068] In S102, through natural language processing (NLP) plot segmentation or chatbot model (Chat Generative Pre-trained Transformer, chatgpt) technology, the entire long video plot copywriting can be split into multiple small segment plot copywritings, for example, 3-4 small segment plot copywritings. That is, the plot text is segmented to obtain segmented plot texts.

[0069] Most of the crawled plot texts are separated by commas, and it is necessary to judge whether the upper and lower sentences describe the content of a complete sentence. The text NLP sentence segmentation model or chatgpt is used to segment the overall plot copywriting to complete the initial sentence segmentation process. In an embodiment of the present invention, a segmented plot text may include multiple sentences of plot text, and in actual application, each sentence of plot text can be processed.

[0070] In S103, the original video may be a film and television drama series, such as the video corresponding to one episode of a TV drama or the video corresponding to a movie. Compared with the target video to be generated, the original video can be understood as a long video, and the target video to be generated can be understood as a short video.

[0071] For each segmented plot text, the video segment in the original video corresponding to the segmented plot text can be determined by determining the starting time of the video corresponding to the segmented plot text.

[0072] In one possible implementation, each segmented plot text and the original video may be input into a plot retrieval model, and the corresponding video start time of each segmented plot text may be retrieved to obtain the video clip corresponding to the segmented plot text.

[0073] For each segmented plot text, the segmented plot text and the original video can be input into the plot retrieval model, and the video start time corresponding to the segmented plot text in the original video can be obtained through the plot retrieval model. The video start time can be used to extract the corresponding video clip from the original video as the video clip corresponding to the segmented plot text.

[0074] In another possible implementation, while inputting the segmented plot texts and original videos into the plot retrieval model, the content information of the dialogue speakers can also be input into the plot retrieval model.

[0075] For example, the segmented plot text description (i.e., segmented plot text), long video (i.e., original video), and speaker content information are input into the plot retrieval model to retrieve the corresponding video start time of each segmented plot text as the video clip corresponding to the segmented plot text.

[0076] In one possible implementation, a segmented plot text may include multiple plot texts. In actual application, each plot text may be processed, and the corresponding video start time of each plot text may be retrieved as the video content corresponding to the plot text.

[0077] For each short plot text, according to each plot description in it, such as "X1 was grounded by X2, but still wanted to go out." In this text, X1 and X2 are two different character names, use TTS to generate the content voice with the required tone, and save it in audio formats such as wav or mp3. Among them, a short plot text is also a segmented plot text, and each plot description in a short plot text is also each plot text in a segmented plot text.

[0078] When performing plot retrieval on a segment of plot text (such as a query plot sentence), the plot can be matched based on the visual image, or the plot can be matched based on the semantic understanding of the dialogue content of the character's lines, or a combination of the two (video images combined with semantic understanding to match the plot).

[0079] Among them, the content information of the speaker of the lines can be used to assist plot retrieval, and plot retrieval is to determine the video start time corresponding to the segmented plot text.

[0080] The speaker content information of the lines may include the dialogues of the characters in the plot, for example:

[0081] Character name A: xxxxx (lines)

[0082] Character name B: XXXX

[0083] Character name A: XXXXX

[0084] According to the dialogues of the characters in the plot, it can assist in understanding what this video content is about and better help with plot retrieval.

[0085] Among them, a plot retrieval model can be trained in advance based on multiple training samples. A training sample pair can include multiple sample plot texts and sample videos, and video segment tags or video start time period tags in the sample videos corresponding to each sample plot text, or it can also include the content information of the line speaker. In this way, when determining the video start time period corresponding to each segmented plot text, each segmented plot text and the original video can be input into the plot retrieval model, or each segmented plot text, the original video, and the content information of the line speaker can be input into the trained plot retrieval model to obtain the video segments or video start times corresponding to each segmented plot text.

[0086] For each segmented plot text, the segmented plot text can be converted from text to speech through a TTS model, and the obtained speech content is the explanatory speech corresponding to the segmented plot text. The explanatory speech can be in audio formats such as waveform sound files (wav) or mp3 format (Moving Picture Experts Group Audio Layer 3).

[0087] In one implementable way, for each segmented plot text, the video segments in the original video corresponding to each plot text in the segmented plot text can be determined, and the explanatory speech corresponding to each plot text can be generated. Among them, the video frame of the video segment corresponding to a plot text is used as the explanatory video frame corresponding to the plot text, and the explanatory speech corresponding to the plot text is the explanatory speech corresponding to the explanatory video frame.

[0088] In S104, after determining the video segments in the original video corresponding to the segmented plot text and obtaining the explanatory speech corresponding to the segmented plot text for the segmented plot text, the video frames in the video segments can be extracted as the explanatory video frames corresponding to the segmented plot text, and the explanatory speech corresponding to the segmented plot text can be used as the explanatory speech corresponding to the explanatory video frames.

[0089] In S105, the segmentation time sequence, that is, the segmentation time sequence for segmenting the plot text in S102 to obtain the segmented plot text. The segmentation time sequence can also be understood as the plot order of the copywriting, specifically the order of each segmented plot text in the plot text.

[0090] In an alternative embodiment, as Figure 2 shown, before S105, the video generation method provided by the embodiments of the present invention may further include:

[0091] S201, using the method of analyzing plot coherence to determine the key plot point statements corresponding to the segmented plot text, or using the ChatGPT large model to extract the key plot point statements corresponding to the segmented plot text.

[0092] Among them, using the method of analyzing plot coherence to determine the key plot point statements corresponding to the plot text includes:

[0093] Sequentially select a sentence in the segmented plot text as a candidate content; for each candidate content, delete the candidate content from the segmented plot text, judge whether it affects the smoothness of understanding the semantic content and the degree of influence, and determine the corresponding score based on whether it affects the smoothness of understanding the semantic content and the degree of influence; based on the scores of each candidate content, select the candidate content ranked in the front according to the score order as the selected key plot point statements.

[0094] It can be understood that the higher the score, the more important the candidate content is. In this way, based on the scores of each candidate content, select the candidate content with a higher score according to the score order as the selected key plot point statements. In this way, according to the candidate content and the corresponding scores, sort by score and select the top N highlight statements. For example, select a highlight statement "This is called XX. I can't stand it anymore and directly shout at the storyteller." "XX" is the character name.

[0095] For the ChatGPT method, that is, using the ChatGPT large model to extract the key plot point statements.

[0096] Briefly understood, the important, key, and highlight plot statements can be selected by using plot coherence analysis or ChatGPT, that is, the key plot point statements are extracted as the points to be inserted with the original sound.

[0097] S202, determine the original content to be inserted based on the key plot point statements. The original content is the video content extracted from the original video based on the key plot point statements, including the original picture and the original audio.

[0098] The video content corresponding to the key plot point statements in the original video can be determined through the plot retrieval model according to the key plot point statements. This video content can be understood as the original content to be inserted.

[0099] In one implementation, the key plot point statements and the original video are input into the plot retrieval model. The starting time of the video corresponding to the key plot point statements in the original video is obtained through the plot retrieval model, and the corresponding video segment is extracted from the original video using the starting time of the video as the video content extracted from the original video based on the key plot point statements.

[0100] In one implementation, based on the determined key plot points (which can also be understood as key plot point statements) and the corresponding video intervals (the video content corresponding to the key plot point statements, that is, the video segments), combined with the face information, line speakers, scene location information, and the close correlation between the lines and the plot content within the corresponding video intervals, within the corresponding video intervals, split out M (for example, M < 5s) seconds of original content as the content to be inserted with the original sound, that is, the original content to be inserted.

[0101] After determining the video segment corresponding to the key plot point statement according to the determined key plot points, considering the temporal integrity of the video content, combined with the scene location information and the close correlation between the lines and the plot content; considering the matching of the plot content, considering the face information and line speakers within the corresponding interval, based on this, extract the original content of a preset duration from the video interval. The preset duration can be determined according to actual needs, etc.

[0102] After determining the video segment corresponding to the key plot point statement (which can be understood as the key video segment), for example, there are multiple shots in this segment. When splitting M seconds, the shots cannot be chopped up, and the temporal integrity needs to be considered. At this time, the time point information of the scene and the shots needs to be considered, etc.

[0103] After determining the video segment corresponding to the key plot point statement (which can be understood as the key video segment), the information of the face and the speaker can also be considered. For example, this plot content: It is a fierce argument between character A and character B about divorce and child custody. The split video needs to be the picture content containing the semantic dialogue of the relevant characters' lines and be inserted as the original sound.

[0104] In this case, S105 may include:

[0105] S1051, according to the splitting time sequence, splice the explanatory video pictures and the original pictures corresponding to each segmented plot text to obtain a video track, splice the explanatory voices corresponding to the explanatory video pictures and the original audio corresponding to the original pictures corresponding to each segmented plot text to obtain an audio track; synthesize the video track and the audio track to obtain the target video.

[0106] The original picture can be understood as the video track without sound in the above original content.

[0107] In one implementation, during the process of splicing the explanatory video images corresponding to each segmented plot text and the original images in the original content corresponding to the key plot point statements, if a segmented plot text is a key plot point statement, then during the splicing process, the explanatory video image corresponding to this segmented plot text can be replaced with this original image, and then, according to the segmentation time sequence, the explanatory video images and original images corresponding to each segmented plot text are spliced respectively.

[0108] The original audio can be understood as the audio track without images in the above-mentioned original content.

[0109] In one implementation, during the process of splicing the explanatory speech corresponding to the explanatory video images corresponding to each segmented plot text and the original audio corresponding to the original images, if a segmented plot text is a key plot point statement, then during the splicing process, the explanatory speech corresponding to this segmented plot text can be superimposed on this original audio to obtain the audio track corresponding to this segmented plot text, and then, according to the segmentation order, the explanatory speech corresponding to the explanatory video images and the original audio corresponding to the original images corresponding to each segmented plot text are spliced respectively.

[0110] The method of synthesizing the video track and the audio track refers to the method of synthesizing the image and the audio in the related technology to obtain the video, and the embodiments of the present invention do not limit this.

[0111] In an optional embodiment, the original content further includes;

[0112] Such as Figure 3 As shown, S202 includes S2021, determining the original content to be inserted based on the key plot point statements, and the original content is the video content extracted from the original video based on the key plot point statements, including the original image, the original audio, and the original text.

[0113] According to the segmentation time sequence of the plot text segmentation, synthesizing the explanatory video images and the explanatory speech corresponding to each segmented plot text respectively to obtain the target video, including:

[0114] S1052: According to the segmentation time sequence, splicing the explanatory video images and the original images corresponding to each segmented plot text respectively to obtain the video track, splicing the explanatory speech corresponding to the explanatory video images and the original audio corresponding to the original images corresponding to each segmented plot text respectively to obtain the audio track, and splicing each segmented plot text and the original text respectively to obtain the subtitle track;

[0115] Synthesizing the video track and the audio track to obtain the target video, including:

[0116] Synthesizing the video track, the audio track, and the subtitle track to obtain the target video.

[0117] Among them, the original text can be the lines in the video content corresponding to the key plot points in the original video, specifically, it can be subtitles.

[0118] Based on the above embodiments, a subtitle track is also synthesized, so that the target video includes subtitles.

[0119] In an optional embodiment, as Figure 4 shown, after determining the video segments corresponding to the segmented plot text in the original video and using the segmented plot text as the commentary text, and converting the commentary text into speech in the TTS manner to obtain the commentary speech corresponding to the segmented plot text, the following steps are further included:

[0120] S401: For each segmented plot text, align the video segment with the commentary speech.

[0121] And, perform steps S104 and S105 on the aligned video segment and commentary speech.

[0122] In an implementable manner, the alignment of the video segment and the commentary speech can be achieved in the following way:

[0123] If the duration of the video segment is greater than the duration of the commentary speech, then contract the video segment to the same duration as the commentary speech;

[0124] If the duration of the video segment is less than the duration of the commentary speech, then expand the video segment to the same duration as the commentary speech.

[0125] The duration of the commentary speech is S seconds, and the length of the video segment corresponding to this sentence of commentary speech is T seconds. If T == S, the duration of this sentence of commentary and the video segment are exactly the same, and there is no need to crop or expand the video interval segment, which is the same as the duration of the commentary;

[0126] If T > S, that is, the duration of the video segment is greater than the duration of the commentary speech, it can also be understood that the duration of the video picture is greater than the duration of the commentary speech. At this time, it is necessary to perform iterative contraction on the duration of the video picture until it is contracted to S seconds, which is the same as the duration of the commentary speech, and then it ends.

[0127] Regarding the way of iterative contraction:

[0128] 1. First, calculate T - S, that is, the duration by which T is longer than S, which can also be understood as the duration that needs to be cut off;

[0129] 2. Divide T into the first preset number n1 of small segments, where n1 can be determined according to actual needs;

[0130] 3. Calculate the duration m1 that needs to be cut off for each small segment, that is, m1 = (T - S) / n1;

[0131] For example, for the above video clip, the starting time point is A→B, and the duration is 7s, that is, T = 7s; now only 5s is needed, that is, the duration of the commentary voice S = 5s, then 2s needs to be cut off, that is, it is calculated that T - S = 2s, and T = 7s is divided into 10 small segments, that is, n1 = 10, then each small segment needs to be cut off 2s / 10 = 0.2s on average.

[0132] 4. Cut off m1 from each of the n1 small segments. Specifically, m1 in each small segment can be selected according to actual needs. For example, the first m1 duration, the last m1 duration, etc. of each small segment are all acceptable;

[0133] 5. Splice the n1 small segments after cutting off m1 obtained in step 4 in the order before cutting, that is, the iteratively shrunk video clip is obtained.

[0134] If T < S, that is, the duration of the video clip is less than the duration of the commentary voice, it can also be understood that the duration of the video picture is less than the duration of the commentary voice. At this time, the video picture needs to be iteratively expanded to S seconds, which is the same as the duration of the commentary voice, and then it ends.

[0135] The process of iterative supplementation can be understood as the reverse process of the above iterative contraction, which is specifically as follows:

[0136] 1. First calculate S - T, that is, the duration by which S is longer than T, which can also be understood as the duration that needs to be expanded;

[0137] 2. Cut T into a second preset number n2 of small segments, where n2 can be determined according to actual needs;

[0138] 3. Calculate the duration m2 that each small segment needs to be expanded, that is, m2 = (S - T) / n2;

[0139] For example, for the above video clip, the starting time point is C→D, and the duration is 5s, that is, T = 5s; now 7S is needed, that is, the duration of the commentary voice S = 7S, then 2s needs to be expanded, that is, it is calculated that S - T = 2s, and T = 5s is divided into 10 small segments, that is, n2 = 10, then each small segment needs to be expanded 2s / 10 = 0.2s on average.

[0140] 4. Expand each of the n2 small segments by m2. The m2 expanded for each small segment can be selected according to actual needs. For example, select the first m2 duration of a small segment as the expanded m2.

[0141] 5. Splice the n2 small segments after expanding m2 obtained in step 4 in the order before cutting, that is, the iteratively expanded video clip is obtained.

[0142] In another implementable manner, the video clip and the commentary voice can also be aligned in the following way:

[0143] Keep the duration of the video clip unchanged, and adjust the playback speed of the commentary voice based on the ratio between the duration of the video clip and the duration of the commentary voice. The playback speed is used to ensure that the duration of playing the commentary voice is the same as the duration of the video clip.

[0144] For example, if the video clip is 5 seconds and the commentary voice is 10s, then the playback speed of the commentary voice can be calculated as 5s / 10s = 0.5. Simply put, play the commentary voice at 0.5 times the speed.

[0145] If the video clip is 10 seconds and the commentary voice is 5s, then the playback speed of the commentary voice can be calculated as 10s / 5s = 2. Simply put, play the commentary voice at 2 times the speed.

[0146] In an alternative embodiment, as Figure 5 shown, before aligning the video clip and the commentary voice for each segmented plot text, it may further include:

[0147] S501: If the segmented plot text is a key plot point statement, then remove the content from the time point corresponding to the key plot point statement to the end of the video clip in the video clip to obtain the video clip after removal.

[0148] S401 specifically includes:

[0149] S4011: For each segmented plot text, align the video clip after removal and the commentary voice.

[0150] Simply put, if the corresponding commentary copy is exactly the time period when the original sound needs to be inserted for the key plot, in the current video frame, after removing all the content from the start time point of the corresponding original sound to the end of this video segment, then perform the above processing of aligning the duration of the commentary voice and the video frame.

[0151] In the embodiment of the present invention, splice each sentence of the above plot text corresponding video frames (the above commentary video frames) + inserted original frames into a complete video track in the order of splitting of the commentary copy (that is, the sequence of each segmented plot text in the plot copy text); according to the starting time of the voice corresponding to the above commentary frames and inserted original frames, splice the commentary voice and the inserted original sound (the original audio corresponding to the original frames) into a complete audio track in the order of the commentary copy; according to the voice text corresponding to the above commentary video frames (that is, each sentence of the plot text in each segmented plot text) and the lines corresponding to the inserted original frames (original text), splice them into a complete subtitle track in the order of the commentary copy; combine the above video track, audio track, and subtitle track into one to produce the short video creation material with commentary + original sound inserted at key plot points.

[0152] If background music is needed, for example, in a suspense movie to set off the suspenseful atmosphere, a little background audio of the suspense genre with a slightly lower volume can be added on the audio track.

[0153] If you also want to shorten the video duration and capture users' time, without affecting the viewing experience, you can slightly speed up the overall short video creation materials. For example: for an original 60s material video, when played at 1.2 times the speed, the video can be compressed to 50s, with a faster rhythm.

[0154] In this way, the embodiments of the present invention can realize short video content of the plot type that automatically completes the interspersed explanation + original sound according to the plot copywriting.

[0155] According to the automatically crawled plot summary, the original plot summary is first segmented into several small segments of the plot (the above segmented plot text) using a plot segmentation module or ChatGPT to form the initial segmented plot copywriting commentary. At the same time, analyze the most important or relatively exciting key plot point sentences in each small segment of the plot content. For each segmented small segment of the plot, use the plot retrieval model to locate the start time of the video content corresponding to each sentence of the plot copywriting. Then, for each sentence of the plot content, use the TTS text-to-speech function to complete the explanatory voice content, and align the voice explanation duration with the corresponding video content duration. Secondly, at the key plot points and within the time period of the video content corresponding to this sentence of the plot copywriting, implant the original video content around information such as the plot characters, line dialogue content, and line speakers. Finally, splice these series of small segments together in the order of the plot copywriting to form the initial material content, and finally add background sound as the final short video content material output.

[0156] The embodiments of the present invention can efficiently and automatically produce short video content materials of the plot type with explanatory and original sound interspersed for long videos of movies and TV dramas, automatically generate voice explanations for the commentary text, erase the original video subtitles, add subtitles of a specified style, add background music, and complete the final output.

[0157] As a form of manifestation of using artificial intelligence technology to generate content (AI Generated Content, AIGC) for creation materials, automated production can greatly reduce labor costs and efficiently produce short video content, which can greatly enrich the content manifestation form of the materials. It plays an important role in distribution, recommendation, drainage, and planting grass, thereby having a very positive and positive effect on increasing the viewing duration of long videos, consuming long video content, and watching more good dramas, and promoting the creation of better and high-quality video content on long video platforms.

[0158] The embodiments of the present invention also provide a video generation device, as Figure 6 shown, which may include:

[0159] An acquisition module 601, configured to acquire a plot text, where the plot text is used to represent the plot summary information of the original video;

[0160] A splitting module 602, configured to split the plot text to obtain segmented plot texts;

[0161] A first determination module 603, configured to, for each segmented plot text, determine a video segment in the original video corresponding to the segmented plot text, and use the segmented plot text as an explanatory text, and convert the explanatory text into speech in a text-to-speech (TTS) manner to obtain an explanatory speech corresponding to the segmented plot text; extract the video frame in the video segment as the explanatory video frame corresponding to the segmented plot text, and use the explanatory speech corresponding to the segmented plot text as the explanatory speech corresponding to the explanatory video frame;

[0162] A synthesis module 604, configured to synthesize the explanatory video frames and explanatory speeches respectively corresponding to each segmented plot text in the chronological order of splitting the plot text to obtain a target video.

[0163] Optionally, the first determination module 603 is specifically configured to input the segmented plot text and the original video into a plot retrieval model, obtain the video start time corresponding to the segmented plot text in the original video through the plot retrieval model, and use the video start time to extract the corresponding video segment from the original video as the video segment corresponding to the segmented plot text, where the plot retrieval model is pre-trained according to multiple training samples, and a training sample pair includes multiple sample plot texts and sample videos, and the video start time period labels in the sample videos corresponding to each sample plot text.

[0164] Optionally, the apparatus further includes:

[0165] A second determination module, configured to, before synthesizing the explanatory video frames and explanatory speeches respectively corresponding to each segmented plot text in the chronological order of splitting the plot text to obtain a target video, use the method of plot coherence analysis to determine the key plot point statements corresponding to the segmented plot text, or use the ChatGPT large model to extract the key plot point statements corresponding to the segmented plot text; determine the original content to be inserted based on the key plot point statements, where the original content is the video content extracted from the original video based on the key plot point statements, including original frames and original audio;

[0166] The synthesis module 604 is specifically configured to, in the chronological order of splitting, splice the explanatory video frames and original frames respectively corresponding to each segmented plot text to obtain a video track, splice the explanatory speeches corresponding to the explanatory video frames and the original audio corresponding to the original frames respectively corresponding to each segmented plot text to obtain an audio track; synthesize the video track and the audio track to obtain a target video.

[0167] Optionally, the second determination module is specifically configured to sequentially select a sentence in the segmented plot text as a candidate content; for each candidate content, delete the candidate content from the segmented plot text, determine whether it affects the smoothness of understanding the semantic content and the degree of influence, and determine the corresponding score based on whether it affects the smoothness of understanding the semantic content and the degree of influence; based on the scores of each candidate content, select the candidate content ranked at the front in the order of scores as the key plot point sentences selected.

[0168] Optionally, the original content further includes original text;

[0169] The synthesis module 604 is specifically configured to splice the explanatory video pictures and the original pictures corresponding to each segmented plot text in sequence of segmentation to obtain a video track, splice the explanatory voice corresponding to the explanatory video pictures and the original audio corresponding to the original pictures corresponding to each segmented plot text to obtain an audio track, and splice each segmented plot text and the original text to obtain a subtitle track; synthesize the video track, the audio track and the subtitle track to obtain the target video.

[0170] Optionally, the device further includes:

[0171] The alignment module is configured to, after determining the video segment corresponding to the segmented plot text in the original video and using the segmented plot text as the explanatory text to convert the explanatory text into voice by means of text-to-speech (TTS) to obtain the explanatory voice corresponding to the segmented plot text, align the video segment with the explanatory voice for each segmented plot text.

[0172] Optionally, the alignment module is specifically configured to, if the duration of the video segment is greater than the duration of the explanatory voice, shrink the video segment to the same duration as the explanatory voice; if the duration of the video segment is less than the duration of the explanatory voice, expand the video segment to the same duration as the explanatory voice.

[0173] Optionally, the alignment module is specifically configured to keep the duration of the video segment unchanged and adjust the playback speed of the explanatory voice based on the ratio between the duration of the video segment and the duration of the explanatory voice, and the playback speed is used to ensure that the duration of playing the explanatory voice is the same as the duration of the video segment.

[0174] Optionally, the device further includes:

[0175] The removal module is configured to, before aligning the video segment with the explanatory voice for each segmented plot text, if the segmented plot text is a key plot point sentence, remove the content from the time point corresponding to the key plot point sentence to the end of the video segment in the video segment to obtain the video segment after removal;

[0176] An alignment module, specifically used to align the removed video segments with the commentary voice for each segmented plot text.

[0177] An embodiment of the present invention also provides an electronic device, as Figure 7 shown, including a processor 701, a communication interface 702, a memory 703, and a communication bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0178] The memory 703 is used to store a computer program;

[0179] The processor 701, when executing the program stored on the memory 703, implements the method steps of the above video generation method.

[0180] The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0181] The communication interface is used for communication between the above terminal and other devices.

[0182] The memory may include a Random Access Memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0183] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0184] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the video generation method described in any one of the above embodiments is implemented.

[0185] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a target video is stored, and the target video is obtained by implementing the method steps of the video generation method described in any one of the above embodiments.

[0186] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, and when it runs on a computer, it causes the computer to execute the video generation method described in any one of the above embodiments.

[0187] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state disk (SSD)).

[0188] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0189] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer-readable storage medium and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the partial description of the method embodiments for the relevant parts.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A video generation method, characterized in that: include: Acquire a plot text, where the plot text is used to represent plot summary information of the original video; Segment the plot text to obtain segmented plot text; For each segmented plot text, determine a video segment in the original video corresponding to the segmented plot text, use the segmented plot text as a commentary text, and convert the commentary text into speech using a text-to-speech (TTS) method to obtain a commentary speech corresponding to the segmented plot text; Extracting the video screen in the video clip as the narration video screen corresponding to the segmented plot text, and using the narration voice corresponding to the segmented plot text as the narration voice corresponding to the narration video screen; According to the segmentation sequence of the plot text, the narration video images and the narration voices corresponding to each segmented plot text are synthesized to obtain a target video.

2. The method according to claim 1, characterized in that Determining a video segment in the original video corresponding to the segmented plot text includes: The segmented plot text and the original video are input into a plot retrieval model, and the video start time corresponding to the segmented plot text in the original video is obtained through the plot retrieval model, and the corresponding video segment is extracted from the original video using the video start time as the video segment corresponding to the segmented plot text, wherein the plot retrieval model is pre-trained based on multiple training samples, and a training sample pair includes multiple sample plot texts and sample videos, as well as video start time segment labels in sample videos corresponding to each sample plot text.

3. The method according to claim 1, characterized in that Before synthesizing the narration video images and the narration voices corresponding to each segmented plot text according to the segmentation sequence of the plot text to obtain the target video, the method further includes: Determine the key plot point sentences corresponding to the segmented plot text by using the plot coherence analysis method, or extract the key plot point sentences corresponding to the segmented plot text by using the chatgpt large model; Determining native content to be inserted based on the key plot point sentence, wherein the native content is video content extracted from the original video based on the key plot point sentence, including native images and native audio; The method of synthesizing the narration video image and the narration voice corresponding to each segmented plot text according to the segmentation time sequence of the plot text to obtain a target video includes: According to the segmentation sequence, the narration video screen and the native screen corresponding to each segmented plot text are spliced ​​to obtain a video track, and the narration voice corresponding to the narration video screen corresponding to each segmented plot text and the native audio corresponding to the native screen are spliced ​​to obtain an audio track; The video track and the audio track are synthesized to obtain a target video.

4. The method according to claim 3, characterized in that The method of using plot coherence analysis to determine key plot point sentences corresponding to the plot text includes: Selecting one sentence in the segmented plot text in turn as a candidate content; For each candidate content, delete the candidate content from the segmented plot text, determine whether the candidate content affects the fluency of understanding the semantic content and the degree of influence, and determine a corresponding score based on whether the candidate content affects the fluency of understanding the semantic content and the degree of influence; Based on the score of each candidate content, the candidate content ranked first is selected in order of score as the selected key plot point sentence.

5. The method according to claim 3, characterized in that: The native content also includes native text; The method of synthesizing the narration video image and the narration voice corresponding to each segmented plot text according to the segmentation time sequence of the plot text to obtain a target video includes: According to the segmentation sequence, the narration video screen and the native screen corresponding to each segmented plot text are spliced ​​to obtain a video track, the narration voice corresponding to the narration video screen corresponding to each segmented plot text and the native audio corresponding to the native screen are spliced ​​to obtain an audio track, and each segmented plot text and the native text are spliced ​​to obtain a subtitle track; The video track and the audio track are synthesized to obtain a target video, The video track, the audio track and the subtitle track are synthesized to obtain a target video.

6. The method according to claim 1, characterized in that After determining the video segments corresponding to the segmented plot text in the original video, and using the segmented plot text as the commentary text, converting the commentary text into speech using a text-to-speech (TTS) method to obtain the commentary speech corresponding to the segmented plot text, the method further includes: For each segmented plot text, the video clip is aligned with the narration voice.

7. The method according to claim 6, characterized in that The aligning the video clip with the commentary voice includes: If the duration of the video clip is longer than the duration of the commentary voice, shrinking the video clip to be the same as the duration of the commentary voice; If the duration of the video segment is shorter than the duration of the commentary voice, the video segment is extended to be the same as the duration of the commentary voice.

8. The method according to claim 7, characterized in that The aligning the video clip with the commentary voice includes: The duration of the video clip is kept unchanged, and based on the ratio between the duration of the video clip and the duration of the commentary voice, the playback speed of the commentary voice is adjusted, wherein the playback speed is used to ensure that the duration of playing the commentary voice is the same as the duration of the video clip.

9. The method according to claim 6, characterized in that Before aligning the video clip with the narration voice in each segmented plot text, the method further includes: If the segmented plot text is a key plot point sentence, then removing the content from the time point corresponding to the key plot point sentence to the end of the video segment to obtain a post-removal video segment; For each segmented plot text, aligning the video clip with the commentary voice includes: For each segmented plot text, the removed video segment is aligned with the narration voice.

10. A video generating device, characterized in that: include: An acquisition module, used to acquire a plot text, where the plot text is used to represent plot summary information of an original video; A splitting module is used to segment the plot text to obtain segmented plot text; The first determination module is used to determine, for each segmented plot text, a video segment in the original video corresponding to the segmented plot text, and use the segmented plot text as an explanation text, and convert the explanation text into speech by using a text-to-speech (TTS) method to obtain an explanation speech corresponding to the segmented plot text; extract a video screen in the video segment as an explanation video screen corresponding to the segmented plot text, and use the explanation speech corresponding to the segmented plot text as an explanation speech corresponding to the explanation video screen; The synthesis module is used to synthesize the narration video images and the narration voices corresponding to each segmented plot text according to the segmentation sequence of the plot text to obtain a target video.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a target video, and the target video is obtained by implementing the method steps of any one of claims 1 to 9.

Images