Video generation method and related device

Abstract

The invention discloses a video generation method and a related device, and relates to the field of video processing, and the method comprises the steps: obtaining a first video; wherein the first video is a to-be-edited video, the first video is edited to obtain a second video, the first video has a plurality of plot lines, and the plot lines are main lines or branch lines; generating a description text set based on the first video; wherein the description text set comprises a plurality of plot line description texts, and each plot line description text describes one plot line; selecting one plot line description text from the description text set as a target text; searching a video indicated by the target text from the first video to obtain a second video; wherein the second video comprises the video content of a story line. According to the invention, the probability of plot chaos can be reduced.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video generation method and related apparatus. Background Technology

[0002] In today's fast-paced lifestyle, many viewers don't have enough time to watch long, complete videos, but still want to quickly grasp the content. Therefore, how to generate shorter videos from longer ones has become a major concern. Currently, one method uses a brief video synopsis as query text, searching for matching video segments within the complete video. Another method matches videos based on the video's complete plot description. Both methods are prone to producing videos with confusing storylines. Summary of the Invention

[0003] In view of the above problems, this application provides a video generation method and related apparatus, which can reduce the probability of plot confusion. The specific solution is as follows:

[0004] The first aspect of this application provides a video generation method, including:

[0005] Obtain a first video; wherein, the first video is a video to be edited, the first video is edited to obtain a second video, the first video has multiple storylines, the storylines are the main storyline or the sub-storylines;

[0006] Based on the first video, a set of descriptive texts is generated; wherein, the set of descriptive texts includes multiple plot line descriptive texts, and each plot line descriptive text describes one plot line;

[0007] Select one of the plotline description texts from the set of description texts as the target text;

[0008] The second video is obtained by searching for the video indicated by the target text in the first video; wherein the second video includes video content of the plot line.

[0009] In one possible implementation, selecting one of the plotline description texts from the set of description texts as the target text includes:

[0010] Obtain the first editing constraint; wherein, the first editing constraint is a constraint on obtaining the video content included in the target video, and the target video includes one or more of the second videos;

[0011] Texts that meet the first editing constraint conditions are filtered from the set of descriptive texts to obtain filtered texts; wherein, the filtered texts include at least one of the plotline descriptive texts;

[0012] One of the plotline description texts in the filtered text is taken as a target text; wherein the number of target texts is the same as the number of plotline description texts included in the filtered text.

[0013] In one possible implementation, generating a set of descriptive text based on the first video includes:

[0014] Based on the first video, a scene text set is generated; wherein, the scene text set includes multiple scene description texts, and the scene description texts are texts describing the video content of any scene in the first video;

[0015] Obtain the second editing constraint; wherein, the second editing constraint is the constraint for obtaining the video content included in the multiple storylines in the first video;

[0016] Based on the scene text set, a large language model is invoked to generate the descriptive text set that conforms to the second editing constraint.

[0017] In one possible implementation, the first video includes a plurality of sub-videos with identifiers indicating the temporal order in which the sub-videos appear in the first video;

[0018] Based on the first video, a set of descriptive text is generated, including:

[0019] Based on each sub-video, a sub-text set with the aforementioned identifier is generated; wherein, the sub-text set includes at least one of the aforementioned plotline description texts, and the plotline description texts in the sub-text set are texts describing the plotline of the sub-video;

[0020] Arrange the sub-text sets according to the order indicated by the identifiers to generate the descriptive text set;

[0021] Selecting a plotline description text from the set of description texts as the target text includes:

[0022] Select one of the plotline description texts from the set of description texts, determine the identifier of the sub-text set to which the selected plotline description text belongs, and obtain the target text with the identifier;

[0023] The step of finding the video indicated by the target text from the first video to obtain the second video includes:

[0024] Based on the identifier of the target text, determine the sub-video indicated by the target text;

[0025] From the sub-videos indicated by the target text, find the video whose video content represents the target text, and obtain the second video.

[0026] In one possible implementation, the step of searching for the video indicated by the target text in the first video to obtain the second video includes:

[0027] According to the various stages of the development of the first video's plot, the target text is divided into multiple sub-texts with a time sequence; wherein, each sub-text describes the video content of one stage.

[0028] Find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segment in the first video to obtain the segment list of the subtext; wherein, the segment description information is the video content information describing a video segment of the first video, and the semantic relevance represents the similarity between the segment description information and the subtext;

[0029] The fragment lists of each subtext are merged according to the time sequence, and the video fragments pointed to by the description information of each merged fragment are merged to obtain the second video.

[0030] In one possible implementation, generating a set of scene texts based on the first video includes:

[0031] The first video is subjected to segment boundary detection to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point;

[0032] Extract video frames before and after the segment boundary from the first video;

[0033] Based on the video frames, scene switching points are detected on the boundaries of the segments, and the segments belonging to the scene switching points are taken as target boundaries; wherein, the scene switching point is the boundary point between one scene and another scene;

[0034] The first video is divided into multiple scenes based on the target boundary;

[0035] Generate video content information describing each of the aforementioned scenes to obtain the scene text set.

[0036] A second aspect of this application provides a video generation system, comprising:

[0037] The video acquisition module is used to acquire a first video; wherein the first video is a video to be edited, and the first video is edited to obtain a second video, and the first video has multiple plot lines, wherein the plot lines are the main plot or the subplots.

[0038] A text generation module is used to generate a set of descriptive texts based on the first video; wherein, the set of descriptive texts includes multiple plot line descriptive texts, and each plot line descriptive text describes one plot line;

[0039] The selection module is used to select one of the plotline description texts from the set of description texts as the target text;

[0040] A video generation module is used to find the video indicated by the target text in the first video to obtain a second video; wherein the second video includes video content of the plot line.

[0041] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video generation method described in the first aspect or any implementation thereof.

[0042] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0043] The memory is used to store computer programs;

[0044] The processor is used to execute the computer program so that the electronic device can implement the video generation method of the first aspect or any implementation thereof.

[0045] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the video generation method described in the first aspect or any implementation thereof.

[0046] By employing the above technical solution, the video generation method and related apparatus provided in this application acquire a first video with multiple storylines; based on the first video, generate a set of descriptive texts, the set of descriptive texts including multiple storyline descriptive texts, each storyline descriptive text describing one storyline; select a storyline descriptive text from the set of descriptive texts as a target text; search for the video indicated by the target text in the first video to obtain a second video, the second video including video content of one storyline. This application generates multiple storyline descriptive texts from the first video, each storyline descriptive text describing only one storyline, and selects one storyline descriptive text to generate the second video, thus realizing the second video edited from the first video describing video content of one storyline. Since a storyline consists of a series of related storylines, it can reduce the probability of video plot confusion caused by multiple storylines intermingling, and improve the quality of the generated video. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0048] Figure 1 A flowchart of a video generation method provided in this application;

[0049] Figure 2 Flowchart of another video generation method provided in this application;

[0050] Figure 3 A structural diagram of a video generation system provided in this application;

[0051] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0052] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0053] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0054] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0055] Currently, one method of video generation uses a concise video synopsis as query text, searching for matching video segments within the complete video. Another method involves matching videos based on a complete plot description. However, neither video synopses nor complete plot descriptions focus on a single storyline, such as a main plot or a subplot. A plotline consists of a series of related events, which can be either main or subplots. These main or subplots outline the development of the story, connecting events and characters in the video in a logical and sequential manner. Video content related to a main or subplot is typically complete, logical, and highly readable. Video synopses and complete plot descriptions, lacking focus on a single storyline, are more prone to plot confusion compared to videos generated from a plotline description.

[0056] Based on this, this application provides a video generation method to solve the above problems.

[0057] Reference Figure 1 , Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this application, such as... Figure 1 As shown in the embodiment of this application, a video generation method may include steps 101 to 104, which are described in detail below.

[0058] Step 101: Obtain the first video; where the first video is the video to be edited, the first video is edited to obtain the second video, the first video has multiple plot lines, the plot lines are the main plot or the side plots.

[0059] The first video is the video to be edited. It includes multiple storylines, which form the framework and structure of the plot. These storylines connect events, character actions, and emotional changes in the video according to a certain logic and sequence, guiding the viewer's understanding of the content. The first video can be a TV series or movie, which often includes multiple storylines. If it's a TV series, the first video could be one episode, multiple episodes, or the entire series. Storylines can be main or subplots. The main storyline is the dominant or overarching line, such as a plotline related to the main character or a major event. Subplots are auxiliary tasks designed around the main storyline, usually supplementing it. They can be plotlines related to supporting characters or minor events. Of course, the main storyline can also refer to the essential core plot chain or theme that drives the story forward, such as the story's central mystery or conflict. Side quests can also be secondary plots that unfold around the main storyline to enrich the events and characters. For example, a side quest could be an adventure, a coming-of-age story, or a romantic story. Events within the main storyline can be events that contribute to character development, plot twists, or conflicts.

[0060] Step 102: Based on the first video, generate a set of descriptive texts; wherein, the set of descriptive texts includes multiple plot line descriptive texts, and each plot line descriptive text describes a plot line.

[0061] The first video contains multiple storylines. Multiple storyline description texts are generated based on the first video, with each description text describing one storyline. The multiple storyline description texts in the set can describe all the storylines in the first video, or they can describe only a portion of the storylines. For example, if the first video is a movie, the set of description texts could describe all the movie's storylines, or it could describe only a portion of the movie's storylines. Similarly, if the first video is a TV series with multiple episodes, the set of description texts could describe the storyline of one episode, or it could describe the storyline of all the episodes.

[0062] When generating a set of descriptive text based on a first video, one approach is to extract video frames, then use an image description model to obtain descriptions of the image content from the extracted frames, and finally extract multiple plotline descriptive texts. Another approach is to obtain video dialogue through speech-to-text conversion, and then use a large language model to generate multiple plotline descriptive texts based on the video dialogue.

[0063] In one possible implementation, a set of descriptive text is generated based on the first video, including:

[0064] Based on the first video, a scene text set is generated; wherein, the scene text set includes multiple scene description texts, and the scene description texts are the video content describing any scene in the first video;

[0065] Obtain the second editing constraint; wherein, the second editing constraint is the constraint on obtaining the video content included in multiple storylines in the first video;

[0066] Based on the scene text set, a large language model is invoked to generate a set of descriptive text that meets the second clipping constraints.

[0067] When generating a set of descriptive texts based on the first video, the first video can be first split into scenes to obtain videos of each scene. Then, corresponding scene description texts can be generated for each scene video. Multiple scene description texts can be generated based on the obtained set of all scene description texts, thus obtaining a set of scene texts.

[0068] Then, the second editing constraint is obtained. This second editing constraint is a constraint on the video content included in multiple storylines in the first video. Specifically, it constrains (requires or limits) the video content included in the multiple storylines edited from the first video. This second editing constraint can extract multiple storylines from multiple scene description texts. Calling a large language model can generate multiple storyline description texts, each describing the specific content of one storyline.

[0069] In addition to constraining (requiring or limiting) the video content included in the storyline, the second editing constraint may also include temporal constraints, text description constraints, and text output format constraints. Specifically, the temporal constraint may include arranging the video description text of multiple storylines according to the chronological order in which scenes occur in the video; the text description constraint may include using general language for description, such as: who did what, and what result or state it led to; and the text output format constraint may include outputting the description text of each storyline using a numerical list.

[0070] By calling upon the large language model to understand the scene description text according to each scene in the video, plot description text that meets the second editing constraints can be generated. The powerful text understanding and summarization capabilities of the large language model can be used to refine the complex scene description text and output several concise plot description texts that can contain only the core conflict and development.

[0071] Optionally, based on the first video, a set of scene text is generated, including:

[0072] Perform segment boundary detection on the first video to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point;

[0073] Extract video frames before and after the segment boundaries from the first video;

[0074] Scene transition point detection is performed on the boundaries of video segments based on video frames, and the boundaries of segments belonging to scene transition points are taken as target boundaries; where a scene transition point is the boundary point between one scene and another scene.

[0075] The first video is divided into multiple scenes based on the target boundaries;

[0076] Generate video content information describing each scene to obtain a set of scene texts.

[0077] When performing segment boundary detection on the first video, a shot detection algorithm can be used to obtain the timestamps of the segment boundaries in the first video. Based on the start and end times of the segment boundary timestamps, the first video can be divided into multiple segments, each segment representing a shot. After obtaining the segment boundaries of each segment in the first video, one or more video frames before and after the segment boundaries can be extracted. For example, N video frames before the segment boundaries and M video frames after the segment boundaries can be extracted. N and M can be the same or different.

[0078] After obtaining each video frame by identifying the boundaries of each segment, it can be determined whether these segments belong to the same scene. Segments of the same scene are aggregated to form a single scene, and then the video content information of this scene is described to obtain scene description information. When detecting scene transition points based on video frames and segment boundaries, scene detection algorithms can be used to find the segment boundaries of scene transition points, obtaining target boundaries. These target boundaries are then used to divide the first video into multiple scenes. The scene transition point is the boundary between one scene and another. This boundary can be the end point of the last segment of the previous scene or the start point of the first segment of the next scene. When generating video content information describing each scene, a multimodal large language model can be used to process the video frames and dialogue information of the scene to obtain information describing the scene.

[0079] This application decomposes a first video into multiple scenes, each scene including multiple segments, and each segment including multiple image frames. Based on generating a set of descriptions for each segment, it detects the scene to which each segment belongs and obtains scene description information by combining the set of description information of segments belonging to the same scene. This can transform the video from a frame sequence into a narrative scene sequence, so as to generate a set of descriptive text that meets the second editing constraints.

[0080] Step 103: Select a plot description text from the set of description texts as the target text.

[0081] Selecting a single plotline description text from the set of description texts ensures that each text describes only one plotline. Since a plotline is either the main storyline or a subplot, compared to editing videos using a synopsis or a complete plot description (which may contain multiple plotlines and lack logical coherence, leading to a chaotic video narrative), selecting a single plotline description text as the target text reduces the probability of recalling other plotline content, such as numerous irrelevant subplot segments. This avoids issues like a lack of focus in the final video and reduces the likelihood of a chaotic narrative in subsequent videos.

[0082] In one possible implementation, a plotline description text is selected as the target text from a set of description texts, including:

[0083] Obtain the first clip constraint; wherein, the first clip constraint is the constraint on obtaining the video content included in the target video, and the target video includes one or more second videos;

[0084] The selected texts are obtained by filtering the texts that meet the constraints of the first clip from the set of descriptive texts; wherein the selected texts include at least one plotline descriptive text.

[0085] One plotline description text from the filtered text is used as a target text; the number of target texts is the same as the number of plotline description texts included in the filtered text.

[0086] The first editing constraint is a constraint on obtaining the video content included in the target video. The target video includes one or more second videos. The first editing constraint is a constraint on obtaining the video content that the user expects. The first editing constraint corresponds to the type of content of the second video. For example, the first editing constraint can be one or more conditions such as the video content and the protagonist's growth, the plot turning point, or the occurrence of conflict, thereby filtering out the text describing the plot line.

[0087] If the selected text consists of only one plotline description text, then there is also only one target text, and a second video is generated from this single target text. If the selected text consists of multiple plotline description texts, then there are multiple target texts, and the number of target texts is the same as the number of plotline description texts, ensuring that each target text describes one plotline. When there are multiple target texts, each target text generates one second video, but because there are multiple target texts, the number of second videos is also multiple, the same as the number of target texts. Each second video includes the video content of one plotline, and these multiple second videos collectively satisfy the first editing constraint condition.

[0088] When filtering texts that meet the first editing constraint from the descriptive text set, a large language model can be used for filtering. Specifically, based on the descriptive text set and the first editing constraint, a prompt word can be generated to instruct the large language model to filter texts that meet the first editing constraint from multiple plotline descriptive texts. Based on this prompt word, the large language model selects the texts that satisfy the first editing constraint from the descriptive text set.

[0089] By using the first editing constraint as the second video generation condition, the desired video can be generated according to different user needs. This first editing constraint might filter out target videos with multiple storylines. During video generation, all storyline description texts that satisfy this condition are first identified (i.e., the filtered texts). Then, one storyline description text is selected to ensure that the generated second video is a video with a single storyline. Even if the first editing constraint indicates the selection of videos with multiple storylines, a second video related to that storyline will still be generated by selecting one storyline description text. This increases the probability of storyline confusion due to multiple storylines. Step 104 is repeated for each storyline description text to obtain multiple second videos.

[0090] This application uses a single plotline description text as a target text from text that meets the first editing constraint. The number of target texts equals the number of plotline description texts. Each target text generates one second video, and the number of second videos equals the number of target texts. Each second video contains video content based on a single plotline required by the user. Multiple second videos collectively satisfy the user's editing needs. By splitting the user's editing needs into plotlines and generating multiple second videos that meet those needs, this application reduces the probability of plot confusion caused by multiple plotlines being mixed into a single video.

[0091] Step 104: Locate the video indicated by the target text in the first video to obtain the second video; wherein, the second video includes video content with a storyline.

[0092] To find the video indicated by the target text in the first video, one approach is to use a cross-modal retrieval model to match text with video. In practice, the target text and the first video can be encoded into high-dimensional feature vectors, and then the similarity between the text vector and the vectors of each video segment can be calculated in the same feature space to find the most matching video segment, thus obtaining the second video. Optionally, the cross-modal retrieval model can be based on a pre-trained model such as CLIP (Contrastive Language-Image Pre-Training). Another approach is to use a large language model to match text with video, where the target text is input into the large language model as prompt words to find the second video.

[0093] If the first editing constraint is to obtain the video content included in multiple second videos, then after obtaining one second video, another plot description text can be selected from the filtered text, and then the video indicated by the text can be found in the first video to obtain another second video, until multiple second videos in the target video are generated to meet the requirements of the first editing constraint.

[0094] This application generates multiple plotline description texts from a first video, with each plotline description text describing only one plotline. A second video is generated by selecting one plotline description text. This ensures that the second video, edited from the first video, describes as much content as possible from a single plotline, reducing the recall of a large amount of video content from other plotlines unrelated to that single plotline. Compared to directly matching the target video with a plot summary or a complete plot description, which can recall a large amount of content unrelated to a single plotline and lead to a chaotic and unfocused video plot, this application selects a single plotline description text as the target text and generates a second video based on this text that describes only one plotline. This ensures that the generated second video is as relevant to one plotline as possible and unrelated to other plotlines, reducing the probability of the generated video being mixed with multiple plotlines and causing plot confusion, thus improving the quality of the generated video.

[0095] In one possible implementation, the second video is obtained by searching for the video to which the target text is indicated in the first video, including:

[0096] Based on the various stages of the first video's plot development, the target text is broken down into multiple sub-texts with a temporal sequence; each sub-text describes the video content of one stage.

[0097] Find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segments in the first video to obtain the segment list of the subtext; where the segment description information is the video content information describing a video segment of the first video, and semantic relevance represents the similarity between the segment description information and the subtext.

[0098] The fragment lists of each subtext are merged in chronological order, and the video segments pointed to by the description information of each merged fragment are combined to obtain the second video.

[0099] This application addresses the issue that the first video's plot description, which includes multiple plot lines such as the main storyline and subplots, lacks a single plot line to focus on. Before obtaining the second video by searching for the video indicated by the target text in the first video, the application first generates target text describing each plot line. Then, it splits the target text according to the various stages of the first video's plot development, generates a list of segments for each subtext, and finally merges the video segments pointed to by the segment description information according to the time sequence to obtain the second video.

[0100] If the first video contains only one storyline, the second video can be obtained by directly searching for the video indicated by the target text in the first video. When the second video is obtained, the target text can be split into multiple sub-texts with time sequence according to the various stages of the story development in the first video. After generating a list of sub-text fragments, they can be merged according to time sequence. Then, the video fragments pointed to by the description information of the merged fragments can be merged to obtain the second video.

[0101] The above method effectively reduces the problem of recalling a large amount of video content unrelated to a single storyline, resulting in chaotic plots and a lack of focus, which is currently caused by directly matching videos using complete video descriptions or brief video summaries. Furthermore, by structurally splitting the target text describing a storyline according to the various stages of the first video's plot development, it is beneficial to improve the accuracy of video matching, further reducing the probability of recalling videos unrelated to the storyline, and further reducing the probability of chaotic second video content caused by the mixing of multiple storylines.

[0102] The first video, according to its plot development, can include multiple stages. A stage refers to a segment in the development of something, divided according to specific criteria, and stages are sequential. For example, the first video can be divided into three stages: beginning, middle, and end. The video content of a stage can be at least one video clip. According to the chronological order of the stages, the video clips corresponding to each stage can be strung together in time to form a complete video.

[0103] The target text describes the video content of the second video, and it can describe the video content of one or more stages of the plot development of the first video. When generating the second video, to more accurately extract video segments from the first video, the target text can be split according to the stages of the plot development, resulting in multiple sub-texts with a temporal sequence. Each sub-text describes the video content of one stage. Because these sub-texts are temporally ordered, video segment extraction can be more targeted, extracting video segments from the corresponding temporal stage. This avoids the problem of temporal mismatch between the content described in the target text and the extracted video segments, improving the accuracy of video generation. It can also solve the problem where the video synopsis cannot reflect the plot structure such as the beginning, climax, and ending, and the target text lacks temporal division. For example, text originally describing the beginning of the plot might be mistakenly interpreted as the ending content when extracting video segments, leading to inaccurate extraction of video segments from the ending and resulting in inaccurate video content that does not match the target text.

[0104] Based on the various stages of the first video's plot development, the target text is broken down into multiple sub-texts arranged chronologically, with each sub-text describing a stage of the second video. For example, the first video includes three stages: Stage 1, Stage 2, and Stage 3. These three stages are arranged chronologically as Stage 1, Stage 2, and Stage 3, where Stage 1 can be the beginning stage, Stage 2 can be the middle stage, and Stage 3 can be the ending stage. The target text describes the video content information of these three stages. This target text can be split into three sub-texts: sub-text 1, sub-text 2, and sub-text 3. Sub-text 1 describes the video content information of stage 1, sub-text 2 describes the video content information of stage 2, and sub-text 3 describes the video content information of stage 3. The temporal sequence of each sub-text has a mapping relationship with the temporal sequence of the stages. That is, sub-text 1 corresponds to stage 1, and the temporal sequence of sub-text 1 is consistent with the temporal sequence of stage 1; sub-text 2 corresponds to stage 2, and the temporal sequence of sub-text 2 is consistent with the temporal sequence of stage 2; and sub-text 3 corresponds to stage 3, and the temporal sequence of sub-text 3 is consistent with the temporal sequence of stage 3. These three sub-texts are arranged in temporal order as sub-text 1, sub-text 2, and sub-text 3.

[0105] When generating videos, directly matching the entire target text is difficult and may result in disjointed and inaccurate video content. Therefore, it is advisable to first break down the target into multiple sequential sub-texts according to the various stages of the first video's plot development. This helps determine the content structure of the first video, such as the beginning, middle or climax, and end of the video, providing a temporal reference for improving the accuracy of subsequent video segment extraction and thus enhancing the overall accuracy of video generation.

[0106] In some examples, the target text is broken down into multiple time-series subtexts according to the various stages of the first video's plot development, including:

[0107] Using the large language model and the thought chain prompts, the target text is processed according to the various stages of the plot development in the first video, resulting in multiple sub-texts with a temporal sequence.

[0108] This embodiment can transform a vague editing task into a series of precise, executable machine instructions through a chain-of-thought (CoT) prompt. CoT simulates the human thought process; when a person receives a video editing task, their thought process involves breaking down the input target text according to the three stages of the story: beginning, climax, and ending, obtaining descriptive text for each stage. Referring to the human thought process, the video generation method provided in this embodiment can pre-generate CoT prompts. The chain-of-thought prompt instructs a large language model (such as a multimodal large model) to process the data according to the following steps:

[0109] Step A: Identify the various stages of the plot development in the first video where the video content described by the target text is located, and determine the order of each stage in the first video to obtain the time sequence of the stages;

[0110] Step B: Divide the target text into multiple sub-texts according to the identified stages, and use the temporal sequence of the stages as the temporal sequence of the sub-texts according to the mapping relationship between the sub-texts and the stages.

[0111] The target text represents multiple stages of the video content. These stages can be all stages of the first video or only some stages of the first video. Each stage has its own order in the first video. For example, in the first video, stage 1 appears first, then stage 2, and finally stage 3. According to the plot development of the first video, the order of these stages in the first video is stage 1, stage 2, and stage 3. Therefore, the time sequence of stage 1 can be time sequence 1, the time sequence of stage 2 can be time sequence 2, and the time sequence of stage 3 can be time sequence 3. Time sequence 1 is earlier than time sequence 2, and time sequence 2 is earlier than time sequence 3. Therefore, the time sequence of stage 1 is earlier than the time sequence of stage 2, and the time sequence of stage 2 is earlier than the time sequence of stage 3.

[0112] The target text is then divided into multiple subtexts according to the identified stages. If three stages are identified, it can be divided into three subtexts, each corresponding to one stage, and subtexts with temporal order are generated. Specifically, this can be based on the mapping relationship between subtexts and stages. If subtext 1 indicates that stage 1 is being searched, then subtext 1 has a mapping relationship with stage 1; if subtext 2 indicates that stage 2 is being searched, then subtext 2 has a mapping relationship with stage 2; if subtext 3 indicates that stage 3 is being searched, then subtext 3 has a mapping relationship with stage 3. Since the temporal order of stage 1 is earlier than that of stage 2, and the temporal order of stage 2 is earlier than that of stage 3, when the temporal order of the stages is used as the temporal order of the subtexts, the temporal order of subtext 1 is earlier than that of subtext 2, and the temporal order of subtext 2 is earlier than that of subtext 3.

[0113] Based on this, by leveraging the thought chain capability of the large language model, when splitting the target text into multiple sub-texts with temporal sequence, the large language model can process the target text by referring to the thought chain prompts to obtain multiple sub-texts with temporal sequence.

[0114] This application leverages the Chain of Thought (CoT) capability of a large language model. It first allows the large language model to think about and decompose the target text, which contains complex descriptive information, into several simpler subtexts representing only one stage. Then, it matches the content of each stage of the video, such as matching the beginning, climax, and ending. This structured decomposition makes the target of each video match clearer and more specific, thereby improving the accuracy of video matching.

[0115] When extracting a second video matching the target text from a first video, the target text is split into multiple sub-texts with a temporal sequence, and then the corresponding video segment is extracted from the first video for each sub-text. In this application, when extracting video segments from sub-texts, the semantic relevance of the sub-texts and video segment descriptions is first performed, and then the matched video segment descriptions are mapped to the video segments. The segment description information is the video content information of a video segment from the first video. Since the first video includes multiple video segments, description information for each video segment can be generated, obtaining a mapping relationship between video segments and segment description information. After obtaining the segment description information corresponding to the sub-text through the aforementioned semantic relevance matching method, the video segment corresponding to the sub-text can be obtained through the mapping relationship between the segment description information and the video segment. The difference between segment description information and target text lies in the fact that target text describes a storyline, specifically the overall content of the second video related to that storyline. Segment description information, on the other hand, describes the content of a single segment of the video. For example, if the second video comprises multiple segments, the target text provides a general description of all these segments, while the segment description information describes the content of each individual segment. Segment description information, compared to target text, reveals more detailed information about the video. This segment description information can cover all segments of the first video or only a portion of them.

[0116] After splitting the target text into multiple subtexts, for each subtext, segment description information with semantic correspondence is searched. This process allows segment description information of a certain stage of video content to be found from various segments of the first video, while segment description information that does not belong to that stage of video content is excluded. This avoids the problem of disordered time sequence in the extracted video. In other words, by splitting the target text according to the development of the plot, subtexts with a time sequence are obtained, which means that the story structure of the video (such as the beginning, climax, and ending) is understood. On this basis, video segments are matched for each subtext, which can make the final edited video content consistent with the video content described by the target text, improve the matching accuracy, and avoid the problem of plot confusion caused by the generation of video due to the reversal of time sequence due to the lack of understanding of the story structure (for example, the target text describes the beginning of the plot, but the final generated second video describes the end of the plot).

[0117] If the target text is split into three subtexts, namely subtext 1, subtext 2, and subtext 3, then the segment description information with the highest semantic relevance to subtext 1 can be found, followed by the segment description information with the highest semantic relevance to subtext 2, and then the segment description information with the highest semantic relevance to subtext 3. Then, the segment description information found from these three subtexts is sorted according to the order of the video segments in the first video, resulting in a segment list for each subtext. This segment list includes segment description information arranged chronologically, and may also include the start and end times of the video segments indicated by the segment description information. For example, the segment description information with a semantic correspondence to subtext 1 includes segment description information 1-1 and segment description information 1-2. In the first video, segment description information 1-1 appears first, followed by segment description information 1-2. Therefore, when sorting segment description information 1-1 and segment description information 1-2, segment description information 1-1 is sorted first, followed by segment description information 1-2. For example, if the fragment description information that has a semantic correspondence with subtext 2 includes fragment description information 2-1 and fragment description information 2-2, and in the first video, fragment description information 2-2 appears first and then fragment description information 2-1 appears, then when sorting fragment description information 2-1 and fragment description information 2-2, fragment description information 2-2 should be sorted first and then fragment description information 2-1.

[0118] In one possible implementation, the fragment description information that has the highest semantic relevance to the subtext is found, including:

[0119] From the set of fragment description information, find the fragment description information that has the highest semantic relevance to the subtext.

[0120] The fragment description information in this fragment description information set can be fragment description information of all fragments of the first video, or fragment description information related to the target text.

[0121] If the fragment description information set is fragment description information related to the target text, then the video generation method provided in this application further includes generating the fragment description information set, the method for generating the fragment description information set including:

[0122] From the set of scene description information, search for video content information describing the scene indicated by the target text to obtain candidate scene description information;

[0123] The video segment pointed to by the candidate scene description information is determined to obtain the first segment;

[0124] The fragment description information describing the first segment is aggregated to obtain a fragment description information set.

[0125] A video scene refers to multiple semantically consecutive video segments that constitute a complete narrative or expressive unit. A video scene is the basic unit of video content analysis, used to divide story segments or specific situations. For a video, one video includes multiple scenes, and a scene can include one or more segments, or a single frame or multiple frames. When generating a set of segment description information, video content information describing the scene indicated by the target text can be searched from the scene description information set to obtain candidate scene description information. This candidate scene description information can be for one scene or for multiple scenes. If the target text indicates one scene, the candidate scene description information is for one scene; if the target text indicates multiple scenes, the subsequent scene description information is for multiple scenes.

[0126] When searching for video content information describing the scene indicated by the target text from the scene description information set, and obtaining candidate scene description information, a large language model can be used to find the candidate scene description information. Specifically, a prompt word including the scene description information set and the target text can be generated. The prompt word is input into the large language model (such as LLM, Qwen3, GPT-4, etc.). The large language model can generate candidate scene description information based on the semantic relationship between the target text and the scene description information.

[0127] After obtaining the candidate scene description information, the video segments to which these scene descriptions point are determined, resulting in the first segment. This first segment can be a segment of one or more scenes. The video content information describing the first segment is then aggregated to obtain a set of segment description information. Each segment description information can be processed using a multimodal large language model to analyze the images and dialogue of the video segment, generating information describing that segment.

[0128] This application filters out scene descriptions relevant to the scene corresponding to the target text, eliminating those irrelevant and further narrowing the search scope, thus improving the accuracy of video clip extraction. Furthermore, leveraging the powerful long-text understanding capabilities of a large language model, it can quickly filter out several semantically related candidate scene descriptions from the scene description set, improving the speed of video generation.

[0129] In one possible implementation, the fragment description information that has the highest semantic relevance to the subtext is found, including:

[0130] Using either method one or method two, find the descriptive information of the segment that has the highest semantic relevance to the subtext:

[0131] Method 1: Call the large language model to find the segment description information that has the highest semantic relevance to the subtext;

[0132] Method 2: Calculate the semantic similarity between the fragment description information and the subtext, and take the fragment description information with the highest semantic similarity as the fragment description information with the highest semantic relevance to the subtext.

[0133] When searching for the segment description information that is most semantically relevant to the subtext, two methods can be used: one is to search using a large language model, and the other is to search using a cross-modal retrieval model.

[0134] For method one, it involves using a large language model for searching. Specifically, a large language model can be invoked to search for the segment description information with the highest semantic relevance to the subtext within the segment description information set. When searching for the segment description information with the highest semantic relevance to the subtext using the large language model, prompt words containing both the subtext and the segment description information set can be generated. This allows the large language model to find the segment description information that has a semantic correspondence with the subtext from the segment description information set, thus obtaining the segment description information with the highest semantic relevance to the subtext.

[0135] For Method 2, a cross-modal retrieval model is used for searching. Specifically, the semantic similarity between each fragment description in the fragment description information set and the subtext is calculated. The fragment description with the highest semantic similarity is considered the fragment description with the highest semantic relevance to the subtext. Specifically, the cross-modal retrieval model searches for fragment description information that has a semantic correspondence with the subtext. The semantic similarity between each fragment description and the subtext is calculated by encoding each fragment description and the subtext to obtain feature vectors. The similarity between the feature vector of the subtext and the feature vector of the fragment description is calculated. The similarity is then sorted in descending order, and K fragment descriptions are selected in descending order of similarity, where K can be a positive number greater than or equal to 1. This yields one or more fragment descriptions, and the fragment description with the highest semantic similarity is considered the fragment description with the highest semantic relevance to the subtext. Optionally, the cross-modal retrieval model is CLIP (Contrastive Language-Image Pre-Training).

[0136] After finding the segment description information that has the highest semantic relevance to the subtext, the found segment description information can be sorted according to the order of the video segments in the first video to obtain a segment list of the subtext, so as to obtain the final second video based on the segment list of each subtext.

[0137] The list of fragments for a subtext corresponds to the subtext. Since the subtext has a temporal order, the resulting list of fragments for the subtext also has a temporal order, and the temporal order of the fragments for the subtext is consistent with the temporal order of the subtext. The fragment lists of each subtext can be merged according to the temporal order. For example, the fragment list corresponding to subtext 1 is fragment list 1, and the fragment list corresponding to subtext 2 is fragment list 2. According to the temporal order, subtext 1 comes before subtext 2, so fragment list 1 comes before fragment list 2. Merging fragment list 1 and fragment list 2 of the subtext gives fragment list 1 and fragment list 2 of the subtext. Then, the video segments pointed to by the segment description information in the segment list 1 of the subtext can be merged with the video segments pointed to by the segment description information in the segment list 2 of the subtext. Assuming that the video segments pointed to by the segment description information in the segment list 1 of the subtext are video segment a and video segment b, with video segment a preceding video segment b, and the video segments pointed to by the segment description information in the segment list 2 of the subtext are video segment c and video segment d, with video segment c preceding video segment d, then the second video obtained after merging is arranged in chronological order as video segment a, video segment b, video segment c, and video segment d.

[0138] The second video's plot development has multiple stages with a temporal sequence, meaning that each stage in the second video has its own time order within the video. A stage can be one or more video segments in the second video. These stages can reflect the narrative structure of the video content and represent the temporal sequence of the video segments. By combining the video segments of the second video from each stage according to the temporal sequence, the second video can be obtained.

[0139] When merging the video segments pointed to by the combined segment description information, since the segment list of the subtext has a temporal order, and the description information of each subtext in the segment list of the subtext also has a temporal order, the generated second video is obtained by arranging the segments indicated by each segment description information in the order of the first video. In this way, the generated second video maintains the continuity in temporal order and avoids the problems of plot confusion and video content incoherence caused by the reversal of the order of video segments. The generated second video is more logical and readable.

[0140] In one possible approach, the video generation method provided in this application, such as... Figure 2 As shown, the method includes:

[0141] Step 201: Obtain the first video; wherein the first video includes multiple sub-videos with identifiers, the identifiers indicating the temporal order in which the sub-videos appear in the first video.

[0142] Step 202: Based on each sub-video, generate a set of sub-texts with identifiers, and arrange the sub-text sets in the order indicated by the identifiers to generate a set of descriptive texts; wherein, each sub-text set includes at least one plotline descriptive text, and the plotline descriptive text in the sub-text set is the text that describes the plotline of the sub-video.

[0143] Step 203: Select a plotline description text from the description text set, determine the identifier of the sub-text set to which the selected plotline description text belongs, and obtain the target text with the identifier.

[0144] Step 204: Based on the identifier of the target text, determine the sub-video indicated by the target text, and from the sub-video indicated by the target text, find the video whose video content represents the target text to obtain the second video.

[0145] The first video includes multiple sub-videos, each with a unique identifier that distinguishes it from the others. This identifier reflects the sub-video's position within the first video. For example, if the first video includes three sub-videos: sub-video 1, sub-video 2, and sub-video 3, and these three sub-videos appear in the first video in the order of sub-video 1, sub-video 2, and sub-video 3, then sub-video 1 is identified by identifier 1, sub-video 2 by identifier 2, and sub-video 3 by identifier 3. Identifier 1 indicates that sub-video 1 appears first in the first video, identifier 2 indicates that sub-video 2 appears second in the first video, and identifier 3 indicates that sub-video 3 appears third in the first video.

[0146] When generating a set of sub-texts with identifiers based on each sub-video, a sub-video generates a sub-text set corresponding to its identifier. For example, when generating the sub-text set for sub-video 1, the identifier of that sub-text set is also 1. The method of generating sub-text sets from sub-videos is similar to... Figure 1 Step 102 is similar and will not be repeated here. After obtaining each sub-text set, the sub-text sets are arranged in the order indicated by the identifiers to generate a description text set. This description text set includes multiple sub-text sets, and the order of the sub-text sets is consistent with the order of the corresponding sub-videos. For example, if the identifier of sub-text set 1 is 1, the identifier of sub-text set 2 is 2, and the identifier of sub-text set 3 is 3, then the sub-text sets are arranged in the order indicated by the identifiers to generate the description text sets as sub-text set 1, sub-text set 2, and sub-text set 3.

[0147] When selecting a plotline description text from a set of description texts, it's necessary to determine the identifier of the sub-text set to which the selected plotline description text belongs. For example, if a plotline description text is selected from sub-text set 1, then the identifier of the target text will be 1. The method for selecting a plotline description text from a set of description texts is similar to... Figure 1 The method of step 103 shown is similar and will not be repeated here.

[0148] Finally, the sub-video indicated by the target text can be determined based on its identifier. Since there is a correspondence between the target text identifier and the sub-video identifier, the corresponding sub-video can be directly found using the target text identifier. For example, if the target text identifier is 1, and the identifier of sub-video 1 is also 1, then the sub-video indicated by the target text can be determined to be sub-video 1. The method for finding the video whose video content represents the target text from the sub-videos indicated by the target text is similar to... Figure 1 The method of step 104 shown is similar and will not be repeated here.

[0149] By identifying and locating the correct video segment, we can avoid the situation where other video segments are located that are not part of the current storyline, thus preventing the generation of a video with a single storyline. This reduces the probability of multiple storylines being mixed into the generated video, which could lead to plot confusion.

[0150] Figure 2 The video generation method shown is similar to... Figure 1 The difference in the video generation methods shown is that, Figure 2 The method described herein can trace the video corresponding to the target text back to the sub-video within the first video through identification, thereby finding the second video from the corresponding sub-video. This solves the problems of existing technologies that generate videos solely from a synopsis. Because the synopsis is disconnected from the video content, it's impossible to determine which part of the video the synopsis describes, making it difficult to locate the correct video segment. This results in a confusing plot and a video that doesn't match the expected content, ultimately failing to meet requirements.

[0151] In one possible implementation, the video generation method provided in this application, after obtaining the second video, further includes:

[0152] The second video is processed by removing empty shots, shortening the silence time between two adjacent lines of dialogue, removing redundant lines, playing the video at double speed, adding transitions, changing the aspect ratio of video frames, and adding elements to obtain the processed second video; wherein, the elements include text and / or audio.

[0153] When removing empty shots from the second video, visual analysis algorithms can be used to detect empty shots, such as continuous images without a subject or images that do not change. If the duration of an empty shot exceeds a first duration threshold, the empty shot removal operation will be performed.

[0154] When shortening the silence duration between two adjacent lines of dialogue in the second video, if the silence duration between the two lines exceeds the second duration threshold, the silence duration will be reduced by a preset duration, such as 0.5 seconds or other durations.

[0155] When removing redundant dialogue from the second video, a large language model can be used to generate prompts that include both the target text and the video dialogue. Based on these prompts, the large language model is called for processing. Specifically, the similarity between the dialogue and the target text can be compared, and multiple lines can be selected as initial dialogue in descending order of similarity. Then, one or more checks can be performed on the initial dialogue, including causal relationship, temporal relationship, and completeness, to obtain the dialogue that passes the checks. The consistency of the checked dialogue can then be compared with the original dialogue in the video, and the number of dialogues can also be controlled. Finally, the dialogue after removing redundancy can be output in a formatted manner, and the output dialogue can be arranged in chronological order.

[0156] When performing video speed-up processing on the second video, non-dialogue videos that are primarily action-oriented (such as chasing, walking, or scene transitions) can be sped up, for example, to 1.5 times the original video speed or other multiples, to increase the pace.

[0157] It can also change the aspect ratio of video frames, such as converting a landscape video to a portrait video. It can also add transitions and elements to the second video. These elements can be text or audio, such as background music or narration.

[0158] This application addresses one or more aspects that affect the quick understanding of video content, such as excessively long empty shots, meaningless pauses between characters, and slow narrative pace. This can prevent the final video from failing to meet user needs, reduce the occurrence of dull and dragging visuals, help improve video information density and accelerate narrative pace, ensure the rhythm and watchability of the finished product, and make the final product conform to the viewing habits of users who want to quickly understand the plot, thereby improving user satisfaction.

[0159] The above describes a video generation method provided by an embodiment of this application. The following will describe a system that performs the above video generation method.

[0160] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a video generation system provided in an embodiment of this application. Figure 3 As shown, the video generation system includes:

[0161] The video acquisition module 301 is used to acquire a first video; wherein, the first video is a video to be edited, and the first video is edited to obtain a second video. The first video has multiple plot lines, which are the main plot or subplots.

[0162] The text generation module 302 is used to generate a set of descriptive texts based on the first video; wherein the set of descriptive texts includes multiple plot line descriptive texts, and each plot line descriptive text describes a plot line.

[0163] The selection module 303 is used to select a plot description text as the target text from the set of description texts.

[0164] The video generation module 304 is used to find the video indicated by the target text in the first video and obtain the second video; wherein the second video includes video content with a storyline.

[0165] In one possible implementation, module 303 is selected, specifically for:

[0166] Obtain the first clip constraint; wherein, the first clip constraint is the constraint on obtaining the video content included in the target video, and the target video includes one or more second videos;

[0167] The selected texts are obtained by filtering the texts that meet the constraints of the first clip from the set of descriptive texts; wherein the selected texts include at least one plotline descriptive text.

[0168] One plotline description text from the filtered text is used as a target text; the number of target texts is the same as the number of plotline description texts included in the filtered text.

[0169] In one possible implementation, the text generation module 302 is specifically used for:

[0170] Based on the first video, a scene text set is generated; wherein, the scene text set includes multiple scene description texts, and the scene description texts are the video content describing any scene in the first video;

[0171] Obtain the second editing constraint; wherein, the second editing constraint is the constraint on obtaining the video content included in multiple storylines in the first video;

[0172] Based on the scene text set, a large language model is invoked to generate a set of descriptive text that meets the second clipping constraints.

[0173] In one possible implementation, the first video includes multiple sub-videos with identifiers indicating the order in which the sub-videos appear in the first video.

[0174] Text generation module 302 is specifically used for:

[0175] Based on each sub-video, a set of sub-texts with identifiers is generated. The sub-text sets are arranged in the order indicated by the identifiers to generate a set of descriptive texts. Each sub-text set includes at least one plotline descriptive text, which is a text describing the plotline of the sub-video.

[0176] Select module 303, specifically for:

[0177] Select a plotline description text from the set of description texts, determine the identifier of the sub-text set to which the selected plotline description text belongs, and obtain the target text with the identifier.

[0178] Video generation module 304 is specifically used for:

[0179] Based on the identifier of the target text, determine the sub-video indicated by the target text;

[0180] From the sub-videos indicated by the target text, find the video whose video content represents the target text to obtain the second video.

[0181] In one possible implementation, the video generation module 304 is specifically used for:

[0182] Based on the various stages of the first video's plot development, the target text is broken down into multiple sub-texts with a temporal sequence; each sub-text describes the video content of one stage.

[0183] Find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segments in the first video to obtain the segment list of the subtext; where the segment description information is the video content information describing a video segment of the first video, and semantic relevance represents the similarity between the segment description information and the subtext.

[0184] The fragment lists of each subtext are merged in chronological order, and the video segments pointed to by the description information of each merged fragment are combined to obtain the second video.

[0185] In one possible implementation, the text generation module 302 is also used for:

[0186] Perform segment boundary detection on the first video to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point;

[0187] Extract video frames before and after the segment boundaries from the first video;

[0188] Scene transition point detection is performed on the boundaries of video segments based on video frames, and the boundaries of segments belonging to scene transition points are taken as target boundaries; where a scene transition point is the boundary point between one scene and another scene.

[0189] The first video is divided into multiple scenes based on the target boundaries;

[0190] Generate video content information describing each scene to obtain a set of scene texts.

[0191] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0192] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0193] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0194] This electronic device is capable of implementing the video generation method described above.

[0195] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the video generation methods provided in this application.

[0196] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the video generation methods provided in this application.

[0197] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0199] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0200] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A video generation method, characterized in that, include: Obtain a first video; wherein, the first video is a video to be edited, the first video is edited to obtain a second video, the first video has multiple storylines, the storylines are the main storyline or the sub-storylines; Based on the first video, a set of descriptive texts is generated; wherein, the set of descriptive texts includes multiple plot line descriptive texts, and each plot line descriptive text describes one plot line; Select one of the plotline description texts from the set of description texts as the target text; The second video is obtained by searching for the video indicated by the target text in the first video; wherein the second video includes video content of the plot line.

2. The video generation method according to claim 1, characterized in that, Selecting a plotline description text from the set of description texts as the target text includes: Obtain the first editing constraint; wherein, the first editing constraint is a constraint on obtaining the video content included in the target video, and the target video includes one or more of the second videos; Texts that meet the first editing constraint conditions are filtered from the set of descriptive texts to obtain filtered texts; wherein, the filtered texts include at least one of the plotline descriptive texts; One of the plotline description texts in the filtered text is taken as a target text; wherein the number of target texts is the same as the number of plotline description texts included in the filtered text.

3. The video generation method according to claim 1, characterized in that, The step of generating a descriptive text set based on the first video includes: Based on the first video, a scene text set is generated; wherein, the scene text set includes multiple scene description texts, and the scene description texts are texts describing the video content of any scene in the first video; Obtain the second editing constraint; wherein, the second editing constraint is the constraint for obtaining the video content included in the multiple storylines in the first video; Based on the scene text set, a large language model is invoked to generate the descriptive text set that conforms to the second editing constraint.

4. The video generation method according to any one of claims 1 to 3, characterized in that, The first video includes multiple sub-videos with identifiers, the identifiers indicating the temporal order in which the sub-videos appear in the first video; Based on the first video, a set of descriptive text is generated, including: Based on each of the sub-videos, a sub-text set with the identifier is generated; wherein, the sub-text set includes at least one of the plotline description texts, and the plotline description texts in the sub-text set are texts describing the plotline of the sub-video; Arrange the sub-text sets according to the order indicated by the identifiers to generate the descriptive text set; Selecting a plotline description text from the set of description texts as the target text includes: Select one of the plotline description texts from the set of description texts, determine the identifier of the sub-text set to which the selected plotline description text belongs, and obtain the target text with the identifier; The step of finding the video indicated by the target text from the first video to obtain the second video includes: Based on the identifier of the target text, determine the sub-video indicated by the target text; From the sub-videos indicated by the target text, find the video whose video content represents the target text, and obtain the second video.

5. The video generation method according to any one of claims 1 to 3, characterized in that, The step of finding the video indicated by the target text from the first video to obtain the second video includes: According to the various stages of the development of the first video's plot, the target text is divided into multiple sub-texts with a time sequence; wherein, each sub-text describes the video content of one stage. Find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segments in the first video to obtain the segment list of the subtext; wherein, the segment description information is the video content information describing a video segment of the first video, and the semantic relevance represents the similarity between the segment description information and the subtext; The fragment lists of each subtext are merged according to the time sequence, and the video fragments pointed to by the description information of each merged fragment are merged to obtain the second video.

6. The video generation method according to claim 3, characterized in that, The step of generating a scene text set based on the first video includes: The first video is subjected to segment boundary detection to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point; Extract video frames before and after the segment boundary from the first video; Based on the video frames, scene switching points are detected on the boundaries of the segments, and the segments belonging to the scene switching points are taken as target boundaries; wherein, the scene switching point is the boundary point between one scene and another scene; The first video is divided into multiple scenes based on the target boundary; Generate video content information describing each of the aforementioned scenes to obtain the scene text set.

7. A video generation apparatus, characterized in that, include: The video acquisition module is used to acquire a first video; wherein the first video is a video to be edited, and the first video is edited to obtain a second video, and the first video has multiple plot lines, wherein the plot lines are the main plot or the subplots. A text generation module is used to generate a set of descriptive texts based on the first video; wherein, the set of descriptive texts includes multiple plot line descriptive texts, and each plot line descriptive text describes one plot line; The selection module is used to select one of the plotline description texts from the set of description texts as the target text; A video generation module is used to find the video indicated by the target text in the first video to obtain a second video; wherein the second video includes video content of the plot line.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video generation method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the video generation method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the video generation method as described in any one of claims 1 to 6.

Images