Video generation method and related device

Abstract

The invention discloses a video generation method and a related device, and relates to the field of video editing, and the method comprises the steps: obtaining a target text and a first video; splitting the target text into a plurality of sub-texts with a time sequence according to each stage of the development of the first video plot; searching fragment description information with the highest semantic correlation with the sub-texts, and sorting the searched fragment description information according to the sequence of the video fragments in the first video to obtain a fragment list of the sub-texts; and merging the fragment lists of the sub-texts according to a time sequence, and merging the video fragments pointed by the merged fragment description information to obtain a second video. The video generation accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of video editing technology, and in particular to a video generation method and related apparatus. Background Technology

[0002] Currently, the method for obtaining a second video from a first video is as follows: extract related video segments from the first video using video description text, and then synthesize the extracted video segments into a second video. When extracting related videos using video description text, a video synopsis is often used as the video description text. Since the video synopsis is a general description, this can easily lead to the generated second video failing to accurately represent the video content described in the video description text, resulting in low accuracy in video generation. Summary of the Invention

[0003] In view of the above problems, this application provides a video generation method and related apparatus, which can improve the accuracy of video generation. The specific solution is as follows:

[0004] The first aspect of this application provides a video generation method, including:

[0005] Obtain target text and a first video; wherein the first video includes multiple video segments, the target text is text describing the video content of a second video, and the second video is at least one video segment of the first video;

[0006] According to the various stages of the development of the first video's plot, the target text is divided into multiple sub-texts with a temporal sequence; wherein, each sub-text describes the video content of one stage.

[0007] Find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segment in the first video to obtain the segment list of the subtext; wherein, the segment description information is the video content information describing a video segment of the first video, and the semantic relevance represents the similarity between the segment description information and the subtext;

[0008] The fragment lists of each subtext are merged according to the time sequence, and the video fragments pointed to by the description information of each merged fragment are merged to obtain the second video.

[0009] One possible implementation also includes:

[0010] From the scene description information set, video content information describing the scene indicated by the target text is searched to obtain candidate scene description information; wherein, the scene description information set includes multiple scene description information, and the scene description information is video content information describing the scene of the first video;

[0011] The video segment pointed to by the candidate scene description information is determined to obtain the first segment;

[0012] The fragment description information describing the first fragment is aggregated to obtain a fragment description information set;

[0013] The process of finding the segment description information that has the highest semantic relevance to the subtext includes:

[0014] From the set of fragment description information, find the fragment description information that has the highest semantic relevance to the subtext.

[0015] One possible implementation also includes:

[0016] Perform segment boundary detection on the first video to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point;

[0017] Extract video frames before and after the segment boundary from the first video;

[0018] Based on the video frames, scene switching points are detected on the boundaries of the segments, and the segments that belong to scene switching points are taken as target boundaries; wherein, the scene switching point is the boundary point between one scene and another scene;

[0019] The first video is divided into multiple scenes based on the target boundary;

[0020] Generate video content information describing each of the aforementioned scenes to obtain a set of scene description information.

[0021] In one possible implementation, splitting the target text into multiple temporally ordered sub-texts according to the various stages of the first video's plot development includes:

[0022] Based on the large language model and the thought chain prompts, the target text is processed according to the various stages of the plot development of the first video to obtain multiple sub-texts with temporal sequence;

[0023] The thought chain prompt is used to instruct the large language model to process the language in the following steps:

[0024] Identify the various stages of the plot development of the video content described by the target text in the first video, and determine the order of each stage in the first video to obtain the temporal sequence of the stages;

[0025] The target text is divided into multiple sub-texts according to the identified stages, and the time sequence of the stages is used as the time sequence of the sub-texts according to the mapping relationship between the sub-texts and the stages.

[0026] In one possible implementation, finding the segment description information that has the highest semantic relevance to the subtext includes:

[0027] Using either method one or method two, find the segment description information that has the highest semantic relevance to the subtext:

[0028] Method 1: Call the large language model to find the segment description information that has the highest semantic relevance to the subtext;

[0029] Method 2: Calculate the semantic similarity between the fragment description information and the subtext, and at least select the fragment description information with the highest semantic similarity as the fragment description information with the highest semantic relevance to the subtext.

[0030] In one possible implementation, after obtaining the list of fragments of the sub-texts and before merging the lists of fragments of each sub-text according to the chronological order, the method further includes:

[0031] Obtain a second segment; wherein the second segment is at least partially identical in video content to the video segment pointed to by the segment description information in the segment list;

[0032] Based on the start and end points of the second segment, the segment description information in the segment list is adjusted so that the segment list of the adjusted subtext includes the segment description information of the second segment;

[0033] The step of merging the list of fragments of each of the sub-texts according to the chronological order, and merging the video fragments pointed to by the description information of each merged fragment to obtain the second video includes:

[0034] The adjusted list of subtext segments is merged according to the chronological order, and the video segments pointed to by the description information of each merged segment are merged to obtain the second video with the second segment.

[0035] A second aspect of this application provides a video generation apparatus, comprising:

[0036] An acquisition module is used to acquire target text and a first video; wherein the first video includes multiple video segments, the target text is text describing the video content of a second video, and the second video is at least one video segment of the first video;

[0037] The text splitting module is used to split the target text into multiple sub-texts with a time sequence according to the various stages of the development of the first video plot; wherein, each sub-text describes the video content of one stage;

[0038] The search module is used to find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segments in the first video to obtain the segment list of the subtext; wherein, the segment description information is the video content information describing a video segment of the first video, and the semantic relevance represents the similarity between the segment description information and the subtext;

[0039] The merging module is used to merge the fragment lists of each of the sub-texts according to the time sequence, and merge the video fragments pointed to by the description information of each merged fragment to obtain the second video.

[0040] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video generation method described in the first aspect or any implementation thereof.

[0041] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program so that the electronic device can implement the video generation method of the first aspect or any implementation thereof.

[0044] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the video generation method described in the first aspect or any implementation thereof.

[0045] Using the above technical solution, the video generation method and related apparatus provided in this application acquire target text and a first video, and according to the various stages of the plot development of the first video, split the target text into multiple sub-texts with a temporal sequence; find the segment description information with the highest semantic relevance to the sub-text, sort the found segment description information according to the order of the video segments in the first video, and obtain a segment list of the sub-text; merge the segment lists of each sub-text according to the temporal sequence, and merge the video segments pointed to by the merged segment description information to obtain a second video. This application breaks down the target text into multiple sub-texts with a temporal sequence according to the various stages of the first video's plot development. Each sub-text describes the video content of one stage. Then, it searches for segment description information for each sub-text and merges the found segment description information in chronological order to obtain the video corresponding to the merged segment description information. This avoids the situation where the content of the final second video is inconsistent with the video content indicated by the target text due to the lack of a breakdown of the first video's plot development stage indicated by the target text, thus improving the accuracy of video generation. In addition, the second video generated after merging the segment list of the sub-texts according to the temporal sequence of the sub-texts also has a temporal sequence, which can maintain the temporal continuity of the found second video and avoid the problem of plot confusion due to the reversal of the order of video segments. The generated second video is more logical and readable. Attached Figure Description

[0046] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 A flowchart of a video generation method provided in this application;

[0048] Figure 2 A structural diagram of a video generation system provided in this application;

[0049] Figure 3 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0050] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0051] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0052] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0053] In the field of multimedia content understanding and intelligent creation, the main application scenarios for automatically locating and extracting the core plot of long videos (such as TV series and movies) include: intelligent trailer and plot summary generation, content promotion and marketing, video retrieval and navigation, and character storyline extraction. In the intelligent trailer and plot summary generation scenario, it is necessary to automatically edit video clips that summarize the main storyline from the entire series. In the content promotion and marketing scenario, it is necessary to quickly generate core plot clips for social media dissemination to attract users. In the video retrieval and navigation scenario, it is necessary for users to not only search by keywords but also to directly jump to relevant videos by entering a plot description. In the character storyline extraction scenario, it is necessary to automatically edit out the complete growth line or emotional line of a specific character to meet users' personalized viewing needs.

[0054] Currently, text-to-video matching typically employs a one-step matching method, which directly searches for videos with relevant content within the entire video using a video description text. This one-step matching method struggles to accurately understand the complex correspondence between abstract narratives and specific video content when dealing with complex plots or video description texts spanning long periods. Consequently, the generated video content may not accurately reflect the video content described in the video description text, resulting in low accuracy in the matched videos.

[0055] The problem arises primarily because video description text is typically a highly summarized and abstract summary. For example, if only the video synopsis is used as the video description text, the plot content represented by the synopsis is quite complex, corresponding to multiple segments distributed across different scenes and spanning a long period of time in the video. Treating the entire video synopsis as a single query leads to the retrieval of a large number of semantically fragmented but unrelated video segments that cannot form a coherent story. For instance, the video description text describes the beginning of the plot, but the final generated video describes the end of the plot, reducing the video's logic and readability and failing to meet the user's needs for the generated video.

[0056] Based on this, this application provides a video generation method to solve the problem of inconsistent and inaccurate matching results when matching video description text with long span and high generalization.

[0057] Reference Figure 1 , Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this application, such as... Figure 1 As shown in the embodiment of this application, a video generation method may include steps 101 to 104, which are described in detail below.

[0058] Step 101: Obtain the target text and the first video; wherein, the first video includes multiple video segments, the target text is text describing the video content of the second video, and the second video is at least one video segment of the first video.

[0059] The video generation method involved in this application can be a method of obtaining a second video from a first video by editing target text. The first video includes multiple video segments, and the second video is one or more video segments of the first video. A video segment is a shot. A shot refers to a continuous, independent and complete part recorded during video shooting from the start of recording to the stop of recording. Video segments can be the most basic and indivisible unit of video.

[0060] To obtain the second video, target text and the first video can be acquired. The target text describes the content of the second video. This target text is a general description of the overall content of the second video, and can be a summary of the video's content. For example, the target text could be a video synopsis, a video content outline, or a plot line description. The first video includes the second video. The target text can be descriptive information about the content of the second video. Based on this descriptive information, the corresponding video can be extracted from the first video to obtain the second video. For example, if the target text is a description of a plot line in the first video, extracting the video content related to that plot line from the first video yields the second video. Alternatively, if the target text is a synopsis of the first video's plot, summarizing the main events in the first video, extracting the video content related to that synopsis from the first video yields the second video.

[0061] Step 102: According to the various stages of the first video's plot development, the target text is divided into multiple sub-texts with a time sequence; where each sub-text describes the video content of one stage.

[0062] The first video, according to its plot development, can include multiple stages. A stage refers to a segment in the development of something, divided according to specific criteria, and stages are sequential. For example, the first video can be divided into three stages: beginning, middle, and end. The video content of a stage can be at least one video clip. According to the chronological order of the stages, the video clips corresponding to each stage can be strung together in time to form a complete first video.

[0063] The target text describes the video content of the second video, and this target text can describe the video content of one or more stages of the plot development of the first video. For example, if the target text is a video plot summary, the plot summary can cover the video content of multiple stages of the first video. When generating the second video, in order to more accurately extract video segments from the first video, the target text can be split according to the stages of plot development, resulting in multiple sub-texts with a time sequence, each sub-text describing the video content of one stage. Because the sub-texts obtained in this way are time-sequential, video segment extraction can be more targeted to extract video segments from the video stage corresponding to the time sequence, avoiding the problem of time sequence mismatch between the content described by the target text and the extracted video segments, thus improving the accuracy of video generation. This can solve the problem that when the video plot summary cannot reflect the plot structure such as the beginning, climax, and ending of the plot, and the target text does not have a time sequence division, the text that originally described the beginning of the plot may be mistakenly identified as the content of the ending of the plot when extracting video segments, resulting in inaccurate extracted video segments and the generated video content not matching the target text.

[0064] Based on the various stages of the first video's plot development, the target text is broken down into multiple sub-texts arranged chronologically, with each sub-text describing a stage of the second video. For example, the first video includes three stages: Stage 1, Stage 2, and Stage 3. These three stages are arranged chronologically as Stage 1, Stage 2, and Stage 3, where Stage 1 can be the beginning stage, Stage 2 can be the middle stage, and Stage 3 can be the ending stage. The target text describes the video content information of these three stages. This target text can be split into three sub-texts: sub-text 1, sub-text 2, and sub-text 3. Sub-text 1 describes the video content information of stage 1, sub-text 2 describes the video content information of stage 2, and sub-text 3 describes the video content information of stage 3. The temporal sequence of each sub-text has a mapping relationship with the temporal sequence of the stages. That is, sub-text 1 corresponds to stage 1, and the temporal sequence of sub-text 1 is consistent with the temporal sequence of stage 1; sub-text 2 corresponds to stage 2, and the temporal sequence of sub-text 2 is consistent with the temporal sequence of stage 2; and sub-text 3 corresponds to stage 3, and the temporal sequence of sub-text 3 is consistent with the temporal sequence of stage 3. These three sub-texts are arranged in temporal order as sub-text 1, sub-text 2, and sub-text 3.

[0065] When generating videos, directly matching the entire target text is difficult and may result in disjointed and inaccurate video content. Therefore, it is advisable to first break down the target into multiple sequential sub-texts according to the various stages of the first video's plot development. This helps determine the content structure of the first video, such as the beginning, middle or climax, and end of the video, providing a temporal reference for improving the accuracy of subsequent video segment extraction and thus enhancing the overall accuracy of video generation.

[0066] In some examples, the target text is broken down into multiple time-series subtexts according to the various stages of the first video's plot development, including:

[0067] Using the large language model and the thought chain prompts, the target text is processed according to the various stages of the plot development in the first video, resulting in multiple sub-texts with a temporal sequence.

[0068] This embodiment can transform a vague editing task into a series of precise, executable machine instructions through a chain-of-thought (CoT) prompt. CoT simulates the human thought process; when a person receives a video editing task, their thought process involves breaking down the input target text according to the three stages of the story: beginning, climax, and ending, obtaining descriptive text for each stage. Referring to the human thought process, the video generation method provided in this embodiment can pre-generate CoT prompts. The chain-of-thought prompt instructs a large language model (such as a multimodal large model) to process the data according to the following steps:

[0069] Step A: Identify the various stages of the plot development in the first video where the video content described by the target text is located, and determine the order of each stage in the first video to obtain the time sequence of the stages;

[0070] Step B: Divide the target text into multiple sub-texts according to the identified stages, and use the temporal sequence of the stages as the temporal sequence of the sub-texts according to the mapping relationship between the sub-texts and the stages.

[0071] The target text represents multiple stages of the video content. These stages can be all stages of the first video or only some stages of the first video. Each stage has its own order in the first video. For example, in the first video, stage 1 appears first, then stage 2, and finally stage 3. According to the plot development of the first video, the order of these stages in the first video is stage 1, stage 2, and stage 3. Therefore, the time sequence of stage 1 can be time sequence 1, the time sequence of stage 2 can be time sequence 2, and the time sequence of stage 3 can be time sequence 3. Time sequence 1 is earlier than time sequence 2, and time sequence 2 is earlier than time sequence 3. Therefore, the time sequence of stage 1 is earlier than the time sequence of stage 2, and the time sequence of stage 2 is earlier than the time sequence of stage 3.

[0072] The target text is then divided into multiple subtexts according to the identified stages. If three stages are identified, it can be divided into three subtexts, each corresponding to one stage, and subtexts with temporal order are generated. Specifically, this can be based on the mapping relationship between subtexts and stages. If subtext 1 indicates that stage 1 is being searched, then subtext 1 has a mapping relationship with stage 1; if subtext 2 indicates that stage 2 is being searched, then subtext 2 has a mapping relationship with stage 2; if subtext 3 indicates that stage 3 is being searched, then subtext 3 has a mapping relationship with stage 3. Since the temporal order of stage 1 is earlier than that of stage 2, and the temporal order of stage 2 is earlier than that of stage 3, when the temporal order of the stages is used as the temporal order of the subtexts, the temporal order of subtext 1 is earlier than that of subtext 2, and the temporal order of subtext 2 is earlier than that of subtext 3.

[0073] Based on this, by leveraging the thought chain capability of the large language model, when splitting the target text into multiple sub-texts with temporal sequence, the large language model can process the target text by referring to the thought chain prompts to obtain multiple sub-texts with temporal sequence.

[0074] This application leverages the Chain of Thought (CoT) capability of a large language model. It first allows the large language model to think about and decompose the target text, which contains complex descriptive information, into several simpler subtexts representing only one stage. Then, it matches the content of each stage of the video, such as matching the beginning, climax, and ending. This structured decomposition makes the target of each video match clearer and more specific, thereby improving the accuracy of video matching.

[0075] Step 103: Find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segments in the first video to obtain the segment list of the subtext; where the segment description information is the video content information describing a video segment of the first video, and semantic relevance represents the similarity between the segment description information and the subtext.

[0076] When extracting a second video matching the target text from a first video, the target text is split into multiple sub-texts with a temporal sequence, and then the corresponding video segment is extracted from the first video for each sub-text. In this application, when extracting video segments from sub-texts, the semantic relevance of the sub-texts and video segment descriptions is first performed, and then the matched video segment descriptions are mapped to the video segments. The segment description information is the video content information of a video segment from the first video. Since the first video includes multiple video segments, description information for each video segment can be generated, obtaining a mapping relationship between video segments and segment description information. After obtaining the segment description information corresponding to the sub-text through the aforementioned semantic relevance matching method, the video segment corresponding to the sub-text can be obtained through the mapping relationship between the segment description information and the video segment. The difference between segment description information and target text is that target text describes the overall content of the second video, while segment description information describes the video content of a single segment. For example, if the second video includes multiple video segments, the target text is a general description of all these video segments, while the segment description information describes the content of each individual video segment. Segment description information, compared to target text, better reflects the detailed content of the video. These segment descriptions can be descriptions of all segments in the first video, or descriptions of some segments in the first video.

[0077] After splitting the target text into multiple subtexts in step 102, for each subtext, segment description information with semantic correspondence is searched. This process can find segment description information of a certain stage of video content from various segments of the first video, while excluding segment description information that is not part of that stage of video content. This avoids the problem of temporal disorder in the extracted video. That is, by splitting the target text according to the development of the plot, subtexts with temporal order are obtained, which means that the story structure of the video (such as the beginning, climax, and ending) is understood. On this basis, video segments are matched for each subtext, which can make the final edited video content consistent with the video content described by the target text, improve the matching accuracy, and avoid the problem of plot confusion caused by the generation of video due to the reversal of the time sequence due to the lack of understanding of the story structure (for example, the target text describes the beginning of the plot, but the final generated second video describes the end of the plot).

[0078] If the target text is split into three subtexts, namely subtext 1, subtext 2, and subtext 3, then the segment description information with the highest semantic relevance to subtext 1 can be found, followed by the segment description information with the highest semantic relevance to subtext 2, and then the segment description information with the highest semantic relevance to subtext 3. Then, the segment description information found from these three subtexts is sorted according to the order of the video segments in the first video, resulting in a segment list for each subtext. This segment list includes segment description information arranged chronologically, and may also include the start and end times of the video segments indicated by the segment description information. For example, the segment description information with a semantic correspondence to subtext 1 includes segment description information 1-1 and segment description information 1-2. In the first video, segment description information 1-1 appears first, followed by segment description information 1-2. Therefore, when sorting segment description information 1-1 and segment description information 1-2, segment description information 1-1 is sorted first, followed by segment description information 1-2. For example, if the fragment description information that has a semantic correspondence with subtext 2 includes fragment description information 2-1 and fragment description information 2-2, and in the first video, fragment description information 2-2 appears first and then fragment description information 2-1 appears, then when sorting fragment description information 2-1 and fragment description information 2-2, fragment description information 2-2 should be sorted first and then fragment description information 2-1.

[0079] In one possible implementation, the fragment description information that has the highest semantic relevance to the subtext is found, including:

[0080] From the set of fragment description information, find the fragment description information that has the highest semantic relevance to the subtext.

[0081] The fragment description information in this fragment description information set can be fragment description information of all fragments of the first video, or fragment description information related to the target text.

[0082] If the fragment description information set is fragment description information related to the target text, then the video generation method provided in this application further includes generating the fragment description information set, the method for generating the fragment description information set including:

[0083] From the scene description information set, search for video content information describing the scene indicated by the target text to obtain candidate scene description information; wherein, the scene description information set includes multiple scene description information, and the scene description information is video content information describing the scene of the first video;

[0084] The video segment pointed to by the candidate scene description information is determined to obtain the first segment;

[0085] The fragment description information describing the first segment is aggregated to obtain a fragment description information set.

[0086] A video scene refers to multiple semantically consecutive video segments that constitute a complete narrative or expressive unit. A video scene is the basic unit of video content analysis, used to divide story segments or specific situations. For a video, one video includes multiple scenes, and a scene can include one or more segments, or a single frame or multiple frames. When generating a set of segment description information, video content information describing the scene indicated by the target text can be searched from the scene description information set to obtain candidate scene description information. This candidate scene description information can be for one scene or for multiple scenes. If the target text indicates one scene, the candidate scene description information is for one scene; if the target text indicates multiple scenes, the subsequent scene description information is for multiple scenes.

[0087] When searching for video content information describing the scene indicated by the target text from the scene description information set, and obtaining candidate scene description information, a large language model can be used to find the candidate scene description information. Specifically, a prompt word including the scene description information set and the target text can be generated. The prompt word is input into the large language model (such as LLM, Qwen3, GPT-4, etc.). The large language model can generate candidate scene description information based on the semantic relationship between the target text and the scene description information.

[0088] After obtaining candidate scene description information, the video segments to which these scene descriptions point are determined, resulting in a first segment. This first segment can be a segment from one or more scenes. The video content information describing the first segment is aggregated to obtain a segment description information set. This set can include one or more sub-description information sets. The segment description information of segments within a sub-description information set represents the segment description information of segments within a single scene. Different sub-description information sets represent the segment description information of segments within different scenes. Each segment description information can be processed using a multimodal large language model to analyze the images and dialogue of the video segment, generating information describing that segment.

[0089] This application filters out scene descriptions relevant to the scene corresponding to the target text, eliminating those irrelevant and further narrowing the search scope, thus improving the accuracy of video clip extraction. Furthermore, leveraging the powerful long-text understanding capabilities of a large language model, it can quickly filter out several semantically related candidate scene descriptions from the scene description set, improving the speed of video generation.

[0090] In one possible implementation, methods for obtaining the set of scene description information include:

[0091] Perform segment boundary detection on the first video to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point;

[0092] Extract video frames before and after the segment boundaries from the first video;

[0093] Scene transition point detection is performed on the boundaries of video segments based on video frames, and the boundaries of segments belonging to scene transition points are taken as target boundaries; where a scene transition point is the boundary point between one scene and another scene.

[0094] The first video is divided into multiple scenes based on the target boundaries;

[0095] Generate video content information describing each scene to obtain a set of scene description information.

[0096] When performing segment boundary detection on the first video, a shot detection algorithm can be used to obtain the timestamps of the segment boundaries in the first video. Based on the start and end times of the segment boundary timestamps, the first video can be divided into multiple segments, each segment representing a shot. After obtaining the segment boundaries of each segment in the first video, one or more video frames before and after the segment boundaries can be extracted. For example, N video frames before the segment boundaries and M video frames after the segment boundaries can be extracted. N and M can be the same or different.

[0097] After obtaining each video frame by identifying the boundaries of each segment, it can be determined whether these segments belong to the same scene. Segments belonging to the same scene are grouped into a single scene, and the video content information of this scene is then described to obtain scene description information. When detecting scene transition points based on video frames and segment boundaries, scene detection algorithms can be used to find the segment boundaries of scene transition points, obtaining target boundaries. These target boundaries are then used to divide the first video into multiple scenes. The scene transition point is the boundary between one scene and another. This boundary can be the end point of the last segment of the previous scene or the start point of the first segment of the next scene. When generating video content information describing each scene, a multimodal large language model can be used to process the video frames and dialogue information of the scene to obtain information describing the scene.

[0098] This application decomposes a first video into multiple scenes, each scene comprising multiple segments, and each segment comprising multiple image frames. Based on generating a set of descriptions for each segment, it detects the scene to which each segment belongs and obtains scene description information by combining the description information sets of segments belonging to the same scene. This allows the video to be transformed from a frame sequence into a narrative scene sequence during video segment extraction, facilitating the search for scenes indicated by the target text. This reduces the number of video segments while achieving precise matching with the target text-indicated scene, improving the accuracy and efficiency of video generation. Furthermore, compared to current methods that only use video summaries for video editing without generating scene description sets for each scene, this application segments the video by scene, performing a coarse-grained scene-level screening to quickly locate candidate scenes related to the target text-indicated scene. Then, it breaks down the target text into sub-texts corresponding to different stages of the first video, enabling fine-grained segment-level matching within candidate scenes. This achieves refined video segment extraction from coarse to fine, resolving the semantic gap between abstract target text descriptions and specific video content.

[0099] In one possible implementation, the fragment description information that has the highest semantic relevance to the subtext is found, including:

[0100] Using either method one or method two, find the descriptive information of the segment that has the highest semantic relevance to the subtext:

[0101] Method 1: Call the large language model to find the segment description information that has the highest semantic relevance to the subtext;

[0102] Method 2: Calculate the semantic similarity between the fragment description information and the subtext, and take the fragment description information with the highest semantic similarity as the fragment description information with the highest semantic relevance to the subtext.

[0103] When searching for the segment description information that is most semantically relevant to the subtext, two methods can be used: one is to search using a large language model, and the other is to search using a cross-modal retrieval model.

[0104] For method one, it involves using a large language model for searching. Specifically, a large language model can be invoked to search for the segment description information with the highest semantic relevance to the subtext within the segment description information set. When searching for the segment description information with the highest semantic relevance to the subtext using the large language model, prompt words containing both the subtext and the segment description information set can be generated. This allows the large language model to find the segment description information that has a semantic correspondence with the subtext from the segment description information set, thus obtaining the segment description information with the highest semantic relevance to the subtext.

[0105] For Method 2, a cross-modal retrieval model is used for searching. Specifically, the semantic similarity between each fragment description in the fragment description information set and the subtext is calculated. The fragment description with the highest semantic similarity is considered the fragment description with the highest semantic relevance to the subtext. Specifically, the cross-modal retrieval model searches for fragment description information that has a semantic correspondence with the subtext. The semantic similarity between each fragment description and the subtext is calculated by encoding each fragment description and the subtext to obtain feature vectors. The similarity between the feature vector of the subtext and the feature vector of the fragment description is calculated. The similarity is then sorted in descending order, and K fragment descriptions are selected in descending order of similarity, where K can be a positive number greater than or equal to 1. This yields one or more fragment descriptions, and the fragment description with the highest semantic similarity is considered the fragment description with the highest semantic relevance to the subtext. Optionally, the cross-modal retrieval model is CLIP (Contrastive Language-Image Pre-Training).

[0106] After finding the segment description information that has the highest semantic relevance to the subtext, the found segment description information can be sorted according to the order of the video segments in the first video to obtain a segment list of the subtext, so as to obtain the final second video based on the segment list of each subtext.

[0107] Step 104: Merge the fragment lists of each subtext in chronological order, and merge the video fragments pointed to by the description information of each merged fragment to obtain the second video.

[0108] The list of fragments for a subtext corresponds to the subtext. Since the subtext has a temporal order, the resulting list of fragments for the subtext also has a temporal order, and the temporal order of the fragments for the subtext is consistent with the temporal order of the subtext. The fragment lists of each subtext can be merged according to the temporal order. For example, the fragment list corresponding to subtext 1 is fragment list 1, and the fragment list corresponding to subtext 2 is fragment list 2. According to the temporal order, subtext 1 comes before subtext 2, so fragment list 1 comes before fragment list 2. Merging fragment list 1 and fragment list 2 of the subtext gives fragment list 1 and fragment list 2 of the subtext. Then, the video segments pointed to by the segment description information in the segment list 1 of the subtext can be merged with the video segments pointed to by the segment description information in the segment list 2 of the subtext. Assuming that the video segments pointed to by the segment description information in the segment list 1 of the subtext are video segment a and video segment b, with video segment a preceding video segment b, and the video segments pointed to by the segment description information in the segment list 2 of the subtext are video segment c and video segment d, with video segment c preceding video segment d, then the second video obtained after merging is arranged in chronological order as video segment a, video segment b, video segment c, and video segment d.

[0109] The second video's plot development has multiple stages with a temporal sequence, meaning that each stage in the second video has its own time order within the video. A stage can be one or more video segments in the second video. These stages can reflect the narrative structure of the video content and represent the temporal sequence of the video segments. By combining the video segments of the second video from each stage according to the temporal sequence, the second video can be obtained.

[0110] When merging the video segments pointed to by the combined segment description information, since the segment list of the subtext has a temporal order, and the description information of each subtext in the segment list of the subtext also has a temporal order, the generated second video is obtained by arranging the segments indicated by each segment description information in the order of the first video. In this way, the generated second video maintains the continuity in temporal order and avoids the problems of plot confusion and video content incoherence caused by the reversal of the order of video segments. The generated second video is more logical and readable.

[0111] In one possible implementation, after obtaining the list of subtext fragments and before merging the lists of subtext fragments in chronological order, the following steps are also included:

[0112] Obtain the second segment; wherein the second segment is at least partially identical in video content to the video segment pointed to by the segment description information in the segment list;

[0113] Based on the start and end points of the second segment, the segment description information in the segment list is adjusted so that the segment list of the adjusted subtext includes the segment description information of the second segment.

[0114] After obtaining the list of subtext fragments, there may be instances where the fragment descriptions in the fragment list indicate missing dialogue, or where additional video content needs to be added. In such cases, the fragment boundaries in the fragment descriptions can be adjusted. Specifically, a second fragment can be obtained, where at least part of the video content overlaps with the video content of the fragment indicated by the fragment description. This second fragment can be related to the speaker in the video fragment indicated by the fragment description, or it can be another video fragment not appearing in the video fragment indicated by the fragment description, such as a high-scoring video fragment reflecting the user's level of liking for that video fragment.

[0115] When adjusting the segment description information in the segment list based on the start and end points of the second segment, the start and end points of the second segment can be compared with the start and end points of the video segment pointed to by the segment description information. If the start point of the second segment is earlier than the start point of the video segment pointed to by the segment description information, then the start point of the second segment is used as the adjusted start point. If the end point of the second segment is later than the end point of the video segment pointed to by the segment description information, then the end point of the second segment is used as the adjusted end point, thus obtaining the adjusted subtext segment list. This ensures that the adjusted subtext segment list includes the segment description information of the second segment.

[0116] In one possible implementation, the list of fragments from each subtext is merged chronologically, and the video fragments pointed to by the description information of each merged fragment are then combined to obtain a second video, including:

[0117] The adjusted list of subtext fragments is merged in chronological order, and the video fragments pointed to by the description information of each merged fragment are merged to obtain a second video with a second fragment.

[0118] When generating the second video, if the segment description information in the segment list of the subtext is adjusted based on the start and end points of the second segment, in order to include the second segment in the second video, the segment list of the adjusted subtext can be merged in chronological order, and the video segments pointed to by the merged segment description information can be merged to obtain the second video with the second segment.

[0119] After obtaining the list of subtext fragments, this application can avoid incomplete fragments by adjusting the start and end points of the fragments, thus ensuring the continuity between the various fragments of the generated video. It can also increase the user's liking for the fragments and help improve the readability of the generated video.

[0120] Optionally, after obtaining the second video, audio, text, and other information can be added to the second video to enrich its content.

[0121] The video generation method provided in this application divides the first video into scenes. A large language model is used to quickly locate candidate scenes associated with the target text. Then, the large language model's thought chain is used to divide the target text into multiple sub-texts with a temporal sequence. Each sub-text describes a stage of the video content, thus splitting the target text according to the video's content structure, such as into three stages: beginning, climax, and ending. This avoids inconsistencies between the final second video and the target text's indicated video content due to a lack of segmentation in the first video's plot development stage, improving the accuracy of video generation. Next, fine-grained matching at the segment level is performed on each sub-text within the candidate scenes to obtain a list of sub-text segments. Finally, these sub-text segment lists are merged chronologically, and segment boundaries can be adjusted to ensure the coherence of the generated video content. The generated video maintains temporal coherence, avoiding plot confusion caused by reversing the order of video segments, thus enhancing the video's logic and readability.

[0122] This invention proposes a multi-stage main storyline matching method based on a large language model. This method mimics the thought process of a human editor, decomposing the complex matching task into three stages: "coarse screening, fine selection, and refinement." First, the video is segmented by scene, and a large language model (LMM) is used for scene-level coarse screening to quickly locate candidate scenes related to the target text. Then, the chain-of-thought (CoT) capability of the large language model is used to decompose the main storyline into structured parts such as "beginning, climax, and ending," and fine-grained matching at the fragment level is performed within the candidate scenes, solving the semantic gap between the abstract target text description and the specific video content. Finally, the matching results are finely calibrated by combining the video's metadata (such as filter information and speaker information) to ensure that the final output storyline segments are both concise and coherent. Using the method provided in this application improves the efficiency and accuracy of video generation, and the final generated video maintains the content and temporal coherence between video segments, improving the logic and readability of the generated video.

[0123] Furthermore, the method provided in this application can automatically generate video content that matches texts such as plot outlines and character storylines, reducing manual editing costs and providing text-to-video matching in terms of artificial intelligence (such as plot assistants and intelligent drama tracking), greatly enhancing the user's drama-watching experience.

[0124] The above describes a video generation method provided by an embodiment of this application. The following will describe a system that performs the above video generation method.

[0125] Please see Figure 2 , Figure 2 This is a schematic diagram of a video generation device provided in an embodiment of this application. Figure 2 As shown, the video generation apparatus includes:

[0126] The acquisition module 201 is used to acquire target text and a first video; wherein the first video includes multiple video segments, the target text is text describing the video content of a second video, and the second video is at least one video segment of the first video;

[0127] The text splitting module 202 is used to split the target text into multiple sub-texts with a time sequence according to the various stages of the development of the first video's plot; wherein, each sub-text describes the video content of one stage.

[0128] The search module 203 is used to find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segments in the first video to obtain the segment list of the subtext; wherein, the segment description information is the video content information describing a video segment of the first video, and the semantic relevance characterizes the similarity between the segment description information and the subtext.

[0129] The merging module 204 is used to merge the fragment lists of each sub-text in chronological order, and to merge the video fragments pointed to by the description information of each merged fragment to obtain the second video.

[0130] In one possible implementation, the video generation system provided in this application further includes:

[0131] The scene filtering module is used to search for video content information of the scene indicated by the target text from the scene description information set to obtain candidate scene description information; wherein, the scene description information set includes multiple scene description information, and the scene description information is video content information describing the scene of the first video; determine the video segment pointed to by the candidate scene description information to obtain the first segment; and aggregate the segment description information describing the first segment to obtain the segment description information set.

[0132] Optionally, module 203 is used specifically for:

[0133] From the set of fragment description information, find the fragment description information that has the highest semantic relevance to the subtext.

[0134] In one possible implementation, the video generation system provided in this application further includes:

[0135] The scene description information set generation module is used to perform segment boundary detection on the first video to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point; extract video frames before and after the segment boundary from the first video; perform scene switching point detection on the segment boundary based on the video frames, and take the segment boundary belonging to the scene switching point as the target boundary; wherein, the scene switching point is the dividing point between one scene and another scene; divide the first video into multiple scenes based on the target boundaries; generate video content information describing each scene to obtain the scene description information set.

[0136] In one possible implementation, the text splitting module 202 is specifically used for:

[0137] Using the large language model and the thought chain prompts, the target text is processed according to the various stages of the plot development in the first video, resulting in multiple sub-texts with a temporal sequence.

[0138] Among them, the thought chain prompt is used to instruct the large language model to process it in the following steps:

[0139] Identify the various stages of the plot development in the first video where the video content described by the target text is located, determine the order of each stage in the first video, and obtain the time sequence of the stages.

[0140] The target text is divided into multiple sub-texts according to the identified stages, and the temporal sequence of the stages is used as the temporal sequence of the sub-texts according to the mapping relationship between the sub-texts and the stages.

[0141] In one possible implementation, the lookup module 203 is specifically used for:

[0142] Using either method one or method two, find the descriptive information of the segment that has the highest semantic relevance to the subtext:

[0143] Method 1: Call the large language model to find the segment description information that has the highest semantic relevance to the subtext;

[0144] Method 2: Calculate the semantic similarity between the fragment description information and the subtext, and take the fragment description information with the highest semantic similarity as the fragment description information with the highest semantic relevance to the subtext.

[0145] In one possible implementation, the video generation system provided in this application further includes:

[0146] The adjustment module is used to obtain a second segment after obtaining the segment list of each subtext and before merging the segment lists of each subtext in chronological order; wherein the second segment is at least partially the same as the video segment pointed to by the segment description information in the segment list; based on the start and end points of the second segment, the segment description information in the segment list is adjusted so that the segment list of the adjusted subtext includes the segment description information of the second segment.

[0147] Optional, merging module 204, specifically used for:

[0148] The adjusted list of subtext fragments is merged in chronological order, and the video fragments pointed to by the description information of each merged fragment are merged to obtain a second video with a second fragment.

[0149] This application also provides an electronic device in its embodiments. (See reference...) Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0150] like Figure 3 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. When the electronic device is powered on, the RAM 303 also stores various programs and data required for the operation of the electronic device. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0151] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, memory cards, hard drives, etc.; and communication devices 309. Communication device 309 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0152] This electronic device is capable of implementing the video generation method described above.

[0153] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the video generation methods provided in this application.

[0154] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the video generation methods provided in this application.

[0155] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0157] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0158] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A video generation method, characterized in that, include: Obtain target text and a first video; wherein the first video includes multiple video segments, the target text is text describing the video content of a second video, and the second video is at least one video segment of the first video; According to the various stages of the development of the first video's plot, the target text is divided into multiple sub-texts with a temporal sequence; wherein, each sub-text describes the video content of one stage. Find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segment in the first video to obtain the segment list of the subtext; wherein, the segment description information is the video content information describing a video segment of the first video, and the semantic relevance represents the similarity between the segment description information and the subtext; The fragment lists of each subtext are merged according to the time sequence, and the video fragments pointed to by the description information of each merged fragment are merged to obtain the second video.

2. The video generation method according to claim 1, characterized in that, Also includes: From the scene description information set, video content information describing the scene indicated by the target text is searched to obtain candidate scene description information; wherein, the scene description information set includes multiple scene description information, and the scene description information is video content information describing the scene of the first video; The video segment pointed to by the candidate scene description information is determined to obtain the first segment; The fragment description information describing the first fragment is aggregated to obtain a fragment description information set; The process of finding the segment description information that has the highest semantic relevance to the subtext includes: From the set of fragment description information, find the fragment description information that has the highest semantic relevance to the subtext.

3. The video generation method according to claim 2, characterized in that, Also includes: Perform segment boundary detection on the first video to obtain the segment boundaries of the first video; wherein, the segment boundary includes the segment start point and the segment end point; Extract video frames before and after the segment boundary from the first video; Based on the video frames, scene switching points are detected on the boundaries of the segments, and the segments that belong to scene switching points are taken as target boundaries; wherein, the scene switching point is the boundary point between one scene and another scene; The first video is divided into multiple scenes based on the target boundary; Generate video content information describing each of the aforementioned scenes to obtain a set of scene description information.

4. The video generation method according to claim 1, characterized in that, The step of splitting the target text into multiple time-series sub-texts according to the various stages of the first video's plot development includes: Based on the large language model and the thought chain prompts, the target text is processed according to the various stages of the plot development of the first video to obtain multiple sub-texts with temporal sequence; The thought chain prompt is used to instruct the large language model to process the language in the following steps: Identify the various stages of the plot development of the video content described by the target text in the first video, and determine the order of each stage in the first video to obtain the temporal sequence of the stages; The target text is divided into multiple sub-texts according to the identified stages, and the time sequence of the stages is used as the time sequence of the sub-texts according to the mapping relationship between the sub-texts and the stages.

5. The video generation method according to any one of claims 1 to 4, characterized in that, The process of finding the segment description information that has the highest semantic relevance to the subtext includes: Using either method one or method two, find the segment description information that has the highest semantic relevance to the subtext: Method 1: Call the large language model to find the segment description information that has the highest semantic relevance to the subtext; Method 2: Calculate the semantic similarity between the fragment description information and the subtext, and take the fragment description information with the highest semantic similarity as the fragment description information with the highest semantic relevance to the subtext.

6. The video generation method according to any one of claims 1 to 4, characterized in that, After obtaining the list of fragments of the sub-text, and before merging the lists of fragments of each sub-text according to the chronological order, the method further includes: Obtain a second segment; wherein the second segment is at least partially identical in video content to the video segment pointed to by the segment description information in the segment list; Based on the start and end points of the second segment, the segment description information in the segment list is adjusted so that the segment list of the adjusted subtext includes the segment description information of the second segment; The step of merging the list of fragments of each of the sub-texts according to the chronological order, and merging the video fragments pointed to by the description information of each merged fragment to obtain the second video includes: The adjusted list of subtext segments is merged according to the chronological order, and the video segments pointed to by the description information of each merged segment are merged to obtain the second video with the second segment.

7. A video generation apparatus, characterized in that, include: An acquisition module is used to acquire target text and a first video; wherein the first video includes multiple video segments, the target text is text describing the video content of a second video, and the second video is at least one video segment of the first video; The text splitting module is used to split the target text into multiple sub-texts with a time sequence according to the various stages of the development of the first video plot; wherein, each sub-text describes the video content of one stage; The search module is used to find the segment description information with the highest semantic relevance to the subtext, and sort the found segment description information according to the order of the video segments in the first video to obtain the segment list of the subtext; wherein, the segment description information is the video content information describing a video segment of the first video, and the semantic relevance represents the similarity between the segment description information and the subtext; The merging module is used to merge the fragment lists of each of the sub-texts according to the time sequence, and merge the video fragments pointed to by the description information of each merged fragment to obtain the second video.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video generation method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the video generation method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the video generation method as described in any one of claims 1 to 6.

Images