Video generation method and related device

Abstract

The invention provides a video generation method and a related device, and the method comprises the steps: obtaining the editing information of a first video, the editing information comprises an editing time period and a first plot description text of a second video, the editing time period is a time period of the first video, and each video frame in the second video is a video frame in the editing time period; splitting the video clip in the editing time period into a plurality of sub-clips, obtaining a second plot description text of each sub-clip, and determining the similarity between the first plot description text and the second plot description text of each sub-clip; obtaining the sub-fragments of which the similarity meets a preset condition from all the sub-fragments; and according to the sequence of the sub-fragments of which the similarities meet the preset condition in the first video, splicing all the sub-fragments of which the similarities meet the preset condition to generate a second video, thereby realizing automatic generation of the second video according to the first plot description text of the second video. The editing efficiency is improved; and the content in the second video can accurately express the intention of the user.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video generation method and related apparatus. Background Technology

[0002] Currently, there are two main methods for generating a second video from a first video: one is purely manual editing: human editors, after understanding the content of the first video, use editing tools such as Adobe Premiere and CapCut to manually locate, crop, splice, and adjust the first video to obtain a second video; the other is a fully automatic highlight detection algorithm: the fully automatic highlight detection algorithm identifies exciting segments in the first video and generates a second video from these segments. The fully automatic highlight detection algorithm is based on the low-order features of the audio data in the first video, and extracts exciting segments from the first video according to these low-order features, such as extracting segments of laughter, applause, and screams from the first video as exciting segments. Alternatively, the fully automatic highlight detection algorithm can identify segments where there are shot changes or significant screen movement as exciting segments.

[0003] The two methods for generating a second video described above have the following drawbacks:

[0004] Manual editing is inefficient and costly: generating a second video may take tens of minutes or even hours, which cannot meet the needs of large-scale, timely content production.

[0005] The fully automated highlight detection algorithm cannot understand user intent, which can be represented by the narrative text. User intent refers to the editing theme the user currently wants to create. For example, if the narrative text is "Guest A complains that Guest B sings badly," it means the user wants to edit a second video of "Guest A complaining that Guest B sings badly." However, the content of the second video generated by the fully automated highlight detection algorithm is less relevant to the user intent, reducing accuracy.

[0006] Therefore, current methods of video generation cannot simultaneously achieve both "editing efficiency" and "accurately conveying user intent". Summary of the Invention

[0007] In view of the above problems, this application provides a video generation method and related apparatus to achieve a balance between "editing efficiency" and "accurately conveying human intent." The specific solution is as follows:

[0008] The first aspect of this application provides a video generation method, including:

[0009] Obtain editing information of the first video, the editing information including editing time period and first plot description text of the second video, the editing time period being a time period of the first video and each video frame in the second video being a video frame in the editing time period;

[0010] The video segment of the aforementioned editing time period is divided into multiple sub-segments;

[0011] Obtain the second plot description text for each sub-segment, and determine the similarity between the first plot description text and the second plot description text for each sub-segment;

[0012] Obtain the sub-segments whose similarity satisfies the preset conditions from all sub-segments;

[0013] The second video is generated by splicing together all the sub-segments that meet the preset similarity criteria in the first video according to their order.

[0014] In one possible implementation, determining the similarity between the first plot description text and the second plot description text for each sub-segment includes:

[0015] Obtain the first text feature of the first plot description text, and obtain the second text feature of the second plot description text of each sub-segment;

[0016] For any sub-segment, calculate the similarity between the first text feature and the second text feature of that sub-segment.

[0017] In one possible implementation, the editing information further includes the video style attribute of the second video; after determining the similarity between the first plot description text and the second plot description text of each sub-segment, the method further includes:

[0018] If the sub-segment possesses the aforementioned video style attribute, then increase the similarity of the sub-segment.

[0019] If the sub-segment does not possess the aforementioned video style attribute, then the similarity of the sub-segment is reduced.

[0020] In one possible implementation, the editing information further includes the video style attribute of the second video; determining the similarity between the first plot description text and the second plot description text of each sub-segment includes:

[0021] Obtain the first text feature of the first plot description text, and obtain the second text feature of the second plot description text of each sub-segment;

[0022] For any sub-segment, calculate the similarity between the first text feature and the second text feature of the sub-segment, and identify whether the sub-segment has the video style attribute to obtain the identification result of the sub-segment;

[0023] Based on the similarity between the first text feature and the second text feature of the sub-segment, and the recognition result of the sub-segment, the similarity between the first plot description text and the second plot description text of the sub-segment is obtained.

[0024] In one possible implementation, after obtaining the sub-segments whose similarity satisfies the preset condition from all sub-segments, the method further includes:

[0025] For any sub-segment whose similarity meets the preset conditions, if the text in the sub-segment has a problem of missing lines, determine the start time and end time of the sentence to which the text belongs. The missing lines mean that the text in the sub-segment is part of a sentence in the video segment, so that the meaning expressed by the text in the sub-segment is different from the meaning expressed by the text in the video segment.

[0026] The sub-segment located between the start time point and the end time point is re-extracted from the first video, and the re-extracted sub-segment is used to replace the original sub-segment.

[0027] In one possible implementation, after obtaining the sub-segments whose similarity satisfies the preset condition from all sub-segments, the method further includes:

[0028] Output the segment identifier of each sub-segment whose similarity satisfies the preset conditions;

[0029] Obtain the processing result of the user on the sub-segment based on the segment identifier, wherein the processing result includes deleting the sub-segment whose similarity meets the preset condition or retaining the sub-segment whose similarity meets the preset condition;

[0030] Based on the processing results of each sub-segment, determine which sub-segments will be retained.

[0031] In one possible implementation, the method further includes: obtaining a thought chain prompt, the thought chain prompt being used to instruct a preset multimodal large model to be processed according to the following steps:

[0032] Step A: Compare the semantic relevance of the second plot description text and the first plot description text of each sub-segment to output a relevance score for each sub-segment. The relevance score is used to represent the similarity between the second plot description text and the first plot description text of the sub-segment.

[0033] Step B: Select sub-segments that meet the preset conditions based on their relevance scores;

[0034] The step of determining the similarity between the first plot description text and the second plot description text of each sub-segment, and obtaining the sub-segments whose similarity satisfies the preset conditions from all sub-segments, includes:

[0035] The preset multimodal large model processes the second plot description text of each sub-segment with reference to the thought chain prompts to obtain the sub-segments that meet the preset conditions selected by the preset multimodal large model.

[0036] In one possible implementation, the thought chain prompt is also used to instruct the preset multimodal large model to be processed in the following steps:

[0037] Step C: Divide the input video clip into multiple sub-segments, and for each sub-segment, generate a second plot description text for that sub-segment;

[0038] Step D: Output a list of segments, which includes the segment identifiers of all selected sub-segments, and the segment identifiers of the sub-segments are arranged in order in the first video;

[0039] The process of splitting the video segment of the edited time period into multiple sub-segments, obtaining the second plot description text for each sub-segment, and determining the similarity between the first plot description text and the second plot description text of each sub-segment; and obtaining sub-segments from all sub-segments whose similarity satisfies a preset condition, includes:

[0040] The video clip is input into the preset multimodal large model, which processes the video clip to obtain the clip list output by the preset multimodal large model.

[0041] A second aspect of this application provides a video generation apparatus, comprising:

[0042] The first acquisition unit is used to acquire editing information of the first video, the editing information including an editing time period and a first plot description text of the second video, the editing time period being a time period of the first video and each video frame in the second video being a video frame in the editing time period.

[0043] A splitting unit is used to split a video segment of the edited time period into multiple sub-segments;

[0044] The second acquisition unit acquires the second plot description text of each sub-segment and determines the similarity between the first plot description text and the second plot description text of each sub-segment.

[0045] The third acquisition unit is used to acquire the sub-segments whose similarity satisfies the preset conditions from all sub-segments;

[0046] The generation unit is used to splice all the sub-segments that meet the preset similarity conditions in the first video according to the order of the sub-segments that meet the preset similarity conditions in the first video, so as to generate the second video.

[0047] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0048] The memory is used to store computer programs;

[0049] The processor is used to execute the computer program so that the electronic device can implement the video generation method of the first aspect or any implementation thereof.

[0050] A fourth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video generation method described in the first aspect or any implementation thereof.

[0051] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the video generation method described in the first aspect or any implementation thereof.

[0052] By employing the above technical solution, the video generation method and related apparatus provided in this application obtain editing information of a first video, including an editing time period and a first plot description text of the second video. The editing time period is a time segment of the first video, and each video frame in the second video is a video frame within the editing time period. The video segment of the editing time period is split into multiple sub-segments, and the second plot description text of each sub-segment is obtained. The similarity between the first plot description text and the second plot description text of each sub-segment is determined. Sub-segments whose similarity meets preset conditions are obtained from all sub-segments. All sub-segments whose similarity meets preset conditions are spliced ​​together according to the order of the sub-segments in the first video to generate the second video. This achieves automatic generation of the second video based on the first plot description text of the second video, improving editing efficiency. Furthermore, the first plot description text can represent user intent. If the second plot description text of a sub-segment in the second video is similar to the first plot description text, it indicates that the sub-segment in the second video can represent user intent, enabling the content in the second video to accurately express user intent, thereby balancing "editing efficiency" and "accurately conveying user intent". Attached Figure Description

[0053] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0054] Figure 1 A flowchart of a video generation method provided in this application;

[0055] Figure 2 Another flowchart of a video generation method provided in this application;

[0056] Figure 3 Another flowchart of a video generation method provided in this application;

[0057] Figure 4 A schematic diagram of a video generation device provided in this application;

[0058] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0059] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0060] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0061] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to such processes, methods, systems, products, or apparatus.

[0062] Currently, video generation methods include manual video editing and automatic identification of exciting segments using fully automated highlight detection algorithms to generate videos from those segments. However, current methods cannot simultaneously achieve both "editing efficiency" and "accurately conveying human intent."

[0063] To address the aforementioned technical problems, this application provides a video generation method and related apparatus. The method involves acquiring editing information of a first video, including an editing time period and a first plot description text of a second video. The editing time period is a segment of the first video, and each video frame in the second video is a video frame within that editing time period. The method further involves splitting the video segment of the editing time period into multiple sub-segments, acquiring the second plot description text of each sub-segment, and determining the similarity between the first plot description text and the second plot description text of each sub-segment. Sub-segments whose similarity satisfies a preset condition are then acquired from all sub-segments. Finally, all sub-segments whose similarity satisfies the preset condition are concatenated according to their order in the first video to generate the second video. This method automatically generates the second video based on the first plot description text, improving editing efficiency. Furthermore, since the first plot description text can represent user intent, if the second plot description text of a sub-segment in the second video is similar to the first plot description text, it indicates that the sub-segment in the second video can represent user intent. This allows the content in the second video to accurately express user intent, thus balancing "editing efficiency" with "accurately conveying user intent."

[0064] The following description, in conjunction with the accompanying drawings, illustrates a video generation method and related apparatus provided in an embodiment of this application. Please refer to... Figure 1 This illustrates an optional flow of the video generation method provided in this application embodiment, which may include the following steps:

[0065] S101. Obtain the editing information of the first video, which includes the editing time period and the first plot description text of the second video. The first plot description text is an overview text used to express the user's intent. The overview text can describe the user's intent in concise and precise language.

[0066] In this context, the editing time segment is a specific time period within the first video, and each video frame in the second video is a video frame within that editing time segment. In other words, the editing time segment indicates the point in the first video from which the editing begins, thus indicating the source of the second video. The editing time segment can include a start time point and an end time point; editing takes place from the start time point to the end time point.

[0067] In some examples, after watching the first video, the user summarizes the initial plot description text that expresses their intent and specifies the editing timeframe. Thus, the user is no longer a mere editor performing tedious editing operations, but rather plays the role of a director, responsible for top-level creative decisions: determining the editing timeframe and the initial plot description text for the second video. Furthermore, creative decisions also include the preset duration of the second video, which indicates the length of the generated second video and can be recorded in the editing information. The electronic device that invokes the video generation method then acts as an intelligent editor capable of deeply understanding the director's intent, responsible for all editing work to generate a second video that expresses the user's intentions.

[0068] In some examples, the first video input is fed into a first preset description text generation model to obtain a first plot description text output by the first preset description text generation model. The first preset description text generation model can be trained based on historical videos and historical plot description texts of historical videos. The training process and architecture of the first preset description text generation model are not limited in this embodiment.

[0069] S102. Divide the video clip of the editing time period into multiple sub-segments.

[0070] In some examples, the editing time period may include a start time point and an end time point. In this case, the start time point and end time point are found in the first video, and the video segment from the start time point to the end time point is extracted.

[0071] In this embodiment, the video segment is randomly split to obtain multiple sub-segments of the video segment, or the video segment is split according to the lines in the video segment, with each sub-segment including at least one line of lines, or the video segment is split according to the shot, such as calling a shot detection model (such as the TransNetv2 model) to detect shot transition points in the video segment and splitting the video segment by shot according to the shot transition points, or splitting according to a specific duration so that each sub-segment has the same duration.

[0072] S103, Obtain the second plot description text for each sub-fragment.

[0073] The second plot description text can be an overview of the core content of a sub-segment, expressing the theme / idea / user intent of the core content. This second plot description text can be generated by a second preset description text generation model. Specifically, the sub-segment is input into the second preset description text generation model, which outputs the second plot description text. This model is trained based on historical sub-segments and their historical plot description texts. The specific process and structure of the second preset description text generation model are not elaborated here; for example, the second preset description text generation model can be, but is not limited to, the CLIP4Caption model. In some examples, the second plot description text can be summarized and written by the user after watching the sub-segment, but using the second preset description text generation model to generate the second plot description text achieves automatic generation, reduces generation costs, and improves timeliness.

[0074] Furthermore, the content of the second plot description text is similar to that of the first plot description text. For example, both the first and second plot description texts can include the behavior of the object. For instance, the first plot description text can include the characters involved in the second video and their behavior, and the second plot description text also needs to include the characters involved in the sub-segment and their behavior. Thus, by using the first and second plot description texts, it can be determined whether the description content in the two texts is related, thereby determining whether the sub-segment can be a sub-segment in the second video.

[0075] S104. Determine the similarity between the first plot description text and the second plot description text of each sub-segment.

[0076] In this embodiment, similarity is used to indicate whether the second plot description text of a sub-segment is related to the first plot description text of the second video. The second plot description text is a summary text of the content in the sub-segment, so similarity can be used to indicate whether the content in the sub-segment is related to the first plot description text of the second video. It can be understood that the higher the similarity, the more relevant the content in the sub-segment is to the first plot description text of the second video, and thus the sub-segment can be used to generate the second video.

[0077] In one possible implementation, the similarity between the first plot description text and the second plot description text of the sub-segment can be a binary classification result, which indicates whether the first plot description text and the second plot description text of the sub-segment are similar or dissimilar. The binary classification result can be obtained based on whether the theme of the clip corresponding to the first plot description text and the theme corresponding to the second plot description text of the sub-segment are the same. If the theme of the clip corresponding to the first plot description text and the theme corresponding to the second plot description text of the sub-segment are the same, a binary classification result indicating that the first plot description text and the second plot description text of the sub-segment are similar is obtained; if the theme of the clip corresponding to the first plot description text (i.e., the user intent) and the theme corresponding to the second plot description text of the sub-segment are different, a binary classification result indicating that the first plot description text and the second plot description text of the sub-segment are dissimilar is obtained. The theme corresponding to the second plot description text of the sub-segment can be simply referred to as the theme of the sub-segment. In this embodiment, the methods for determining whether the theme of the sub-segment and the clip theme are the same may include, but are not limited to, the following:

[0078] If the first and second plot description texts contain the same keywords, then the sub-segment's theme and the edit's theme are determined to be the same; otherwise, the sub-segment's theme and the edit's theme are determined to be different. Alternatively, if the first and second plot description texts form a question-and-answer relationship, then the sub-segment's theme and the edit's theme are determined to be the same; otherwise, the sub-segment's theme and the edit's theme are determined to be different. Alternatively, the first and second plot description texts can be input into a theme prediction model, and the output of the theme prediction model can be used to determine whether the two themes are the same. The theme prediction model can be trained based on historical plot description texts and their corresponding themes; the specific process will not be elaborated further.

[0079] In one possible implementation, this embodiment may determine whether the first plot description text and the second plot description text of the sub-segment are similar in the following ways:

[0080] If the first and second plot description texts contain the same keywords, then the first plot description text and the second plot description text of the sub-segment are considered similar; otherwise, they are considered dissimilar. Alternatively, if the first and second plot description texts form a question-and-answer relationship, then they are considered similar; otherwise, they are considered dissimilar. Alternatively, the first and second plot description texts can be input into a topic prediction model to obtain the clip topic to which the first plot description text belongs and the topic topic to which the second plot description text belongs, and then it can be determined whether these two topics are the same. The topic prediction model can be trained based on historical plot description texts and their corresponding topics; the specific process will not be elaborated further.

[0081] In one possible implementation, determining the similarity between the first plot description text and the second plot description text for each sub-segment includes:

[0082] Obtain the first text feature of the first plot description text, and obtain the second text feature of the second plot description text of each sub-segment; for any sub-segment, calculate the similarity between the first text feature and the second text feature of that sub-segment.

[0083] For any one of the first and second plot description texts, a text feature extraction model (such as bge-m3) is used to generate text features for that plot description text. The text feature extraction model can be trained based on historical plot description texts and their historical text features. The specific process and architecture of the text feature extraction model will not be detailed in this embodiment. In some examples, the content of the first and second plot description texts is similar, making the input to the text feature extraction model similar. This allows the text feature model to accurately extract the text features of each plot description text, thereby improving the accuracy of the similarity score.

[0084] In this embodiment, the similarity between text features can be any one of the following: cosine similarity between text features, distance between text features, and Pearson correlation coefficient between text features.

[0085] Optionally, the editing information may also include the video style attribute of the first video, such as specifying the video style attribute as fast-paced or humorous. The video style attribute can be applied to determining the similarity between the first plot description text and the second plot description text of the sub-segment. One feasible approach is to, after determining the similarity between the first plot description text and the second plot description text of the sub-segment, increase the similarity of the sub-segment if it possesses a video style attribute, and decrease its similarity if it does not. The extent of the increase or decrease is not limited in this embodiment.

[0086] Another feasible approach is to obtain the first text features of the first plot description text and the second text features of the second plot description text for each sub-segment. For any sub-segment, the similarity between the first text features and the second text features of the sub-segment is calculated to identify whether the sub-segment has a video style attribute, thus obtaining the identification result of the sub-segment. Based on the similarity between the first text features and the second text features of the sub-segment, and the identification result of the sub-segment, the similarity between the first plot description text and the second plot description text of the sub-segment is obtained. If the sub-segment has a video style attribute, the identification result of the sub-segment can be recorded as 1; if the sub-segment does not have a video style attribute, the identification result of the sub-segment can be recorded as 0. Then, a preset calculation algorithm (such as a weighted algorithm) is used to process the similarity between the first text features and the second text features of the sub-segment, and the identification result of the sub-segment, to obtain the comprehensive score of the sub-segment. The comprehensive score of the sub-segment is the similarity between the first plot description text and the second plot description text of the sub-segment.

[0087] Taking a fast-paced video style as an example, a fast-paced style tends to select shorter segments with more visual variations, i.e., segments with high information density. When calculating similarity using the fast-paced attribute, the similarity of sub-segments with fast pacing may be greater than that of sub-segments without fast pacing. This allows for the selection of sub-segments with fast pacing as much as possible, thereby generating a second video with high information density and richer visual variations.

[0088] S105. Obtain sub-segments whose similarity meets the preset conditions from all sub-segments.

[0089] In one possible implementation, obtaining sub-segments whose similarity meets a preset condition includes: sorting them from high to low similarity, selecting from the highest-ranked sub-segment until the total duration of all selected sub-segments is close to or equal to the preset duration of the second video, where the preset duration of the second video can be recorded in the editing information; or, selecting a preset number of sub-segments from the highest-ranked sub-segment, where the value of the preset number is not limited in this embodiment; or, obtaining sub-segments whose similarity is greater than a preset similarity threshold, where the value of the preset similarity threshold is not limited in this embodiment.

[0090] In one possible implementation, obtaining sub-segments whose similarity satisfies a preset condition includes: retaining the first and last sub-segments of the video segment; selecting sub-segments whose similarity satisfies a preset condition from the remaining sub-segments of the video segment based on the similarity between the first plot description text and the second plot description text of the remaining sub-segments; and ensuring that all obtained sub-segments satisfy any one of the following conditions:

[0091] The total duration of all acquired sub-segments is close to or equal to the preset duration of the second video, or the number of acquired sub-segments is not less than the preset number, or the similarity of the sub-segments selected based on similarity is greater than the preset similarity threshold.

[0092] S106. Following the order of the sub-segments that meet the preset similarity conditions in the first video, splice all the sub-segments that meet the preset similarity conditions to generate the second video.

[0093] The order of sub-segments in the first video can be determined based on the start and / or end time of the sub-segment in the first video. After all sub-segments that meet the preset similarity conditions in the order of the sub-segments in the first video are processed, the second video of the first video is obtained, thereby completing the editing of the first video according to the first plot description text of the second video.

[0094] In some examples, when splicing all sub-segments that meet the preset similarity criteria, a transition effect is added between two adjacent sub-segments. The transition effect completes the switching between two adjacent sub-segments, thereby reducing the abruptness of the switching between two adjacent sub-segments and improving the coherence between two adjacent sub-segments.

[0095] The transition effects include, but are not limited to, black fade-in / fade-out, white fade-in / fade-out, and dissolve. If two adjacent sub-segments are sub-segments from different shots within the same scene, the transition effect used for the two adjacent sub-segments is dissolve; if two adjacent sub-segments are sub-segments from different scenes, the transition effect used for the two adjacent sub-segments is fade-in / fade-out. Furthermore, the color tone of the two adjacent sub-segments is determined. If the color tone of the two adjacent sub-segments is darker, a black fade-in / fade-out is used; if the color tone of the two adjacent sub-segments is brighter, a white fade-in / fade-out is used. Whether two adjacent sub-segments are from the same scene or different scenes can be determined by a scene recognition model. The scene recognition model can be determined based on historical sub-segments and the scenes to which the historical sub-segments belong. The specific process and the architecture of the scene recognition model are not limited in this embodiment.

[0096] By employing the above technical solution, the video generation method provided in this application embodiment can obtain the editing information of a first video, including an editing time period and a first plot description text of a second video. The editing time period is a time segment of the first video, and each video frame in the second video is a video frame within the editing time period. The video segment of the editing time period is split into multiple sub-segments, and the second plot description text of each sub-segment is obtained. The similarity between the first plot description text and the second plot description text of each sub-segment is determined, and sub-segments whose similarity meets preset conditions are obtained from all sub-segments. According to the order of similarity in the first video that meets the preset similarity conditions, all sub-segments whose similarity meets the preset conditions are spliced ​​together to generate a second video. This achieves automatic generation of the second video based on the first plot description text of the second video, improving editing efficiency. Furthermore, the first plot description text can represent user intent. If the second plot description text of a sub-segment in the second video is similar to the first plot description text, it indicates that the sub-segment in the second video can represent user intent, enabling the content in the second video to accurately express user intent, thereby balancing "editing efficiency" and "accurately conveying user intent".

[0097] Please see Figure 2 This illustrates another optional process of a video generation method provided in an embodiment of this application, which may include the following steps:

[0098] S201. Obtain the editing information of the first video. The editing information includes the editing time period and the first plot description text of the second video. The editing time period is a time period of the first video and each video frame in the second video is a video frame in the editing time period.

[0099] S202. Divide the video clip of the editing time period into multiple sub-segments.

[0100] S203, Obtain the second plot description text for each sub-fragment.

[0101] S204. Determine the similarity between the first plot description text and the second plot description text of each sub-segment.

[0102] S205. Obtain sub-segments whose similarity meets the preset conditions from all sub-segments.

[0103] Steps S201 to S205 can be referred to as steps S101 to S105 above, and will not be described in detail here.

[0104] S206. For any sub-segment whose similarity meets the preset conditions, if there is a problem of missing lines in the text of the sub-segment, determine the start time and end time of the sentence to which the text belongs.

[0105] In this context, "missing dialogue" refers to a situation where the text in a sub-segment is only a portion of a sentence in a video clip, causing the meaning of the text in the sub-segment to differ from its meaning in the video clip. This missing dialogue occurs because the text is truncated during the process of splitting the video clip into multiple sub-segments. This truncation includes both dialogue truncation and semantic truncation. Dialogue truncation refers to the removal of a single line of dialogue during the segmentation process, while semantic truncation refers to the removal of a single sentence. A single sentence can include at least two lines of dialogue.

[0106] In this embodiment, text truncation indicates that the text is a partial content of a line of dialogue. Therefore, if the text is a partial content of the line to which it belongs, it is determined that the text has a truncation problem. The process may include: checking the start and end times of the text; if the start time of the text is after the start time of a line of dialogue, and the end time of the text is before the end time of the same line of dialogue, then it is said that the text is a partial content of the line to which it belongs.

[0107] Semantic truncation in text indicates that the text is a complete line of dialogue, but that complete line is only a part of a sentence. Normally, a sentence ends with a period, and lines of dialogue within a sentence end with commas, question marks, exclamation marks, or other punctuation marks. If the last punctuation mark in the text is not a period, it confirms that the text is only a part of a sentence, thus indicating a semantic truncation problem.

[0108] S207. Extract the sub-segment located between the start time point and the end time point from the first video, and use the extracted sub-segment to replace the original sub-segment.

[0109] In this embodiment, by re-extracting sub-segments using the start and end times of the sentence to which the text belongs, it can be ensured that the beginning and end of a sentence are within the sub-segment, thereby making the content in the sub-segments of the generated second video complete and avoiding the impact of incomplete content on the understanding of the second video.

[0110] S208. According to the order of the sub-segments that meet the preset similarity conditions in the first video, splice all the segments that meet the preset similarity conditions to generate the second video.

[0111] In this embodiment, for any sub-segment whose similarity meets the preset conditions, if there is a problem of missing lines in the text of the sub-segment, the start time and end time of the sentence to which the text belongs are determined, and the sub-segment located between the start time and the end time is re-extracted from the first video. The re-extracted sub-segment is used to replace the original sub-segment, which can ensure that the beginning and end of a sentence are within the sub-segment, thereby making the content in the sub-segment of the generated second video complete and avoiding the impact of incomplete content on the understanding of the second video.

[0112] Please see Figure 3 This illustrates another optional process of a video generation method provided in an embodiment of this application, which may include the following steps:

[0113] S301. Obtain the editing information of the first video. The editing information includes the editing time period and the first plot description text of the second video. The editing time period is a time period of the first video and each video frame in the second video is a video frame in the editing time period.

[0114] S302. Divide the video clip of the editing time period into multiple sub-segments.

[0115] S303, Obtain the second plot description text for each sub-fragment.

[0116] S304. Determine the similarity between the first plot description text and the second plot description text of each sub-segment.

[0117] S305. Obtain sub-segments whose similarity meets the preset conditions from all sub-segments.

[0118] Steps S301 to S305 can be referred to as steps S101 to S105 above, and will not be described in detail here.

[0119] S306. Output the segment identifier of each sub-segment whose similarity meets the preset conditions.

[0120] S307. Obtain the processing results of the sub-segments by the user based on the segment identifier, wherein the processing results include deleting sub-segments whose similarity meets the preset conditions or retaining sub-segments whose similarity meets the preset conditions.

[0121] S308. Based on the processing results of each sub-segment, determine the sub-segments to be retained.

[0122] S309. Sort all the retained sub-segments according to their order in the first video to generate the second video.

[0123] The segment identifier of a sub-segment is used to point to a unique sub-segment. For example, the segment identifier of a sub-segment can be determined according to the order of the sub-segment in the first video. For example, the segment identifier of a sub-segment can be 1, 2, 3, etc. The earlier the sub-segment is in the first video, the smaller its segment identifier value is; or, the segment identifier of a sub-segment can be the start time point or end time point of the sub-segment in the first video.

[0124] After the segment identifiers are output, the user decides which segments to delete and / or which to keep. The kept segments are used to generate a second video, thereby improving flexibility and personalization of the second video.

[0125] In some examples, this embodiment can transform a vague editing task into a series of precise, executable machine instructions through a chain-of-thought (CoT) prompt. CoT is used to simulate the thought process of a human editor. When a human editor receives an editing task, their thought process is as follows:

[0126] (1) Review the materials: First, watch the first few minutes of material (i.e., the first video) specified by the director.

[0127] (2) Marking: During the viewing process, mark the content related to the editing theme on the timeline. The content related to the editing theme includes, but is not limited to, key dialogues, core actions and important reactions.

[0128] (3) Rough cut assembly: splice all the marked segments together in order to get the rough cut video.

[0129] (4) Fine-tuning: Check if the rough cut video has timed out. If it has timed out (i.e., the length of the rough cut video is longer than the preset length), then the segments in the rough cut video will be cut until the length is equal to the preset length, or the length is less than the preset length but close to the preset length. The cutting process is as follows: human editors delete unimportant scene segments in the rough cut video. If the length of the rough cut video is still longer than the preset length after deleting unimportant scene segments, then unimportant sub-segments in the rough cut video will be deleted.

[0130] Research revealed that the deleted scene segments and sub-segments were all segments with low relevance to the editing theme. Therefore, this embodiment selects sub-segments based on their relevance to the editing theme. The relevance of a sub-segment to the editing theme can be represented by the similarity between the second plot description text of the sub-segment and the first plot description text of the second video. The first plot description text represents the user's intent, which is the editing theme the user currently wants to edit. Furthermore, a scene segment may contain many sub-segments unrelated to the editing theme, and a scene segment may be quite long. The second video generated based on scene segments may have poor accuracy and require a long processing time (e.g., after deleting scene segments, sub-segments must also be deleted). Therefore, this embodiment preferably generates the second video based on sub-segments.

[0131] Referring to the thought process of human editors, the video generation method provided in this application embodiment can pre-generate CoT cues. The thought chain cues are used to instruct the preset multimodal large model to process according to the following steps:

[0132] Step A: Compare the semantic relevance of the second plot description text and the first plot description text of each sub-segment, and output a relevance score for each sub-segment. The relevance score is used to represent the similarity between the second plot description text and the first plot description text of the sub-segment.

[0133] Step B: Select sub-segments that meet the preset conditions based on their relevance scores.

[0134] Therefore, the pre-defined multimodal large model processes the second plot description text of each sub-segment based on the thought chain prompts to obtain the sub-segments that meet the pre-defined conditions selected by the pre-defined multimodal large model.

[0135] Furthermore, the mind chain prompt is also used to instruct the preset multimodal large model to be processed in the following steps:

[0136] Step C: Divide the input video clip into multiple sub-segments, and for each sub-segment, generate a second plot description text for that sub-segment;

[0137] Step D: Output a list of segments, which includes the segment identifiers of all selected sub-segments, arranged in the order they appear in the first video.

[0138] Therefore, the video clips are input into a preset multimodal large model, which processes the video clips to obtain a list of clips output by the preset multimodal large model. The preset multimodal large model then completes the splitting of the video clips, generates the second plot description text of the sub-segments in the video clips, obtains the similarity between the second plot description text and the first plot description text of the sub-segments, selects the sub-segments that meet the preset conditions, and outputs the clip identifiers of all selected sub-segments in the form of a clip list.

[0139] Therefore, this embodiment does not simply identify sub-segments related to the editing theme, but rather seeks the optimal solution under multiple constraints. This ensures that the final output is not a rough compilation of materials, but a carefully arranged second video that meets multiple constraints, including editing time periods and a first plot description text, as well as a preset duration. The corresponding preset multimodal model can automatically segment, describe, match, and edit the specified video segments based on CoT prompts. Finally, under the premise of satisfying all multiple constraints, a segment list is output. This segment list records the segment identifiers of all sub-segments. A second video with a coherent plot and a tight rhythm can be generated by using the sub-segments corresponding to the segment identifiers.

[0140] In some instances, the mind chain prompt also instructs the editing of a second video with a total duration close to the preset duration. During the editing process, the original first and last sub-segments are retained. From all remaining sub-segments, selections are made based on similarity from highest to lowest. The segment identifiers of the selected sub-segments are added to a segment list until the total duration of all sub-segments in the segment list is close to or equal to the preset duration, thus generating a second video with a duration of the preset duration or close to it. For example, the preset duration could be 30 to 60 seconds. Using the above video generation method, a second video with a duration of 30 to 60 seconds can be generated from the first video, essentially creating a short video for the first video, facilitating its promotion.

[0141] The above describes a video generation method provided by an embodiment of this application. The following describes the apparatus for performing the above video generation method.

[0142] Please see Figure 4 , Figure 4 This is a schematic diagram of a second video generation apparatus provided in an embodiment of this application. For a description of each unit in the second video generation apparatus shown in Figure 4, please refer to the above method embodiment. Figure 4 As shown, the second video generation device may include: a first acquisition unit 10, a splitting unit 20, a second acquisition unit 30, a third acquisition unit 40, and a generation unit 50.

[0143] The first acquisition unit 10 is used to acquire the editing information of the first video. The editing information includes the editing time period and the first plot description text of the second video. The editing time period is a time period of the first video and each video frame in the second video is a video frame in the editing time period.

[0144] Splitting unit 20 is used to split a video segment of an edited time period into multiple sub-segments.

[0145] The second acquisition unit 30 acquires the second plot description text of each sub-segment and determines the similarity between the first plot description text and the second plot description text of each sub-segment.

[0146] The third acquisition unit 40 is used to acquire sub-segments whose similarity meets preset conditions from all sub-segments.

[0147] The generation unit 50 is used to splice all the sub-segments that meet the preset similarity conditions in the first video according to the order of the sub-segments that meet the preset similarity conditions, so as to generate the second video.

[0148] In one possible implementation, the second acquisition unit 30 is used to acquire the first text features of the first plot description text and the second text features of the second plot description text of each sub-segment; for any sub-segment, the similarity between the first text features and the second text features of the sub-segment is calculated.

[0149] In one possible implementation, the editing information also includes the video style attribute of the second video; the second acquisition unit 30 is further configured to increase the similarity of the sub-segments if the sub-segments have a video style attribute, and decrease the similarity of the sub-segments if the sub-segments do not have a video style attribute.

[0150] In one possible implementation, the editing information also includes the video style attribute of the second video; the second acquisition unit 30 is used to acquire the first text feature of the first plot description text and the second text feature of the second plot description text of each sub-segment; for any sub-segment, calculate the similarity between the first text feature and the second text feature of the sub-segment, identify whether the sub-segment has a video style attribute, so as to obtain the identification result of the sub-segment; based on the similarity between the first text feature and the second text feature of the sub-segment and the identification result of the sub-segment, obtain the similarity between the first plot description text and the second plot description text of the sub-segment.

[0151] In one possible implementation, the video generation apparatus further includes an extraction unit, which is used to determine the start and end times of the sentence to which the text belongs if the text in any sub-segment that meets the preset similarity conditions has a problem of missing lines. The missing lines refer to the text in the sub-segment being part of a sentence in the video segment, such that the meaning expressed by the text in the sub-segment is different from the meaning expressed by the text in the video segment. The sub-segment located between the start and end times is re-extracted from the first video, and the re-extracted sub-segment is used to replace the original sub-segment.

[0152] In one possible implementation, the video generation apparatus further includes an output unit and a processing unit. The output unit outputs segment identifiers for each sub-segment whose similarity meets preset conditions. The processing unit acquires the processing results of the user on the sub-segments based on the segment identifiers, wherein the processing results include deleting or retaining sub-segments whose similarity meets preset conditions; and determines the retained sub-segments based on the processing results for each sub-segment.

[0153] In one possible implementation, the video generation device further includes: a thought chain acquisition unit, used to acquire thought chain prompts, which instruct a preset multimodal large model to process the following steps: Step A: compare the semantic relevance of the second plot description text and the first plot description text of each sub-segment to output a relevance score for each sub-segment, whereby the relevance score represents the similarity between the second plot description text and the first plot description text of the sub-segment; Step B: select sub-segments that meet preset conditions according to the relevance score; correspondingly, the second acquisition unit and the third acquisition unit are used to trigger the preset multimodal large model to process the second plot description text of each sub-segment with reference to the thought chain prompts to obtain the sub-segments that meet the preset conditions selected by the preset multimodal large model.

[0154] In one possible implementation, the thought chain prompt is also used to instruct the preset multimodal large model to process the video clips as follows: Step C: Split the input video clips into multiple sub-segments, and generate a second plot description text for each sub-segment; Step D: Output a list of segments, which includes the segment identifiers of all selected sub-segments, and the segment identifiers of the sub-segments are arranged in the order of the first video; Correspondingly, the splitting unit, the second acquisition unit, and the third acquisition unit are used to trigger the preset multimodal large model to input the video clips into the preset multimodal large model, and the preset multimodal large model processes the video clips to obtain the list of segments output by the preset multimodal large model.

[0155] This application also provides an electronic device in its embodiments. (See reference...) Figure 5The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0156] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404. The processing unit 401 can be considered as a processor in the electronic device, and the ROM 402, RAM 403, and storage device 408 can be considered as memory in the electronic device. The memory is used to store computer programs; the processor is used to execute the computer programs to enable the electronic device to implement any of the video generation methods provided in the embodiments of this application.

[0157] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0158] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the video generation methods provided in this application.

[0159] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the video generation methods provided in this application.

[0160] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0162] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0163] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A video generation method, characterized in that, include: Obtain editing information of the first video, the editing information including editing time period and first plot description text of the second video, the editing time period being a time period of the first video and each video frame in the second video being a video frame in the editing time period; The video segment of the aforementioned editing time period is divided into multiple sub-segments; Obtain the second plot description text for each sub-segment, and determine the similarity between the first plot description text and the second plot description text for each sub-segment; Obtain the sub-segments whose similarity satisfies the preset conditions from all sub-segments; The second video is generated by splicing together all the sub-segments that meet the preset similarity criteria in the first video according to their order.

2. The method according to claim 1, characterized in that, The determination of the similarity between the first plot description text and the second plot description text of each sub-segment includes: Obtain the first text feature of the first plot description text, and obtain the second text feature of the second plot description text of each sub-segment; For any sub-segment, calculate the similarity between the first text feature and the second text feature of that sub-segment.

3. The method according to claim 2, characterized in that, The editing information also includes the video style attributes of the second video; after determining the similarity between the first plot description text and the second plot description text of each sub-segment, the method further includes: If the sub-segment possesses the aforementioned video style attribute, then increase the similarity of the sub-segment. If the sub-segment does not possess the aforementioned video style attribute, then the similarity of the sub-segment is reduced.

4. The method according to claim 1, characterized in that, The editing information also includes the video style attributes of the second video; determining the similarity between the first plot description text and the second plot description text of each sub-segment includes: Obtain the first text feature of the first plot description text, and obtain the second text feature of the second plot description text of each sub-segment; For any sub-segment, calculate the similarity between the first text feature and the second text feature of the sub-segment, and identify whether the sub-segment has the video style attribute to obtain the identification result of the sub-segment; Based on the similarity between the first text feature and the second text feature of the sub-segment, and the recognition result of the sub-segment, the similarity between the first plot description text and the second plot description text of the sub-segment is obtained.

5. The method according to claim 1, characterized in that, After obtaining the sub-segments whose similarity satisfies the preset condition from all sub-segments, the method further includes: For any sub-segment whose similarity meets the preset conditions, if the text in the sub-segment has a problem of missing lines, determine the start time and end time of the sentence to which the text belongs. The missing lines mean that the text in the sub-segment is part of a sentence in the video segment, so that the meaning expressed by the text in the sub-segment is different from the meaning expressed by the text in the video segment. The sub-segment located between the start time point and the end time point is re-extracted from the first video, and the re-extracted sub-segment is used to replace the original sub-segment.

6. The method according to claim 1, characterized in that, After obtaining the sub-segments whose similarity satisfies the preset condition from all sub-segments, the method further includes: Output the segment identifier of each sub-segment whose similarity satisfies the preset conditions; Obtain the processing result of the user on the sub-segment based on the segment identifier, wherein the processing result includes deleting the sub-segment whose similarity meets the preset condition or retaining the sub-segment whose similarity meets the preset condition; Based on the processing results of each sub-segment, determine which sub-segments will be retained.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: obtaining thought chain prompts, which are used to instruct a preset multimodal large model to be processed according to the following steps: Step A: Compare the semantic relevance of the second plot description text and the first plot description text of each sub-segment to output a relevance score for each sub-segment. The relevance score is used to represent the similarity between the second plot description text and the first plot description text of the sub-segment. Step B: Select sub-segments that meet the preset conditions based on their relevance scores; The step of determining the similarity between the first plot description text and the second plot description text of each sub-segment, and obtaining the sub-segments whose similarity satisfies the preset conditions from all sub-segments, includes: The preset multimodal large model processes the second plot description text of each sub-segment with reference to the thought chain prompts to obtain the sub-segments that meet the preset conditions selected by the preset multimodal large model.

8. The method according to claim 7, characterized in that, The thought chain prompt is also used to instruct the preset multimodal large model to be processed in the following steps: Step C: Divide the input video clip into multiple sub-segments, and for each sub-segment, generate a second plot description text for that sub-segment; Step D: Output a list of segments, which includes the segment identifiers of all selected sub-segments, and the segment identifiers of the sub-segments are arranged in order in the first video; The process involves splitting the video segment of the edited time period into multiple sub-segments, obtaining the second plot description text for each sub-segment, and determining the similarity between the first plot description text and the second plot description text for each sub-segment. The sub-segments whose similarity satisfies the preset conditions are obtained from all sub-segments, including: The video clip is input into the preset multimodal large model, which processes the video clip to obtain the clip list output by the preset multimodal large model.

9. A video generation apparatus, characterized in that, include: The first acquisition unit is used to acquire editing information of the first video, the editing information including an editing time period and a first plot description text of the second video, the editing time period being a time period of the first video and each video frame in the second video being a video frame in the editing time period. A splitting unit is used to split a video segment of the edited time period into multiple sub-segments; The second acquisition unit acquires the second plot description text of each sub-segment and determines the similarity between the first plot description text and the second plot description text of each sub-segment. The third acquisition unit is used to acquire the sub-segments whose similarity satisfies the preset conditions from all sub-segments; The generation unit is used to splice all the sub-segments that meet the preset similarity conditions in the first video according to the order of the sub-segments that meet the preset similarity conditions in the first video, so as to generate the second video.

10. An electronic device, characterized in that, The electronic device includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the video generation method as described in any one of claims 1 to 8.

Images