Video processing method, apparatus, device, storage medium and product

Abstract

The present disclosure provides a video processing method, device, equipment, storage medium and product, and relates to the technical field of computers. The method comprises the following steps: dividing an original video into a plurality of video clips, determining plot description information corresponding to the plurality of video clips respectively, screening a highlight video clip from the plurality of video clips based on the plot description information by using a first large language model, determining a starting editing clip and an ending editing clip corresponding to the highlight video clip from the plurality of video clips based on the plot description information by using a second large language model, and performing editing processing on the original video according to the starting editing clip and the ending editing clip to obtain an edited video. By using the above technical solution, the editing effect can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to video processing methods, apparatus, devices, storage media, and products. Background Technology

[0002] With the rapid development of internet technology, users can watch a variety of videos on internet platforms, including longer videos such as movies, TV series, web series, and short dramas. For longer videos, edited videos can be created by extracting highlights, allowing users to efficiently understand the main content of the video.

[0003] Video editing is primarily done by professional editors, requiring high levels of skill and experience, and is inefficient and costly. To save labor and time, automated editing solutions have emerged. Currently, automated editing solutions typically model the problem as a video segment selection problem or a next shot selection problem, and then train the software by assigning pre-defined labels to each shot as features. This end-to-end approach lacks an understanding of the displayed video content, resulting in poor editing quality. Summary of the Invention

[0004] This disclosure provides video processing methods, apparatus, devices, storage media, and products, which can optimize existing video processing solutions for video editing.

[0005] In a first aspect, embodiments of this disclosure provide a video processing method, including:

[0006] The original video is divided into multiple video segments;

[0007] Determine the plot description information corresponding to each of the multiple video segments;

[0008] Using the first major language model based on the plot description information, highlight video clips are selected from the multiple video clips;

[0009] Using the second language model based on the plot description information, the starting and ending clips corresponding to the highlight video clip are determined from the multiple video clips;

[0010] The original video is edited based on the starting clip and the ending clip to obtain an edited video.

[0011] Secondly, embodiments of this disclosure also provide a video processing apparatus, including:

[0012] The segmentation module is used to divide the original video into multiple video segments;

[0013] The description information determination module is used to determine the plot description information corresponding to the plurality of video segments respectively;

[0014] The segment filtering module is used to filter out highlight video segments from the multiple video segments based on the plot description information using the first language model;

[0015] The segment determination module is used to determine the start and end clip segments corresponding to the highlight video segment from the plurality of video segments based on the plot description information using the second language model.

[0016] The editing module is used to edit the original video according to the starting editing segment and the ending editing segment to obtain an edited video.

[0017] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0018] One or more processors;

[0019] Storage device for storing one or more programs.

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method provided in the embodiments of this disclosure.

[0021] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the video processing method provided in embodiments of this disclosure.

[0022] Fifthly, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the video processing method provided in this disclosure.

[0023] The video processing solution provided in this disclosure divides the original video into multiple video segments, determines the plot description information corresponding to each video segment, uses a first language model based on the plot description information to filter out highlight video segments from the multiple video segments, and uses a second language model based on the plot description information to determine the start and end clips corresponding to the highlight video segments from the multiple video segments. The original video is then edited based on the start and end clips to obtain the edited video. By adopting the above technical solution, the editing effect can be effectively improved. Attached Figure Description

[0024] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0025] Figure 1 This is a schematic flowchart of a video processing method provided in an embodiment of the present disclosure;

[0026] Figure 2 This is a schematic flowchart of another video processing method provided in an embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram of a video editing process provided in an embodiment of the present disclosure;

[0028] Figure 4 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of the present disclosure;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0033] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0034] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0035] Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of the present disclosure. This embodiment is applicable to video editing, such as editing film and television dramas. The method can be executed by a video processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, such as a mobile terminal like a mobile phone, tablet computer, or personal digital assistant, or a device like a personal computer (PC) or server.

[0036] like Figure 1 As shown, the method includes:

[0037] Step 101: Divide the original video into multiple video segments.

[0038] In this embodiment, the original video is the video to be edited, and the specific video type is not limited. The original video can be, for example, a movie, TV series, web series, or short drama, and can be a complete video or part of a video from a TV series; specifically, it can be a single episode or multiple consecutive episodes. The original video can also be a user-shot video; it can also be other forms of video. It should be noted that the video data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data, such as video editing) shall comply with the requirements of relevant laws, regulations, and provisions.

[0039] For example, when segmenting the original video, segmentation can be done by shot or by scene, meaning different video segments can correspond to different scenes. The definition of a scene can be set according to actual needs. For example, a scene typically covers the development of events over a period of time and revolves around a specific theme or plot. From the beginning to the end of the scene, the time is continuous; for example, a phone call scene may last for several minutes, during which the communication between the people in the call takes place uninterrupted. A scene can also correspond to an event, such as the phone call event mentioned above. Optionally, a model can be used to identify scenes in the original video, and then the original video can be segmented according to the identified scenes, with each scene's video content corresponding to a video segment. Optionally, video frames or shots in the original video can be aggregated to obtain multiple video segments corresponding to different scenes. Segmenting the original video based on scenes, that is, using scenes as the smallest unit of operation, can better ensure the integrity of the plot and improve the processing efficiency of subsequent large language models compared to conventional segmentation based on video frames or shots.

[0040] Step 102: Determine the plot description information corresponding to each of the multiple video segments.

[0041] For example, for each of multiple video segments, plot description information corresponding to the current video can be determined based on the current video segment itself and / or its associated information. The plot description information can specifically be a textual summary and narration of the plot content within the video segment, clearly and concisely conveying information such as the story's development, key times, and character relationships. This can include key elements such as characters, events, time, location, conflicts and turning points, and emotions and themes. Optionally, input data corresponding to the current video segment can be determined based on the current video segment itself and / or its associated information. This input data is then fed into a preset model (such as a large language model, specifically a multimodal large language model), and the corresponding plot description information is determined based on the output of the preset model.

[0042] The current video clip itself can be the video data of the current video clip, specifically including video frames and / or audio data. The associated information of the current video clip can include information extracted from the current video clip using a preset algorithm and / or script information obtained from the original video production company, etc. Information extracted from the current video clip can include, for example, dialogue, subtitles, and character portraits.

[0043] Step 103: Using the first language model based on the plot description information, select highlight video clips from the multiple video clips.

[0044] The Large Language Model (LLM) involved in this embodiment can be any type of large language model, without any specific limitation. The sample data volume and model parameters of the large language model can be massive, for example, possessing billions or even hundreds of billions of parameters, and are trained on large-scale text data or other modal data. This embodiment does not limit the model training process. Any two of the first large language model and other large language models involved in this step (such as the preset model mentioned above, the second large language model, and the third large language model mentioned later) can be the same or different large language models, without specific limitation.

[0045] For example, a highlight video clip can be a video clip that meets the characteristics of a highlight plot. These characteristics can be determined based on dimensions such as the excitement of the plot, visual impact, emotional appeal, and the significance of plot twists. The plot description information corresponding to multiple video clips can be input into a first language model. The first language model, based on its understanding of the plot description information, will select video clips corresponding to plot description information that meet the highlight plot characteristics as highlight video clips. Optionally, to improve the first language model's ability to select highlight video clips, prompt information can be provided, such as the first preset prompt information. The first preset prompt information can include preset plot descriptions that meet the highlight plot characteristics, such as "the protagonist is engaged in a fierce battle or conflict with the enemy or antagonist" or "a major secret is revealed (such as the protagonist learning about their background)." To enhance the reference value of the first preset prompt information, multiple preset plot descriptions can be included. The first language model can refer to the preset plot descriptions in the first preset prompt information, match the preset plot descriptions with the plot description information corresponding to the current video clip, and thus select highlight video clips from multiple video clips. The specific number of highlight video clips is not limited. Each highlight video clip may include at least one video segment, i.e., one or more video segments. When multiple video segments are included, they are usually temporally consecutive. Optionally, original videos of different themes may correspond to the same or different first preset prompts, which can be set according to actual needs. If they correspond to the same first preset prompt, the first preset prompt may include a preset plot description that matches the highlight plot characteristics of the videos of different themes.

[0046] Step 104: Using the second language model based on the plot description information, determine the starting and ending clips corresponding to the highlight video clip from the multiple video clips.

[0047] For example, after determining the highlight video segment, based on ensuring the continuity of the plot development in the edited video or for other reasons, this step can select appropriate entry and exit points for the highlight video segment, that is, determine the starting and ending edit segments. The selection range of the starting edit segment can be a video segment that is temporally preceding the highlight video segment or the first video segment within the highlight video segment; the selection range of the ending edit segment can be a video segment that is temporally following the highlight video segment or the last video segment within the highlight video segment. Optionally, the selection range of the starting and / or ending edit segments can be further constrained according to the expected length of the edited video. For example, the selection range of the starting edit segment can be a video segment that is temporally preceding the highlight video segment and whose time difference from the highlight video segment is less than a first time difference threshold, or the first video segment within the highlight video segment; the selection range of the ending edit segment can be a video segment that is temporally following the highlight video segment and whose time difference from the highlight video segment is less than a second time difference threshold, or the last video segment within the highlight video segment. The first time difference threshold and the second time difference threshold are determined based on the expected time length, such as being less than or equal to the expected time length.

[0048] For example, the plot description information corresponding to each video segment within the selection range of the starting clip, as well as the plot description information corresponding to the current highlight video segment, can be input into the second language model. The second language model, based on its understanding of each plot description, will then select a suitable video segment as the starting clip. Similarly, the plot description information corresponding to each video segment within the selection range of the ending clip, as well as the plot description information corresponding to the current highlight video segment, can be input into the second language model. The second language model, based on its understanding of each plot description, will then select a suitable video segment as the ending clip. Alternatively, the plot description information corresponding to each video segment within the selection range of the starting clip, the ending clip, and the current highlight video segment can all be input into the second language model. The second language model, based on its understanding of each plot description, will then select a suitable video segment as both the starting and ending clips.

[0049] Optionally, to improve the ability of the second language model to determine the starting and ending clips, prompts can be provided to the second language model. Prompts can be set separately for the starting and ending clips, such as second preset prompts and third preset prompts, respectively. The second preset prompt can include the selection range of the starting clip or other prompts, and the third preset prompt can include the selection range of the ending clip or other prompts. Other prompts in the second preset prompt can include, for example, preset plot descriptions that satisfy the characteristics of the opening plot (which can be called preset starting plot descriptions). These opening plot characteristics can be features that attract the user's attention, such as "a dispute occurs between the main characters" or "the main character and a supporting character compete for a spot." Similarly, other prompts in the third preset prompt can include preset plot descriptions that satisfy the characteristics of the ending plot (which can be called preset ending plot descriptions). These ending plot characteristics can be features that leave the audience wanting more or create suspense, such as "the misunderstanding between the male and female leads has just been resolved" or "the competition result is about to be revealed." Optionally, original videos of different themes can correspond to the same or different second preset prompts, which can be set according to actual needs. If the same second preset prompt information is used, a preset plot description that matches the opening plot characteristics of videos of different themes can be included in the second preset prompt information. Optionally, original videos of different themes can correspond to the same or different third preset prompt information, which can be set according to actual needs. If the same third preset prompt information is used, a preset plot description that matches the ending plot characteristics of videos of different themes can be included in the third preset prompt information.

[0050] Optionally, if there are multiple highlight video clips, the corresponding start and end clips can be determined independently for each highlight video clip, or the start and end clips corresponding to each highlight video clip can be determined sequentially according to the time order of the highlight video clips.

[0051] Step 105: Edit the original video according to the starting clip and the ending clip to obtain the edited video.

[0052] For example, after determining the start and end clips of the highlight video segment, the editing range can be determined based on the start and end clips. The editing range can be a continuous editing range relative to the original video, or it can include multiple sub-editing ranges at intervals. There is no specific limitation. The original video is then edited according to the editing range to obtain the edited video.

[0053] The video processing method provided in this disclosure divides the original video into multiple video segments, determines the plot description information corresponding to each video segment, uses a first large language model to filter out highlight video segments from the multiple video segments based on the plot description information, and uses a second large language model to determine the start and end clips corresponding to the highlight video segments from the multiple video segments based on the plot description information. The original video is then edited based on the start and end clips to obtain an edited video. By adopting the above technical solution, after determining the plot description information of the segmented video segments, highlight video segments are first filtered out using a large language model based on the plot description information. Then, the start and end clips of the highlight video segments are determined using the large language model, i.e., the entry and exit points for editing the highlight video segments. Finally, editing is performed based on the start and end clips. Compared with existing end-to-end model processing tasks, this method ensures that the large language model has a clear understanding of the video content, effectively improving the editing effect.

[0054] In some embodiments, the step of using a first language model to filter highlight video segments from the plurality of video segments based on the plot description information includes: for each of the plurality of video segments, using the first language model to determine the plot score corresponding to the current video segment based on the plot description information and first preset prompt information, wherein the first preset prompt information includes at least one preset plot description and preset scores corresponding to the at least one preset plot description; filtering video segments from the plurality of video segments whose plot scores are greater than a preset score threshold to obtain candidate highlight segments; and determining highlight video segments based on the candidate highlight segments. Thus, in addition to the preset plot description included in the first preset prompt information, a preset score corresponding to the preset plot description is added, which can improve the prompting effect of the first preset prompt information, enabling the first language model to better quantify the degree to which video segments meet the highlight plot features, thereby achieving the effect of automatically scoring the highlight degree of video segments using the first language model. The plot score determined by the first language model can more accurately determine the highlight degree of each video segment, thereby more accurately filtering highlight video segments.

[0055] For example, as illustrated above, the first preset prompt can be further optimized to "3 points for a scene where the protagonist engages in a fierce battle or conflict with an enemy or antagonist," and "2 points for a scene where a major secret is revealed (such as the protagonist learning about their past)." This allows the first language model to match the scene description information corresponding to the current video segment with the preset scene description in the first preset prompt. For instance, it analyzes whether the scene description information matches the features of the preset scene description. The more features matched, the higher the scene score. If none are matched, the scene can be identified as an ordinary scene, and the scene score could be, for example, 0 points. Optionally, to improve the accuracy of the scene score, each video segment can be scored multiple times. For example, for each of the multiple video segments, the first language model uses the scene description information and the first preset prompt to determine the initial scene score multiple times. The final scene score is then determined based on the statistical values ​​(such as the average or median) of the multiple determined initial scene scores.

[0056] For example, after determining the plot score corresponding to each video segment, video segments can be filtered based on a preset score threshold. For instance, if the preset score threshold is 0, video segments with plot scores greater than 0 are filtered out as candidate highlight segments. After determining the candidate highlight segments, they can be directly identified as highlight video segments, or further processing can be performed to obtain the final highlight video segment.

[0057] In some embodiments, determining the highlight video segment based on the candidate highlight segments includes: aggregating the candidate highlight segments to obtain highlight segments to be sorted, wherein at least two temporally consecutive candidate highlight segments are aggregated into the same highlight segment to be sorted; for each highlight segment to be sorted, determining an aggregated plot score corresponding to the current highlight segment to be sorted based on the plot scores corresponding to the candidate highlight segments included in the current highlight segment to be sorted; sorting the highlight segments to be sorted based on the aggregated plot scores, and determining the highlight video segment based on the sorting result. Therefore, considering the possibility of temporally consecutive candidate highlight segments, aggregating the candidate highlight segments before determining the aggregated plot score can more accurately determine the highlight video segment.

[0058] For example, for at least two temporally consecutive candidate highlight segments, they can be aggregated into the same unsorted highlight segment. Independent candidate highlight segments can be identified as a single unsorted highlight segment. Specifically, starting from the first candidate highlight segment in chronological order, the first candidate highlight segment is identified as the current candidate highlight segment. It is then determined whether there are any temporally consecutive candidate highlight segments following the current candidate highlight segment. If not, the current candidate highlight segment is identified as an unsorted highlight segment. If it does, the search continues until the last non-temporally consecutive target candidate highlight segment is found. The current candidate highlight segment is then merged into a single unsorted highlight segment, and the target candidate highlight segment is taken as the new current candidate highlight segment. The steps of determining whether there are any temporally consecutive candidate highlight segments following the current candidate highlight segment are repeated until the last candidate highlight segment is aggregated into a single unsorted highlight segment.

[0059] For example, if the highlight segments to be sorted include only one candidate highlight segment, the aggregate plot score of the highlight segments to be sorted is the plot score of the included candidate highlight segments; if the highlight segments to be sorted include at least two candidate highlight segments, the aggregate plot score of the highlight segments to be sorted is the sum or weighted sum of the plot scores of the included candidate highlight segments.

[0060] For example, after determining the aggregate plot score corresponding to each highlight segment to be sorted, they can be sorted in descending order according to the aggregate plot score, and the preset number of highlight segments to be sorted at the top (such as the K highlight segments to be sorted with the highest aggregate plot scores) can be determined as highlight video segments.

[0061] In some embodiments, the method of determining the starting clip corresponding to the highlight video clip from the plurality of video clips based on the plot description information using a second language model includes: for each highlight video clip, determining the starting clip corresponding to the current highlight video clip from the candidate starting clips using the second language model based on the plot description information corresponding to the current highlight video clip, the plot description information of the candidate starting clip corresponding to the current highlight video clip, and a second preset prompt information. The second preset prompt information includes at least one preset starting plot description, and the candidate starting clips include at least one of the following: video clips whose plot score is less than or equal to the preset score threshold that precedes the current highlight video clip in time; the first video clip of the highlight video clip that precedes the current highlight video clip in time; and the first video clip of the current highlight video clip. Therefore, by filtering video clips that can be candidates for starting clips based on the aforementioned determined plot scores, and by utilizing the capabilities of the second language model combined with prompt information containing preset starting plot descriptions, suitable video clips as starting clips can be quickly and accurately determined from the candidate starting clips, improving the efficiency and accuracy of automated editing and further enhancing the editing effect.

[0062] For example, assuming a preset score threshold of 0, video clips with a plot score of 0 that precede the current highlight video clip can be considered as candidates. Furthermore, to broaden the candidate range without disrupting the continuity of the highlight video clips, the first video clip of the highlight video clip preceding the current highlight video clip and / or the first video clip of the current highlight video clip can also be considered as candidates. Optionally, the candidate range of the starting clip can be further constrained based on the expected length of the edited video. For example, candidate starting clips include video clips that precede the current highlight video clip, have a time difference of less than a first time difference threshold from the current highlight video clip, and have a plot score less than or equal to the preset score threshold; they may also include the first video clip of the highlight video clip preceding the current highlight video clip, with a time difference of less than the first time difference threshold from the current highlight video clip. Optionally, to enhance the prompting effect of the second preset prompt information, it can include multiple preset starting plot descriptions. Optionally, the corresponding starting clip determined by the second language model for the current highlight video clip can be one or more, without specific limitation.

[0063] In some embodiments, using a second language model based on the plot description information, determining the ending clip corresponding to the highlight video clip from the plurality of video clips includes: for each highlight video clip, using the second language model based on the plot description information corresponding to the current highlight video clip, the plot description information of the candidate ending clips corresponding to the current highlight video clip, and third preset prompt information, determining the ending clip corresponding to the current highlight video clip from the candidate ending clips, wherein the third preset prompt information includes at least one preset ending plot description, and the candidate ending clips include at least one of the following: video clips whose plot score is less than or equal to the preset score threshold that is temporally after the current highlight video clip, the last video clip of the highlight video clips that is temporally before and after the current highlight video clip, and the last video clip of the current highlight video clip. Therefore, by filtering video clips that can be candidates for ending clips based on the aforementioned determined plot scores, and by utilizing the capabilities of the second language model combined with prompt information containing preset ending plot descriptions, suitable video clips as ending clips can be quickly and accurately determined from the candidate ending clips, improving the efficiency and accuracy of automated editing and further enhancing the editing effect.

[0064] For example, assuming a preset score threshold of 0, video segments with a plot score of 0 that follow the current highlight video segment can be considered as candidates. Furthermore, to broaden the candidate range without disrupting the continuity of highlight video segments, the last video segment of a highlight video segment that follows the current highlight video segment and / or the last video segment of the current highlight video segment can also be considered as candidates. Optionally, the candidate range for ending clips can be further constrained based on the expected length of the edited video. For example, candidate ending clips include video segments that follow the current highlight video segment in time, have a time difference of less than a second time difference threshold from the current highlight video segment, and have a plot score less than or equal to the preset score threshold; they may also include the last video segment of a highlight video segment that follows the current highlight video segment in time, has a time difference of less than the second time difference threshold from the current highlight video segment. Optionally, to enhance the prompting effect of the third preset prompt information, it can include multiple preset ending plot descriptions. Optionally, the corresponding ending clip segment determined by the second language model for the current highlight video segment can be one or more, without specific limitation.

[0065] In some embodiments, the step of editing the original video based on the starting clip and the ending clip to obtain an edited video includes: determining a candidate editing range based on the starting clip and the ending clip; using a third language model, determining the segments to be deleted within the candidate editing range based on plot description information corresponding to the video segments included in the candidate editing range; determining a target editing range based on the candidate editing range and the segments to be deleted; and editing the original video based on the target editing range to obtain the edited video. Thus, by first determining the candidate editing range based on the starting clip and the ending clip, and then utilizing the capabilities of the third language model to select segments that can be deleted from the candidate editing range based on plot description information within the overall plot, such as video segments whose correlation with each highlight video segment in the candidate editing range is lower than a preset correlation level, specifically, scenes from story subplots outside the highlight video segments, or deleting video segments that do not affect the overall plot understanding, the plot of the edited video is ensured to be concise.

[0066] For example, the starting clips in each highlight video segment can be compared, and the earliest starting clip can be determined as the first video segment in the candidate clip range. The ending clips in each highlight video segment can be compared, and the latest starting clip can be determined as the last video segment in the candidate clip range.

[0067] In some embodiments, at least one of the highlight video segments has at least two corresponding start clips and / or at least two end clips. That is, for a given highlight video segment, multiple editing schemes may be determined, each with different start clips and / or end clips. This allows for a wider variety of editing time ranges, increasing the diversity of edited videos.

[0068] In some embodiments, determining candidate editing ranges based on the starting clip and the ending clip includes: for each highlight video clip, determining a first editing range corresponding to the current highlight video clip based on the starting clip and the ending clip corresponding to the current highlight video clip; arranging and combining the first editing ranges to obtain multiple sets of second editing ranges; for each set of second editing ranges, deduplicating duplicate video clips within the current second editing range to obtain a third editing range corresponding to the current second editing range; and determining candidate editing ranges based on the third editing ranges. This allows for the determination of multiple candidate editing ranges, further improving the diversity of edited videos.

[0069] For highlight video clips with at least two corresponding start clips and / or at least two end clips, multiple first clip ranges can be determined. If a large number of highlight video clips can be identified with multiple first clip ranges, a large number of second clip ranges can be obtained by permuting and combining the first clip ranges. During the process of determining the start and end clips for each highlight video clip, different highlight video clips are independent. Therefore, duplicate video clips may exist in the permuted and combined second clip ranges. Duplicate video clips can be deduplicated to obtain the corresponding third clip ranges. Finally, candidate clip ranges are determined based on the third clip ranges. For example, N-1 video clips from each group of N duplicate video clips within the second clip range can be deleted from the second clip range to obtain the third clip range, which is then determined as the candidate clip range.

[0070] In some embodiments, determining the segments to be deleted within the candidate editing range using a third language model, based on plot description information corresponding to the video segments included in the candidate editing range, includes: using the third language model to determine the segments to be deleted within the candidate editing range based on plot description information corresponding to the video segments included in the candidate editing range and fourth preset prompt information, wherein the fourth preset prompt information includes a segment identifier of the candidate deleted segment. Thus, by providing prompt information to the third language model, the candidate deleted segments are suggested, thereby improving the accuracy of the segments to be deleted selected by the third language model.

[0071] Optionally, the candidate deleted segments include video segments whose plot score is less than or equal to the preset score threshold. Thus, video segments with lower plot scores are provided as candidates for deletion to the large language model, enabling it to more reasonably and accurately select segments to be deleted. For example, if the preset score threshold is 0, then a segment identifier for the candidate deleted segment is added to the plot description information of video segments with a plot score of 0 within the candidate editing range, allowing the third large language model to clearly define the selection range of segments that can be deleted.

[0072] Figure 2 This is a flowchart illustrating another video processing method provided in an embodiment of the present disclosure. The embodiments of the present disclosure are optimized based on the various optional solutions in the above embodiments. Figure 3 This is a schematic diagram of a video editing process provided in an embodiment of the present disclosure, which can be combined with... Figure 2 and Figure 3 The technical solutions of the embodiments of this disclosure will be understood.

[0073] Specifically, the method includes the following steps:

[0074] Step 201: Divide the original video into multiple video segments, where different video segments correspond to different scenes.

[0075] like Figure 3 As shown, for ease of explanation, the original video is assumed to be divided into 10 video segments.

[0076] Step 202: Determine the plot description information corresponding to each of the multiple video segments.

[0077] Step 203: For each video segment among multiple video segments, use the first language model to determine the plot score corresponding to the current video segment based on the plot description information and the first preset prompt information. The first preset prompt information includes at least one preset plot description and a preset score corresponding to each preset plot description.

[0078] like Figure 3 As shown, assume that robot icons represent large language models, and the first robot icon represents the first large language model. The first large language model is used to determine the plot score corresponding to each video segment. Assume... Figure 3 As shown, the plot scores corresponding to the 1st to 10th video segments in chronological order are 0, 2, 0, 3, 4, 2, 0, 3, 1 and 0, respectively.

[0079] Step 204: Select video clips with plot scores greater than a preset score threshold from multiple video clips to obtain candidate highlight clips.

[0080] For example, suppose the preset score threshold is 0, such as Figure 3 As shown, video clips with a plot score greater than 0 are represented by gray triangles, such as... Figure 3 The 2nd, 4th, 5th, 6th, 8th, and 9th video clips are also candidate highlight clips; video clips with a plot score of 0 are represented by black triangles, such as... Figure 3 The 1st, 3rd, 7th, and 10th video segments can be categorized as non-candidate highlight segments.

[0081] Step 205: Aggregate the candidate highlight fragments to obtain highlight fragments to be sorted, wherein at least two temporally consecutive candidate highlight fragments are aggregated into the same highlight fragment to be sorted.

[0082] like Figure 3As shown, the video clips adjacent to the second video clip are all non-candidate highlight clips. Therefore, the second video clip itself is identified as a highlight clip to be sorted, let's call it highlight clip 1. The fourth, fifth, and sixth video clips are three consecutive candidate highlight clips, which can be aggregated into the same highlight clip to be sorted, let's call it highlight clip 2. The eighth and ninth video clips are two consecutive candidate highlight clips, which can be aggregated into the same highlight clip to be sorted, let's call it highlight clip 3.

[0083] Step 206: For each highlight segment to be sorted, determine the aggregate plot score corresponding to the current highlight segment to be sorted based on the plot scores corresponding to the candidate highlight segments contained in the current highlight segment to be sorted.

[0084] For example, for highlight segment 1 to be sorted, which only includes the second video segment, the corresponding aggregated plot score is the plot score corresponding to the second video segment, which is 2; for highlight segment 2 to be sorted, which includes the fourth, fifth, and sixth video segments, the corresponding aggregated plot score is the sum of the plot scores corresponding to these three video segments, which is 9; for highlight segment 3 to be sorted, which includes the eighth and ninth video segments, the corresponding aggregated plot score is the sum of the plot scores corresponding to these two video segments, which is 4.

[0085] Step 207: Sort the highlight segments to be sorted based on the aggregate plot score, and determine the highlight video segments according to the sorting results.

[0086] like Figure 3 As shown, after sorting highlight clip 1, highlight clip 2, and highlight clip 3 based on aggregate plot scores, the sorting result is: highlight clip 2, highlight clip 3, and highlight clip 1. Assuming the preset quantity is 3, all three highlight clips can be identified as highlight video clips; if the preset quantity is 2, highlight clip 2 and highlight clip 3 can be identified as highlight video clips.

[0087] Step 208: For each highlight video segment in the highlight video segment, use the second language model to determine the starting clip segment corresponding to the current highlight video segment from the candidate starting segments based on the plot description information corresponding to the current highlight video segment, the plot description information of the candidate starting segments corresponding to the current highlight video segment, and the second preset prompt information.

[0088] The second preset prompt information includes at least one preset starting plot description. The candidate starting segment includes a video segment whose plot score is less than or equal to the preset score threshold that is located before the current highlight video segment in time, the first video segment of the highlight video segment that is located before the current highlight video segment in time, and the first video segment of the current highlight video segment.

[0089] like Figure 3 As shown, the second robot icon represents the second major language model. This second major language model is used to determine the starting and ending clips corresponding to each highlight video segment. For example... Figure 3 As shown, taking the current highlight video clip as the first highlight video clip as an example, the video clips marked with equilateral triangles below are the candidate starting clips corresponding to the first highlight video clip, the video clips marked with gray equilateral triangles below are the starting clips corresponding to the first highlight video clip determined by the second language model, and the video clips marked with white equilateral triangles below were not determined as starting clips by the second language model.

[0090] Step 209: For each highlight video segment in the highlight video segment, use the second language model to determine the ending clip corresponding to the current highlight video segment from the candidate ending clips based on the plot description information corresponding to the current highlight video segment, the plot description information of the candidate ending clip corresponding to the current highlight video segment, and the third preset prompt information.

[0091] The third preset prompt information includes at least one preset ending plot description, and the candidate ending segment includes at least one of the following: a video segment whose plot score is less than or equal to the preset score threshold that is located after the current highlight video segment in time; the last video segment of the highlight video segments that is located before and after the current highlight video segment in time; and the last video segment of the current highlight video segment.

[0092] like Figure 3 As shown, taking the current highlight video clip as the first highlight video clip as an example, the video clips marked with inverted triangles below are the candidate ending clips corresponding to the first highlight video clip, the video clips marked with gray inverted triangles below are the ending clips corresponding to the first highlight video clip determined by the second language model, and the video clips marked with white inverted triangles below were not determined as ending clips by the second language model.

[0093] Step 210: Determine the candidate clip range based on the starting clip and the ending clip.

[0094] For example, the method for determining the range of candidate clips can be found in the relevant description above. Figure 3As shown, taking one of the candidate clip ranges as an example, the candidate clip range is from the 1st video clip to the 9th video clip.

[0095] Step 211: Using the third language model, based on the plot description information and the fourth preset prompt information corresponding to the video segments included in the candidate editing range, determine the segments to be deleted included in the candidate editing range.

[0096] The fourth preset prompt information includes segment identifiers for candidate deleted segments, which include video segments whose plot score is less than or equal to the preset score threshold. Optionally, candidate deleted segments do not include the first and last video segments within the candidate clip range.

[0097] like Figure 3 As shown, the third robot icon represents the third language model. This third language model is used to determine the segments to be deleted within the candidate clip range. The candidate segments to be deleted can be the 3rd and 7th video segments marked with scissors icons below. The 7th video segment marked with a black scissors icon below was identified as a segment to be deleted by the third language model; the 3rd video segment marked with a white scissors icon below was not identified as a segment to be deleted by the third language model.

[0098] Step 212: Determine the target editing range based on the candidate editing range and the segments to be deleted, and perform editing processing on the original video based on the target editing range to obtain the edited video.

[0099] like Figure 3 As shown, the segments to be deleted are removed from the candidate clip range to obtain the target clip range, which includes the 1st to the 6th video segments and the 8th to the 9th video segments.

[0100] The video processing method provided in this embodiment determines the plot description information of the segmented video segments. First, it uses a large language model to determine the plot score corresponding to each video segment based on the plot description information and prompt information containing preset plot descriptions and preset scores that conform to highlight plot features. Candidate highlight segments are then selected based on the plot scores. These candidate highlight segments are aggregated and sorted to select the final highlight video segments. Next, the large language model, along with second preset prompt information containing preset starting plot descriptions that conform to beginning plot features and third preset prompt information containing preset ending plot descriptions that conform to ending plot features, determines the starting and ending editing segments of each highlight video segment, i.e., the cut scene for editing the highlight video segment. After cutting out the scene and determining the candidate editing range based on the starting and ending clips, the large language model is used again to determine the clips to be deleted from the perspective of the overall plot, that is, to perform scene simplification processing, thereby ensuring the compactness of the plot of the video clips within the target editing range. By breaking down the automated editing task into three tasks: highlight positioning, cut-in and cut-out point selection, and scene simplification, the editing problem is transformed into a well-defined sub-problem that is more friendly to the large language model. Compared with the existing end-to-end model processing tasks, it can ensure that the large language model has a clear understanding of the video content, which improves the smoothness, excitement, and viewing experience of the edited video compared with the existing solutions, and further effectively improves the editing effect of automated video editing.

[0101] Figure 4 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the device includes:

[0102] The segmentation module 401 is used to segment the original video into multiple video segments;

[0103] Description information determination module 402 is used to determine the plot description information corresponding to the plurality of video segments respectively;

[0104] The segment filtering module 403 is used to filter out highlight video segments from the plurality of video segments based on the plot description information using the first language model;

[0105] The segment determination module 404 is used to determine the start and end clip segments corresponding to the highlight video segment from the plurality of video segments based on the plot description information using the second language model.

[0106] The editing module 405 is used to edit the original video according to the starting editing segment and the ending editing segment to obtain an edited video.

[0107] The video processing apparatus provided in this embodiment determines the plot description information of the segmented video segments, firstly using a large language model to filter out highlight video segments based on the plot description information, and then using the large language model to determine the start and end clipping segments of the highlight video segments, that is, the entry and exit points for editing the highlight video segments. Finally, it performs editing processing based on the start and end clipping segments. Compared with existing end-to-end model processing tasks, it can ensure that the large language model has a display understanding of the video content, effectively improving the editing effect.

[0108] Optionally, the fragment filtering module includes:

[0109] The plot score determination unit is used to determine the plot score of each of the plurality of video segments by using a first language model based on the plot description information and the first preset prompt information corresponding to the current video segment. The first preset prompt information includes at least one preset plot description and a preset score corresponding to each of the at least one preset plot description.

[0110] The candidate highlight segment determination unit is used to filter out video segments whose plot score is greater than a preset score threshold from the plurality of video segments to obtain candidate highlight segments;

[0111] A highlight video clip determination unit is used to determine a highlight video clip based on the candidate highlight clips.

[0112] Optionally, the highlight video clip determination unit includes:

[0113] The fragment aggregation subunit is used to aggregate the candidate highlight fragments to obtain highlight fragments to be sorted, wherein at least two candidate highlight fragments that are temporally consecutive are aggregated into the same highlight fragment to be sorted.

[0114] The aggregation score determination subunit is used to determine the aggregation plot score corresponding to the current highlight segment for each highlight segment to be sorted, based on the plot scores corresponding to the candidate highlight segments contained in the current highlight segment to be sorted.

[0115] The highlight video clip determination subunit is used to sort the highlight clips to be sorted based on the aggregated plot score, and determine the highlight video clips according to the sorting result.

[0116] Optionally, the segment determination module includes a start clip segment determination unit and / or an end clip segment determination unit; wherein:

[0117] The starting clip determination unit is used to determine the starting clip corresponding to the current highlight video clip for each highlight video clip in the highlight video clip by using a second language model based on the plot description information corresponding to the current highlight video clip, the plot description information of the candidate starting clip corresponding to the current highlight video clip, and a second preset prompt information. The second preset prompt information includes at least one preset starting plot description, and the candidate starting clip includes at least one of the following: a video clip whose plot score is less than or equal to the preset score threshold that is located before the current highlight video clip in time, the first video clip of the highlight video clip that is located before the current highlight video clip in time, and the first video clip of the current highlight video clip.

[0118] The end clip determination unit is used to determine the end clip corresponding to the current highlight video segment from the candidate end clips for each highlight video segment in the highlight video segments, using a second language model based on the plot description information corresponding to the current highlight video segment, the plot description information of the candidate end clips corresponding to the current highlight video segment, and a third preset prompt information. The third preset prompt information includes at least one preset end plot description, and the candidate end clips include at least one of the following: video segments whose plot score is less than or equal to the preset score threshold that are located after the current highlight video segment in time; the last video segment of the highlight video segments that are located before and after the current highlight video segment in time; and the last video segment of the current highlight video segment.

[0119] Optionally, the editing module includes:

[0120] A candidate clip range determination unit is used to determine a candidate clip range based on the starting clip segment and the ending clip segment;

[0121] The segment to be deleted unit is used to determine the segments to be deleted within the candidate editing range by using a third language model, based on the plot description information and the fourth preset prompt information corresponding to the video segments included in the candidate editing range. The fourth preset prompt information includes the segment identifier of the candidate deleted segment, and the candidate deleted segment includes video segments whose plot score is less than or equal to the preset score threshold.

[0122] The target editing range determination unit is used to determine the target editing range based on the candidate editing range and the segment to be deleted;

[0123] The editing unit is used to edit the original video based on the target editing range to obtain an edited video.

[0124] Optionally, at least one of the highlight video segments has at least two corresponding start clip segments and / or at least two end clip segments;

[0125] The candidate clip range determination unit includes:

[0126] The first editing range determination subunit is used to determine the first editing range corresponding to the current highlight video segment for each highlight video segment based on the start and end editing segments corresponding to the current highlight video segment.

[0127] The second editing range determination subunit is used to arrange and combine the first editing range to obtain multiple sets of second editing ranges;

[0128] The third editing range determination subunit is used to perform deduplication on the repeated video segments in the current second editing range for each group of second editing ranges, so as to obtain the third editing range corresponding to the current second editing range.

[0129] The candidate clip range determination subunit is used to determine the candidate clip range based on the third clip range.

[0130] The video processing apparatus provided in this disclosure can execute the video processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0131] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0132] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 5 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 5 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0133] like Figure 5As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0134] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0135] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0136] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0137] The electronic device provided in this embodiment and the video processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0138] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the video processing method provided in the above embodiments.

[0139] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the video processing method provided in the above embodiments.

[0140] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0141] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0142] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: segment the original video into multiple video segments; determine plot description information corresponding to each of the multiple video segments; use a first language model to filter out highlight video segments from the multiple video segments based on the plot description information; use a second language model to determine the start and end clips corresponding to the highlight video segments from the multiple video segments based on the plot description information; and edit the original video according to the start and end clips to obtain an edited video.

[0143] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0145] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a module does not necessarily limit the module itself; for example, a segmentation module can also be described as "a module that segments an original video into multiple video segments".

[0146] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0147] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] According to one or more embodiments of this disclosure, a video processing method is provided, comprising:

[0149] The original video is divided into multiple video segments;

[0150] Determine the plot description information corresponding to each of the multiple video segments;

[0151] Using the first major language model based on the plot description information, highlight video clips are selected from the multiple video clips;

[0152] Using the second language model based on the plot description information, the starting and ending clips corresponding to the highlight video clip are determined from the multiple video clips;

[0153] The original video is edited based on the starting clip and the ending clip to obtain an edited video.

[0154] According to one or more embodiments of this disclosure, the step of using a first large language model to filter highlight video segments from the plurality of video segments based on the plot description information includes:

[0155] For each of the plurality of video segments, a plot score corresponding to the current video segment is determined using a first large language model based on the plot description information and a first preset prompt information corresponding to the current video segment. The first preset prompt information includes at least one preset plot description and a preset score corresponding to each of the at least one preset plot description.

[0156] From the plurality of video clips, video clips with plot scores greater than a preset score threshold are selected to obtain candidate highlight clips;

[0157] The highlight video segment is determined based on the candidate highlight segments.

[0158] According to one or more embodiments of this disclosure, determining the highlight video segment based on the candidate highlight segments includes:

[0159] The candidate highlight fragments are aggregated to obtain highlight fragments to be sorted, wherein at least two candidate highlight fragments that are temporally consecutive are aggregated into the same highlight fragment to be sorted.

[0160] For each highlight segment to be sorted, the aggregate plot score corresponding to the current highlight segment to be sorted is determined based on the plot scores corresponding to the candidate highlight segments contained in the current highlight segment to be sorted.

[0161] The unsorted highlight segments are sorted based on the aggregated plot score, and the highlight video segments are determined according to the sorting results.

[0162] According to one or more embodiments of this disclosure, determining the start and end clips corresponding to the highlight video clip from the plurality of video clips based on the plot description information using a second language model includes:

[0163] For each highlight video segment in the highlight video segments, the starting clip segment corresponding to the current highlight video segment is determined from the candidate starting segments using the second language model based on the plot description information corresponding to the current highlight video segment, the plot description information of the candidate starting segments corresponding to the current highlight video segment, and the second preset prompt information. The second preset prompt information includes at least one preset starting plot description, and the candidate starting segments include at least one of the following: video segments whose plot score is less than or equal to the preset score threshold that are time-preceding the current highlight video segment, the first video segment of the highlight video segment that is time-preceding the current highlight video segment, and the first video segment of the current highlight video segment; and / or,

[0164] For each highlight video segment in the highlight video segments, the second language model is used to determine the ending clip corresponding to the current highlight video segment from the candidate ending segments based on the plot description information corresponding to the current highlight video segment, the plot description information of the candidate ending segments corresponding to the current highlight video segment, and the third preset prompt information. The third preset prompt information includes at least one preset ending plot description, and the candidate ending segments include at least one of the following: video segments whose plot score is less than or equal to the preset score threshold that are located after the current highlight video segment in time; the last video segment of the highlight video segments that are located before and after the current highlight video segment in time; and the last video segment of the current highlight video segment.

[0165] According to one or more embodiments of this disclosure, the step of editing the original video based on the starting clip segment and the ending clip segment to obtain an edited video includes:

[0166] The candidate clip range is determined based on the starting clip and the ending clip;

[0167] Using the third language model, based on the plot description information and the fourth preset prompt information corresponding to the video segments included in the candidate editing range, the segments to be deleted included in the candidate editing range are determined. The fourth preset prompt information includes the segment identifier of the candidate deleted segment, and the candidate deleted segment includes video segments whose plot score is less than or equal to the preset score threshold.

[0168] The target editing range is determined based on the candidate editing range and the segment to be deleted;

[0169] The original video is edited based on the target editing range to obtain an edited video.

[0170] According to one or more embodiments of this disclosure, at least one of the highlight video segments has at least two corresponding start clip segments and / or at least two end clip segments;

[0171] The process of determining the candidate clip range based on the starting clip and the ending clip includes:

[0172] For each highlight video clip, determine the first clip range corresponding to the current highlight video clip based on the starting clip and ending clip corresponding to the current highlight video clip;

[0173] Arrange and combine the first editing range to obtain multiple sets of second editing ranges;

[0174] For each group of second clip ranges, duplicate video segments within the current second clip range are deduplicated to obtain the third clip range corresponding to the current second clip range;

[0175] The candidate clip range is determined based on the third clip range.

[0176] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0177] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0178] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video processing method, characterized in that, include: The original video is divided into multiple video segments; Determine the plot description information corresponding to each of the multiple video segments; Using the first major language model based on the plot description information, highlight video clips are selected from the multiple video clips; Using the second language model based on the plot description information, the starting and ending clips corresponding to the highlight video clip are determined from the multiple video clips; The original video is edited based on the starting clip and the ending clip to obtain an edited video.

2. The method according to claim 1, characterized in that, The step of using the first major language model to filter highlight video clips from the multiple video clips based on the plot description information includes: For each of the plurality of video segments, a plot score corresponding to the current video segment is determined using a first large language model based on the plot description information and a first preset prompt information corresponding to the current video segment. The first preset prompt information includes at least one preset plot description and a preset score corresponding to each of the at least one preset plot description. From the plurality of video clips, video clips with plot scores greater than a preset score threshold are selected to obtain candidate highlight clips; The highlight video segment is determined based on the candidate highlight segments.

3. The method according to claim 2, characterized in that, The step of determining the highlight video segment based on the candidate highlight segments includes: The candidate highlight fragments are aggregated to obtain highlight fragments to be sorted, wherein at least two candidate highlight fragments that are temporally consecutive are aggregated into the same highlight fragment to be sorted. For each highlight segment to be sorted, the aggregate plot score corresponding to the current highlight segment to be sorted is determined based on the plot scores corresponding to the candidate highlight segments contained in the current highlight segment to be sorted. The unsorted highlight segments are sorted based on the aggregated plot score, and the highlight video segments are determined according to the sorting results.

4. The method according to claim 2, characterized in that, The step of using a second language model based on the plot description information to determine the start and end clips corresponding to the highlight video clip from the multiple video clips includes: For each highlight video segment in the highlight video segments, the starting clip segment corresponding to the current highlight video segment is determined from the candidate starting segments using the second language model based on the plot description information corresponding to the current highlight video segment, the plot description information of the candidate starting segments corresponding to the current highlight video segment, and the second preset prompt information. The second preset prompt information includes at least one preset starting plot description, and the candidate starting segments include at least one of the following: video segments whose plot score is less than or equal to the preset score threshold that are time-preceding the current highlight video segment, the first video segment of the highlight video segment that is time-preceding the current highlight video segment, and the first video segment of the current highlight video segment; and / or, For each highlight video segment in the highlight video segments, the second language model is used to determine the ending clip corresponding to the current highlight video segment from the candidate ending segments based on the plot description information corresponding to the current highlight video segment, the plot description information of the candidate ending segments corresponding to the current highlight video segment, and the third preset prompt information. The third preset prompt information includes at least one preset ending plot description, and the candidate ending segments include at least one of the following: video segments whose plot score is less than or equal to the preset score threshold that are located after the current highlight video segment in time; the last video segment of the highlight video segments that are located before and after the current highlight video segment in time; and the last video segment of the current highlight video segment.

5. The method according to claim 2, characterized in that, The step of editing the original video based on the starting clip and the ending clip to obtain an edited video includes: The candidate clip range is determined based on the starting clip and the ending clip; Using the third language model, based on the plot description information and the fourth preset prompt information corresponding to the video segments included in the candidate editing range, the segments to be deleted included in the candidate editing range are determined. The fourth preset prompt information includes the segment identifier of the candidate deleted segment, and the candidate deleted segment includes video segments whose plot score is less than or equal to the preset score threshold. The target editing range is determined based on the candidate editing range and the segment to be deleted; The original video is edited based on the target editing range to obtain an edited video.

6. The method according to claim 5, characterized in that, At least one of the aforementioned highlight video segments has at least two corresponding start clip segments and / or at least two end clip segments; The process of determining the candidate clip range based on the starting clip and the ending clip includes: For each highlight video clip, determine the first clip range corresponding to the current highlight video clip based on the starting clip and ending clip corresponding to the current highlight video clip; Arrange and combine the first editing range to obtain multiple sets of second editing ranges; For each group of second clip ranges, duplicate video segments within the current second clip range are deduplicated to obtain the third clip range corresponding to the current second clip range; The candidate clip range is determined based on the third clip range.

7. A video processing apparatus, characterized in that, include: The segmentation module is used to divide the original video into multiple video segments; The description information determination module is used to determine the plot description information corresponding to the plurality of video segments respectively; The segment filtering module is used to filter out highlight video segments from the multiple video segments based on the plot description information using the first language model; The segment determination module is used to determine the start and end clip segments corresponding to the highlight video segment from the plurality of video segments based on the plot description information using the second language model. The editing module is used to edit the original video according to the starting editing segment and the ending editing segment to obtain an edited video.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method as described in any one of claims 1-6.

9. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the video processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the video processing method as described in any one of claims 1-6.

Images