Video editing method and device, electronic equipment and storage medium

Abstract

The invention discloses a video editing method and device, electronic equipment and a storage medium. The method comprises the following steps: segmenting a to-be-edited video to obtain a plurality of video clips; understanding text generation and subtitle generation are carried out on each video clip, and a clip understanding text and a clip subtitle of each video clip are obtained; determining a target description text based on fragment understanding texts and / or fragment subtitles corresponding to a plurality of target video fragments selected from the plurality of video fragments; and performing video synthesis based on the plurality of target video clips and the target description text to obtain a target video corresponding to the to-be-edited video. According to the method provided by the invention, the video editing efficiency is relatively high, and the video editing cost is relatively low.

Description

Technical Field

[0001] This application relates to the field of electronic information technology, and more specifically, to a video editing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rise of short videos, self-media, and streaming media platforms, video content creation and dissemination have become important channels for information dissemination and social interaction. How to quickly and accurately extract valuable content from a large amount of video material and perform intelligent editing has become an important research topic in the field of video processing.

[0003] Currently, the editing process typically involves editors viewing the video frame by frame, selecting key segments based on their subjective judgment, and creating descriptive text. Finally, editors manually edit the video based on the selected segments and the created descriptive text. However, this video editing method is inefficient. Summary of the Invention

[0004] In view of this, embodiments of this application propose a video editing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, embodiments of this application provide a video editing method, the method comprising: segmenting the video to be edited to obtain multiple video segments; generating comprehension text and subtitles for each video segment to obtain comprehension text and subtitles for each video segment; determining target description text based on the comprehension text and / or subtitles corresponding to multiple target video segments selected from the multiple video segments; and performing video synthesis based on the multiple target video segments and the target description text to obtain a target video corresponding to the video to be edited.

[0006] Secondly, embodiments of this application provide a video editing apparatus, comprising: a segmentation module for segmenting a video to be edited to obtain multiple video segments; a generation module for generating comprehension text and subtitles for each video segment to obtain comprehension text and subtitles for each video segment; a determination module for determining target description text based on the comprehension text and / or subtitles corresponding to multiple target video segments selected from the multiple video segments; and a video acquisition module for performing video synthesis based on the target video segment sequence and the target description text to obtain a target video corresponding to the video to be edited.

[0007] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; the memory stores computer-readable instructions, which, when executed by the processor, implement the above-described method.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the above-described method.

[0009] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the method described above.

[0010] This application provides a video editing method, apparatus, electronic device, and storage medium. First, the video to be edited is segmented to obtain multiple video segments. For each video segment, comprehension text and subtitles are generated, resulting in comprehension text and subtitles for each segment. Then, based on the comprehension text and subtitles of each target video segment, descriptive text is generated to obtain target descriptive text. Finally, based on the multiple target video segments and target descriptive texts, the target video corresponding to the video to be edited is determined. This achieves automatic determination of target video segments and target descriptive text based on the video to be edited, and further, determines the target video corresponding to the video to be edited based on the target video segments and target descriptive text. This achieves the goal of automatic video editing, eliminating reliance on manual editing by editors, improving editing efficiency, and reducing the labor costs of the editing process. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram is shown illustrating the application scenarios applicable to the embodiments of this application; Figure 2 A flowchart of a video editing method according to an embodiment of this application is shown; Figure 3 A schematic diagram illustrating a video clip acquisition process according to an embodiment of this application is shown; Figure 4 A schematic diagram illustrating a process for obtaining fragmented text understanding according to an embodiment of this application is shown; Figure 5 It shows Figure 2 A flowchart of step S120 in one embodiment is shown in the corresponding example; Figure 6 A schematic diagram of a video editing process according to an embodiment of this application is shown; Figure 7A block diagram of a video editing apparatus according to one embodiment of this application is shown; Figure 8 A structural block diagram of an electronic device for performing a video editing method according to an embodiment of this application is shown. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the application. It should be noted that "multiple" as used herein refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0016] Please refer to Figure 1 The diagram illustrates an application scenario applicable to the embodiments of this application. This application scenario includes a terminal 110, a server 120, and a database 130.

[0017] Terminal 110 can be, for example, a smartphone, tablet, e-book reader, music player, wearable device, smart home device, in-vehicle terminal, etc. Terminal 110 has a video editing client installed. This video editing client can edit the video to be edited to obtain the edited target video. Alternatively, the video editing client can send a video editing request so that server 120 can edit the video to be edited based on the video editing request to obtain the edited target video.

[0018] The server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0019] In some implementations, terminal 110 can generate and send a video editing request to server 120 (via a video editing client). Server 120 then edits the video to be edited based on the video editing request to obtain the target video. Finally, server 120 returns the target video to terminal 110 so that terminal 110 can display the edited target video.

[0020] In some implementations, the server 120 itself can respond to the video editing operation triggered by the user locally, perform editing processing on the video to be edited, and obtain the edited target video.

[0021] Alternatively, the terminal 110 itself can respond to the video editing operation triggered by the user locally, perform editing processing on the video to be edited, and obtain the edited target video.

[0022] Of course, when the terminal 110 implements the video editing process of this application, after the server 120 obtains the object detection model, it can send the object detection model to the terminal 110, and the terminal 110 stores the object detection model so that the terminal can implement video editing.

[0023] To more clearly explain the solution of this application, the following embodiments all use electronic devices as the execution subject of the video editing method.

[0024] See also Figure 2 , Figure 2 The flowchart illustrates a video editing method according to an embodiment of this application. The method is applied to an electronic device, which may be... Figure 1 The terminal 110 or server 120 in the middle, the method may include: S110. The video to be edited is segmented to obtain multiple video segments.

[0025] In this application, the video to be edited refers to a video that needs to be edited. It can be an independent video or a video segment selected from a longer video. The video to be edited can be a video captured by a camera or other acquisition device, a video generated by a neural network model, or a video created by a video producer, etc.

[0026] In some implementations, the video to be edited can be evenly divided into multiple video segments. For example, a 1-hour video can be evenly divided into six 10-minute video segments.

[0027] In other implementations, the video to be edited can be segmented based on the visual differences between adjacent video frames to obtain multiple video segments. These visual differences include, but are not limited to, differences in color distribution and brightness histograms, and can be determined using the open-source scenedetect library.

[0028] Specifically, a difference threshold can be set based on requirements. If the visual difference between adjacent video frames in the video to be edited is greater than the difference threshold, it is determined that the scene of the next video frame of the adjacent video frame has changed. The point where the adjacent video frame is cut is determined as a video segmentation point. In this way, multiple video segmentation points are determined, and the video to be edited is segmented using multiple video segmentation points to obtain multiple video segments. Thus, each video segment after segmentation is an independent scene segment, and different video segments may be video segments targeting different scenes.

[0029] In some other implementations, the video to be edited may be segmented to obtain multiple initial video segments; for each initial video segment, highlight video segments may be extracted from the initial video segment based on the motion intensity between adjacent video frames in the initial video segment; non-highlight video segments may be determined based on the other video segments in the multiple initial video segments excluding the highlight video segments; and multiple video segments may be determined based on the highlight video segments and the non-highlight video segments.

[0030] This can be done by evenly dividing the video to be edited into multiple initial video segments, or by dividing the video to be edited based on the visual differences between adjacent video frames to obtain multiple initial video segments.

[0031] After this, for each initial video segment, optical flow can be calculated using a Python computer vision library to obtain the average amplitude of the motion vectors (i.e., the optical flow field) between adjacent video frames in the initial video segment, which is used as the motion intensity between adjacent frames in the initial video segment. Here, the average amplitude can be the average value of the motion vectors of each pixel in the video frames of the initial video segment.

[0032] In some implementations, for each initial video segment, segments with motion intensity higher than a preset motion intensity threshold are identified as highlight video segments corresponding to that initial video segment. This process is repeated for all initial video segments to identify all highlight video segments. The preset motion intensity threshold can be set based on requirements and is not limited in this application.

[0033] In some other implementations, for each initial video segment, a segment (including at least one video frame) with a motion intensity higher than a preset motion intensity threshold is identified as the key segment corresponding to the initial video segment. Then, with the key segment as the center, the segment is extended forward and backward by a specified duration (e.g., 5s or 6s) to obtain the highlight video segment corresponding to the initial video segment. In this way, all initial video segments are traversed to identify all highlight video segments.

[0034] For each initial video segment, the non-highlight video segment corresponding to that initial video segment is determined based on the portion of that initial video segment excluding the highlight video segments.

[0035] Specifically, if the duration of video segments other than the highlight video segments in the initial video clip is short, then the video segments other than the highlight video segments are taken as non-highlight video segments. For example, if three highlight video segments are extracted from an initial video clip that is 2 minutes long, and four video segments remain, each of which does not exceed 10 seconds, then these four remaining video segments are taken as four non-highlight video segments.

[0036] If the video segments other than the highlight segments in the initial video clip are relatively long, these segments can be uniformly divided to obtain non-highlight video segments. For example, if a 2-minute initial video clip has one highlight segment, leaving two remaining segments, each at least 30 seconds long, these two segments can be uniformly divided to obtain four non-highlight video segments. The length of the video segments can be determined using a duration threshold. For example, segments longer than the threshold are considered longer, while segments shorter than the threshold are considered shorter. The duration threshold could be, for example, 20 seconds.

[0037] This ensures that all non-highlight video clips are of reasonable length, facilitating subsequent unified processing and avoiding difficulties in storyline construction caused by excessively long video clips.

[0038] The set of highlight video clips is called the highlight video clip set, and the set of non-highlight video clips is called the non-highlight video clip set. The video clips in the highlight video clip set and the non-highlight video clip set are used as multiple video clips after the video to be edited is segmented. In this way, the highlight part (highlight video clip) and the normal part (non-highlight video clip) can be identified from the video to be edited, which makes it easier to select the required video clips for subsequent editing and improves the efficiency of video editing.

[0039] For example, the process of acquiring multiple video clips is as follows: Figure 3As shown, firstly, scene change detection is performed on the input video to be edited (that is, the visual differences between adjacent video frames in the video to be edited) to slice the video to be detected into multiple initial video segments. Then, highlight detection is performed on each initial video segment to identify highlight video segments. All highlight video segments are then aggregated to obtain a set of highlight video segments. For the parts other than highlight video segments, uniform segmentation is performed to obtain non-highlight video segments. All non-highlight video segments are then aggregated to obtain a set of non-highlight video segments.

[0040] S120. Generate the understanding text and subtitles for each video segment separately to obtain the segment understanding text and segment subtitles for each video segment.

[0041] For each video segment, a large language model can be used to generate understood text based on that video segment, resulting in segment-level understood text for that video segment. For example, the instruction "Generate understood text for this video segment" and the video segment can be input into a large language model to obtain the segment-level understood text output by the large language model for that video segment.

[0042] Similarly, subtitles can be generated based on the video clip using a large language model to obtain segment subtitles for that video clip. For example, the instruction "Generate subtitles for this video clip" and the video clip can be input into a large language model to obtain segment subtitles output by the large language model for the video clip.

[0043] Among them, Large Language Model (LLM) refers to a deep learning model trained on a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on various topics by being trained on huge datasets. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, and to a certain extent simulate the human language cognition and generation process.

[0044] Of course, users can set preset theme information based on their needs. Preset theme information is used to indicate the theme that the edited target video should conform to or the content that the edited target video should include. Preset theme information can be words or sentences. For example, the preset theme information can be "tourism", so the edited target video should conform to the theme of tourism. Or, the preset theme information can be "skills to improve skiing skills", so the edited target video should include the content of "skills to improve skiing skills".

[0045] The generated subtitles can be in standard subtitle file formats (such as SRT and ASS formats), making them easy to use directly in subsequent video editing processes.

[0046] It is worth mentioning that when generating fragment comprehension text and fragment subtitles, the input instruction information can include preset theme information so that the generated fragment comprehension text and fragment subtitles conform to the preset theme information.

[0047] S130. Determine the target description text based on the segment understanding text and / or segment subtitles corresponding to each of the multiple target video segments selected from multiple video segments.

[0048] In some implementations, some segments may be randomly selected from multiple video clips as multiple target video segments.

[0049] In some other implementations, all the multiple video segments can be acquired as multiple target video segments.

[0050] In some other implementations, the editing mode corresponding to the video to be edited can be obtained; multiple target video segments can be selected from multiple video segments based on the editing mode; the multiple target video segments include at least one highlight video segment.

[0051] For example, if the editing mode is highlight editing mode, all highlight video clips are acquired as multiple target video clips. As another example, if the editing mode is regular narrative mode, the target highlight video clip and associated non-highlight video clips are acquired as multiple target video clips; the target highlight video clips are selected based on the segment understanding text and / or segment subtitles of each highlight video clip. Here, "associated with the target highlight video clip" can refer to a relationship with the target highlight video band in dimensions such as narrative logic, visual effects, and chronological order.

[0052] A large language model can be used to analyze the content and thematic logic of each highlight video clip based on its individual segment understanding text and / or segment subtitles. This allows for the selection of the highlight video clip that best represents the preset thematic information and the core of the video to be edited, which is then designated as the target highlight video clip. The large language model used here for selecting the target highlight video clip may differ from or be the same as the aforementioned large language model.

[0053] Then, using a large language model, at least one non-highlight video segment associated with the target highlight video segment is selected from the non-highlight video segments as the associated non-highlight video segment, and the selected target highlight video segment and the associated non-highlight video segment are obtained as multiple selected target video segments.

[0054] Then, the target description text is determined based on the segment understanding text and / or segment subtitles corresponding to the selected multiple target video segments.

[0055] One approach is to input the segment understanding texts of each target video segment together (in any order) into a large language model, which then generates descriptive text based on these texts. Similarly, another approach is to input the segment subtitles of each target video segment together (in any order) into a large language model, which then generates descriptive text based on these subtitles.

[0056] Alternatively, the segment understanding text and segment subtitles of each target video segment can be combined to obtain the combined text of each target video segment. Then, the combined texts of all target video segments are input together (in no particular order) into the large language model, which generates the target description text based on the combined texts of each target video segment.

[0057] Of course, in order to ensure that the generated target description text meets the user's theme requirements, the preset theme information and the information used to generate the target description text (that is, the segment understanding text, segment subtitles or combined text of the aforementioned target video segments) can be input into the large language model to prompt the large language model to generate target description text that meets the preset theme information.

[0058] Optionally, in some implementations, S130 may include: sorting multiple target video segments to obtain a target video segment sequence; and determining target description text based on the segment understanding text and / or segment subtitles corresponding to each target video segment in the target video segment sequence.

[0059] After identifying multiple target video segments, the segments can be sorted according to their position in the video to be edited, resulting in a sequence of target video segments.

[0060] Alternatively, a large language model can be used to understand the narrative logic and visual fluency of the segment understanding text and / or segment subtitles based on each target video segment, determine the narrative order corresponding to each target video segment, and then sort multiple target video segments according to the narrative order to obtain a target video segment sequence.

[0061] For example, the order of the positions of multiple target video segments a1, a2, a3, and a4 in the video to be edited is a1-a2-a3-a4. The narrative order determined by the large language model is a3-a2-a1-a4. Therefore, the sequence of target video segments composed of multiple target video segments a1, a2, a3, and a4 is a3a2a1a4.

[0062] In some embodiments, after obtaining the target video segment sequence, the fourth language model can be used to update the order of each target video segment in the target video segment sequence based on at least one of the following: content coherence, thematic consistency, and emotional progression.

[0063] The fourth major language model refers to the model used to optimize the target video segment sequence. It can be any major language model, without limitation here. Of course, it can be the same as the major language models mentioned above. For example, the major language model could be GPT-4 or Deepseek, etc.

[0064] Therefore, compared to the target video segment sequence before the update, the order of each target video segment in the updated target video segment sequence may change, and the updated target video segment sequence has better visual smoothness and narrative clarity.

[0065] Correspondingly, the corresponding segment understanding texts of each target video segment in the target video segment sequence can be sorted according to the arrangement order of each target video segment in the target video segment sequence to obtain an understanding text sequence. Then, the large language model generates the descriptive text based on the understanding text sequence to obtain the target descriptive text.

[0066] Of course, in order to ensure that the generated target description text meets the user's topic requirements, the preset topic information and the understood text sequence can be input into the large language model together to prompt the large language model to generate target description text that meets the preset topic information.

[0067] Because fragment understanding provides complete semantic background and contextual information for fragments, it ensures that large language models have sufficient factual basis during reasoning, avoids fabricating content or deviating from the actual scene and plot, and ensures that the generated target description text is more accurate.

[0068] Alternatively, the subtitles corresponding to each target video segment in the target video segment sequence can be sorted according to their arrangement order to obtain a subtitle sequence. Then, a large language model can be used to generate descriptive text based on the subtitle sequence to obtain the target descriptive text.

[0069] Of course, in order to ensure that the generated target description text meets the user's theme requirements, the preset theme information and subtitle sequence can also be input into the large language model to prompt the large language model to generate target description text that meets the preset theme information.

[0070] Because the subtitles enhance the key information of each video segment, they help the large speech model highlight key events or emotional turning points in the segments when generating storylines, maintaining the compactness and clarity of the narrative, and making the generated target description text clearer and more accurate.

[0071] In some other embodiments, S130 may include: constructing a sequence of target video segments based on multiple target video segments; combining the segment understanding text and segment subtitles of each target video segment in the target video segment sequence to obtain a combined text for each target video segment; and generating a descriptive text for the target video segment sequence based on the combined text of each target video segment in the target video segment sequence using a third language model to obtain the target descriptive text for the target video segment sequence. In other words, the segment understanding text and segment subtitles of the target video clip can be combined to obtain a combined information, which serves as the combined text of the target video clip. Then, according to the arrangement order of each target video clip in the target video clip sequence, the combined texts corresponding to each target video clip in the target video clip sequence can be sorted to obtain a combined text sequence. Finally, a large language model is used to generate descriptive text based on the combined text sequence to obtain the target descriptive text.

[0072] Of course, in order to ensure that the generated target description text meets the user's topic requirements, the preset topic information and the combined text sequence can be input into the large language model to prompt the large language model to generate target description text that meets the preset topic information.

[0073] Furthermore, the aforementioned generation of descriptive text for the target video segment sequence using a third language model based on the combined text of each target video segment in the target video segment sequence includes: generating descriptive text for the target video segment sequence using a third language model based on a prompt template, preset topic information, and the combined text of each target video segment in the target video segment sequence; the prompt template is used to prompt the third language model to generate target descriptive text conforming to a preset format without expanding fictional content. That is, the prompt template is mainly used to prompt the third language model not to fabricate content, to generate target descriptive text based on the given combined text sequence of each target video segment, and to ensure that the output target descriptive text conforms to a preset format.

[0074] The third major language model refers to the model used to generate descriptive text, which can be any major language model; no specific limitation is made here. Of course, it can be the same as the major language models mentioned above. For example, a major language model could be GPT-4 or Deepseek, etc.

[0075] The preset format can be a format set based on requirements. This preset format can clearly display the text of each target video segment in the target video segment sequence in the target description text. For example, the preset format is "video segment 1: text 1, video segment 2: text 2, video segment 3: text 3, ..., video segment t: text t". This makes it easy to determine the text of each target video segment in each target video segment sequence based on the target description text.

[0076] Based on the target description text generated by the aforementioned process, the third model naturally connects the text of each target video segment according to the time sequence or narrative order, avoiding jumps or breaks in the narrative and ensuring coherence. Furthermore, it analyzes the emotional changes between target video segments, guides the storyline to progress in emotional tone, achieves emotional progression, and always revolves around the theme of the final film set by the user's needs, without deviating from the core and ensuring thematic consistency.

[0077] S140. Based on multiple target video segments and target description text, perform video synthesis to obtain the target video corresponding to the video to be edited.

[0078] Multiple target video segments can be combined to form a target video segment sequence (the specific process is described in the previous embodiments and will not be repeated here), and the target video segment sequence can be combined into a single video according to the order of the target video segments in the target video segment sequence, which is then used as the target video. The target description text can be added to the target video in the form of a brief introduction or narration.

[0079] Alternatively, the target video segments can be combined into a single video according to their order, serving as the target video. At the same time, subtitles or voice-overs can be generated from the target description text and added to the target video.

[0080] In this embodiment, the video to be edited is first segmented into multiple video segments. For each video segment, comprehension text and subtitles are generated, resulting in comprehension text and subtitles for each segment. Then, based on the comprehension text and subtitles of each target video segment, descriptive text is generated to obtain target descriptive text. Finally, based on multiple target video segments and target descriptive texts, the target video corresponding to the video to be edited is determined. This achieves the goal of automatically determining target video segments and target descriptive text based on the video to be edited, and further determining the target video corresponding to the video to be edited based on the target video segments and target descriptive text. This achieves the goal of automatically editing the video to be edited, eliminating reliance on manual editing by editors, improving editing efficiency, and reducing the labor costs of the editing process.

[0081] Secondly, by combining histogram change detection and optical flow, video scene detection and highlight segment extraction are achieved to determine the required target video segments from the video to be edited. This method does not rely on deep learning models to determine target video segments, thus improving the efficiency of target video segmentation. Furthermore, the determined target video segments are of reasonable length, without excessively long segments, facilitating efficient video editing.

[0082] In some embodiments, such as Figure 4 As shown, S120 includes: S121. For each video segment, based on the length of the video segment, extract keyframes from the video segment to obtain multiple keyframes corresponding to the video segment.

[0083] For any given video segment, a number of keyframes are adaptively extracted based on the segment's length. The frame extraction strategy includes: if the video segment is short (e.g., less than 10 seconds), all video frames within the segment are used as keyframes; if the video segment is long, keyframes are sampled at fixed time intervals (e.g., 1 frame per second) or evenly across the total number of frames. This ensures that the selected keyframes cover the dynamic changes of the video segment while controlling the frame count to avoid redundancy and reduce the computational burden of subsequent processing, while preserving the main visual semantic information of the video segment.

[0084] Of course, the length of the video clip can also be determined based on a given duration threshold. For example, if the duration of the video clip is longer than the given duration threshold, the video clip is determined to be longer; if the duration of the video clip is not longer than the given duration threshold, the video clip is determined to be shorter. The given duration threshold is, for example, 10 seconds.

[0085] S122. Generate description text for each keyframe to obtain the frame description text for each keyframe.

[0086] For each selected keyframe, a neural network model can be used to generate descriptive text for the keyframe. This generates a concise and accurate natural language description for each keyframe, serving as its frame description text. The frame description text retains the main information, action features, and scene background elements of each frame. By iterating through all keyframes, the individual frame description text for each keyframe can be obtained.

[0087] The neural network model here can be a quantized and optimized BLIP (Bootstrapping Language-Image Pretraining) model. By quantizing the BLIP model, a quantized BLIP model is obtained, effectively compressing the number of parameters in the BLIP model. During inference, the memory usage of the BLIP model is significantly reduced, and the overall memory usage of the BLIP model is kept within a controllable range (approximately 10GB). At the same time, after quantization, the BLIP model can still maintain a high semantic understanding accuracy, enabling efficient inference even in resource-constrained environments. It supports batch frame processing, thereby accelerating the generation of frame description text and improving the overall system throughput.

[0088] S123. Based on the frame description text of each keyframe in the video segment, determine the segment understanding text and segment subtitles for each video segment.

[0089] In some implementations, the segment understanding text can be generated by a large language model based on the frame description text of each keyframe selected in the video segment. For example, the instruction "Generate understanding text based on the frame description text of each keyframe selected in the video segment" and the frame description text of each keyframe selected in the video segment are input into the large language model to obtain the segment understanding text output by the large language model for the video segment.

[0090] Similarly, subtitles can be generated by using a large language model based on the frame description text of each keyframe selected in the video clip, thus obtaining the segment subtitles for the video clip. For example, the instruction "Generate subtitles based on the frame description text of each keyframe selected in the video clip" and the frame description text of each keyframe selected in the video clip can be input into a large language model to obtain the segment subtitles output by the large language model for the video clip.

[0091] Of course, similarly to the above, preset topic information can also be added to the instruction information here, so that the large language model can generate fragment understanding text and fragment subtitles based on the instruction information including preset topic information.

[0092] In some other embodiments, S123 may further include: arranging the frame description text corresponding to each keyframe in the video segment based on the order of their positions to obtain a sequence of description text corresponding to the video segment; generating understanding text based on the sequence of description text corresponding to the video segment using a first language model to obtain segment understanding text corresponding to the video segment; and determining segment subtitles corresponding to the video segment based on the segment understanding text corresponding to the video segment.

[0093] The first large language model refers to the large language model used for generating and understanding text. It can be any type of large language model, without limitation here. Of course, it can be the same as the large language model mentioned above. For example, the large language model could be GPT-4 or Deepseek, etc.

[0094] In other words, the frame description text of each keyframe in the video clip must first be sorted to obtain the description text sequence corresponding to the video clip. Then, the first language model can generate the understanding text based on the description text sequence corresponding to the video clip to obtain the segment understanding text corresponding to the video clip. Specifically, the instruction information "generate understanding text based on the description text sequence corresponding to the video clip" and the description text sequence corresponding to the video clip can be input into the first language model to obtain the segment understanding text output by the first language model for the video clip.

[0095] In some other implementations, the first language model can be used to generate understood text based on preset topic information and a sequence of descriptive text corresponding to the video segment, thus obtaining the segment-understood text corresponding to the video segment. In other words, instruction information including preset topic information and a sequence of descriptive text corresponding to the video segment can be input into the first language model, which then performs tasks such as semantic fusion, topic extraction, and emotion recognition to obtain the segment-understood text output by the first language model for the video segment.

[0096] Semantic fusion refers to integrating information from different frames to understand the dynamic changes and scene development within a video clip. Topic extraction refers to identifying the central event or theme of a video clip, such as "a person jogging in a forest." Emotion recognition refers to inferring the emotional tone of the scene depicted in the video clip, such as joy, tension, or warmth.

[0097] In summary, the first major language model can output a structurally complete and semantically coherent understood text for a video clip. This understood text serves as the segment's understood text, which clearly describes the core content and semantic thread of the video clip.

[0098] For example, the process of obtaining fragment understanding text is as follows: Figure 5 As shown, firstly, the input video segment is processed by frame extraction to obtain N key frames. Then, the quantized Blip model is used to generate descriptive text to obtain the frame description text for each of the N key frames. After that, the first language model is used to generate understanding text based on the frame description text of each of the N key frames to obtain the segment understanding text.

[0099] After obtaining the segment understanding text of a video clip, subtitles for the video clip can be generated based on the segment understanding text. For example, by using a second large language model to further process the text based on the segment understanding text corresponding to the video clip, segment subtitles can be obtained. Here, the second large language model refers to the large language model used for subtitle generation; it can be any large language model, without limitation here. Of course, it can be the same as the first large language model mentioned above.

[0100] Of course, similarly, when processing text by understanding the corresponding segments of video clips using the second language model, preset topic information can also be input into the second language model so that the segment subtitles output by the second language model conform to the preset topic information.

[0101] For example, the video editing process in this application is as follows: Figure 6 As shown.

[0102] First, the video to be edited is acquired, and multiple video segments are segmented using a scene detection algorithm (that is, the method in S110). Then, video understanding is performed on each video segment based on preset theme information to obtain the segment understanding text corresponding to each video segment.

[0103] Then, based on the segment understanding text corresponding to the video segment and the preset topic information, the segment subtitles corresponding to the video segment are determined by using a large language model.

[0104] Next, the large language model selects target video segments based on the segment subtitles and segment understanding text corresponding to the video segments, and generates a storyline based on the segment subtitles and segment understanding text of the selected target video segments. This results in target description text that matches the target video segment sequence formed by the target video segments. The target description text can clearly and accurately indicate the development of the storyline of the target video formed by the target video segment sequence.

[0105] Finally, video synthesis is performed based on the target video segment sequence and the corresponding target description text to obtain the edited target video, thereby realizing the editing of the video to be edited.

[0106] In this embodiment, the segment understanding text of the video segment is determined based on the key frames sampled in the video segment. The selected key frames not only cover the dynamic changes of the video segment, but also control the number of selected key frames to avoid redundancy and improve video editing efficiency.

[0107] See also Figure 7 , Figure 7 This illustration shows a block diagram of a video editing apparatus according to an embodiment of the present application. The video editing apparatus 700 includes: The segmentation module 710 is used to segment the video to be edited, resulting in multiple video segments; The generation module 720 is used to generate the understanding text and subtitles for each video segment, so as to obtain the segment understanding text and segment subtitles for each video segment. The determination module 730 is used to determine the target description text based on the segment understanding text and / or segment subtitles corresponding to each of the multiple target video segments selected from multiple video segments; The video acquisition module 740 is used to synthesize video based on the target video segment sequence and the target description text to obtain the target video corresponding to the video to be edited.

[0108] Optionally, the generation module 720 is further configured to, for each video segment, extract keyframes from the video segment based on the length of the video segment to obtain multiple keyframes corresponding to the video segment; generate descriptive text for each keyframe to obtain frame description text for each keyframe; and determine segment understanding text and segment subtitles for each video segment based on the frame description text for each keyframe in the video segment.

[0109] Optionally, the generation module 720 is further configured to arrange the frame description text corresponding to each keyframe in the video segment based on the order of their positions in the video segment, thereby obtaining a sequence of description text corresponding to the video segment; generate understanding text based on the sequence of description text corresponding to the video segment using the first language model, thereby obtaining segment understanding text corresponding to the video segment; and determine the segment subtitles corresponding to the video segment based on the segment understanding text corresponding to the video segment.

[0110] Optionally, the generation module 720 is also used to generate understanding text based on preset topic information and the descriptive text sequence corresponding to the video segment through the first language model, so as to obtain the segment understanding text corresponding to the video segment.

[0111] Optionally, the generation module 720 is also used to perform text reprocessing based on the segment understanding text corresponding to the video segment through the second language model to obtain the segment subtitles corresponding to the video segment.

[0112] Optionally, the determining module 730 is also used to construct a sequence of target video segments based on multiple target video segments; combine the segment understanding text and segment subtitles of each target video segment in the sequence of target video segments to obtain the combined text of each target video segment; and generate the target description text of the sequence of target video segments by using a third language model based on the combined text of each target video segment in the sequence of target video segments.

[0113] Optionally, the determining module 730 is further configured to generate descriptive text for the target video segment sequence by using a third language model based on a prompt template, preset topic information, and combined text of each target video segment in the target video segment sequence; the prompt template is used to prompt the third language model to generate target descriptive text conforming to a preset format without expanding fictional content.

[0114] Optionally, the determining module 730 is further configured to update the order of each target video segment in the target video segment sequence by using a fourth language model, based on at least one of the following: content coherence, thematic consistency, and emotional progression relationship of each target video segment in the target video segment sequence.

[0115] Optionally, the segmentation module 710 is further configured to segment the video to be edited to obtain multiple initial video segments; for each initial video segment, extract highlight video segments from the initial video segment based on the motion intensity between adjacent video frames in the initial video segment; determine non-highlight video segments based on other video segments excluding highlight video segments from the multiple initial video segments; and determine multiple video segments based on highlight video segments and non-highlight video segments.

[0116] Optionally, the determining module 730 is further configured to obtain the editing mode corresponding to the video to be edited; and to filter multiple target video segments from multiple video segments based on the editing mode; the multiple target video segments include at least one highlight video segment.

[0117] Optionally, the determining module 730 is further configured to, if the editing mode is highlight editing mode, acquire all highlight video clips as multiple target video clips; if the editing mode is regular narrative mode, acquire the target highlight video clip and the associated non-highlight video clips associated with the target highlight video clip as multiple target video clips; the target highlight video clip is a highlight video clip selected based on the segment understanding text and / or segment subtitles of each highlight video clip.

[0118] Optionally, the segmentation module 710 is also used to segment the video to be edited based on the visual differences between adjacent video frames in the video to be edited, so as to obtain multiple initial video segments.

[0119] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0120] Figure 8 A structural block diagram of an electronic device for performing a video editing method according to an embodiment of this application is shown. The electronic device may be... Figure 1The terminal (110) or server (120), etc., should be noted. Figure 8 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0121] like Figure 8 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0122] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0123] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0124] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0126] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0127] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0128] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the electronic device to perform the methods of any of the above embodiments.

[0129] In the embodiments of this application, the terms "module" or "unit" refer to a part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0130] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for embodiment.

[0131] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause an electronic device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0132] Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A video editing method, characterized in that, The method includes: The video to be edited is segmented to obtain multiple video clips; For each video segment, the understanding text and subtitles are generated separately to obtain the segment understanding text and segment subtitles for each video segment. Based on the segment understanding text and / or segment subtitles corresponding to each of the multiple target video segments selected from the multiple video segments, the target description text is determined; Based on the multiple target video segments and the target description text, video synthesis is performed to obtain the target video corresponding to the video to be edited.

2. The method according to claim 1, characterized in that, The step of generating comprehension text and subtitles for each video segment to obtain comprehension text and subtitles for each video segment includes: For each video segment, keyframes are extracted based on the length of the video segment to obtain multiple keyframes corresponding to the video segment; For each of the keyframes, a description text is generated to obtain the frame description text for each keyframe. Based on the frame description text of each keyframe in the video segment, the segment understanding text and segment subtitles for each video segment are determined.

3. The method according to claim 2, characterized in that, The process of determining the segment understanding text and segment subtitles for each video segment based on the frame description text of each keyframe in the video segment includes: Based on the chronological order of the keyframes in the video segment, the frame description text corresponding to each keyframe in the video segment is arranged to obtain the description text sequence corresponding to the video segment. Using the first major language model, the understanding text is generated based on the descriptive text sequence corresponding to the video segment, thus obtaining the segment understanding text corresponding to the video segment; Based on the segment understanding text corresponding to the video segment, the segment subtitle corresponding to the video segment is determined.

4. The method according to claim 3, characterized in that, The process of generating comprehension text based on the descriptive text sequence corresponding to the video segment using the first major language model, to obtain the segment comprehension text corresponding to the video segment, includes: Using the first major language model, the understanding text is generated based on preset topic information and the descriptive text sequence corresponding to the video segment, thus obtaining the segment understanding text corresponding to the video segment.

5. The method according to claim 3, characterized in that, The step of determining the segment subtitles corresponding to the video segment based on the segment understanding text includes: The second language model is used to reprocess the text based on the segment understanding text corresponding to the video segment to obtain the segment subtitles corresponding to the video segment.

6. The method according to claim 1, characterized in that, The step of determining the target description text based on the segment understanding text and / or segment subtitles corresponding to each of the multiple target video segments selected from the multiple video segments includes: Based on the multiple target video segments, a target video segment sequence is constructed; The segment understanding text and segment subtitles of each target video segment in the target video segment sequence are combined to obtain the combined text for each target video segment. The target descriptive text of the target video segment sequence is obtained by generating descriptive text based on the combined text of each target video segment in the target video segment sequence using the third major language model.

7. The method according to claim 6, characterized in that, The generation of target description text for the target video segment sequence using a third language model based on the combined text of each target video segment in the target video segment sequence, includes: Using the third language model, descriptive text is generated based on prompt templates, preset topic information, and combined text of each target video segment in the target video segment sequence, resulting in target descriptive text for the target video segment sequence; the prompt template is used to prompt the third language model to generate target descriptive text conforming to a preset format without expanding fictional content.

8. The method according to claim 6, characterized in that, Before combining the segment understanding text and segment subtitles of each target video segment in the target video segment sequence to obtain the combined text for each target video segment, the method further includes: Using the fourth language model, the order of each target video segment in the target video segment sequence is updated based on at least one of the following: content coherence, thematic consistency, and emotional progression.

9. The method according to claim 1, characterized in that, The video to be edited is segmented to obtain multiple video segments, including: The video to be edited is segmented to obtain multiple initial video segments; For each initial video segment, a highlight video segment is extracted from the initial video segment based on the motion intensity between adjacent video frames in the initial video segment; Based on the video segments other than the highlight video segments from the plurality of initial video segments, non-highlight video segments are determined; The plurality of video segments are determined based on the highlight video segments and the non-highlight video segments.

10. The method according to claim 9, characterized in that, Before determining the target description text based on the segment understanding text and / or segment subtitles corresponding to each of the multiple target video segments selected from the plurality of video segments, the method further includes: Obtain the editing mode corresponding to the video to be edited; Multiple target video segments are selected from the plurality of video segments based on the editing mode; the plurality of target video segments include at least one highlight video segment.

11. The method according to claim 10, characterized in that, The step of selecting multiple target video segments from the multiple video segments based on the editing mode includes: If the editing mode is the highlight editing mode, all highlight video clips are obtained as the multiple target video clips; If the editing mode is a regular narrative mode, the target highlight video clip and the associated non-highlight video clips associated with the target highlight video clip are obtained as the plurality of target video clips; the target highlight video clip is a highlight video clip selected based on the segment understanding text and / or segment subtitles of each of the highlight video clips.

12. The method according to claim 9, characterized in that, The process of segmenting the video to be edited yields multiple initial video segments, including: Based on the visual differences between adjacent video frames in the video to be edited, the video to be edited is segmented to obtain multiple initial video segments.

13. A video editing device, characterized in that, The device includes: The segmentation module is used to segment the video to be edited, resulting in multiple video segments. The generation module is used to generate understanding text and subtitles for each video segment, so as to obtain the segment understanding text and segment subtitles for each video segment. The determination module is used to determine the target description text based on the segment understanding text and / or segment subtitles corresponding to each of the multiple target video segments selected from the multiple video segments; The video acquisition module is used to perform video synthesis based on the target video segment sequence and the target description text to obtain the target video corresponding to the video to be edited.

14. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-12.

Images