A video editing method, apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a video clipping method and device, electronic equipment and storage medium, and relate to the technical field of video clipping. The method comprises: constructing at least one video frame group corresponding to a target video to be clipped based on the target video; each video frame group contains a plurality of video frames in the target video, and the content of the segment formed by the plurality of video frames can represent an object behavior of a predetermined clipping object existing in the target video; calling a visual language large model to generate content description data of each video frame group; calling a predetermined large language model based on the content description data of each video frame group to generate caption data of each video frame group; and performing content synthesis processing based on each video frame group and the caption data of each video frame group to clip a video about the predetermined clipping object. The present application can generate a video that matches the video content and has interestingness.

Description

Technical Field

[0001] This application relates to the field of video editing technology, and in particular to a video editing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rise of social networks and short video platforms, the number of videos uploaded by users has exploded. Before uploading a video, users need to edit it to create the final product.

[0003] Current video editing methods mainly involve analyzing the feature data of each video frame to evaluate the quality of each frame, and then automatically editing and compositing the frames that meet the quality requirements to generate a video.

[0004] However, as the demand for engaging and entertaining videos increases, the videos generated by related technologies often lack sufficient appeal. Therefore, how to edit and generate videos that match the content and are also engaging is a pressing issue that needs to be addressed. Summary of the Invention

[0005] The purpose of this application is to provide a video editing method, apparatus, electronic device, and storage medium to edit and generate videos that match the video content and are entertaining. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a video editing method, the method comprising:

[0007] Based on the target video to be edited, at least one video frame group corresponding to the target video is constructed; wherein, each video frame group contains multiple video frames in the target video, and the fragment content formed by the multiple video frames can characterize an object behavior of the predetermined editing object in the target video.

[0008] The visual language model is invoked to generate content description data for each of the constructed video frame groups.

[0009] Based on the content description data of each video frame group, a predefined large language model is invoked to generate the caption data for each video frame group; wherein, the caption data for each video frame group is: description data describing the behavior of the object represented by the video frame group from a first-person perspective.

[0010] Based on each video frame group and its respective accompanying text data, content synthesis processing is performed to edit a video about the predetermined editing object.

[0011] Optionally, constructing at least one group of video frames corresponding to the target video based on the target video to be edited includes:

[0012] The target video to be edited is subjected to frame extraction to obtain various candidate video frames;

[0013] According to the frame order, the candidate video frames are grouped; wherein, each video frame in the grouped video frame group contains an object of the target category, and the content of any segment formed by the grouped video frame group can characterize an object behavior of the target category in the target video; the target category is the category to which the predetermined clip object belongs;

[0014] Based on the video frame groups obtained from the grouping, at least one video frame group corresponding to the target video is determined.

[0015] Optionally, grouping the candidate video frames according to frame order includes:

[0016] Step 1: According to the frame order, select the video frame with the earliest timestamp from each candidate video frame as the current video frame, and create a new video frame group with empty content.

[0017] Step 2: Detect whether there is an object of the target category in the current video frame. If it exists, proceed to step 4; otherwise, proceed to step 3.

[0018] Step 3: Select the next candidate video frame as the current video frame and return to step 2;

[0019] Step 4: Determine if the current video frame group is empty. If it is empty, add the current video frame to the current video frame group and proceed to Step 5. If it is not empty, determine if the time difference between the current video frame and the first video frame in the current video frame group is less than the interval threshold. If it is less, add the current video frame to the current video frame group and proceed to Step 5. If it is not less, proceed to Step 6. The first video frame in the current video frame group is the video frame with the latest timestamp in the current video frame group.

[0020] Step 5: Determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the first time threshold. If it is not greater, take the next candidate video frame as the current video frame and return to step 2. If it is greater, execute step 7. The second video frame in the current video frame group is the video frame with the earliest timestamp in the current video frame group.

[0021] Step 6: Determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold. If it is greater, proceed to step 7. If it is not greater, clear the current video frame group, add the current video frame to the current video frame group, and take the next candidate video frame as the current video frame, and return to step 2.

[0022] Step 7: Determine the current video frame group as a divided video frame group, create a new video frame group with empty content as the current video frame group, and select the next candidate video frame as the current video frame, and return to step 2.

[0023] Step 8: If the time difference between the first video frame and the second video frame in the current video frame group in Step 5 is not greater than the first time threshold, and there is no next candidate video frame for the current video frame, determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold. If it is greater, determine the current video frame group as a completed video frame group and the division ends; otherwise, discard the current video frame group and the division ends. Also, determine each of the currently divided video frame groups as the grouped video frame groups.

[0024] Optionally, the predetermined large language model is a model trained and optimized according to a predetermined optimization method; wherein, the predetermined optimization method includes:

[0025] Determine a visual dataset; wherein each piece of visual data in the visual dataset is an image or a video clip;

[0026] Each visual data in the visual dataset is input into the visual language model to generate content description data corresponding to each visual data.

[0027] Input the content description data corresponding to each visual data into the first language model to obtain the corresponding text data for each content description data.

[0028] Each content description data and its corresponding accompanying text data are used as a sample data to train the second large language model, thereby obtaining the predetermined large language model; wherein, the number of parameters of the first large language model is greater than the number of parameters of the second large language model.

[0029] Optionally, the step of generating caption data for each video frame group by calling a predetermined large language model based on the content description data of each constructed video frame group includes:

[0030] When there are multiple video frame groups, based on the content description data of each video frame group, the video frame groups are filtered according to a predetermined filtering rule; wherein, the predetermined filtering rule is to select video frame groups whose content description data does not meet the similarity condition.

[0031] The predefined large language model is invoked to generate the caption data for each of the selected video frame groups.

[0032] Optionally, the predetermined editing object is any object belonging to the target category; correspondingly, determining at least one video frame group corresponding to the target video based on the video frame group obtained by grouping includes: determining the video frame group obtained by grouping as at least one video frame group corresponding to the target video.

[0033] or,

[0034] The predetermined editing object is a specified object belonging to the target category; correspondingly, determining at least one video frame group corresponding to the target video based on the video frame groups obtained by grouping includes:

[0035] For each group of video frames obtained from grouping, determine the feature information of the objects contained in each video frame in the video frame group, and identify whether the feature information of the objects contained in each video frame matches the feature information of the specified object.

[0036] Based on the number of successfully matched video frames in each video frame and the total number of video frames, it is determined whether the video frame group contains the specified object. If so, the video frame group is determined as the video frame group corresponding to the target video.

[0037] Optionally, the method further includes:

[0038] Based on the text data of each video frame group, generate the voice-over data for each video frame group.

[0039] The process of performing content synthesis based on each video frame group and its respective caption data to edit a video about the predetermined editing object includes:

[0040] For each video frame group, the earliest timestamp of the video frame in the video frame group is used as the start timestamp, and the latest timestamp of the video frame in the video frame group is used as the end timestamp. Video segments are extracted from the target video corresponding to the video frame group. The extracted video segments, the text data and dubbing data of the video frame group are synthesized to obtain the synthesized video segment corresponding to the video frame group.

[0041] The obtained composite video segments are spliced ​​together, and background music is added to the spliced ​​video to edit a video about the predetermined editing object.

[0042] Secondly, embodiments of this application provide a video editing apparatus, the apparatus comprising:

[0043] A construction module is used to construct at least one video frame group corresponding to the target video based on the target video to be edited; wherein each video frame group contains multiple video frames in the target video, and the fragment content formed by the multiple video frames can characterize an object behavior of the predetermined editing object in the target video.

[0044] The first generation module is used to call the large visual language model to generate content description data for each of the constructed video frame groups.

[0045] The second generation module is used to generate the caption data for each video frame group based on the content description data of each video frame group and by calling a predefined large language model. The caption data for each video frame group is: description data describing the behavior of the object represented by the video frame group from a first-person perspective.

[0046] The compositing module is used to perform content compositing processing based on each video frame group and the accompanying text data of each video frame group, so as to edit a video about the predetermined editing object.

[0047] Thirdly, embodiments of this application provide an electronic device, including:

[0048] Memory, used to store computer programs;

[0049] A processor, when executing a program stored in memory, implements any of the video editing methods described above.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the video editing methods described above.

[0051] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the video editing methods described above.

[0052] Beneficial effects of the embodiments in this application:

[0053] The video editing method provided in this application, in order to edit a predetermined editing object, first constructs at least one video frame group corresponding to the target video. Each video frame group contains multiple video frames from the target video, and the fragment content formed by the multiple video frames can represent an object behavior of the predetermined editing object in the target video. That is, by analyzing at least one video frame group, this application can analyze the object behavior of the predetermined editing object. Through a large visual language model, this application can understand and analyze the object behavior of the predetermined editing object represented by the video frame group, and the generated content description data matches the video content. Furthermore, the accompanying text data of each video frame group generated by the predetermined large language model also matches the video content. Moreover, the accompanying text data of each video frame group is a description of the object behavior represented by the video frame group from a first-person perspective. The edited video can describe the object behavior of the predetermined editing object from a first-person perspective, making the edited video more interesting. Therefore, this application can edit and generate videos that match the video content and are interesting.

[0054] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0056] Figure 1 A flowchart illustrating a video editing method provided in an embodiment of this application;

[0057] Figure 2 This is a schematic diagram illustrating a process for grouping candidate video frames according to an embodiment of this application.

[0058] Figure 3 A flowchart illustrating the process of filtering video frame groups that match a specified object, as provided in an embodiment of this application.

[0059] Figure 4 A schematic diagram illustrating the generation of content description data through a large visual language model, as provided in an embodiment of this application.

[0060] Figure 5 A schematic diagram of accompanying data generated from visual data using a first language model, provided as an embodiment of this application.

[0061] Figure 6A schematic diagram illustrating the training of a second language model, provided for an embodiment of this application;

[0062] Figure 7 A schematic diagram illustrating the generation of caption data for video frame groups using a predetermined large language model, as provided in an embodiment of this application.

[0063] Figure 8 This is a schematic diagram of the structure of a video editing device provided in an embodiment of this application;

[0064] Figure 9 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0066] Current video editing methods primarily analyze the feature data of each video frame to evaluate its quality, and then automatically edit and synthesize frames that meet the quality requirements to generate a video. These methods require extracting and analyzing feature data frame by frame, which is time-consuming and involves a large amount of redundant operations. Furthermore, evaluating quality based on feature data makes it difficult to understand the semantic information within video frames, easily leading to the omission of key moments.

[0067] Based on this, embodiments of this application provide a video editing method, apparatus, electronic device, and storage medium for editing and generating videos that match the video content and are entertaining.

[0068] The following section first introduces a video editing method provided in an embodiment of this application.

[0069] The video editing method provided in this application can be applied to electronic devices, such as mobile phones, computers, or servers. This application does not limit the specific form of the electronic device. The video editing method provided in this application can be applied to video editing scenarios, specifically scenarios where one or more target videos to be edited are edited to generate a video; and the duration of the edited video can be less than or equal to the total duration of the target videos to be edited.

[0070] Furthermore, the execution entity of the video editing method provided in this application embodiment can be a video editing device. For example, the video editing device can be functional software running on a terminal device, such as video editing tool software; additionally, the video editing device can also be a plugin for an existing client, for example, a plugin in a video playback client.

[0071] This application provides a video editing method that may include the following steps:

[0072] Based on the target video to be edited, at least one video frame group corresponding to the target video is constructed; wherein, each video frame group contains multiple video frames in the target video, and the fragment content formed by the multiple video frames can represent an object behavior of the predetermined editing object in the target video.

[0073] The visual language model is invoked to generate content description data for each of the constructed video frame groups.

[0074] Based on the content description data of each video frame group, a predefined large language model is invoked to generate the caption data for each video frame group; wherein, the caption data for each video frame group is: description data describing the behavior of the object represented by the video frame group from a first-person perspective.

[0075] Based on each video frame group and its respective accompanying text data, content synthesis processing is performed to edit a video about the predetermined editing object.

[0076] The video editing method provided in this application, in order to edit a predetermined editing object, first constructs at least one video frame group corresponding to the target video. Each video frame group contains multiple video frames from the target video, and the fragment content formed by the multiple video frames can represent an object behavior of the predetermined editing object in the target video. That is, by analyzing at least one video frame group, this application can analyze the object behavior of the predetermined editing object. Through a large visual language model, this application can understand and analyze the object behavior of the predetermined editing object represented by the video frame group, and the generated content description data matches the video content. Furthermore, the accompanying text data of each video frame group generated by the predetermined large language model also matches the video content. Moreover, the accompanying text data of each video frame group is a description of the object behavior represented by the video frame group from a first-person perspective. The edited video can describe the object behavior of the predetermined editing object from a first-person perspective, making the edited video more interesting. Therefore, this application can edit and generate videos that match the video content and are interesting.

[0077] The following description, in conjunction with the accompanying drawings, provides an exemplary video editing method based on an embodiment of this application.

[0078] like Figure 1 As shown in the embodiments of this application, a video editing method may include the following steps:

[0079] S101: Based on the target video to be edited, construct at least one video frame group corresponding to the target video;

[0080] Each video frame group contains multiple video frames from the target video, and the fragment content formed by the multiple video frames can represent the object behavior of the predetermined editing object in the target video.

[0081] The video editing method of this application is for editing a predetermined editing object. This application can generate interesting edited videos by analyzing the object behavior of the predetermined editing object. First, for the video frames in the target video to be edited, at least one group of video frames corresponding to the target video is constructed; the segment content formed by each group of video frames can represent an object behavior of the predetermined editing object in the target video. Therefore, the subsequent analysis of at least one group of video frames can realize the analysis of the object behavior of the predetermined editing object.

[0082] The target video can be one or more, meaning one or more target videos are used as video sources for video editing. Each target video can correspond to at least one group of video frames, and each group of video frames contains multiple video frames corresponding to the target video. Furthermore, the content of any segment formed by the multiple video frames in any group of at least one group of video frames corresponding to each target video can represent the object behavior of the predetermined editing object within the target video. Accordingly, a target video is a video containing the predetermined editing object. It is understandable that a target video can include not only the predetermined editing object but also other objects, which is also reasonable.

[0083] It should be noted that the pre-selected clip object can be any object belonging to the target category, or a specified object belonging to the target category. For example, if the target category is pet, the pre-selected clip object can be any pet object such as a kitten or puppy; or, the pre-selected clip object can be kitten A (i.e., a specified object belonging to the target category).

[0084] There are several ways to construct at least one group of video frames corresponding to the target video to be edited. For example, in one implementation, a large model can be used to analyze the target video to obtain at least one group of video frames. This implementation allows for rapid analysis to obtain at least one group of video frames from the target video.

[0085] For example, in another implementation, based on the target video to be edited, at least one group of video frames corresponding to the target video is constructed, including:

[0086] The target video to be edited is subjected to frame extraction to obtain various candidate video frames;

[0087] According to the frame order, the candidate video frames are grouped; each video frame in the resulting video frame group contains an object of the target category, and the content of any segment formed by the resulting video frame group can represent the object behavior of the target category in the target video; the target category is the category to which the predetermined editing object belongs;

[0088] Based on the video frame groups obtained from the grouping, at least one video frame group corresponding to the target video is determined.

[0089] To accelerate video editing efficiency, reduce redundant manipulations, and lower time costs, this application first performs frame extraction on the target video to obtain candidate video frames. The candidate video frames can be grouped according to frame order. Each video frame in the grouped video frame group contains an object of the target category to which the predetermined editing object belongs. The content of the segment formed by the grouped video frame group can represent the object behavior of the target category object in the target video. That is, the grouped video frame group can be understood as a video frame group about the target category object. Based on the grouped video frame group, at least one video frame group corresponding to the target video and about the predetermined editing object can be determined.

[0090] When performing frame extraction on the target video, frame extraction can be performed at the same time interval, for example, extracting video frames from the target video at intervals of 1 frame every 'a' seconds; alternatively, video frames can be randomly extracted from the target video. This application does not limit the method of frame extraction. Furthermore, the method of grouping the candidate video frames according to frame order will be described in detail in subsequent embodiments and will not be repeated here.

[0091] In this application, the frame-sampling process reduces time costs and unnecessary redundant operations compared to the frame-by-frame analysis method in related technologies. By grouping the candidate video frames according to the frame order, video frame groups of each object in the target category can be obtained, and at least one video frame group corresponding to the target video of the predetermined editing object in the target object can be accurately determined.

[0092] Since the predetermined editing objects are different, the methods for determining at least one video frame group corresponding to the target video are also different. In one implementation of this application, the predetermined editing object is any object belonging to the target category. Accordingly, for example, determining at least one video frame group corresponding to the target video based on the video frame group obtained by grouping includes: determining the video frame group obtained by grouping as at least one video frame group corresponding to the target video.

[0093] or,

[0094] The pre-defined editing object is a specified object belonging to the target category; correspondingly, for example, based on the video frame groups obtained by grouping, at least one video frame group corresponding to the target video is determined, including:

[0095] For each group of video frames obtained from grouping, determine the feature information of the objects contained in each video frame in the video frame group, and identify whether the feature information of the objects contained in each video frame matches the feature information of the specified object.

[0096] Based on the number of successfully matched video frames in each video frame and the total number of video frames, determine whether the video frame group contains the specified object. If so, determine the video frame group as the video frame group corresponding to the target video.

[0097] When the predetermined editing object is any object belonging to the target category, since each video frame in the grouped video frame group contains an object of the target category and can characterize the object behavior of the target category object in the target video, the grouped video frame group can be directly determined as at least one video frame group corresponding to the target video frame.

[0098] When the intended editing object is a specified object belonging to the target category, it is necessary to select a video frame group about the specified object from the grouped video frame groups. First, for each video frame group obtained from the grouping, it is first determined whether the feature information of the object contained in each video frame matches the feature information of the specified object, that is, whether the object contained in each video frame is the specified object. Then, based on the number of successfully matched video frames in each video frame and the total number of video frames, it is determined whether the video frame group contains the specified object. For example, the ratio of the number of successfully matched video frames to the total number of video frames is calculated. If the calculated ratio is greater than a preset judgment threshold (such as 80%), it means that the proportion of video frames in the video frame group that match the feature information of the specified object meets the requirements. The video frame group is considered to contain the specified object and is determined as the video frame group corresponding to the target video.

[0099] It should be noted that feature extraction networks can be used to extract features from each video frame in the video frame group to obtain the feature information of the objects contained in each video frame. Additionally, feature extraction can be performed on images containing a specified object to obtain the feature information of that specified object. During matching and recognition, for each video frame, the similarity between the feature information of the object contained in that video frame and the feature information of the specified object can be calculated. If the similarity is greater than a similarity threshold, the object contained in that video frame is identified as a match with the specified object, and the object contained in that video frame can be considered the specified object. Furthermore, there can be multiple feature information sets for a specified object. For example, there can be multiple images containing a specified object, and the feature extraction network can obtain the feature information of multiple specified objects. For each video frame, the similarity between the feature information of the object contained in that video frame and the feature information of each specified object can be calculated. The maximum calculated similarity is taken as the similarity score between the object contained in that video frame and the specified object, used to determine whether a match has occurred.

[0100] There are various ways to identify whether the feature information of an object contained in a video frame matches the feature information of a specified object, such as Euclidean distance, cross-entropy, etc. This application does not limit this method.

[0101] It should be noted that in some cases, the target video can be a video shot about a specified object. In this case, the video frame group obtained by grouping can be directly determined as at least one video frame group corresponding to the target video.

[0102] In this application, for different situations of the predetermined editing object, a corresponding method can be adopted to select at least one video frame group corresponding to the target video from the video frame groups obtained by grouping, so that the selected at least one video frame group corresponding to the target video can represent the object behavior of the predetermined editing object in the target video, so as to realize video editing for the predetermined editing object.

[0103] Furthermore, for each group of video frames obtained from grouping, before determining the feature information of the objects contained in each video frame of that group, it can be determined whether the number of video frames in that group exceeds a threshold. If so, the video frames in that group are sampled to obtain video frame groups with the number of video frames equal to the threshold. For these video frame groups with the number of video frames equal to the threshold, the feature information of the objects contained in each video frame is then determined. This method can reduce the number of video frames in a group by sampling (uniform sampling or random sampling, etc.), thus improving the efficiency of determining at least one video frame group corresponding to the target video when the number of video frames in the group is too large.

[0104] S102: Call the large visual language model to generate content description data for each of the constructed video frame groups;

[0105] In order to generate edited videos that match the video content, this application calls a large visual language model to perform semantic understanding and analysis on multiple video frames in each video frame group, and generates content description data for each video frame group.

[0106] For example, such as Figure 4 As shown, each video frame group and its content description prompts can be input into the visual language model, enabling the model to perform semantic understanding and analysis based on the prompts and generate content description data for each group. Alternatively, each video frame group and its content description prompts can be input sequentially into the model to generate corresponding content description data. The content description prompts guide the model's semantic understanding and analysis; for example, a prompt might be "Please describe the cat / dog in the video frame in detail, focusing on the cat / dog's behavior." The resulting content description data for a single video frame group could be "A gray cat is being petted by someone nearby."

[0107] S103: Based on the content description data of each video frame group, call the predefined large language model to generate the caption data of each video frame group.

[0108] The accompanying data for each video frame group is: descriptive data describing the behavior of the object represented by the video frame group from a first-person perspective;

[0109] Content description data can be understood as the descriptive data obtained by the visual language big model after semantic understanding and analysis of video frame groups. It describes the behavior of the object represented by the video frame group from a subjective perspective. In order to edit and generate interesting videos, this application calls a predetermined big language model to generate the caption data for each video frame group based on the content description data of each video frame group. Moreover, the caption data of each video frame group is the descriptive data describing the behavior of the object represented by the video frame group from a first-person perspective. For example, if the content description data generated for a video frame group is "A gray cat is being petted by someone next to it", the caption data for the video frame group can be "The owner's hands are so soft, it feels even better to pet it".

[0110] For example, such as Figure 7As shown, the content description data and caption prompts for each video frame group can be input into a predefined large language model so that the model can generate caption data for each video frame group; alternatively, for each video frame group, the content description data and caption prompts can be input into the predefined large language model to generate caption data for that video frame group. The caption prompts guide the predefined large language model to rewrite the content description data into caption data from a first-person perspective; for example, the caption prompt could be "You are a pet content creator, and your task is to rewrite the following description into caption from a pet's perspective."

[0111] For example, in one implementation, based on the content description data of each of the constructed video frame groups, a predefined large language model is invoked to generate caption data for each video frame group, including:

[0112] When there are multiple video frame groups, based on the content description data of each video frame group, the video frame groups are filtered according to a predetermined filtering rule; wherein, the predetermined filtering rule is to select video frame groups whose content description data does not meet the similarity condition.

[0113] The predefined large language model is invoked to generate the caption data for each of the selected video frame groups.

[0114] This application can directly call a predefined large language model to generate caption data for each video frame group. When there are multiple video frame groups, to enhance the entertainment value of the edited video, video frame groups whose content descriptions do not meet the similarity criteria can be selected according to predefined filtering rules; and the predefined large language model can be called to generate caption data for each of the selected video frame groups.

[0115] Among them, video frame groups whose content description data does not meet the similarity condition can be understood as having different object behaviors described by the content description data; that is, this application can select multiple video frame groups that represent different object behaviors from the constructed video frame groups based on the content description data of each of the constructed video frame groups. For example, the object behaviors described by the content description data of each selected video frame group can be: playing, eating, sleeping, interacting, going to the toilet, etc.

[0116] For example, this application can input the content description data of each constructed video frame group, filtering prompts, and filtering examples into a predetermined large language model to filter out various video frame groups. The filtering prompts are used to guide the predetermined large language model to filter out video frame groups that meet the filtering examples from the constructed video frame groups based on their respective content description data.

[0117] For example, the filter prompt could be "You are a pet content creator, and your task is to filter out video frame groups with dissimilar content descriptions from the following video frame groups." Filter examples could be: "1. Pet Playing: The content description shows moments of a pet being active, exploring, or playing fun games, such as a cat / dog running or playing with toys; 2. Pet Eating: The content description shows the behavior of a pet eating; 3. Pet Sleeping: The content description shows peaceful or interesting sleeping postures of a pet; 4. Pet Interaction: The content description shows interactions between pets and people, or between pets, such as pets licking each other's fur or pets playing with people; 5. Pet Using the Toilet: The content description shows the behavior of a pet while learning or eliminating waste." These filter examples can be understood as examples of different object behaviors.

[0118] In this application, when there are multiple video frame groups constructed, video frame groups can be filtered from the constructed video frame groups to select video frame groups whose content description data does not meet the similarity conditions. That is, multiple video frame groups that represent different object behaviors are selected, and a predefined large language model is called to generate the corresponding caption data for each of the filtered video frame groups, so that the edited video can contain multiple different object behaviors and corresponding caption data, thereby improving the interest of the edited video.

[0119] S104: Based on each video frame group and its respective caption data, perform content synthesis processing to edit a video about the predetermined editing object.

[0120] To generate the edited video, content compositing can be performed based on each video frame group (the video frame group that generates the caption data) and the caption data of each video frame group. For example, the video segments corresponding to each video frame group and the caption data of each video frame group can be combined, and then the combined video segments can be spliced ​​together to obtain a video about the predetermined editing object. Alternatively, the video segments corresponding to each video frame group can be spliced ​​together first, and then the spliced ​​video can be combined with the caption data of each video frame group (during the compositing process, the caption data of each video frame group is combined with the corresponding video segments in the spliced ​​video). The video segments corresponding to each video frame group can be extracted from the target video. For example, the earliest timestamp among the timestamps of each video frame in the video frame group can be used as the start timestamp, and the latest timestamp among the timestamps of each video frame in the video frame group can be used as the end timestamp. A video segment can then be extracted from the target video corresponding to that video frame group as the video segment corresponding to that video frame group. It is understandable that content compositing can include background music compositing. That is, content compositing is performed based on each video frame group, the accompanying text data of each video frame group, and the background music. Of course, in other implementations, the edited video can reuse the audio present in the target video. That is, the audio of the video segment corresponding to each video frame group is reused. During compositing, the audio of each video segment is also synthesized. At this time, the video frames and audio frames of the video segments corresponding to each video frame group, as well as the accompanying text data of each video frame group, are all synthesized.

[0121] In addition, to enhance the richness and interest of the edited videos, the video editing method of this application also includes:

[0122] Based on the text data of each video frame group, generate the voice-over data for each video frame group.

[0123] Based on each video frame group and its respective accompanying text data, content compositing is performed to edit a video about a predetermined editing object, including:

[0124] For each video frame group, the earliest timestamp of the video frame in the video frame group is used as the start timestamp, and the latest timestamp of the video frame in the video frame group is used as the end timestamp. Video segments are extracted from the target video corresponding to the video frame group. The extracted video segments, the text data and dubbing data of the video frame group are synthesized to obtain the synthesized video segment corresponding to the video frame group.

[0125] The obtained composite video segments are spliced ​​together, and background music is added to the spliced ​​video to edit a video about the predetermined editing object.

[0126] This application can also generate voice-over data for each video frame group based on the caption data of each video frame group, and during content compositing, extract the video segment corresponding to each video frame group from the target video, and synthesize the video segment, the caption data of the video frame group, and the voice-over data to obtain the synthesized video segment corresponding to the video frame group; then, the synthesized video segments can be spliced ​​together (either according to the time order of each synthesized video segment in the target video, or in a random order, etc.) and background music can be added to edit a video about the predetermined editing object.

[0127] It should be noted that the methods for synthesizing video clips with accompanying text and voice-over data, and for splicing together the synthesized video clips and adding background music, are similar to existing technologies and will not be elaborated upon here. The added background music can be any randomly selected or pre-set background music, and this application does not limit this.

[0128] Alternatively, the voiceover data for each video frame group can be automatically generated by inputting the voiceover data of each video frame group into a text-to-speech (TTS) algorithm.

[0129] As can be seen, this application can generate voice-over data based on the caption data, and synthesize video clips, caption data and voice-over data of the same video frame group to generate a synthesized video clip. Furthermore, after splicing the synthesized video clips and adding background music, a video about a predetermined editing object is obtained. The resulting edited video contains multimodal data of video clips, caption data, voice-over data and background music, making the edited video about the predetermined object more rich and interesting.

[0130] It should be noted that for any target video to be edited, the video of the predetermined editing object can be generated by editing in the above manner. If there are multiple target videos, during the content compositing process, after splicing the various composite video segments of each target video to obtain the corresponding spliced ​​video, the spliced ​​videos of each target video can be spliced ​​again (the splicing order is not limited) to obtain the video of the predetermined editing object for each target video.

[0131] In the technical solution of this application, the acquisition, storage, use, processing, transmission, provision and disclosure of target videos, video frames, video frame groups, content description data, caption data, dubbing data and background music are all carried out with the user's authorization.

[0132] The video editing method provided in this application, in order to edit a predetermined editing object, first constructs at least one video frame group corresponding to the target video. Each video frame group contains multiple video frames from the target video, and the fragment content formed by the multiple video frames can represent an object behavior of the predetermined editing object in the target video. That is, by analyzing at least one video frame group, this application can analyze the object behavior of the predetermined editing object. Through a large visual language model, this application can understand and analyze the object behavior of the predetermined editing object represented by the video frame group, and the generated content description data matches the video content. Furthermore, the accompanying text data of each video frame group generated by the predetermined large language model also matches the video content. Moreover, the accompanying text data of each video frame group is a description of the object behavior represented by the video frame group from a first-person perspective. The edited video can describe the object behavior of the predetermined editing object from a first-person perspective, making the edited video more interesting. Therefore, this application can edit and generate videos that match the video content and are interesting.

[0133] Alternatively, in another embodiment of this application, such as Figure 2 As shown, grouping candidate video frames according to frame order can include the following steps:

[0134] Step 1: According to the frame order, select the video frame with the earliest timestamp from each candidate video frame as the current video frame, and create a new video frame group with empty content.

[0135] When grouping, for the first video frame, according to the frame order, select the video frame with the earliest timestamp (the first video frame) from each candidate video frame as the current video frame, and create a new video frame group with empty content as the current video frame group.

[0136] Step 2: Detect whether there is an object of the target category in the current video frame. If it exists, proceed to step 4; otherwise, proceed to step 3.

[0137] To ensure that every video frame in the resulting video frame group contains an object of the target category, it is possible to detect whether the current frame contains an object of the target category. If it does, step 4 is executed to determine the grouping of video frames containing objects of the target category; otherwise, step 3 is executed to group the next candidate video frames.

[0138] Step 3: Select the next candidate video frame as the current video frame and return to step 2;

[0139] If the current video frame does not contain an object of the target category, then the current video frame is not grouped into a video frame group. Instead, the next candidate video frame is used as the current video frame, and the process returns to step 2 to continue grouping the next candidate video frame.

[0140] Step 4: Determine if the current video frame group is empty. If it is empty, add the current video frame to the current video frame group and proceed to Step 5. If it is not empty, determine if the time difference between the current video frame and the first video frame in the current video frame group is less than the interval threshold. If it is less, add the current video frame to the current video frame group and proceed to Step 5. If it is not less, proceed to Step 6.

[0141] Before adding the current video frame to the current video frame group, it can be determined whether the current video frame group is empty. If it is empty, the current video frame is added to the current video frame group, and step 5 is executed. If it is not empty, it is determined whether the time difference between the current video frame and the first video frame in the current video frame group is less than the interval threshold. If it is less, the current video frame is added to the current video frame group (that is, it is determined whether the time interval between the current video frame and the video frame with the latest timestamp in the current video frame group is less than the longest time interval during which the target category object has not appeared. If it is less, the video frame is added to the current video frame group to ensure the continuity of the object behavior represented by the video frames in the divided video frame groups), and step 5 is executed. If it is not less, step 6 is executed. Here, the first video frame in the current video frame group is the video frame with the latest timestamp in the current video frame group; the interval threshold is used to represent the longest time interval during which the target category object is allowed to not appear.

[0142] Step 5: Determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the first time threshold. If it is not greater, take the next candidate video frame as the current video frame and return to step 2; if it is greater, proceed to step 7.

[0143] After adding the current video frame to the current video frame group, it can be determined whether the current video frame group is a fully divided video frame group. That is, it is determined whether the time difference between the first and second video frames in the current video frame group is greater than the first time threshold. If it is not greater, it means that other video frames can be added to the current video frame group. The next candidate video frame is taken as the current video frame, and the process returns to step 2. If it is greater, it means that the current video frame group has met the maximum duration requirement (that is, during the loop execution, the duration represented by the current video frame group exceeds the first time threshold for the first time), and step 7 is executed. Here, the second video frame in the current video frame group is the video frame with the earliest timestamp in the current video frame group. The first time threshold is used to represent the maximum duration required for any video segment.

[0144] Step 6: Determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold. If it is greater, proceed to step 7. If it is not greater, clear the current video frame group, add the current video frame to the current video frame group, and take the next candidate video frame as the current video frame, and return to step 2.

[0145] If the current video frame group is not empty, and the time difference between the current video frame and the first video frame in the current video frame group is not less than the interval threshold, it indicates that the current video frame and the first video frame are not continuous, and the current video frame is not added to the current video frame group. At this time, it can be determined whether the current video frame group has been divided, that is, whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold. If it is greater, that is, the duration represented by the current video frame group is greater than the required minimum duration, and step 7 is executed to determine the current video frame group as a divided video frame group; if it is not greater, it indicates that the current video frame group is not completely divided. If the duration represented by the video frame group does not meet the minimum duration requirement, the video frames in the current video frame group can be considered as not being relevant. Furthermore, the current video frame and subsequent video frames are not continuous with the first video frame in the current video frame group, and no other video frames can be added to the current video frame group. Therefore, the current video frame group is cleared (i.e., the video frames in the current video frame group are discarded), and the current video frame is added to the cleared current video frame group. The next candidate video frame is then used as the current video frame, and the process returns to step 2. Here, the second time threshold is used to represent the minimum duration required for any video segment.

[0146] Step 7: Determine the current video frame group as a divided video frame group, create a new video frame group with empty content as the current video frame group, and select the next candidate video frame as the current video frame, and return to step 2.

[0147] That is, when the duration represented by the current video frame group exceeds the maximum duration requirement for the first time, or when the time difference between the current video frame and the first video frame is not less than the interval threshold, the duration represented by the current video frame group is greater than the minimum duration requirement. The current video frame group is then determined as a completed video frame group, and a new video frame group with empty content is created as the current video frame group. The next candidate video frame is then used as the current video frame, and the process returns to step 2 for the division of the next video frame group.

[0148] Step 8: If the time difference between the first video frame and the second video frame in the current video frame group in Step 5 is not greater than the first time threshold, and there is no next candidate video frame for the current video frame, determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold. If it is greater, determine the current video frame group as a completed video frame group and the division ends; otherwise, discard the current video frame group and the division ends. Also, determine each of the currently divided video frame groups as the grouped video frame groups.

[0149] If, in step 5, the time difference between the first and second video frames in the current video frame group is not greater than the first time threshold (i.e., the duration of the current video frame group is not greater than the maximum duration requirement), and there is no next video frame for the current video frame, then it can be determined whether the time difference between the first and second video frames in the current video frame group is greater than the second time threshold (i.e., whether the duration of the current video frame group is greater than the minimum duration requirement). If it is greater, it indicates that the current video frame group is the last video frame group that meets the time requirement, and the current video frame group is determined as a video frame group that has been divided, and the division ends. Otherwise, it indicates that the current video frame group is the last video frame group and does not meet the time requirement, and the current video frame group is discarded, and the division ends. In addition, each video frame group that has been divided is determined as a video frame group obtained by grouping.

[0150] As can be seen, the above steps in this application can divide candidate video frames containing objects of the target category, and the divided video frame groups meet the minimum duration requirement and the maximum duration requirement. Furthermore, the video frames in the video frame groups are continuous and can accurately represent the object behavior of the target category.

[0151] In another embodiment of this application, the predetermined large language model is a model trained and optimized according to a predetermined optimization method; wherein, the predetermined optimization method includes:

[0152] Define the visual dataset; where each piece of visual data in the visual dataset is either an image or a video clip;

[0153] Each visual data point in the visual dataset is input into the visual language model to generate content description data corresponding to each visual data point.

[0154] Input the content description data corresponding to each visual data into the first language model to obtain the corresponding text data for each content description data.

[0155] Each content description data and its corresponding accompanying text data are used as a sample data to train the second large language model, thus obtaining the predetermined large language model; wherein, the number of parameters of the first large language model is greater than the number of parameters of the second large language model.

[0156] The pre-defined large language model in this application can be a model optimized and trained according to a predetermined optimization method, so that the caption data of each generated video frame group can more accurately describe the video content, making the generated edited video match the video content. Furthermore, this method can train a pre-defined large language model with a relatively small number of parameters and high accuracy (the accuracy of the pre-defined large language model can be close to that of the first large language model). Compared to directly using the first large language model as the pre-defined large language model, this effectively reduces the storage space occupied and the resource consumption required for model invocation.

[0157] When optimizing training, the sample data required for training can be determined first. This application aims to enable the pre-defined large language model after optimization training to generate more accurate matching text data based on the content description data. Therefore, a piece of content description data and its corresponding matching text data can be used as a piece of sample data.

[0158] When constructing sample data, a visual dataset can be determined first. Each visual data point in the visual dataset is either an image or a video clip. Image data can represent the static behavior of the contained object, while video clips can represent the continuous behavior of the contained object. In other words, the visual data contained in the visual dataset can more comprehensively represent the behavior of the contained object. This application can input each visual data point in the visual dataset into a large visual language model to generate content description data corresponding to each visual data point. Then, the content description data corresponding to each visual data point is input into a first large language model to obtain the corresponding caption data. Afterward, each content description data point and its corresponding sample data can be used to train a second large language model to obtain the predetermined large language model.

[0159] The first language model in this application has a larger number of parameters than the second language model to ensure the accuracy of the corresponding text data for each content description data obtained through the first language model. Furthermore, the corresponding text data for each content description data generated by the first language model can be delivered to annotators for review and adjustment to ensure that the text data conforms to the content described by the corresponding content description data. Moreover, the text data can be rewritten in various ways to optimize the training of the pre-defined large language model and enable it to generate diverse text data.

[0160] like Figure 6As shown, when training the second language model, for each sample data, the content description data in the sample data can be used as input data, and the accompanying text data in the sample data can be used as the ground truth. The loss of the second language model for the output data of the content description data and the accompanying text data can be calculated in any way, such as mean squared error, mean absolute error, etc. The loss characterizes the ability of the second language model to generate accompanying text data for the content description data. The parameters of the second language model are gradually adjusted until the calculated loss meets the requirements, and the predetermined large language model after optimization training is obtained.

[0161] The visual language model in this application focuses on generating content description data, while the first and second large language models focus on language conversion (converting objective content description data into caption data from a first-person perspective). By using the first large language model, which has more parameters, to generate caption data corresponding to the content description data, the accuracy of the generated caption data can be ensured. Subsequently, the second large language model is optimized and trained using the content description data and the corresponding caption data. The resulting pre-defined large language model has stronger language conversion capabilities and can generate more accurate and diverse caption data based on the content description data.

[0162] The video editing method provided in this application will now be described in conjunction with another embodiment.

[0163] This application includes steps such as video frame extraction, pet detection (detection of objects of the target category), segmentation (video segment extraction), pet ReID (used to associate content description data, caption data, and voice-over data, etc.), content understanding (semantic understanding and analysis through a large visual language model), video frame group selection, caption data generation, voice-over data generation, and video segment splicing and synthesis processing, to achieve the generation of edited video from the original target video.

[0164] The specific process of this application embodiment is as follows:

[0165] 1. Input multi-angle image templates for pets (taking pets as the target category as an example) to establish pet identity files;

[0166] 2. For each of the n input target videos, extract video frames from each target video at intervals of 1 frame per 'a' seconds to obtain candidate video frames. These candidate video frames can form a set of video frame sequences. , , 1≤i≤n. Where, Let n be the set of total video frame sequences extracted from n target videos. Let k be the sequence of video frames extracted from the i-th target video, where k is the number of video frames extracted. Let be the k-th video frame extracted from the i-th target video.

[0167] III. For the set of total video frame sequences This paper uses an improved pet detection algorithm (in this application, pets refer to cats and dogs) based on general object detection algorithms (such as YOLOv5, YOLOv8, etc.; YOLO stands for You Only Look Once, YOLOv5 is the fifth version of the YOLO algorithm, and YOLOv8 is the eighth version of the YOLO algorithm) to obtain... Each of the Feature information of the pet in each video frame.

[0168] IV. For According to Figure 2 Group them as shown. , The video frame groups obtained from each group are characterized. This is a group of video frames obtained by grouping. For Finally, the total set of video frame groups is obtained by dividing the n target videos into groups. =flatten( ) = [c1, …, cv], where v = v1, …, or v n The `flatten` function is typically used to "flatten" nested lists (or other iterable objects), that is, to convert multi-level nested structures into single-level structures. For example, `[[1, 2], [3, [4, 5]]]` is converted to `[1, 2, 3, 4, 5]`.

[0169] V. Regarding For each video frame group, if the number of video frames in the video frame group exceeds the maximum limit b (number threshold), then the video frame group is uniformly sampled to obtain a video frame group with b video frames. For example... Figure 3 As shown, for each video frame group (video frame group with or without uniform sampling), based on the pet detection target box of each video frame in the video frame group, step S301 is performed: pet target region cropping, to obtain the pet target region set b. v = [ ,… The feature extraction network (such as Dino, Dinov2, etc.; Dino stands for DETR with Improved DeNoising, meaning it combines the DETR framework with an improved denoising training mechanism; Dinov2 indicates the version iteration, i.e., the second version of Dino) is used to extract features from b. vS302: Feature extraction is performed on each pet target region to obtain the pet target region feature set. v = [ ,…, [This refers to the feature information v1-vx of the objects contained in each video frame.] Accordingly, a feature extraction network is used to perform S302: feature extraction on the multi-angle pet identity templates (identity template image 1...identity template image y) entered in the pet identity file to obtain the pet (a specific pet, corresponding to the specified object mentioned above) identity template feature. id = [ , …, [, i.e., the feature information id1-feature information idy of the specified object. For any pet target region feature Combine it with pet identity template features id Each identity template feature (i.e., any feature information vx and feature information id1 - feature information idy) is sequentially subjected to S303: similarity matching, and each similarity is taken. ... The maximum value in As The similarity to the specified object is calculated by selecting the maximum similarity vx from similarity v1 and similarity vx. Then, S304 is performed: threshold comparison; if... Greater than the threshold If a match is found, the video frame is considered a successful match with the pet designated as the target; the number of successfully matched video frames in the video frame group (i.e., the number of successfully matched video frames) is incremented by 1 (matched_objv); otherwise, the match is considered a failure. v After all pet target areas in the video frame group have completed identity template matching (i.e., after all video frames in the video frame group have been matched), execute S305: calculate the ratio, that is, calculate the ratio of the number of successfully matched frames to the total number of video frames in the video frame group, matched_objv / len(b v ); and based on the calculated ratio, perform S306: threshold comparison; and based on the result of the threshold comparison, execute S307: determine whether a match is found, that is, determine whether the video frame group is a video frame group that matches the specified object; for example, if the ratio of the number of successful matches to the total number of video frames in the video frame group is matched_objv / len(b v () greater than the preset judgment threshold If the video frame group is found to meet the requirements for pet identification, it will be added to the set of successfully matched video frame groups. In the middle. When After all video frame groups have been matched, the final set of successfully matched video frame groups is obtained. =[c1,…,cw],w≤v.

[0170] VI. Regarding Each video frame group in the range [c1,…,cw] is sequentially input into the visual language model along with the content description cues. In this process, a collection of content description data for each video frame group is obtained. =[d1,..,dw].

[0171] VII. The filter suggestions and filter examples are input into a predefined large language model. In the process, a list of selected video frame groups (which can be understood as video frame groups representing exciting content) is obtained: id_list = [id1, … idp]. The filtering prompts require the model to prioritize video frame groups whose content description data does not meet the similarity criteria. A heuristic definition of a set of video frame groups that meet the filtering rules is given in the filtering example.

[0172] 8. In order to generate engaging captions from a pet's perspective, this application's planned large language model This refers to a model trained and optimized according to a predetermined optimization method (e.g., Supervised Fine-Tuning (SFT)). Visual dataset. Composed of a mixture of single images and video clips, for Each piece of visual data is input into the large visual language model. In the process, a visual dataset is obtained. Content description collection .for The content description data of each visual data point is input into the first language model along with the pet-perspective humorous captions (similar to caption prompts). middle( The first language model has more parameters than the second language model. It generates humorous captions from a pet's perspective. During generation, it can also input caption examples (i.e., examples of pet-perspective caption styles) into the first language model, ultimately obtaining a set of caption data corresponding to the visual data. .

[0173] like Figure 5 As shown, for each visual data item, the content description data can be input along with the prompt words for generating captions into the first language model to generate the caption data corresponding to that visual data item; or, to improve the efficiency of caption data generation, the content description data of each visual data item and the prompt words for generating captions can be input into the first language model to generate the caption data corresponding to each visual data item.

[0174] According to ReID and The elements in the dataset are linked one-to-one (i.e., based on the ReID of the visual data, the content description data of the same visual data is associated with the accompanying text data), forming the sample dataset. , It will be handed over to the annotation personnel for annotation and inspection. The annotation process involves verifying whether the content description data in the text matches the corresponding text data (i.e., whether the content they describe is identical), and performing content calibration and diversifying the expression of the text data to obtain the final annotated version. . Used for supervised fine-tuning of large language models .

[0175] The caption prompt can be something like, "You are a pet content creator, and your task is to rewrite the following descriptions into captions from a pet's perspective." Caption examples can include content description data and corresponding caption data. For instance, the content description data could include: "1. A black and white cat is eating in front of its bowl; 2. A tabby cat is being petted while lying on someone's lap..." The corresponding caption data could be: "1. This food smells so good, I can't stop eating it; 2. My owner's hands are so soft, it feels even better to pet them..."

[0176] 9. The collection of content descriptions for video frame groups in id_list The accompanying prompts and examples will be fed into the pre-defined large language model after optimization and training. In the middle, we obtained a set of humorous captions from a pet's perspective, CP = [ ,…, The pre-defined large language model used in step seven is also an optimized and trained model.

[0177] 10. Map the video frame group numbers in id_list to (That is, based on the video frame group number, from) (In the process of determining the filtered video frame groups), a set of filtered video frame groups is obtained. =[ ,…, ] For any filtered group of video frames Obtain its start timestamp in the original target video. and end timestamp Extract from the original video [ , The video segment of the video frame group is obtained from the video segment of the video frame group. .for The selected video frame groups in the process ultimately yield a set of video clips. .

[0178] 11. Input each piece of voiceover data from CP into the text-to-speech (TTS) algorithm to obtain the voiceover data set A = [ ;

[0179] 12. Set the accompanying text data set CP = [ ,…, ] Video clip collection and the dubbing data set A = [ Each element in the process is mapped one-to-one using ReID (that is, ReID is used to associate the caption data, video clips, and dubbing data corresponding to the same video frame group), and then synthesized to obtain a set of composite video clips with captions and dubbing. .Will The video clips are spliced ​​together and background music is added to get the final edited pet video.

[0180] This application significantly reduces the large amount of redundant operations associated with frame-by-frame analysis by sampling and segmenting the target video data (dividing the video frame into groups, removing video frames without pets and discontinuous video frames), while accurately preserving valid video clips containing pets. Simultaneously, this application utilizes a large visual language model and a large language model to collaboratively analyze video frames, effectively capturing and deeply understanding pet behavior, and filtering out engaging and interesting pet video clips that match the video content. Furthermore, this application can generate caption data describing the subject's behavior from a first-person perspective, and generate voice-over data, making the edited pet videos more entertaining.

[0181] Based on the above method embodiments, this application also provides a video editing apparatus, such as... Figure 8 As shown, the device includes:

[0182] The construction module 810 is used to construct at least one video frame group corresponding to the target video based on the target video to be edited; wherein each video frame group contains multiple video frames in the target video, and the fragment content formed by the multiple video frames can characterize an object behavior of the predetermined editing object in the target video.

[0183] The first generation module 820 is used to call the large visual language model to generate content description data for each of the constructed video frame groups.

[0184] The second generation module 830 is used to generate the caption data for each video frame group based on the content description data of each of the constructed video frame groups and by calling a predetermined large language model; wherein, the caption data for each video frame group is: description data describing the behavior of the object represented by the video frame group from a first-person perspective.

[0185] The compositing module 840 is used to perform content compositing processing based on each video frame group and the accompanying text data of each video frame group, so as to edit a video about the predetermined editing object.

[0186] Optionally, the building module includes:

[0187] The frame extraction submodule is used to extract frames from the target video to be edited, and obtain various candidate video frames.

[0188] The grouping submodule is used to group the candidate video frames according to the frame order; wherein, each video frame in the grouped video frame group contains an object of the target category, and the content of any segment formed by the grouped video frame group can characterize an object behavior of the target category in the target video; the target category is the category to which the predetermined clip object belongs;

[0189] The determination submodule is used to determine at least one video frame group corresponding to the target video based on the video frame groups obtained by grouping.

[0190] Optionally, the grouping submodule is specifically used for:

[0191] Step 1: According to the frame order, select the video frame with the earliest timestamp from each candidate video frame as the current video frame, and create a new video frame group with empty content.

[0192] Step 2: Detect whether there is an object of the target category in the current video frame. If it exists, proceed to step 4; otherwise, proceed to step 3.

[0193] Step 3: Select the next candidate video frame as the current video frame and return to step 2;

[0194] Step 4: Determine if the current video frame group is empty. If it is empty, add the current video frame to the current video frame group and proceed to Step 5. If it is not empty, determine if the time difference between the current video frame and the first video frame in the current video frame group is less than the interval threshold. If it is less, add the current video frame to the current video frame group and proceed to Step 5. If it is not less, proceed to Step 6. The first video frame in the current video frame group is the video frame with the latest timestamp in the current video frame group.

[0195] Step 5: Determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the first time threshold. If it is not greater, take the next candidate video frame as the current video frame and return to step 2. If it is greater, execute step 7. The second video frame in the current video frame group is the video frame with the earliest timestamp in the current video frame group.

[0196] Step 6: Determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold. If it is greater, proceed to step 7. If it is not greater, clear the current video frame group, add the current video frame to the current video frame group, and take the next candidate video frame as the current video frame, and return to step 2.

[0197] Step 7: Determine the current video frame group as a divided video frame group, create a new video frame group with empty content as the current video frame group, and select the next candidate video frame as the current video frame, and return to step 2.

[0198] Step 8: If the time difference between the first video frame and the second video frame in the current video frame group in Step 5 is not greater than the first time threshold, and there is no next candidate video frame for the current video frame, determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold. If it is greater, determine the current video frame group as a completed video frame group and the division ends; otherwise, discard the current video frame group and the division ends. Also, determine each of the currently divided video frame groups as the grouped video frame groups.

[0199] Optionally, the predetermined large language model is a model trained and optimized according to a predetermined optimization method; wherein, the predetermined optimization method includes:

[0200] Determine a visual dataset; wherein each piece of visual data in the visual dataset is an image or a video clip;

[0201] Each visual data in the visual dataset is input into the visual language model to generate content description data corresponding to each visual data.

[0202] Input the content description data corresponding to each visual data into the first language model to obtain the corresponding text data for each content description data.

[0203] Each content description data and its corresponding accompanying text data are used as a sample data to train the second large language model, thereby obtaining the predetermined large language model; wherein, the number of parameters of the first large language model is greater than the number of parameters of the second large language model.

[0204] Optionally, the second generation module is specifically used for:

[0205] When there are multiple video frame groups, based on the content description data of each video frame group, the video frame groups are filtered according to a predetermined filtering rule; wherein, the predetermined filtering rule is to select video frame groups whose content description data does not meet the similarity condition.

[0206] The predefined large language model is invoked to generate the caption data for each of the selected video frame groups.

[0207] Optionally, the predetermined editing object is any object belonging to the target category; correspondingly, the determining submodule is used to: determine the video frame group obtained by grouping as at least one video frame group corresponding to the target video;

[0208] or,

[0209] The predetermined clipping object is a specified object belonging to the target category; correspondingly, the determining submodule is used for:

[0210] For each group of video frames obtained from grouping, determine the feature information of the objects contained in each video frame in the video frame group, and identify whether the feature information of the objects contained in each video frame matches the feature information of the specified object.

[0211] Based on the number of successfully matched video frames in each video frame and the total number of video frames, it is determined whether the video frame group contains the specified object. If so, the video frame group is determined as the video frame group corresponding to the target video.

[0212] Optionally, the device further includes:

[0213] The third generation module is used to generate the voice-over data for each video frame group based on the text data of each video frame group.

[0214] The synthesis module is specifically used for:

[0215] For each video frame group, the earliest timestamp of the video frame in the video frame group is used as the start timestamp, and the latest timestamp of the video frame in the video frame group is used as the end timestamp. Video segments are extracted from the target video corresponding to the video frame group. The extracted video segments, the text data and dubbing data of the video frame group are synthesized to obtain the synthesized video segment corresponding to the video frame group.

[0216] The obtained composite video segments are spliced ​​together, and background music is added to the spliced ​​video to edit a video about the predetermined editing object.

[0217] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0218] Memory 901 is used to store computer programs;

[0219] The processor 902 is used to implement any of the video editing methods described above when executing the program stored in the memory 901.

[0220] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0221] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0222] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0223] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0224] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0225] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the video editing methods described above.

[0226] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the video editing methods described above.

[0227] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0228] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0229] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0230] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A video clip method characterized by, The method comprises: Based on the target video to be edited, at least one video frame group corresponding to the target video is constructed; each video frame group contains a plurality of video frames in the target video, and the content of the plurality of video frames can represent an object behavior of a predetermined editing object in the target video; A visual language large model is called to generate content description data of each video frame group; Based on the content description data of each video frame group, a predetermined large language model is called to generate caption data of each video frame group; the caption data of each video frame group is description data describing the object behavior represented by the video frame group in the first person perspective; Based on each video frame group and the caption data of each video frame group, content synthesis processing is performed to edit a video about the predetermined editing object; The at least one video frame group is constructed by grouping each candidate video frame of the target video; the grouping of each candidate video frame of the target video comprises: Step 1: according to the frame sequence, select the video frame with the earliest timestamp from each candidate video frame as the current video frame, and create a new video frame group with empty content; Step 2: detect whether the current video frame contains an object of the target category, if yes, execute step 4, otherwise, execute step 3; Step 3: take the next candidate video frame as the current video frame and return to step 2; Step 4: determine whether the current video frame group is empty, if yes, add the current video frame to the current video frame group and execute step 5; if not, determine whether the time difference between the current video frame and the first video frame in the current video frame group is less than the interval threshold, if yes, add the current video frame to the current video frame group and execute step 5; if not, execute step 6; wherein the first video frame in the current video frame group is the video frame with the latest timestamp in the current video frame group; Step 5: determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the first time threshold, if not, take the next candidate video frame as the current video frame and return to step 2; if yes, execute step 7; wherein the second video frame in the current video frame group is the video frame with the earliest timestamp in the current video frame group; Step 6: determine whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold, if yes, execute step 7; if not, clear the current video frame group, add the current video frame to the current video frame group, and take the next candidate video frame as the current video frame and return to step 2; Step 7: determine the current video frame group as a divided video frame group, create a new video frame group with empty content as the current video frame group, and take the next candidate video frame as the current video frame and return to step 2; Step 8, if the time difference between the first video frame and the second video frame in the current video frame group is not greater than the first time threshold in step 5, and there is no next candidate video frame of the current video frame, it is judged whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold, if yes, the current video frame group is determined as a divided video frame group, and the division is ended, otherwise, the current video frame group is discarded, and the division is ended; and the current divided video frame group is determined as a video frame group obtained by grouping.

2. The method of claim 1, wherein, The at least one video frame group corresponding to the target video is constructed based on the target video to be edited, including: Frame extraction processing is performed on the target video to be edited to obtain each candidate video frame; Grouping is performed on each candidate video frame; wherein each video frame in the video frame group obtained by grouping contains an object of a target category, and the content of any video frame group obtained by grouping forms a segment that can represent an object behavior of the object of the target category existing in the target video; the target category is a category to which the predetermined editing object belongs; Based on the video frame group obtained by grouping, at least one video frame group corresponding to the target video is determined.

3. The method according to any of claims 1-2, characterized in that, The predetermined large language model is a model optimized by a predetermined optimization method; wherein the predetermined optimization method includes: Determine a visual data set; wherein each piece of visual data in the visual data set is image data or a video segment; Each visual data in the visual data set is input into the visual language large model to generate content description data corresponding to each visual data respectively; Each content description data and the corresponding script data of the content description data are taken as a sample data, and the second large language model is trained to obtain the predetermined large language model; wherein the number of parameters of the first large language model is greater than the number of parameters of the second large language model. Based on the content description data of each video frame group constructed, the predetermined large language model is called to generate script data of each video frame group respectively, including:

4. The method according to any one of claims 1-2, characterized in that, In the case where the number of constructed video frame groups is multiple, based on the content description data of each video frame group constructed, video frame groups are selected according to a predetermined selection rule from the constructed video frame groups; wherein the predetermined selection rule is to select video frame groups whose content description data does not meet the similarity condition; The predetermined large language model is called to generate script data of each video frame group selected respectively. The predetermined editing object is any object belonging to a target category; accordingly, based on the video frame group obtained by grouping, at least one video frame group corresponding to the target video is determined, including: the video frame group obtained by grouping is determined as at least one video frame group corresponding to the target video.

5. The method of claim 2, wherein, Or, ​ The predetermined clipping object is a specified object belonging to a target category; correspondingly, the video frame group obtained based on grouping is used to determine at least one video frame group corresponding to the target video, including: For each video frame group obtained by grouping, the feature information of the objects contained in each video frame in the video frame group is determined, and whether the feature information of the objects contained in each video frame matches the feature information of the specified object is identified respectively; Based on the number of matching successful video frames in each video frame and the total number of the video frames, it is judged whether the video frame group is a video frame group containing the specified object, and if so, the video frame group is determined as the video frame group corresponding to the target video.

6. The method according to any one of claims 1-2, characterized in that, The method further includes: Generating the dubbing data of each video frame group according to the script data of each video frame group; The content synthesis processing is performed based on each video frame group and the script data of each video frame group to obtain a video about the predetermined clipping object, including: For each video frame group, the earliest timestamp of the video frames of the video frame group is taken as the starting timestamp, and the latest timestamp of the video frames of the video frame group is taken as the ending timestamp, a video segment is cut from the target video corresponding to the video frame group, and the cut video segment, the script data and the dubbing data of the video frame group are synthesized to obtain a synthesized video segment corresponding to the video frame group; The obtained synthesized video segments are spliced and background music is added to the spliced video to obtain a video about the predetermined clipping object.

7. A video clip device characterized by The device includes: The construction module is configured to construct at least one video frame group corresponding to the target video based on the target video to be clipped; each video frame group contains a plurality of video frames in the target video, and the segment content formed by the plurality of video frames can represent an object behavior of the predetermined clipping object in the target video; The first generation module is configured to call a visual language large model to generate content description data of each constructed video frame group; The second generation module is configured to call a predetermined large language model based on the content description data of each constructed video frame group to generate script data of each video frame group; the script data of each video frame group is description data describing the object behavior represented by the video frame group in the first person perspective; The synthesis module is configured to perform content synthesis processing based on each video frame group and the script data of each video frame group to obtain a video about the predetermined clipping object; The at least one video frame group is constructed by grouping each candidate video frame of the target video; the grouping of each candidate video frame of the target video includes: Step 1: selecting a video frame with the earliest timestamp as the current video frame from each candidate video frame according to the frame sequence, and creating a new video frame group with empty content; Step 2: detecting whether the current video frame contains an object of the target category, if so, executing step 4, otherwise, executing step 3; Step 4: adding the current video frame to the video frame group, and executing step 2. Step 3, taking the next candidate video frame as the current video frame, and returning to step 2; Step 4, judging whether the current video frame group is empty, if empty, adding the current video frame to the current video frame group, and executing step 5; if not empty, judging whether the time difference between the current video frame and the first video frame in the current video frame group is less than the interval threshold, if less than, adding the current video frame to the current video frame group, and executing step 5; if not less than, executing step 6; wherein the first video frame in the current video frame group is the video frame with the latest time stamp in the current video frame group; Step 5, judging whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the first time threshold, if not greater than, taking the next candidate video frame as the current video frame, and returning to execute step 2; if greater than, executing step 7; wherein the second video frame in the current video frame group is the video frame with the earliest time stamp in the current video frame group; Step 6, judging whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold, if greater than, executing step 7; if not greater than, clearing the current video frame group, adding the current video frame to the current video frame group, taking the next candidate video frame as the current video frame, and returning to step 2; Step 7, determining the current video frame group as a divided video frame group, newly building a video frame group with empty content as the current video frame group, taking the next candidate video frame as the current video frame, and returning to step 2; Step 8, if the time difference between the first video frame and the second video frame in the current video frame group in step 5 is not greater than the first time threshold, and there is no next candidate video frame of the current video frame, judging whether the time difference between the first video frame and the second video frame in the current video frame group is greater than the second time threshold, if greater than, determining the current video frame group as a divided video frame group, and ending the division, otherwise, discarding the current video frame group, and ending the division; and determining the current divided video frame groups as the video frame groups obtained by grouping.

8. An electronic device, comprising: Comprise: a memory for storing a computer program; a processor for executing the program stored on the memory to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6.

Images