Video editing method and device, equipment, storage medium and computer program
The video editing method automatically generates expressive videos by sampling frames for attribute analysis, determining highlight templates, and segmenting content, addressing the complexity of manual editing to enhance user experience.
Patent Information
- Application Number
- CN202310666211.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-06-06
AI Technical Summary
Users face challenges in creating high-quality, expressive videos due to the complexity of video editing software, requiring significant time to learn and master, leading to difficulties in producing aesthetically pleasing and engaging content.
A video editing method that samples multiple editing materials to determine attribute information, uses a server to identify target highlight templates and frames, and generates video segments based on these attributes, allowing for automatic video creation without requiring extensive user knowledge.
Enables efficient production of aesthetically rich and expressive videos by automatically generating highlight and regular segments, improving editing efficiency and ensuring consistency with background music and video effects.
Smart Images

Figure CN119094858A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of media technology, and in particular to a video editing method, device, equipment, storage medium and computer program. Background Art
[0002] With the rapid development of multimedia and the Internet, users have higher and higher requirements for the quality of the videos they shoot. They no longer just want to simply record their lives through the videos they shoot, but want to show more beautiful and expressive videos.
[0003] In the related art, users can manually cut, merge, and reassemble the shot video materials through video editing, and then generate video films with different expressiveness.
[0004] However, video editing is a professional field. It usually takes a lot of time to learn and master how to use video editing software. Users often cannot master video editing knowledge in a short period of time, making it difficult to edit high-quality clips from video materials, resulting in users being unable to display more beautiful and expressive videos. Summary of the invention
[0005] The present application provides a video editing method, device, equipment, storage medium and computer program, which can solve the problem in the related art that it is difficult to generate expressive video films. The technical solution is as follows:
[0006] In a first aspect, a video editing method is provided, which is applied to a terminal, and the method includes: sampling multiple editing materials to obtain multiple sampling frames, the editing materials include videos or pictures, determining attribute information of each of the sampling frames, the attribute information indicating characteristics of the sampling frame, sending the attribute information of the multiple sampling frames to a server, so that the server determines a target highlight template and a highlight sampling frame corresponding to each of the multiple highlight shots included in the target highlight template, the highlight sampling frame is a sampling frame in the multiple sampling frames that matches the highlight shot, receiving the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots sent by the server, based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots, determining regular video segments and highlight video segments corresponding to the multiple editing materials, and generating video pieces obtained after the multiple editing materials are edited based on the regular video segments and the highlight video segments.
[0007] Since multiple sampling frames are obtained by sampling multiple editing materials, and the attribute information of multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can constitute the highlight segments in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matching the highlight sampling frames can also be determined, thereby generating corresponding highlight video segments and regular video segments, and finally generating video films. In other words, the video editing method provided by the embodiment of the present application can automatically generate video films without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video films are generated based on regular video segments and highlight video segments, the highlight segments are the most exciting video clips in the video films generated according to the target highlight templates and highlight sampling frames, therefore, the video films can also have rich aesthetics and expressiveness.
[0008] Optionally, the attribute information of the sampling frame includes a semantic feature vector, which indicates the image semantic information of the sampling frame. Optionally, the attribute information of the sampling frame includes a feature label of the sampling frame, and the feature label of the sampling frame includes at least one type of feature label, and different types of feature labels can characterize the features of the sampling frame in different dimensions. Optionally, the feature label can be divided into multiple types such as aesthetic labels, face labels, scene labels, object labels, camera labels, time labels and location labels. Among them, the aesthetic label can characterize the features of the sampling frame in aesthetic dimensions such as composition and color, thereby reflecting the aesthetic value of the sampling frame; the face label can characterize the attributes of the face in the sampling frame; the scene label can characterize the features of the sampling frame in the scene dimension; the object label can characterize the objects contained in the sampling frame; the camera label can characterize the camera movement of the sampling frame; the time label can indicate the shooting time of the sampling frame, and the location label can indicate the shooting location of the sampling frame. Optionally, the feature label of the sampling frame also includes at least one label output by a multi-label model. In this case, at least one label output by the multi-label model can characterize the features of the sampling frame in at least one dimension. The at least one dimension includes an object dimension, a text dimension, and the like.
[0009] By determining the attribute information of the above-mentioned sampling frames, the image features of each sampling frame can be fully known, so that the highlight video segments and regular video segments can be determined more accurately, thereby ensuring that the finished video can also have rich beauty and expressiveness.
[0010] Optionally, the method of determining regular video segments and highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots includes: determining at least one group of regular sampling frames from the sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots based on the attribute information of the multiple sampling frames and the number of the multiple highlight shots; determining regular shots corresponding to each regular sampling frame in the at least one group of regular sampling frames from the multiple editing materials; synthesizing regular shots corresponding to regular sampling frames in the same group of the at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment; determining at least one highlight video segment from the multiple editing materials based on the target highlight template and the highlight sampling frames corresponding to each of the highlight shots.
[0011] Optionally, the method of determining at least one group of regular sampling frames from sampling frames other than highlight sampling frames corresponding to the multiple highlight lenses based on the attribute information of the multiple sampling frames and the number of the multiple highlight lenses comprises: determining a plurality of candidate sampling frames from sampling frames other than highlight sampling frames corresponding to the multiple highlight lenses based on the attribute information of the multiple sampling frames, the plurality of candidate sampling frames having different image contents; clustering the plurality of candidate sampling frames based on the attribute information of the plurality of candidate sampling frames to obtain at least one cluster set, each cluster set including at least one candidate sampling frame; determining the number of regular lenses included in the video film based on the number of the at least one cluster set and the number of the multiple highlight lenses; and selecting a group of regular sampling frames from each cluster set based on the number of the regular lenses and the number of candidate sampling frames in each cluster set to obtain the at least one group of regular sampling frames.
[0012] Since the image contents of the multiple candidate sampling frames are different, it is possible to effectively avoid the existence of pictures with similar contents in the determined regular sampling frames, thereby ensuring that there is no duplication of video pictures in the video film, so that the video film has rich aesthetics and expressiveness.
[0013] Optionally, after sending the attribute information of the multiple sampling frames to the server, the method also includes: receiving target background music sent by the server, the target background music includes at least one music segment, and each of the music segments corresponds to a number of shots; determining the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the multiple highlight shots, including: determining the target music segment from the at least one music segment based on the number of the at least one cluster set, the number of the multiple highlight shots and the number of shots corresponding to each music segment; determining the number of shots corresponding to the target music segment as the number of shots of the video film; and determining the difference between the number of shots of the video film and the number of the multiple highlight shots as the number of regular shots included in the video film.
[0014] Since the number of shots of the video is determined based on the target background music, the target music segment is a fragment of the target background music. In this way, it can ensure that the final video is more consistent with the target background music, and the video is more appealing with the blessing of the target music segment.
[0015] Optionally, after determining at least one group of regular sampling frames from sampling frames other than highlight sampling frames corresponding to the multiple highlight shots based on the attribute information of the multiple sampling frames and the number of the multiple highlight shots, the method further includes: sending the at least one group of regular sampling frames to the server so that the service determines video effect information corresponding to the video film based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each of the highlight shots; receiving the video effect information sent by the server; and generating the video film obtained after video editing of the multiple editing materials based on the regular video segments and the highlight video segments, including: generating the video film based on the regular video segments, the highlight video segments and the video effect information.
[0016] Since the video effect information is determined based on the regular sampling frames and the highlight sampling frames corresponding to each of the highlight shots, the corresponding video effects can be determined for each shot in the video in a more targeted manner, thereby ensuring that the transitions of shots in the final generated video are more natural, and the video can also be more appealing with the addition of video effects.
[0017] Optionally, the method of generating a video film obtained after video editing of the multiple editing materials based on the regular video segments and the highlight video segments includes: displaying a video preview interface, the video preview interface including the regular video segments and the highlight video segments; in response to an editing operation of a target video segment, performing the editing operation on the target video segment, the target video segment being any one of the regular video segments and the highlight video segments; generating the video film based on the regular video segments and the highlight video segments obtained after editing.
[0018] That is to say, after determining the regular video segments and the highlight video segments, the user can replace the unsatisfactory highlight segments and regular segments. In other words, the present application can provide users with more choices so that the final generated video film can meet the user's preferences and requirements.
[0019] In a second aspect, a video editing method is provided, which is applied to a server, and the method comprises: receiving attribute information of each sampling frame among a plurality of sampling frames sent by a terminal, the plurality of sampling frames being obtained by sampling a plurality of editing materials, the attribute information indicating features of the sampling frames, determining a target highlight template and a highlight sampling frame corresponding to each highlight shot among a plurality of highlight shots included in the target highlight template from a plurality of candidate highlight templates based on the attribute information of the plurality of sampling frames, the highlight sampling frame being a sampling frame among the plurality of sampling frames that matches the highlight shot, sending the target highlight template and the highlight sampling frame corresponding to each of the highlight shots to the terminal, so that the terminal determines regular video segments and highlight video segments corresponding to the plurality of editing materials, and generating a video film obtained after video editing of the plurality of editing materials based on the regular video segments and the highlight video segments.
[0020] Since multiple sampling frames are obtained by sampling multiple editing materials, and the attribute information of multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can constitute the highlight segments in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matching the highlight sampling frames can also be determined, thereby generating corresponding highlight video segments and regular video segments, and finally generating video films. In other words, the video editing method provided by the embodiment of the present application can automatically generate video films without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video films are generated based on regular video segments and highlight video segments, the highlight segments are the most exciting video clips in the video films generated according to the target highlight templates and highlight sampling frames, therefore, the video films can also have rich aesthetics and expressiveness.
[0021] Optionally, each of the candidate highlight templates has a matching constraint; determining a target highlight template and a highlight sampling frame corresponding to each highlight lens in the target highlight template based on the attribute information of the multiple sampling frames from the multiple candidate highlight templates includes: determining a matching score between each sampling frame and each highlight lens in each candidate highlight template based on the attribute information of each sampling frame; determining a plurality of first highlight templates and a highlight sampling frame corresponding to each highlight lens in each of the first highlight templates from the multiple candidate highlight templates based on the matching score between each sampling frame and each highlight lens in each candidate highlight template and the matching constraint of each candidate highlight template; selecting at least one first highlight template from the multiple first highlight templates as the target highlight template, and using the highlight sampling frame corresponding to each highlight lens in the at least one first highlight template as the highlight sampling frame corresponding to each highlight lens in the target highlight template.
[0022] The highlight sampling frame corresponding to each highlight shot in the first highlight template is determined by the matching score between each sampling frame and each highlight shot in each candidate highlight template. This ensures that the finally determined highlight sampling frame is more consistent and accurate with the highlight shot, making the finished video have rich beauty and expressiveness.
[0023] Optionally, the attribute information of each of the sampling frames includes a feature label and a semantic feature vector of the sampling frame, and each highlight shot in each of the candidate highlight templates has a label requirement and a semantic feature vector; determining the matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates based on the attribute information of each of the sampling frames includes: for each of the candidate highlight templates, determining the label score between each of the sampling frames and each highlight shot in the candidate highlight template based on the feature label of each of the sampling frames and the label requirement of each highlight shot in the candidate highlight template; determining the similarity score between each of the sampling frames and each highlight shot in the candidate highlight template based on the semantic feature vector of each of the sampling frames and the semantic feature vector of each highlight shot in the candidate highlight template; determining the matching score between each of the sampling frames and each highlight shot in the candidate highlight template based on the label score and the similarity score between each of the sampling frames and each highlight shot in the candidate highlight template.
[0024] The matching score between each sampling frame and each highlight lens in the candidate highlight template is determined by the label score and the similarity score, and the label score is related to the label requirement of each highlight lens, and the similarity score is related to the semantic feature vector. In this way, the final matching score can accurately guarantee the matching of the sampling frame and the highlight lens, thereby ensuring the accuracy of the subsequent highlight sampling frame determined based on the matching score.
[0025] Optionally, based on the matching scores between each sampling frame and each highlight lens in each candidate highlight template and the matching constraints of each candidate highlight template, determining multiple first highlight templates and highlight sampling frames corresponding to each highlight lens in each of the first highlight templates from the multiple candidate highlight templates includes: for each candidate highlight template, based on the matching scores between each sampling frame and each highlight lens in the candidate highlight template and the matching constraints of the candidate highlight template, determining whether each highlight lens in the candidate highlight template has a matching sampling frame; if each highlight lens in the candidate highlight template has a matching sampling frame, and the number of sampling frames matched by the target highlight lens in the candidate highlight template is multiple, then based on the attribute information of the multiple sampling frames matched by the target highlight lens, determining the highlight sampling frame corresponding to the target highlight lens, and using the candidate highlight template as the first highlight template, and the target highlight lens is at least one highlight lens in the candidate highlight template.
[0026] Optionally, the category of each of the candidate highlight templates includes a beginning highlight template, a middle highlight template or an ending highlight template; selecting at least one first highlight template from the multiple first highlight templates as the target highlight template, and using the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template as the highlight sampling frame corresponding to each highlight shot in the target highlight template, including: screening the multiple first highlight templates to obtain multiple second highlight templates, and the highlight sampling frames corresponding to the highlight shots in every two of the second highlight templates do not overlap; if there is overlap in the multiple second highlight templates Among at least two beginning highlight templates and / or at least two ending highlight templates, the beginning highlight template and the ending highlight template with the highest priority among the multiple second highlight templates are selected; the beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the target highlight templates, and the highlight sampling frames corresponding to each highlight shot in the beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the highlight sampling frames corresponding to each highlight shot in the target highlight template.
[0027] Since there is no overlap in the highlight sampling frames corresponding to the highlight shots in every two second highlight templates, it can be ensured that there are no pictures with similar content in the highlight sections, thereby ensuring that there are no repetitive video pictures in the finished video, so that the finished video has rich beauty and expressiveness.
[0028] Optionally, the method further includes: determining target background music from multiple candidate background music based on the attribute information of the multiple sampling frames, the target background music including at least one music segment, each of the music segments corresponding to a number of shots; and sending the target background music to the terminal.
[0029] Since the finished video also includes target background music, and the target background music is determined based on the attribute information of multiple sampling frames, this can ensure that the final target background music is more consistent with multiple materials, and make the finished video more infectious with the addition of the target music segment.
[0030] Optionally, the method further comprises: receiving at least one set of regular sampling frames sent by the terminal;
[0031] Based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each highlight shot in the target highlight template, video effect information corresponding to the video film is determined; and the video effect information is sent to the terminal.
[0032] The video effect information in the finished video is determined by using at least one group of regular sampling frames and the highlight sampling frames corresponding to each highlight lens in the target highlight template. In this way, the corresponding video effect can be determined for each lens in the finished video in a more targeted manner, thereby ensuring that the transition of lenses in the finally generated finished video is more natural, and also making the finished video more appealing with the addition of video effects.
[0033] According to a third aspect, a video editing method is provided, the method comprising: sampling a plurality of editing materials to obtain a plurality of sampling frames, the editing materials comprising videos or pictures, determining attribute information of each of the sampling frames, the attribute information indicating features of the sampling frames, determining a target highlight template and a highlight sampling frame corresponding to each highlight shot in a plurality of highlight shots included in the target highlight template from a plurality of candidate highlight templates based on the attribute information of the plurality of sampling frames, the highlight sampling frame being a sampling frame in the plurality of sampling frames that matches the highlight shot, determining regular video segments and highlight video segments corresponding to the plurality of editing materials based on the plurality of editing materials, the attribute information of the plurality of sampling frames, the target highlight template and the highlight sampling frames corresponding to the plurality of highlight shots, and generating a video film obtained after video editing of the plurality of editing materials based on the regular video segments and the highlight video segments.
[0034] Optionally, each of the candidate highlight templates has a matching constraint condition;
[0035] The determining, based on the attribute information of the multiple sampling frames, from multiple candidate highlight templates, a target highlight template and a highlight sampling frame corresponding to each highlight shot of the multiple highlight shots included in the target highlight template, comprises:
[0036] Based on the attribute information of each of the sampling frames, determining a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates;
[0037] Based on the matching scores between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, determining a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each of the first highlight templates from the plurality of candidate highlight templates;
[0038] At least one first highlight template is selected from the multiple first highlight templates as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template is used as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
[0039] Optionally, the attribute information of each of the sampled frames includes a feature label and a semantic feature vector of the sampled frame, and each highlight shot in each of the candidate highlight templates has a label requirement and a semantic feature vector;
[0040] The determining, based on the attribute information of each of the sampling frames, a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates comprises:
[0041] For each of the candidate highlight templates, based on the feature labels of each of the sampling frames and the label requirements of each highlight shot in the candidate highlight template, determine a label score between each of the sampling frames and each highlight shot in the candidate highlight template;
[0042] Determine a similarity score between each of the sampling frames and each of the highlight shots in the candidate highlight template based on the semantic feature vector of each of the sampling frames and the semantic feature vector of each of the highlight shots in the candidate highlight template;
[0043] Based on the label score and the similarity score between each of the sampled frames and each of the highlight shots in the candidate highlight template, a matching score between each of the sampled frames and each of the highlight shots in the candidate highlight template is determined.
[0044] Optionally, the determining, from the plurality of candidate highlight templates, a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each of the first highlight templates based on a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates and a matching constraint condition of each of the candidate highlight templates, comprises:
[0045] For each of the candidate highlight templates, determining whether each highlight shot in the candidate highlight template has a matching sampling frame based on a matching score between each sampling frame and each highlight shot in the candidate highlight template and a matching constraint condition of the candidate highlight template;
[0046] If each highlight shot in the candidate highlight template has a matching sampling frame, and the number of sampling frames matched by the target highlight shot in the candidate highlight template is multiple, then based on the attribute information of the multiple sampling frames matched by the target highlight shot, the highlight sampling frame corresponding to the target highlight shot is determined, and the candidate highlight template is used as the first highlight template, and the target highlight shot is at least one highlight shot in the candidate highlight template.
[0047] Optionally, the category of each candidate highlight template includes a beginning highlight template, a middle highlight template or an ending highlight template;
[0048] The step of selecting at least one first highlight template from the plurality of first highlight templates as the target highlight template, and using the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template as the highlight sampling frame corresponding to each highlight shot in the target highlight template includes:
[0049] Screening the plurality of first highlight templates to obtain a plurality of second highlight templates, wherein highlight sampling frames corresponding to highlight shots in every two of the second highlight templates do not overlap;
[0050] If there are at least two beginning highlight templates and / or at least two ending highlight templates in the plurality of second highlight templates, selecting the beginning highlight template with the highest priority and the ending highlight template with the highest priority in the plurality of second highlight templates;
[0051] The beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the target highlight templates, and the highlight sampling frames corresponding to each highlight shot in the beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the highlight sampling frames corresponding to each highlight shot in the target highlight template.
[0052] Optionally, the determining the regular video segments and the highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots includes:
[0053] Based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, determining at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots;
[0054] Determine, from the plurality of editing materials, a regular shot corresponding to each regular sampling frame in the at least one group of regular sampling frames;
[0055] Combining regular shots corresponding to regular sampling frames in the same group of the at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment;
[0056] Based on the target highlight template and the highlight sampling frame corresponding to each of the highlight shots, at least one of the highlight video segments is determined from the multiple clip materials.
[0057] Optionally, the determining, based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots comprises:
[0058] Based on the attribute information of the multiple sampling frames, determining multiple candidate sampling frames from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots, wherein the image contents of the multiple candidate sampling frames are different;
[0059] Based on the attribute information of the multiple candidate sampling frames, clustering the multiple candidate sampling frames to obtain at least one cluster set, each of the cluster sets including at least one candidate sampling frame;
[0060] Determine the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots;
[0061] Based on the number of the regular shots and the number of candidate sampling frames in each of the cluster sets, a group of regular sampling frames is selected from each of the cluster sets to obtain the at least one group of regular sampling frames.
[0062] Optionally, after determining the attribute information of each of the sampling frames, the method further includes:
[0063] Based on the attribute information of the plurality of sampling frames, determining target background music from a plurality of candidate background music, the target background music comprising at least one music segment, each of the music segments corresponding to a number of shots;
[0064] The determining the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots comprises:
[0065] Determine a target music segment from the at least one music segment based on the number of the at least one cluster set, the number of the plurality of highlight shots, and the number of shots corresponding to each of the music segments;
[0066] Determine the number of shots corresponding to the target music segment as the number of shots of the completed video;
[0067] The difference between the number of shots in the completed video and the number of the plurality of highlight shots is determined as the number of regular shots included in the completed video.
[0068] Optionally, after determining at least one group of regular sampling frames from sampling frames other than highlight sampling frames corresponding to the multiple highlight shots based on the attribute information of the multiple sampling frames and the number of the multiple highlight shots, the method further comprises:
[0069] Determine video effect information corresponding to the video film based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each highlight shot in the target highlight template;
[0070] The step of generating a video film obtained by performing video editing on the plurality of clipping materials based on the regular video segment and the highlight video segment comprises:
[0071] The video film is generated based on the regular video segments, the highlight video segments and the video effect information.
[0072] Optionally, after determining the attribute information of each sampling frame in the multiple sampling frames, the method further includes:
[0073] Based on the attribute information of the multiple sampling frames, target background music is determined from multiple candidate background music, the target background music includes at least one music segment, and each of the music segments corresponds to a number of shots.
[0074] In a fourth aspect, a video editing device is provided, wherein the video editing device has the function of implementing the video editing method in the first aspect. The video editing device includes at least one module, and the at least one module is used to implement the video editing method provided in the first aspect.
[0075] In a fifth aspect, a video editing device is provided, wherein the video editing device has the function of implementing the video editing method in the second aspect. The video editing device includes at least one module, and the at least one module is used to implement the video editing method provided in the second aspect.
[0076] In a sixth aspect, a video editing device is provided, wherein the video editing device has the function of implementing the video editing method in the third aspect. The video editing device includes at least one module, and the at least one module is used to implement the video editing method provided in the third aspect.
[0077] In a seventh aspect, a terminal is provided, the terminal comprising a processor and a memory, the memory being used to store a computer program for executing the video editing method provided in the first aspect or the third aspect. The processor is configured to execute the computer program stored in the memory to implement the video editing method described in the first aspect or the third aspect.
[0078] Optionally, the terminal may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.
[0079] In an eighth aspect, a server is provided, the server comprising a processor and a memory, the memory being used to store a computer program for executing the video editing method provided in the second aspect or the third aspect. The processor is configured to execute the computer program stored in the memory to implement the video editing method described in the second aspect or the third aspect.
[0080] Optionally, the server may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.
[0081] In a ninth aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed on a terminal, the terminal executes the steps of the video editing method described in the first aspect or the third aspect.
[0082] In a tenth aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed on a server, the server executes the steps of the video editing method described in the second aspect or the third aspect.
[0083] In the eleventh aspect, a computer program product comprising instructions is provided, and when the instructions are executed on a terminal, the terminal executes the steps of the video editing method described in the first aspect or the third aspect. In other words, a computer program is provided, and when the computer program is executed on a terminal, the terminal executes the steps of the video editing method described in the first aspect or the third aspect.
[0084] In a twelfth aspect, a computer program product comprising instructions is provided, and when the instructions are executed on a server, the server executes the steps of the video editing method described in the second aspect or the third aspect. In other words, a computer program is provided, and when the computer program is executed on a server, the server executes the steps of the video editing method described in the second aspect or the third aspect.
[0085] The technical effects obtained in the above-mentioned second to twelfth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0086] Figure 1 is a schematic diagram of a video screen provided in an embodiment of the present application;
[0087] Figure 2 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;
[0088] Figure 3 is a flow chart of a video editing method provided by an embodiment of the present application;
[0089] Figure 4 It is a flowchart of determining a target highlight template and a highlight sampling frame corresponding to each highlight shot of a plurality of highlight shots included in the target highlight template provided by an embodiment of the present application;
[0090] Figure 5 is a flowchart of determining regular video segments and highlight video segments corresponding to the multiple clipping materials provided by an embodiment of the present application;
[0091] Figure 6 is a flow chart of another video editing method provided by an embodiment of the present application;
[0092] Figure 7 is a structural schematic diagram of a video editing device provided in an embodiment of the present application;
[0093] Figure 8 is a structural schematic diagram of another video editing device provided in an embodiment of the present application;
[0094] Fig. 9 is a structural schematic diagram of another video editing device provided in an embodiment of the present application;
[0095] Fig.10 It is a structural block diagram of a terminal provided in an embodiment of the present application;
[0096] Fig.11 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0097] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0098] For ease of understanding, before explaining in detail the video editing method provided in the embodiment of the present application, the nouns, application scenarios and implementation environment involved in the embodiment of the present application are first introduced.
[0099] Video film: refers to the complete video generated by cutting, merging and recombining videos, pictures and other materials.
[0100] Shot: refers to a continuous video clip, that is, the video clip between two editing points. It should be noted that a certain length of picture playback can also form a shot.
[0101] Paragraph: Also called video paragraph, a video can be composed of multiple paragraphs, each of which is composed of multiple shots that are connected in the video and closely related in editing concepts. For a certain paragraph, the multiple shots in the paragraph have obvious differences and boundaries from the adjacent shots before and after the paragraph. According to different editing requirements and purposes, paragraphs can also be divided into multiple types. For example, regular paragraphs, highlight paragraphs, etc.
[0102] It should be noted that a single shot can also constitute a paragraph, and multiple shots can also be combined into one shot through split screen or picture-in-picture, and the combined shot can also be called a paragraph. Figure 1 , Figure 1 The video includes four lenses, which can appear in the same video screen in the form of split screen and constitute a segment.
[0103] Highlight segment: also known as highlight video segment, is a typical segment type. Highlight segment is the most important and exciting video clip in the video, which can attract the audience's attention, make a deep impression on the audience, and even resonate with them emotionally. Highlight segment is usually generated by connecting at least one shot according to specific editing logic and editing techniques.
[0104] According to the placement in the video, the highlight paragraph can be further divided into the beginning highlight paragraph, the ending highlight paragraph and the middle highlight paragraph. Among them, the beginning highlight paragraph is usually located at the beginning of the video, the ending highlight paragraph is located at the end of the video, and the middle highlight paragraph is located in the middle of the video. For example, a group of fast-switching shots are used at the beginning of the video to present an overview of the entire travel process. Such a group of fast-switching shots constitutes the beginning highlight paragraph. For another example, a group of shots with ending meanings (such as backs, ending subtitles, etc.) are used at the end of the video to form the ending highlight paragraph.
[0105] Regular paragraphs: also known as regular video paragraphs, are the most basic and important paragraph type in video production. Regular paragraphs present the development of the plot and the behavior of the characters through the switching and connection of shots. Regular paragraphs usually follow a chronological order, so that the audience can clearly understand the context of the development of the story and the logic of the characters' behavior. In the embodiment of the present application, the paragraphs in the video production other than the highlight paragraphs are called regular paragraphs.
[0106] Shot size: refers to the difference in the size of the subject in the shot due to the different distances between the camera and the subject. Shot sizes can generally be divided into five types: close-up, near shot, mid shot, panoramic shot, and long shot.
[0107] With the rapid development of multimedia and the Internet, users have higher and higher requirements for the quality of the videos they shoot. They no longer just want to simply record their lives through the videos they shoot, but want to show more beautiful and expressive videos.
[0108] Currently, users can manually cut, merge, and reassemble the filmed video materials through video editing, and then generate video films with different expressiveness. Usually, users with editing experience can edit videos with smooth shot transitions, clear story lines, rich layers, and artistic atmosphere. In other words, the quality of the video film is highly dependent on the user's editing knowledge. However, video editing is a professional field. It usually takes a lot of time to learn and master how to use video editing software. Users often cannot master video editing knowledge in a short period of time, making it difficult to edit high-quality clips from video materials, which makes it impossible for users to display more beautiful and expressive videos.
[0109] Based on this, the embodiment of the present application provides a video editing method, which can automatically generate a video film based on the attribute information of multiple sampling frames. Since the multiple sampling frames are obtained by sampling multiple editing materials, and the attribute information of the multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can form the highlight paragraphs in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matched by the highlight sampling frames can also be determined, and then the corresponding highlight video paragraphs and regular video paragraphs are generated, and finally the video film is generated. In other words, the video editing method provided by the embodiment of the present application can automatically generate a video film without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video film is generated based on the regular video paragraph and the highlight video paragraph, the highlight paragraph is the most exciting video segment in the video film generated according to the target highlight template and the highlight sampling frame, therefore, the video film can also have rich aesthetics and expressiveness.
[0110] Next, the implementation environment involved in the embodiments of the present application is introduced.
[0111] Please refer to Figure 2 , Figure 2 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a terminal 10 and a server 20. The terminal 10 can be connected to the server 20 for communication. The communication connection can be a wired connection or a wireless connection, which is not limited in the embodiment of the present application.
[0112] The terminal 10 can sample multiple clip materials to obtain multiple sampling frames, and then determine the attribute information of each sampling frame. The terminal 10 can send the attribute information of the multiple sampling frames to the server 20. The server 20 can receive the attribute information of each sampling frame in the multiple sampling frames sent by the terminal 10, and based on the attribute information of the multiple sampling frames, determine the target highlight template and the highlight sampling frame corresponding to each highlight lens in the multiple highlight lenses included in the target highlight template from multiple candidate highlight templates. Then the server 20 can send the target highlight template and the highlight sampling frame corresponding to each highlight lens to the terminal 10. The terminal 10 receives the target highlight template and the highlight sampling frames corresponding to the multiple highlight lenses sent by the server 20, and based on the multiple clip materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight lenses, determines the regular video segments and the highlight video segments corresponding to the multiple clip materials, and then generates the video pieces obtained after the multiple clip materials are clipped based on the regular video segments and the highlight video segments.
[0113] That is, the terminal 10 is used to determine the attribute information of multiple sampling frames, and determine the regular video segments and highlight video segments corresponding to multiple editing materials and generate a video film. The server 20 is used to determine the target highlight template and the highlight sampling frame corresponding to each of the multiple highlight shots included in the target highlight template. The terminal 10 and the server 20 interact to jointly implement the above-mentioned video editing method. In practical applications, the above-mentioned steps of determining the attribute information of multiple sampling frames, determining the target highlight template and the highlight sampling frame corresponding to each of the multiple highlight shots included in the target highlight template, and determining the regular video segments and highlight video segments corresponding to multiple editing materials and generating a video film can also be performed by any one of the devices in the terminal 10 or the server 20. That is, the above-mentioned video editing method is implemented by any one of the devices in the terminal 10 or the server 20.
[0114] Next, taking the terminal as an example, the process of the terminal 10 implementing the above-mentioned video editing method alone is described. The terminal 10 can sample multiple editing materials to obtain multiple sampling frames, and determine the attribute information of each sampling frame, and then determine the target highlight template and the highlight sampling frame corresponding to each highlight lens in the multiple highlight lenses included in the target highlight template from multiple candidate highlight templates based on the attribute information of the multiple editing materials, the multiple sampling frames, the target highlight template, and the highlight sampling frames corresponding to the multiple highlight lenses, and then generate the video film obtained after the multiple editing materials are edited based on the regular video segment and the highlight video segment.
[0115] Among them, the terminal 10 can be any electronic product that can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as PC (Personal Computer), mobile phone, smart phone, PDA (Personal Digital Assistant), wearable device, PPC (Pocket PC), tablet computer, smart car machine, smart TV, smart speaker, etc.
[0116] Server 20 can be an independent server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms, or a cloud computing service center.
[0117] Those skilled in the art should understand that the above-mentioned terminal 10 and server 20 are only examples, and other existing or future terminals or servers that may appear, if applicable to the embodiments of the present application, should also be included in the protection scope of the embodiments of the present application and are included here by reference.
[0118] It should be noted that the application scenarios and implementation environments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that, with the evolution of the implementation environment, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0119] Next, the video editing method provided in the embodiment of the present application is explained in detail. Figure 3 This is a flowchart of a video editing method provided by an embodiment of the present application. Figure 3 , the method comprises the following steps.
[0120] Step 301: The terminal samples a plurality of clipping materials to obtain a plurality of sampling frames, wherein any one of the plurality of clipping materials includes a video or a picture.
[0121] In some embodiments, for any one of the multiple clipping materials, the terminal may sample the clipping material to obtain at least one sampling frame corresponding to the clipping material. Each of the multiple clipping materials may be sampled in the same manner to obtain at least one sampling frame corresponding to each of the multiple clipping materials, and finally obtain multiple sampling frames.
[0122] Since any one of the multiple clip materials includes a video or a picture, that is, each of the multiple clip materials can be a video or a picture. In the case of different types of clip materials, the terminal implements different methods of sampling the clip materials, which will be introduced below.
[0123] When the clipping material is a picture, the terminal may directly use the clipping material as a sampling frame corresponding to the clipping material.
[0124] In the case where the clip material is a video, the terminal may sample the clip material according to a reference sampling frequency to obtain at least one sampling frame corresponding to the clip material.
[0125] The reference sampling frequency is set in advance and can be adjusted according to different requirements in different situations. For example, the reference sampling frequency can be 1 hertz (Hz). In this case, the clip material can be sampled at a frequency of 1 Hz, that is, one video frame of the clip material is collected every 1 second as a sampling frame.
[0126] In some embodiments, before sampling the clipping material, the terminal can also obtain the clipping material.
[0127] Optionally, the terminal has a camera, and can shoot through the camera of the terminal to obtain videos or pictures, and the terminal can use the shot videos or pictures as the editing material. Of course, the terminal can also receive videos or pictures shot by other devices, and use the videos or pictures shot by the other devices as the editing material. The other devices are devices with shooting functions other than the above-mentioned terminals. In other words, the other devices can send the videos or pictures shot by themselves to the terminal. In actual applications, the terminal can also download videos or pictures from the Internet, and then use the downloaded videos or pictures as editing materials.
[0128] That is to say, the clip material can be shot by the terminal itself, or shot by other devices with shooting functions, or can be videos or pictures downloaded by the terminal from the Internet. Of course, the terminal can also obtain the clip material in other ways, which is not limited in the embodiments of the present application.
[0129] Step 302: The terminal determines attribute information of each sampling frame among a plurality of sampling frames, where the attribute information indicates a feature of the sampling frame.
[0130] For any one of the multiple sampling frames, the terminal can determine the attribute information of the sampling frame. In the same way, each of the multiple sampling frames is processed to determine the attribute information of each sampling frame.
[0131] In some embodiments, the attribute information of the sample frame includes a semantic feature vector, and the semantic feature vector indicates the image semantic information of the sample frame. In this case, the terminal can input the sample frame into a multimodal semantic understanding model to obtain the semantic feature vector of the sample frame output by the multimodal semantic understanding model.
[0132] As an example, the multimodal semantic understanding model can be a contrastive language and image pre-training (CLIP) model, a Transformer-based vision and language pre-trained language representation (vision-and-language BERT, ViLBERT) model, etc., which is not limited to the embodiments of the present application.
[0133] Optionally, the multimodal semantic understanding model can output a semantic feature vector whose data type is the target data type and whose number of dimensions is the target number of dimensions.
[0134] The target data type and the target number of dimensions are pre-set and can be adjusted according to different requirements in different situations. For example, the target data type can be a floating point type (float) and the target number of dimensions can be 512. In this case, the multimodal semantic understanding model can output a 512-dimensional floating point semantic feature vector.
[0135] In some embodiments, the attribute information of the sampling frame includes a feature tag of the sampling frame, and the feature tag of the sampling frame includes at least one type of feature tag, and different types of feature tags can characterize the features of the sampling frame in different dimensions. For example, the feature tags can be divided into multiple types such as aesthetic tags, face tags, scene tags, object tags, camera tags, time tags, and location tags. Among them, the aesthetic tag can characterize the features of the sampling frame in aesthetic dimensions such as composition and color, thereby reflecting the aesthetic value of the sampling frame; the face tag can characterize the attributes of the face in the sampling frame; the scene tag can characterize the features of the sampling frame in the scene dimension; the object tag can characterize the object contained in the sampling frame; the camera tag can characterize the camera movement of the sampling frame; the time tag can indicate the shooting time of the sampling frame, and the location tag can indicate the shooting location of the sampling frame.
[0136] Since there are many types of feature tags, the determination process of each type of feature tag listed above will be introduced below.
[0137] In the case that the feature label of the sample frame includes an aesthetic label, the terminal can determine the aesthetic score of the sample frame and further use the aesthetic score as the aesthetic label of the sample frame.
[0138] In some embodiments, the terminal can input the sample frame into an aesthetic model to obtain an aesthetic score of the sample frame output by the aesthetic model.
[0139] In the case where the feature label of the sampling frame includes a face label, the terminal can determine the attributes of the face in the sampling frame, and then use the attributes of the face as the face label of the sampling frame.
[0140] In some embodiments, the terminal can input the sample frame into a face model to obtain the attributes of the face in the sample frame output by the face model. Optionally, the face attributes include the position of the face in the sample frame, the angle of the face, and the expression of the face, etc. For example, the angle of the face includes the front face, the side face, etc., and the expression of the face includes crying face, smiling face, grimace, etc.
[0141] It should be noted that, when the sampling frame does not include a human face, the face label of the sampling frame can represent that the sampling frame does not include a human face.
[0142] In the case that the feature label of the sampling frame includes a scene label, the terminal can determine the scene of the sampling frame, and then use the scene of the sampling frame as the scene label of the sampling frame.
[0143] In some embodiments, the terminal can input the sampling frame into a scene model to obtain the scene of the sampling frame output by the scene model, and use the scene of the sampling frame as a scene label of the sampling frame.
[0144] In the case that the feature label of the sampling frame includes an object label, the terminal can determine the name of the object in the sampling frame, and then use the name of the object as the object label of the sampling frame.
[0145] In some embodiments, the terminal can input the sampling frame into an object detection model to obtain the name of the object in the sampling frame output by the object detection model, and use the name of the object as the object label of the sampling frame.
[0146] Optionally, the terminal can also determine the position of the object corresponding to the name of the object in the sampling frame, and then store the name of the object and the position of the object in the sampling frame in correspondence. In this case, the object detection model can also output the position of the object in the sampling frame.
[0147] It should be noted that the position of an object in a sampling frame can be represented by the coordinates of the object in the sampling frame, or by the orientation of the object in the sampling frame. For example, the position of an object in a sampling frame can be represented as the object being located in the area surrounded by the four coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) in the sampling frame. For another example, the position of an object in a sampling frame can be represented as the object being located in the center of the sampling frame, which is not limited in the embodiments of the present application.
[0148] When the feature label of the sampling frame includes a camera movement label, the terminal can determine the camera movement direction and camera movement speed of the sampling frame based on the sampling frame and adjacent sampling frames, and use the camera movement direction and camera movement speed of the sampling frame as the camera movement label of the sampling frame. The adjacent sampling frame corresponds to the same editing material as the sampling frame, and is the previous sampling frame or the next sampling frame adjacent to the sampling frame.
[0149] In some embodiments, the terminal can input the sampling frame and adjacent sampling frames into a camera movement model to obtain the camera movement direction and camera movement speed of the sampling frame output by the camera movement model.
[0150] As an example, the camera movement direction of the sampling frame may include upward camera movement, downward camera movement, left camera movement, right camera movement, forward camera movement, and backward camera movement, etc., which is not limited to the embodiments of the present application.
[0151] In the case that the characteristic tag of the sampling frame includes a time tag, the terminal can use the shooting time of the sampling frame as the time tag corresponding to the sampling frame.
[0152] In the case where the feature tag of the sampling frame includes a location tag, the terminal can use the shooting location of the sampling frame as the location tag corresponding to the sampling frame.
[0153] Each of the above-mentioned feature labels can only characterize the features of the sampling frame in one dimension. However, in actual applications, the feature label of the sampling frame also includes at least one label output by the multi-label model. In this case, the at least one label output by the multi-label model can characterize the features of the sampling frame in at least one dimension. The at least one dimension includes an object dimension, a text dimension, and the like. For example, if the image content of the sampling frame 1 includes a person holding a dog and standing at the door of a name called "Time Coffee Shop", then the labels that can be output by the multi-label model for the sampling frame 1 include "person", "dog", and "coffee shop", where the two labels "person" and "dog" are the features of the sampling frame in the object dimension, and "coffee shop" is the feature of the sampling frame in the text dimension. For ease of description, the at least one label output by the multi-label model will be referred to as the first label in the following.
[0154] In some embodiments, the terminal can directly input the sampling frame into the multi-label model to obtain at least one label of the sampling frame output by the multi-label model, and use the at least one label of the sampling frame as the first label of the sampling frame.
[0155] In practical applications, the characteristic labels of the sampling frames may also include other types of labels, which are not listed one by one in the embodiments of the present application.
[0156] Step 303: The terminal sends the attribute information of the multiple sampling frames to the server.
[0157] Step 304: The server receives attribute information of each sampling frame in the multiple sampling frames sent by the terminal, and based on the attribute information of the multiple sampling frames, determines a target highlight template from multiple candidate highlight templates and a highlight sampling frame corresponding to each highlight lens in the multiple highlight lenses included in the target highlight template, where the highlight sampling frame is a sampling frame in the multiple sampling frames that matches the highlight lens.
[0158] In some embodiments, each candidate highlight template has matching constraints, in which case, please refer to Figure 4 , the server can follow Figure 4 Steps 3041 - 3043 shown determine a target highlight template and a highlight sampling frame corresponding to each highlight shot in a plurality of highlight shots included in the target highlight template.
[0159] 3041: Based on the attribute information of each sampling frame, determine a matching score between each sampling frame and each highlight shot in each candidate highlight template.
[0160] In some embodiments, the attribute information of each sample frame includes a feature label and a semantic feature vector of the sample frame, and each highlight shot in each candidate highlight template has a label requirement and a semantic feature vector. In this case, the matching score between each sample frame and each highlight shot in each candidate highlight template can be determined according to the following steps (1) to (3).
[0161] (1) For each candidate highlight template, based on the feature label of each sampling frame and the label requirement of each highlight shot in the candidate highlight template, determine the label score between each sampling frame and each highlight shot in the candidate highlight template.
[0162] For any highlight shot in any one of the multiple candidate highlight templates and any one of the multiple sampling frames, the label requirement of the highlight shot indicates the feature label that the sampling frame matching the highlight shot needs to satisfy, and the label requirement includes at least one sub-label requirement. In this case, the number of sub-label requirements that the feature label of the sampling frame can satisfy is used as the label score between the sampling frame and the highlight shot. Each sampling frame and each highlight shot in the candidate highlight template are processed in the same way to obtain the label score between each sampling frame and each highlight shot in the candidate highlight template.
[0163] As an example, if the label requirements of highlight shot A in candidate highlight template 1 include sub-label requirement 1 and sub-label requirement 2, where sub-label requirement 1 is that the object labels of the sampling frame include "person" and "door", and sub-label requirement 2 is that the scene labels of the sampling frame include "medium shot" or "panorama". The object labels of sampling frame a include "person" and "door", and the scene label includes "panorama", and the object label of sampling frame b includes "person", and the scene label includes "medium shot". In this case, the feature label of sampling frame a meets two sub-label requirements of highlight shot A, namely sub-label requirement 1 and sub-label requirement 2, and the label score between sampling frame a and highlight shot A is 2. The feature label of sampling frame b meets one sub-label requirement of highlight shot A, namely sub-label requirement 2, and the label score between sampling frame b and highlight shot A is 1.
[0164] (2) Based on the semantic feature vector of each sampling frame and the semantic feature vector of each highlight shot in the candidate highlight template, determine the similarity score between each sampling frame and each highlight shot in the candidate highlight template.
[0165] In some embodiments, for any highlight shot in any one of a plurality of candidate highlight templates and any one of a plurality of sampling frames, the semantic feature vector of the highlight shot indicates the image semantic information corresponding to the sampling frame that best matches the highlight shot. In this case, the similarity score between the semantic feature vector of the sampling frame and the semantic feature vector of the highlight shot can be determined based on the semantic feature vector of the sampling frame and the semantic feature vector of the highlight shot, according to the relevant similarity measurement algorithm, or by inputting the semantic feature vector of the sampling frame and the semantic feature vector of the highlight shot into a similarity score calculation model. By processing each sampling frame and each highlight shot in the candidate highlight template in the same manner, the similarity score between each sampling frame and each highlight shot in the candidate highlight template can be obtained.
[0166] (3) Based on the label score and similarity score between each sampling frame and each highlight shot in the candidate highlight template, determine the matching score between each sampling frame and each highlight shot in the candidate highlight template.
[0167] In some embodiments, for any highlight shot in any one of the multiple candidate highlight templates and any one of the multiple sampling frames, the sum of the label score and the similarity score between the sampling frame and the highlight shot can be directly used as the matching score between the sampling frame and the highlight shot. By processing each sampling frame and each highlight shot in the candidate highlight template in the same manner, the matching score between each sampling frame and each highlight shot in the candidate highlight template can be obtained.
[0168] In some other embodiments, the server stores weights corresponding to label scores and similarity scores, respectively. In this case, for any highlight shot in any one of multiple candidate highlight templates and any one of multiple sampling frames, the server multiplies the label score between the sampling frame and the highlight shot by the weight corresponding to the label score as the target label score, multiplies the similarity score between the sampling frame and the highlight shot by the weight corresponding to the similarity score as the target similarity score, and takes the sum of the target label score and the target similarity score as the matching score between the sampling frame and the highlight shot. By processing each sampling frame and each highlight shot in the candidate highlight template in the same manner, the matching score between each sampling frame and each highlight shot in the candidate highlight template can be obtained.
[0169] In summary, for any highlight lens in any one of the multiple candidate highlight templates and any one of the multiple sampling frames, the server can determine the label score and similarity score between the highlight lens and the sampling frame, and then determine the matching score between the highlight lens and the sampling frame based on the label score and the similarity score. In practical applications, the server may also only determine one of the label score and the similarity score, and then directly use the label score or the similarity score as the matching score between the highlight lens and the sampling frame. If only the label score is determined, the attribute information of each sampling frame includes the feature label of the sampling frame, and each highlight lens in each candidate highlight template has a label requirement. If only the similarity score is determined, the attribute information of each sampling frame includes a semantic feature vector, and each highlight lens in each candidate highlight template has a semantic feature vector.
[0170] 3042: Based on the matching scores between each sampling frame and each highlight shot in each candidate highlight template and the matching constraints of each candidate highlight template, determine a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each first highlight template from a plurality of candidate highlight templates.
[0171] Next, the target highlight template and the highlight sampling frame corresponding to each highlight shot in the multiple highlight shots included in the target highlight template may be determined according to the following steps (1)-(2).
[0172] (1) For each candidate highlight template, based on the matching score between each sampling frame and each highlight shot in the candidate highlight template and the matching constraint condition of the candidate highlight template, determine whether each highlight shot in the candidate highlight template has a matching sampling frame.
[0173] In some embodiments, based on the matching score between each sampling frame and each highlight lens in the candidate highlight template and the matching constraint condition of the candidate highlight template, a brute force solution or a bipartite graph matching algorithm can be used to determine whether each highlight lens in the candidate highlight template has a matching sampling frame, that is, to determine whether the candidate highlight template has a solution. Of course, in practical applications, other algorithms can also be used to determine whether the candidate highlight template has a solution, which is not limited in the embodiments of the present application.
[0174] It should be noted that if there is at least one highlight shot in the candidate highlight template that has no matching sampling frame, it is determined that the candidate highlight template has no solution.
[0175] In some embodiments, the matching constraints of the candidate highlight templates include conventional constraints and dynamic constraints. The conventional constraints refer to the constraints that each candidate highlight template must follow in the process of determining whether there is a solution, and the conventional constraints of each candidate highlight template are the same, and the dynamic constraints are determined based on the matching score threshold corresponding to each candidate highlight template.
[0176] As an example, the conventional constraint condition includes: for any one of the multiple candidate highlight templates, the sampled frames matched by each highlight shot in the candidate highlight template are not repeated.
[0177] In some embodiments, the server further stores a matching score threshold corresponding to each candidate highlight template, and the matching score threshold includes a total score threshold and a score threshold corresponding to each shot in the corresponding candidate highlight template. In this case, for any one of the multiple candidate highlight templates, the dynamic constraint conditions of the candidate highlight template include: the matching score between any one of the highlight shots in the candidate highlight template and the matching sampling frame is not less than the score threshold corresponding to the shot, and the sum of the matching scores between each highlight shot in the candidate highlight template and the matching sampling frame is not less than the total score threshold corresponding to the candidate highlight template.
[0178] That is to say, under the constraints of the matching constraints of the above-mentioned candidate highlight templates, the sampling frames matched by each highlight shot in the solvable candidate highlight templates satisfy the above-mentioned dynamic constraints and conventional constraints, that is, the sampling frames matched by each highlight shot have image semantic information corresponding to the sampling frame that best matches the corresponding highlight shot and can meet the label requirements of the corresponding highlight shot, and there is no duplication in the sampling frames matched by each highlight shot.
[0179] For example, candidate highlight template a includes two highlight shots, namely highlight shot a and highlight shot b. If candidate highlight template a has a solution, and the sampling frame matched by highlight shot a is sampling frame A, and the sampling frame matched by highlight shot b is sampling frame B, then the matching score between sampling frame A and highlight shot a is not less than the score threshold corresponding to highlight shot a, the matching score between sampling frame B and highlight shot b is not less than the score threshold corresponding to highlight shot b, and the sum of the matching score between sampling frame A and highlight shot a and the matching score between sampling frame B and highlight shot b is not less than the total score threshold corresponding to candidate highlight template a.
[0180] In some embodiments, the category of each candidate highlight template includes a beginning highlight template, a middle highlight template or an ending highlight template, and the matching constraints of the candidate highlight templates also include a time constraint, and the time constraint of each candidate highlight template is determined based on the category of the corresponding candidate highlight template.
[0181] Optionally, the server may determine a shooting time range of the multiple clip materials, and determine a time constraint condition corresponding to each category of candidate highlight templates based on the shooting time range.
[0182] In some embodiments, the attribute information of the multiple sampling frames includes time tags, and the server can directly use the time range corresponding to the largest time tag and the smallest time tag among the multiple time tags as the shooting time range of the multiple clip materials.
[0183] Since the shooting time ranges of multiple clip materials correspond to different days, the implementation methods of determining the time constraint conditions corresponding to the candidate highlight templates of each category based on the shooting time range are different, which will be introduced separately below. It should be noted that the number of days corresponding to the shooting time range refers to the span of the shooting dates of the shooting time range. If there are two shooting dates in the shooting time range, the number of days corresponding to the shooting time range is determined to be 2 days.
[0184] If the number of days corresponding to the shooting time range of multiple clip materials is 1 day, that is, the multiple clip materials are shot within one day. In this case, since the shooting time of the clip materials is relatively concentrated, the time constraint corresponding to each category of candidate highlight templates can be ignored, that is, the time constraint corresponding to each category of candidate highlight templates is empty. Of course, in actual applications, the time range corresponding to the daytime (sunrise to sunset) in the shooting time range can also be used as the time constraint corresponding to the beginning highlight template and the middle highlight template, and the time range corresponding to after sunset can be used as the time constraint corresponding to the ending highlight template.
[0185] If the number of days corresponding to the shooting time range of multiple editing materials is 2 days, that is, the multiple editing materials were shot within 2 days. In this case, the time range corresponding to the first date in the shooting time range can be used as the time constraint condition corresponding to the beginning highlight template and the middle highlight template, and the time range corresponding to the second date in the shooting time range can be used as the time constraint condition corresponding to the ending highlight template. The first date is earlier than the second date.
[0186] If the number of days corresponding to the shooting time range of multiple editing materials is 3 days, that is, the multiple editing materials were shot within 3 days. In this case, the time range corresponding to the first date in the shooting time range can be used as the time constraint condition corresponding to the beginning highlight template, the time range corresponding to the second date in the shooting time range can be used as the time constraint condition corresponding to the middle highlight template, and the time range corresponding to the third date in the shooting time range can be used as the time constraint condition corresponding to the ending highlight template. The first date is earlier than the second date and the third date, and the second date is earlier than the third date.
[0187] If the number of days corresponding to the shooting time range of multiple editing materials is greater than 3 days, that is, the multiple editing materials were shot within at least four days. In this case, the time range corresponding to x1% of the shooting time range can be used as the time constraint condition corresponding to the beginning highlight template, the time range corresponding to x2% of the shooting time range can be used as the time constraint condition corresponding to the middle highlight template, and the time range corresponding to x3% of the shooting time range can be used as the time constraint condition corresponding to the ending highlight template. The sum of x1, x2 and x3 is not greater than 100, and there is no overlap between the time range corresponding to x1%, the time range corresponding to x2% and the time range corresponding to x3%, and the time range corresponding to x1% is earlier than the time range corresponding to x2% and the time range corresponding to x3%, and the time range corresponding to x2% is earlier than the time range corresponding to x3%.
[0188] Among them, x1, x2 and x3 are pre-set and can be adjusted according to different needs in different situations.
[0189] (2) If each highlight shot in the candidate highlight template has a matching sampling frame, and the number of sampling frames matched by the target highlight shot in the candidate highlight template is multiple, then based on the attribute information of the multiple sampling frames matched by the target highlight shot, the highlight sampling frame corresponding to the target highlight shot is determined, and the candidate highlight template is used as the first highlight template, and the target highlight shot is at least one highlight shot in the candidate highlight template.
[0190] If each highlight shot in the candidate highlight template has a matching sampling frame, and the number of sampling frames matched by the target highlight shot in the candidate highlight template is multiple, that is, the candidate highlight template has multiple solutions, and there is at least one highlight shot in the candidate highlight template that can match multiple sampling frames. In this case, it is necessary to screen the multiple solutions corresponding to the candidate highlight template.
[0191] In some embodiments, the attribute information of the multiple sampling frames matched by the target highlight shot includes an aesthetic label. In this case, for a highlight shot in the target highlight shot, the sampling frame with the largest aesthetic label among the multiple sampling frames matched by the highlight shot is used as the highlight sampling frame corresponding to the highlight shot, and each highlight shot in the target highlight shot is processed in the same manner to obtain a highlight sampling frame corresponding to each highlight shot in the target highlight shot.
[0192] In other embodiments, for a highlight shot in the target highlight shot, the matching score between each sampling frame and the highlight shot can be determined based on the attribute information of multiple sampling frames matched by the highlight shot, and the sampling frame with the largest matching score is used as the highlight sampling frame corresponding to the highlight shot, and each highlight shot in the target highlight shot is processed in the same manner to obtain the highlight sampling frame corresponding to each highlight shot in the target highlight shot. Of course, in practical applications, the highlight sampling frame corresponding to the highlight shot can also be determined by other methods, for example, any one of the multiple sampling frames matched by the highlight shot is used as the highlight sampling frame corresponding to the highlight shot, and the embodiments of the present application are not limited to this.
[0193] Among them, the implementation process of determining the matching score between each sampling frame and the highlight lens based on the attribute information of multiple sampling frames matched by the highlight lens is similar to the implementation process of determining the matching score between each sampling frame and each highlight lens in each candidate highlight template based on the attribute information of each sampling frame in the above text. For details, please refer to the relevant content in the above text, which will not be repeated here.
[0194] 3043: Select at least one first highlight template from the multiple first highlight templates as a target highlight template, and use the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
[0195] In some embodiments, the category of each candidate highlight template includes a beginning highlight template, a middle highlight template or an ending highlight template. In this case, multiple first highlight templates can be screened to obtain multiple second highlight templates, and the highlight sampling frames corresponding to the highlight shots in every two second highlight templates do not overlap. If there are at least two beginning highlight templates and / or at least two ending highlight templates in the multiple second highlight templates, the beginning highlight template with the highest priority and the ending highlight template with the highest priority in the multiple second highlight templates are selected, and the beginning highlight template and the ending highlight template with the highest priority, as well as the middle highlight template in the multiple second highlight templates are determined as the target highlight template, and the highlight sampling frames corresponding to each highlight shot in the beginning highlight template and the ending highlight template with the highest priority, as well as the middle highlight template in the multiple second highlight templates are determined as the highlight sampling frames corresponding to each highlight shot in the target highlight template.
[0196] Although there is no duplication of highlight sampling frames corresponding to different highlight shots in the same first highlight template in the multiple first highlight templates determined above, there may be repeated highlight sampling frames between different first highlight templates. In order to avoid repeated sampling frames in the finally generated highlight video segment, it is necessary to screen the multiple first highlight templates to obtain multiple second highlight templates.
[0197] In some embodiments, the server stores priorities corresponding to a plurality of candidate highlight templates, that is, the server stores priorities corresponding to a plurality of first highlight templates. In this case, the specific implementation process of screening a plurality of first highlight templates to obtain a plurality of second highlight templates includes: counting the number of occurrences of highlight sampling frames corresponding to each highlight lens in the plurality of first highlight templates, and if there is no highlight sampling frame with a number of occurrences greater than 2 in the statistical result, the plurality of first highlight templates are used as a plurality of second highlight templates. If there is a highlight sampling frame with a number of occurrences greater than 2 in the statistical result, the first highlight template with the lowest priority in the first highlight template corresponding to the highlight sampling frame with a number of occurrences greater than 2 is deleted from the plurality of first highlight templates, and the step of counting the number of occurrences of the highlight sampling frame corresponding to each highlight lens in the plurality of first highlight templates is returned to. At this time, the plurality of second highlight templates are the plurality of first highlight templates after deletion.
[0198] If there is no highlight sampling frame with a number of occurrences greater than 2 in the statistical result, it means that there are no repeated highlight sampling frames between the multiple first highlight templates. Therefore, the multiple first highlight templates can be used as multiple second highlight templates. If there is a highlight sampling frame with a number of occurrences greater than 2 in the statistical result, it means that there are repeated highlight sampling frames between the multiple first highlight templates. Therefore. The first highlight template with the lowest priority in the first highlight templates corresponding to the highlight sampling frame with a number of occurrences greater than 2 can be deleted from the multiple first highlight templates, and the step of counting the number of occurrences of the highlight sampling frame corresponding to each highlight lens in the multiple first highlight templates is returned to the above step.
[0199] Since there is usually only one beginning highlight template and / or one ending highlight template in a video, and there may be multiple beginning highlight templates and / or at least two ending highlight templates among the multiple second highlight templates determined above, in order to avoid repeated highlight segments in the finally generated highlight video segment, it is necessary to screen the multiple second highlight templates to obtain the target highlight template.
[0200] In some embodiments, the server stores priorities corresponding to a plurality of candidate highlight templates. In this case, if there are at least two beginning highlight templates and / or at least two ending highlight templates among the plurality of second highlight templates, the beginning highlight template with the highest priority and the ending highlight template with the highest priority among the plurality of second highlight templates are selected, and the beginning highlight template and the ending highlight template with the highest priority, as well as the middle highlight template among the plurality of second highlight templates are determined as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the beginning highlight template and the ending highlight template with the highest priority, as well as the middle highlight template among the plurality of second highlight templates are determined as the highlight sampling frame corresponding to each highlight shot in the target highlight template. If there are not at least two beginning highlight templates and / or at least two ending highlight templates among the plurality of second highlight templates, the plurality of second highlight templates are directly determined as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the plurality of second highlight templates is determined as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
[0201] In summary, when there are at least two beginning highlight templates and / or at least two ending highlight templates among multiple second highlight templates, the beginning highlight template with a high priority and at least two ending highlight templates can be used as target highlight templates. In practical applications, when there are at least two beginning highlight templates and / or at least two ending highlight templates among multiple second highlight templates, any one of the at least two beginning highlight templates and any one of the at least two ending highlight templates can be selected, and the arbitrarily selected beginning highlight template and ending highlight template can be used as target highlight templates, and this embodiment of the present application does not limit this.
[0202] Step 305: The server sends the target highlight template and the highlight sampling frame corresponding to each highlight shot in the multiple highlight shots included in the target highlight template to the terminal.
[0203] Step 306: The terminal receives the target highlight template sent by the server and the highlight sampling frames corresponding to the multiple highlight shots included in the target highlight template, and determines the regular video segments and highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots included in the target highlight template.
[0204] In some embodiments, please refer to Figure 5 , the terminal can Figure 5 Steps 3061-3064 shown determine the regular video segments and highlight video segments corresponding to the multiple clip materials.
[0205] 3061: Based on the attribute information of the multiple sampling frames and the number of the multiple highlight shots, determine at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots.
[0206] In some embodiments, the terminal may determine at least one set of regular sampling frames according to the following steps (1) to (4).
[0207] (1) Based on the attribute information of the multiple sampling frames, multiple candidate sampling frames are determined from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots, and the image contents of the multiple candidate sampling frames are different.
[0208] In some embodiments, the attribute information of each sampling frame includes a semantic feature vector. Based on the semantic feature vectors of the sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots in the multiple sampling frames, the similarity scores between every two sampling frames in the sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots are determined to obtain multiple first scores. Based on the multiple first scores, multiple candidate sampling frames are determined from the sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots.
[0209] For the convenience of description, the sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots in the multiple sampling frames are subsequently referred to as remaining sampling frames.
[0210] The specific implementation process of determining the similarity score between every two sampling frames in the remaining sampling frames includes: for any two sampling frames in the remaining sampling frames, based on the semantic feature vectors of the two sampling frames, determining the similarity score between the two sampling frames. Every two sampling frames in the remaining sampling frames are processed in the same way to obtain the similarity score between every two sampling frames in the remaining sampling frames, that is, multiple first scores.
[0211] The specific implementation method of determining the similarity scores between the two sampling frames based on the semantic feature vectors of the two sampling frames is similar to the above-mentioned implementation method of determining the similarity scores between each sampling frame and each highlight shot in the candidate highlight template based on the semantic feature vector of each sampling frame and the semantic feature vector of each highlight shot in the candidate highlight template. For details, please refer to the relevant content above, which will not be repeated here.
[0212] Based on the multiple first scores, the specific implementation process of determining multiple candidate sampling frames from the remaining sampling frames includes: if there is at least one first score greater than the first similarity threshold among the multiple first scores, then the first target sampling frame in the two sampling frames corresponding to the first target score in the at least one first score in the remaining sampling frames is deleted, and the above step of determining the similarity score between every two sampling frames in the remaining sampling frames is returned. At this time, the remaining sampling frame is the remaining sampling frame after deletion, the first target score is any one of the at least one first score, or the first score with the largest value in the at least one first score, the first target sampling frame is any one of the two sampling frames corresponding to the first target score, or the sampling frame with the smallest aesthetic label in the two sampling frames corresponding to the first target score. If there is no first score greater than the first similarity among the multiple first scores, the remaining sampling frame is used as the multiple candidate sampling frames.
[0213] If at least one first score greater than the first similarity threshold exists among the multiple first scores, it means that there are sampling frames with similar image contents among the remaining sampling frames, and therefore, the first target sampling frame among the two sampling frames corresponding to the first target score in the at least one first score in the remaining sampling frames can be deleted. If there is no first score greater than the first similarity among the multiple first scores, it means that there are no sampling frames with similar image contents among the remaining sampling frames, and therefore, the remaining sampling frames can be used as the multiple candidate sampling frames.
[0214] The first similarity threshold is set in advance and can be adjusted according to different requirements in different situations.
[0215] To sum up, the terminal can delete the sampling frames with smaller image content differences from the remaining sampling frames based on the similarity scores of the two sampling frames, and use the remaining sampling frames after deletion as the multiple candidate sampling frames. In actual applications, the terminal can also delete the remaining sampling frames based on the feature labels of the sampling frames.
[0216] In some embodiments, the feature label of the sampling frame includes a location label. In this case, if there are at least two sampling frames with the same location label in the remaining sampling frames, the sampling frames except the second sampling frame in the at least two sampling frames with the same location label in the remaining sampling frames are deleted to obtain multiple candidate sampling frames, and the second sampling frame is any one of the at least two sampling frames with the same location label, or is the sampling frame with the largest aesthetic label in the at least two sampling frames with the same location label. If there are no sampling frames with the same location label in the remaining sampling frames, the remaining sampling frames are used as the multiple candidate sampling frames.
[0217] In some embodiments, the characteristic label of the sampling frame includes a time label. In this case, the difference of the time label between every two sampling frames in the remaining sampling frames is determined to obtain multiple time difference values. If there is at least one time difference value less than the time difference threshold value in the multiple time difference values, the second target sampling frame in the two sampling frames corresponding to the target time difference value in the at least one time difference value in the remaining sampling frames is deleted, and the step of determining the difference of the time label between every two sampling frames in the remaining sampling frames is returned to the above step. At this time, the remaining sampling frame is the remaining sampling frame after deletion, the target time difference value is any one of the at least one time difference value, or the smallest time difference value in the at least one time difference value, and the second target sampling frame is any one of the two sampling frames corresponding to the target time difference value, or the sampling frame with the smallest aesthetic label in the two sampling frames corresponding to the target time difference value. If there is no time difference value less than the time difference threshold value in the multiple time difference values, the remaining sampling frame is used as the multiple candidate sampling frames.
[0218] In some embodiments, the feature label of the sampling frame includes at least one label output by the multi-label model. In this case, the number of identical labels in the first labels of every two sampling frames in the remaining sampling frames is determined to obtain multiple label numbers. If there is at least one label number greater than the number threshold in the multiple label numbers, the target third sampling frame in the two sampling frames corresponding to the difference in the number of target labels in the at least one label number in the remaining sampling frames is deleted, and the step of determining the number of identical labels in the first labels of every two sampling frames in the remaining sampling frames is returned to the above step. At this time, the remaining sampling frame is the remaining sampling frame after deletion, the target label number is any one of the at least one label number, or the largest label number in the at least one label number, the target third sampling frame is any one of the two sampling frames corresponding to the target label number, or the sampling frame with the smallest aesthetic label in the two sampling frames corresponding to the target label number. If there is no label number greater than the number threshold in the multiple label numbers, the remaining sampling frame is used as the multiple candidate sampling frames.
[0219] Since some of the multiple candidate sampling frames may have unstable lens movement, after determining the multiple candidate sampling frames, the multiple candidate sampling frames can be further screened to obtain multiple screened candidate sampling frames, and the lens movement of the multiple screened candidate sampling frames is stable.
[0220] The specific implementation process of further screening the multiple candidate sampling frames to obtain the multiple candidate sampling frames after screening includes: determining the target shot duration, for any candidate sampling frame among the multiple candidate sampling frames, clipping a video clip with a time length equal to the target shot duration from the multiple clipping materials, the video clip containing the candidate sampling frame, taking the video clip containing the candidate sampling frame as the first shot corresponding to the candidate sampling frame, if the camera speed of the first shot is greater than the camera speed threshold and / or the camera variance is greater than the variance threshold, then deleting the candidate sampling frame from the multiple candidate sampling frames, otherwise, not deleting the candidate sampling frame from the multiple candidate sampling frames, the target shot duration refers to the duration of each shot in the video film. Each candidate sampling frame among the multiple candidate sampling frames is processed in the same way to obtain the multiple candidate sampling frames after screening.
[0221] In some embodiments, the terminal stores the target shot duration, in which case the terminal can directly obtain the target shot duration. In other embodiments, the terminal can obtain the target background music, which includes the shot duration corresponding to the target background music, and the terminal uses the shot duration corresponding to the target background music as the target shot duration.
[0222] Optionally, after the terminal sends the attribute information of the multiple sampling frames to the server, the server can also determine the target background music from multiple candidate background music based on the attribute information of the multiple sampling frames, and the target background music includes the shot duration corresponding to the target background music, and then send the target background music to the terminal.
[0223] The implementation process of determining the target background music from multiple candidate background music based on the attribute information of the multiple sampling frames includes: fusing the attribute information of the multiple sampling frames to obtain fused first information, determining the candidate background music corresponding to the fused first information from multiple candidate background music based on the fused first information, and using the candidate background music corresponding to the fused first information as the target background music. The fused first information can characterize the common features of the multiple sampling frames.
[0224] In some embodiments, the attribute information of the multiple sampling frames includes a first label and / or a semantic feature vector, and the fused information includes a fused first label and / or a fused semantic feature vector. In this case, the union of the first labels of the multiple sampling frames is used as the fused first label, and the average value of the semantic feature vectors of the multiple sampling frames is used as the fused semantic feature vector.
[0225] As an example, the matching recommendation algorithm can be used, or the fused information can be input into the matching recommendation model to obtain the candidate background music corresponding to the fused information. Of course, in practical applications, the candidate background music corresponding to the fused information can also be determined by other methods, which is not limited in the embodiments of the present application.
[0226] In some embodiments, the average value of the camera movement speed of at least one sampling frame included in the first shot is used as the camera movement speed of the first shot, and the variance of the camera movement values corresponding to at least one sampling frame included in the first shot is used as the camera movement variance of the shot, and the camera movement value corresponding to each sampling frame can indicate the camera movement direction of the sampling frame.
[0227] Optionally, the camera movement direction of the sampling frame may include upward camera movement, downward camera movement, left camera movement, right camera movement, forward camera movement and backward camera movement. In this case, the camera movement value corresponding to the sampling frame of upward camera movement and the camera movement value corresponding to the sampling frame of downward camera movement are opposite to each other, the camera movement value corresponding to the sampling frame of left camera movement and the camera movement value corresponding to the sampling frame of right camera movement are opposite to each other, and the camera movement value corresponding to the sampling frame of forward camera movement and the camera movement value corresponding to the sampling frame of backward camera movement are opposite to each other.
[0228] Among them, the time difference threshold, quantity threshold, camera speed threshold and variance threshold are all set in advance, and can be adjusted according to different needs in different situations.
[0229] It should be noted that, if the multiple candidate sampling frames are further screened to obtain multiple screened candidate sampling frames, the multiple candidate sampling frames in the subsequent steps are the multiple screened candidate sampling frames.
[0230] (2) Clustering the multiple candidate sampling frames based on the attribute information of the multiple candidate sampling frames to obtain at least one cluster set, each cluster set including at least one candidate sampling frame.
[0231] In some embodiments, the plurality of candidate sampling frames may be clustered based on the semantic feature vectors of the plurality of candidate sampling frames according to a relevant clustering algorithm to obtain at least one cluster set.
[0232] In some other embodiments, the plurality of candidate sampling frames may be clustered based on the semantic feature vectors and clustering constraints of the plurality of candidate sampling frames according to a relevant clustering algorithm to obtain at least one cluster set.
[0233] For example, the clustering constraint condition includes: for each cluster set, the time label and / or location label of the candidate sampling frames in the cluster set are the same, and / or, the first labels of the candidate sampling frames in the cluster set have an intersection, and / or, the number of identical labels in the first labels of the candidate sampling frames in the cluster set is greater than the cluster label threshold. Of course, in practical applications, the clustering constraint condition may also include other content, which is not limited in the embodiments of the present application.
[0234] Among them, the cluster label threshold is set in advance and can be adjusted according to different needs in different situations.
[0235] (3) Based on the number of the at least one cluster set and the number of the plurality of highlight shots, determine the number of regular shots included in the video.
[0236] There are multiple implementation methods for determining the number of regular shots included in a finished video. Two of the implementation methods are introduced below.
[0237] The first implementation method is to determine the number of shots in the video based on the number of the at least one cluster set and the number of the multiple highlight shots, and determine the difference between the number of shots in the video and the number of the multiple highlight shots as the number of regular shots included in the video.
[0238] Optionally, a value obtained by multiplying the number of the at least one cluster set by the number of reference shots and the sum of the numbers of the plurality of highlight shots is used as the number of shots in the completed video.
[0239] In a second implementation, the terminal is able to obtain target background music, which includes at least one music segment, and each music segment corresponds to a number of shots. In this case, based on the number of the at least one cluster set, the number of multiple highlight shots, and the number of shots corresponding to each music segment, the target music segment is determined from the at least one music segment, the number of shots corresponding to the target music segment is determined as the number of shots of the video, and the difference between the number of shots of the video and the number of multiple highlight shots is determined as the number of regular shots included in the video.
[0240] Optionally, after the terminal sends the attribute information of the multiple sampling frames to the server, the server can also determine the target background music from multiple candidate background music based on the attribute information of the multiple sampling frames, and the target background music includes at least one music segment, each music segment corresponds to a number of shots, and then send the target background music to the terminal.
[0241] The implementation process of determining the target background music from multiple candidate background music based on the attribute information of the multiple sampling frames has been described above. For details, please refer to the relevant content above, which will not be repeated here.
[0242] Based on the number of the at least one cluster set, the number of multiple highlight shots, and the number of shots corresponding to each music segment, a specific implementation process of determining a target music segment from the at least one music segment includes: taking the value of the sum of the value of the number of the at least one cluster set multiplied by the reference shot number and the number of the multiple highlight shots as the first shot number, and taking the music segment corresponding to the number of shots in the at least one shot number that is greater than the first shot number and has the smallest difference with the first shot number as the target music segment.
[0243] (4) Based on the number of the regular shots and the number of candidate sampling frames in each cluster set, a group of regular sampling frames is selected from each cluster set to obtain at least one group of regular sampling frames.
[0244] Based on the number of regular shots and the number of candidate sampling frames in each cluster set, the number of regular sampling frames in each cluster set is determined; based on the number of regular sampling frames in each cluster set, a group of regular sampling frames is selected from each cluster set to obtain at least one group of regular sampling frames.
[0245] For any one of the at least one cluster set, a value obtained by dividing the number of regular shots by the number of candidate sampling frames is multiplied by the number of candidate sampling frames in the cluster set to obtain a first value, and the number of regular sampling frames in the cluster set is determined based on the first value. Each cluster set in the at least one cluster set is processed in the same manner to obtain the number of regular sampling frames in each cluster set.
[0246] In some embodiments, the first value is rounded and the rounded value is used as the number of regular sampling frames in the cluster set. In other embodiments, the multiple reference value with the smallest difference from the first value among multiple multiple reference values is used as the number of regular sampling frames in the cluster set, and the multiple reference value is a positive integer multiple of the reference value.
[0247] Based on the number of regular sampling frames in each cluster set, there are multiple implementations of selecting a group of regular sampling frames from each cluster set, two of which are introduced below.
[0248] A first implementation method: for any one of the at least one cluster set, randomly select candidate sampling frames from the cluster set, the number of which is the number of regular sampling frames in the cluster set, to obtain a group of regular sampling frames corresponding to the cluster set. Each cluster set in the at least one cluster set is processed in the same way to obtain at least one group of regular sampling frames.
[0249] Second implementation: for any one of the at least one cluster set, the candidate sampling frames in the cluster set are sorted in descending order of aesthetic labels, and the first H candidate sampling frames are used as a group of regular sampling frames corresponding to the cluster set. Each cluster set in the at least one cluster set is processed in the same way to obtain at least one group of regular sampling frames, where H is the number of regular sampling frames in the cluster set.
[0250] As an example, if the number of regular sampling frames in cluster set 1 is 4, the candidate sampling frames in cluster set 1 are sorted in descending order of aesthetic labels, and the first 4 candidate sampling frames are used as a group of regular sampling frames corresponding to cluster set 1.
[0251] 3062: Determine, from the multiple clip materials, a regular shot corresponding to each regular sampling frame in at least one group of regular sampling frames.
[0252] For any regular sampling frame in any one of the at least one group of regular sampling frames, a video clip with a time length equal to the time length of the target shot is clipped from the multiple clipping materials, and the video clip contains the regular sampling frame, and the video clip containing the regular sampling frame is used as the regular shot corresponding to the regular sampling frame. Each regular sampling frame in each group of the at least one group of regular sampling frames is processed in the same manner, and finally a regular shot corresponding to each regular sampling frame in the at least one group of regular sampling frames can be obtained.
[0253] 3063: Combining regular shots corresponding to regular sampling frames in the same group of the at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment.
[0254] 3064: Determine at least one highlight video segment from the multiple clip materials based on the target highlight template and the highlight sampling frame corresponding to each highlight shot.
[0255] In some embodiments, for any one of the target highlight templates, the target highlight template includes a shot duration coefficient corresponding to each highlight shot in the target highlight template, and based on the shot duration ratio corresponding to each highlight shot and the highlight sampling frame corresponding to each highlight shot, a highlight video segment corresponding to the target highlight template is determined from the multiple clip materials. Each target highlight template in the target highlight template is processed in the same manner, and finally at least one highlight video segment can be obtained.
[0256] For any highlight shot in the target highlight template, the duration coefficient corresponding to the highlight shot is multiplied by the target shot duration as the shot duration corresponding to the highlight shot, and a video segment having a duration corresponding to the shot duration corresponding to the highlight shot and containing a highlight sampling frame corresponding to the highlight shot is edited from the multiple editing materials, and the video segment is used as the video clip corresponding to the highlight shot. Each highlight shot in the target highlight template is processed in the same way, and finally a video clip corresponding to each highlight shot can be obtained.
[0257] In some embodiments, the target highlight template includes the sequence of video clips corresponding to each highlight shot in the target highlight template. According to the sequence of video clips corresponding to the highlight shots, the video clips corresponding to each highlight shot are merged to obtain the highlight video segment corresponding to the target highlight template.
[0258] Step 307: The terminal generates a video film obtained after video editing of the multiple editing materials based on the regular video segments and the highlight video segments corresponding to the multiple editing materials.
[0259] Based on the regular video segments and the highlight video segments corresponding to the multiple clipping materials, there are multiple implementation methods for generating a video film obtained after the multiple clipping materials are clipped. Two of the implementation methods are introduced below.
[0260] The first implementation method: the terminal directly determines the priorities of the regular video segments and the highlight video segments based on the regular video segments and the highlight video segments corresponding to multiple editing materials, and merges the regular video segments and the highlight video segments in order from high to low priority to obtain a video film. The priorities of the regular video segments and the highlight video segments indicate the sequential positions of the regular video segments and the highlight video segments in the video film.
[0261] For any highlight video segment in the highlight video segment, if the target highlight template category corresponding to the highlight video segment is the beginning highlight segment, the priority of the highlight video segment is determined to be the highest. If the target highlight template category corresponding to the highlight video segment is the ending highlight segment, the priority of the highlight video segment is determined to be the lowest. If the target highlight template category corresponding to the highlight video segment is the middle highlight segment, the average shooting time of the highlight video segment is determined, and based on the average shooting time of the highlight video segment and the average shooting time of each regular video segment in the regular video segment, the priority of the highlight video segment and each regular video segment is determined.
[0262] The average value of the time tags of each sampling frame in the highlight video segment is used as the average shooting time of the highlight video segment. The implementation method of determining the average shooting time of each regular video segment is the same as that of determining the average shooting time of the highlight video segment, which will not be repeated here.
[0263] The average shooting time of the highlight video segment and the average shooting time of each regular video segment in the regular video segment are sorted in order from early to late, and the sorting result is used as the priority result of the highlight video segment and each regular video segment, wherein the video segment with an earlier average shooting time has a higher priority than the video segment with a later average shooting time.
[0264] In some embodiments, the terminal can generate the video clip based on the regular video segments corresponding to the multiple clipping materials, the highlight video segments corresponding to the multiple clipping materials, and the target music segments.
[0265] In this case, the target music includes at least one music segment, and each music segment corresponds to a melody turning point. Based on the regular video segments and the highlight video segments corresponding to the multiple clip materials, the priorities of the regular video segments and the highlight video segments are determined, and the regular video segments and the highlight video segments are merged in order of priority from high to low, and the target music segment is used as background music to obtain a video film, and the priorities of the regular video segments and the highlight video segments indicate the sequential positions of the regular video segments and the highlight video segments in the video film.
[0266] For any highlight video segment in the highlight video segment, if the target highlight template category corresponding to the highlight video segment is the beginning highlight segment, the priority of the highlight video segment is determined to be the highest. If the target highlight template category corresponding to the highlight video segment is the ending highlight segment, the priority of the highlight video segment is determined to be the lowest. If the target highlight template category corresponding to the highlight video segment is the middle highlight segment, the melody turning point corresponding to the target music segment is determined as the average shooting time of the highlight video segment, and the priority of the highlight video segment and each regular video segment is determined based on the average shooting time of the highlight video segment and the average shooting time of each regular video segment in the regular video segment.
[0267] In some embodiments, the terminal can also generate the video film based on the video effect information corresponding to the regular video segments, the highlight video segments and the video film.
[0268] In this case, after the terminal determines at least one group of regular sampling frames, the at least one group of regular sampling frames can also be sent to the server, and then the server receives the at least one group of regular sampling frames sent by the terminal, and based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each highlight shot in the target highlight template, determines the video effect information corresponding to the video film from multiple candidate effect information, and sends the video effect information to the terminal.
[0269] In some embodiments, the candidate effect information includes candidate titles, candidate filters, candidate transitions, and candidate sound effects. The video effect information corresponding to the video film includes a target title, a target filter, a target transition, and a target sound effect. Among them, the target transition includes the transition of the highlight shot and the transition of the regular shot, the target shot refers to the shot corresponding to the regular sampling frame, and the target sound effect includes the sound effect corresponding to the highlight paragraph and the sound effect corresponding to the regular paragraph.
[0270] The server determines the fused second information based on the attribute information of the at least one group of regular sampling frames and the attribute information of the highlight sampling frames corresponding to each highlight shot in the target highlight template, and based on the fused second information, determines the candidate title and candidate filter corresponding to the fused second information from multiple candidate titles and candidate filters, and uses the candidate title and candidate filter corresponding to the fused second information as the target title and target filter.
[0271] The implementation process of determining the fused second information and determining the candidate title and candidate filter corresponding to the fused second information from multiple candidate titles and candidate filters based on the fused second information is similar to the implementation process of determining the fused first information and determining the candidate background music corresponding to the fused first information from multiple candidate background music based on the fused first information. For details, please refer to the relevant content above and will not be repeated here.
[0272] In some embodiments, each candidate transition has a corresponding priority. In this case, for any one of the at least one group of regular sampling frames, based on the attribute information of the regular sampling frame, the matching score between the regular sampling frame and each of the multiple candidate transitions is determined to obtain multiple transition matching scores. The candidate transition with the highest priority among the multiple transition matching scores that is greater than the transition matching score threshold is used as the transition of the regular shot corresponding to the regular sampling frame. Each regular sampling frame can be processed in the same manner, and the transition of the regular shot corresponding to each regular sampling frame can be determined.
[0273] Among them, the transition matching score threshold is set in advance and can be adjusted according to different needs in different situations.
[0274] Based on the attribute information of the regular sampling frame, determining the matching score between the regular sampling frame and each candidate transition in multiple candidate transitions is similar to the implementation process of determining the matching score between each sampling frame and each highlight shot in each candidate highlight template based on the attribute information of each sampling frame. For detailed process, please refer to the relevant content above, which will not be repeated here.
[0275] In some embodiments, each target highlight template includes a transition corresponding to each highlight shot in the target highlight template. In this case, the server can directly determine the transition corresponding to each highlight shot. Of course, the transition corresponding to the highlight shot can also be determined in the manner of determining the transition of a conventional shot as described above, and this embodiment of the application does not limit this.
[0276] In some embodiments, each tag in the first tag has a corresponding priority, and each tag corresponds to a sound effect. In this case, for any regular video segment in the regular video segment, the server can use the sound effect corresponding to the tag with the highest priority in the intersection of the first tags of the sampled frames in the regular video segment as the sound effect corresponding to the regular segment. By processing each regular video segment in the regular video segment in the same manner, the sound effect corresponding to each regular video segment can be determined.
[0277] In some embodiments, each target highlight template includes the sound effect of the target highlight template. In this case, for any highlight video segment in the highlight video segment, the server can determine the sound effect of the target highlight template corresponding to the highlight video segment as the sound effect corresponding to the highlight video segment. Of course, the sound effect corresponding to the highlight video segment can also be determined in the manner of determining the sound effect corresponding to the conventional video segment, which is not limited in the embodiments of the present application.
[0278] The second implementation method: the terminal displays a video preview interface, which includes the regular video segment and the highlight video segment. In response to an editing operation on a target video segment, an editing operation is performed on the target video segment. The target video segment is any one of the regular video segment and the highlight video segment. Based on the regular video segment and the highlight video segment obtained after editing, the video film is generated.
[0279] Since the user may not be satisfied with the automatically generated regular video segments and highlight video segments, the user can trigger an editing operation of the target video segment on the video preview interface. The terminal responds to the editing operation of the target video segment by displaying at least one replacement segment corresponding to the target video segment. The user selects a replacement segment from the at least one replacement segment corresponding to the target video segment and triggers a replacement confirmation operation to instruct the terminal to replace the target video segment with the replacement segment selected by the user.
[0280] If the target video segment is a regular video segment, then determine the cluster set corresponding to the target video segment, select at least one group of sampling frames with the same number of shots as the target video segment from the cluster set except the candidate sampling frames corresponding to the target video segment, determine the shots corresponding to each sampling frame in the at least one group of sampling frames from multiple clip materials based on the at least one group of sampling frames, synthesize the shots corresponding to the sampling frames in the same group in the at least one group of sampling frames into a replacement segment, so as to obtain at least one replacement segment corresponding to the target video segment. If the target video segment is a highlight video segment, then determine the category of the target highlight template corresponding to the target video segment, determine at least one other first highlight template of the same category as the target highlight template from multiple first highlight templates, determine at least one other second highlight template of the same category as the target highlight template from multiple second highlight templates, and determine at least one replacement segment corresponding to the target video segment from multiple clip materials based on the at least one first highlight template and / or at least one second highlight template.
[0281] The implementation method of determining at least one replacement segment corresponding to the target video segment from multiple editing materials based on the at least one first highlight template and / or at least one second highlight template is similar to the implementation method of determining at least one highlight video segment from the multiple editing materials based on the target highlight template and the highlight sampling frames corresponding to each highlight shot. For the detailed process, please refer to the relevant content above, which will not be repeated here.
[0282] Based on the regular video segments and highlight video segments obtained after editing, the implementation process of generating the video film is similar to the first implementation method mentioned above. For the detailed implementation process, please refer to the relevant content above, which will not be repeated here.
[0283] In actual applications, the user can also modify the duration of the target shot, that is, modify the duration of the target shot to a positive integer multiple of the duration of the target shot. In this case, the duration of each shot in the regular video segment and the highlight video segment corresponding to the multiple clip materials is the modified duration of the target shot. Since the duration of each shot is usually greater than the target shot duration before the modification after the modification, the duration of the video clip when it is played is a positive integer multiple of the target music segment. In this case, the audio of the target background music with a duration of a positive integer multiple of the target music segment after the start time of the target music segment is used as the background music of the video clip.
[0284] Since the multiple sampling frames are obtained by sampling the multiple editing materials, and the attribute information of the multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can form the highlight paragraphs in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matched by the highlight sampling frames can also be determined, and then the corresponding highlight video paragraphs and regular video paragraphs are generated, and finally the video film is generated. In other words, the video editing method provided by the embodiment of the present application can automatically generate video films without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video film is generated based on the regular video paragraphs and the highlight video paragraphs, the highlight paragraphs are the most exciting video clips in the video film generated according to the target highlight template and the highlight sampling frame, therefore, the video film can also have rich aesthetics and expressiveness. Moreover, the embodiment of the present application can also realize the replacement of the highlight paragraphs and the regular paragraphs, that is, provide more choices for users, so that the finally generated video film can meet the user's preferences and requirements.
[0285] The above embodiment is to implement the above video editing method by interaction between the terminal and the server. In practical applications, the above video editing method can also be implemented by any device in the terminal or the server. Figure 6 , Figure 6 FIG. 1 is a flow chart of another video editing method provided in an embodiment of the present application. The method comprises the following steps.
[0286] Step 601: Sampling a plurality of clipping materials to obtain a plurality of sampling frames, wherein any one of the plurality of clipping materials includes a video or a picture.
[0287] The implementation of this step can refer to the implementation of the terminal sampling multiple clip materials to obtain multiple sample frames in step 301 in the above-mentioned embodiment. Different from the above-mentioned embodiment, the execution subject of this embodiment can also be a server. In the case where the execution subject is a server, the multiple clip materials can be sent by the terminal to the server, or can be pre-stored in the server. Of course, they can also be obtained by the server in other ways, and the embodiment of the present application does not limit this.
[0288] Step 602: Determine attribute information of each sampling frame in a plurality of sampling frames, where the attribute information indicates a feature of the sampling frame.
[0289] The implementation method of this step can refer to the relevant method introduced in step 302 of the aforementioned embodiment, and the embodiment of the present application will not be repeated here.
[0290] Step 603: Based on the attribute information of each sampling frame in the multiple sampling frames, determine a target highlight template from multiple candidate highlight templates and a highlight sampling frame corresponding to each highlight shot in the multiple highlight shots included in the target highlight template, and the highlight sampling frame is a sampling frame in the multiple sampling frames that matches the highlight shot.
[0291] The implementation method of this step can refer to the relevant methods introduced in steps 3041-3043 in the aforementioned embodiment, and the embodiments of this application will not be repeated here.
[0292] Step 604: based on multiple editing materials, attribute information of each sampling frame in multiple sampling frames, target highlight template and highlight sampling frames corresponding to multiple highlight shots, determine regular video segments and highlight video segments corresponding to the multiple editing materials.
[0293] The implementation method of this step can refer to the relevant methods introduced in steps 3061-3064 in the aforementioned embodiment, and the embodiments of the present application will not be repeated here.
[0294] Step 605: Based on the regular video segments and the highlight video segments corresponding to the plurality of clipping materials, a video film is generated after the plurality of clipping materials are clipped.
[0295] The implementation of this step can refer to the implementation of step 307 in the above embodiment, in which the terminal generates a video film obtained after video editing of the multiple clipping materials based on the regular video segments and highlight video segments corresponding to the multiple clipping materials. Different from the above embodiment, the execution subject of this embodiment can also be a server. In the case where the execution subject is a server, for the second implementation of step 307, the server can send the regular video segment and the highlight video segment to the terminal so that the terminal can display a video preview interface, and the user can trigger an editing operation of the target video segment on the video preview interface. In response to the editing operation of the target video segment, the terminal sends the target video segment to the server, the server receives the target video segment and determines at least one replacement segment corresponding to the target video segment, and sends the at least one replacement segment corresponding to the target video segment to the terminal, and the terminal displays the at least one replacement segment corresponding to the target video segment. The user selects a replacement segment from the at least one replacement segment corresponding to the target video segment and triggers a replacement determination operation to instruct the terminal to replace the target video segment with the replacement segment selected by the user, and the terminal sends the replacement segment selected by the user to the server, and the server receives the replacement segment selected by the user to obtain the regular video segment and the highlight video segment obtained after editing, and generates the video film based on the regular video segment and the highlight video segment obtained after editing.
[0296] Since the multiple sampling frames are obtained by sampling the multiple editing materials, and the attribute information of the multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can form the highlight paragraphs in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matched by the highlight sampling frames can also be determined, and then the corresponding highlight video paragraphs and regular video paragraphs are generated, and finally the video film is generated. In other words, the video editing method provided by the embodiment of the present application can automatically generate video films without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video film is generated based on the regular video paragraphs and the highlight video paragraphs, the highlight paragraphs are the most exciting video clips in the video film generated according to the target highlight template and the highlight sampling frame, therefore, the video film can also have rich aesthetics and expressiveness. Moreover, the embodiment of the present application can also realize the replacement of the highlight paragraphs and the regular paragraphs, that is, provide more choices for users, so that the finally generated video film can meet the user's preferences and requirements. In addition, in the embodiment of the present application, the above-mentioned video editing method can be implemented by only one device without the need for interaction between the terminal and the server, thereby saving communication time and greatly improving the efficiency of video editing.
[0297] Figure 7 is a schematic diagram of the structure of a video editing device provided by an embodiment of the present application. The video editing device can be implemented as part or all of a terminal by software, hardware, or a combination of both. Figure 7 The device includes: a sampling module 701, a first determining module 702, a first sending module 703, a first receiving module 704, a second determining module 705 and a generating module 706.
[0298] The sampling module 701 is used to sample multiple clip materials to obtain multiple sampling frames, and the clip materials include videos or pictures. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0299] The first determination module 702 is used to determine the attribute information of each sampling frame, where the attribute information indicates the characteristics of the sampling frame. The detailed implementation process refers to the corresponding contents in the above embodiments, which will not be repeated here.
[0300] The first sending module 703 is used to send the attribute information of the multiple sampling frames to the server, so that the server determines the target highlight template and the highlight sampling frame corresponding to each highlight lens in the multiple highlight lenses included in the target highlight template, and the highlight sampling frame is the sampling frame matching the highlight lens in the multiple sampling frames. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0301] The first receiving module 704 is used to receive the target highlight template and highlight sampling frames corresponding to multiple highlight shots sent by the server. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0302] The second determination module 705 is used to determine the regular video segments and the highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0303] The generation module 706 is used to generate a video film obtained by video editing based on the regular video segments and the highlight video segments corresponding to the multiple editing materials. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0304] Optionally, the second determining module 705 is specifically configured to:
[0305] Based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, determining at least one group of regular sampling frames from the sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots;
[0306] Determine, from a plurality of editing materials, a regular shot corresponding to each regular sampling frame in at least one group of regular sampling frames;
[0307] Combining regular shots corresponding to regular sampling frames in the same group of at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment;
[0308] At least one highlight video segment is determined from a plurality of clip materials based on a target highlight template and a highlight sampling frame corresponding to each highlight shot.
[0309] Optionally, the second determining module 705 is specifically configured to:
[0310] Based on the attribute information of the plurality of sampling frames, a plurality of candidate sampling frames are determined from the sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots, wherein the image contents of the plurality of candidate sampling frames are different;
[0311] Based on the attribute information of the plurality of candidate sampling frames, clustering the plurality of candidate sampling frames to obtain at least one cluster set, each cluster set including at least one candidate sampling frame;
[0312] Determine the number of regular shots included in the video based on the number of at least one cluster set and the number of multiple highlight shots;
[0313] Based on the number of regular shots and the number of candidate sampling frames in each cluster set, a group of regular sampling frames is selected from each cluster set to obtain at least one group of regular sampling frames.
[0314] Optionally, the video editing device further includes:
[0315] A second receiving module is used to receive target background music sent by the server, where the target background music includes at least one music segment, and each music segment corresponds to a number of shots;
[0316] The second determining module 705 is specifically used for:
[0317] Determining a target music segment from at least one music segment based on the number of at least one cluster set, the number of the plurality of highlight shots, and the number of shots corresponding to each music segment;
[0318] The number of shots corresponding to the target music segment is determined as the number of shots of the completed video;
[0319] The difference between the number of shots in the finished video and the number of the plurality of highlight shots is determined as the number of regular shots included in the finished video.
[0320] Optionally, the video editing device further includes:
[0321] A second sending module is used to send at least one group of regular sampling frames to the server, so that the service determines the video effect information corresponding to the video film based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each highlight lens;
[0322] The third receiving module is used to receive the video effect information sent by the server;
[0323] The generation module 706 is specifically used for:
[0324] Generate video clips based on regular video segments, highlight video segments and video effect information.
[0325] Optionally, the generating module 706 is specifically used for:
[0326] Display a video preview interface, which includes a regular video segment and a highlight video segment;
[0327] In response to an editing operation on a target video segment, performing an editing operation on the target video segment, where the target video segment is any one of a regular video segment and a highlight video segment;
[0328] Based on the regular video segments and highlight video segments obtained after editing, a video film is generated.
[0329] Since the multiple sampling frames are obtained by sampling the multiple editing materials, and the attribute information of the multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can form the highlight paragraphs in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matched by the highlight sampling frames can also be determined, and then the corresponding highlight video paragraphs and regular video paragraphs are generated, and finally the video film is generated. In other words, the video editing method provided by the embodiment of the present application can automatically generate video films without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video film is generated based on the regular video paragraphs and the highlight video paragraphs, the highlight paragraphs are the most exciting video clips in the video film generated according to the target highlight template and the highlight sampling frame, therefore, the video film can also have rich aesthetics and expressiveness. Moreover, the embodiment of the present application can also realize the replacement of the highlight paragraphs and the regular paragraphs, that is, provide more choices for users, so that the finally generated video film can meet the user's preferences and requirements.
[0330] It should be noted that: the video editing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when performing video editing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video editing device provided in the above embodiment and the video editing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0331] Figure 8 is a schematic diagram of the structure of a video editing device provided by an embodiment of the present application. The video editing device can be implemented as part or all of a server by software, hardware, or a combination of both. Figure 8 The device includes: a first receiving module 801, a first determining module 802 and a first sending module 803.
[0332] The first receiving module 801 is used to receive attribute information of each of the multiple sampling frames sent by the terminal, where the multiple sampling frames are obtained by sampling multiple clip materials, and the attribute information indicates the characteristics of the sampling frames. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0333] The first determination module 802 is used to determine, from multiple candidate highlight templates, a target highlight template and a highlight sampling frame corresponding to each highlight lens in the multiple highlight lenses included in the target highlight template based on the attribute information of the multiple sampling frames, wherein the highlight sampling frame is a sampling frame matching the highlight lens in the multiple sampling frames. The detailed implementation process refers to the corresponding content in the above-mentioned embodiments, which will not be repeated here.
[0334] The first sending module 803 is used to send the target highlight template and the highlight sampling frame corresponding to each highlight shot to the terminal, so that the terminal determines the regular video segments and the highlight video segments corresponding to the multiple clipping materials, and generates a video film obtained after the multiple clipping materials are clipped based on the regular video segments and the highlight video segments. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0335] Optionally, each candidate highlight template has a matching constraint;
[0336] The first determining module 802 is specifically used for:
[0337] Based on the attribute information of each sampled frame, determining a matching score between each sampled frame and each highlight shot in each candidate highlight template;
[0338] Based on the matching score between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, determine a plurality of first highlight templates and a highlight sampling frame corresponding to each highlight shot in each first highlight template from a plurality of candidate highlight templates;
[0339] At least one first highlight template is selected from the multiple first highlight templates as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template is used as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
[0340] Optionally, the attribute information of each sampled frame includes a feature label and a semantic feature vector of the sampled frame, and each highlight shot in each candidate highlight template has a label requirement and a semantic feature vector;
[0341] The first determining module 802 is specifically used for:
[0342] For each candidate highlight template, based on the feature label of each sampling frame and the label requirement of each highlight lens in the candidate highlight template, determine the label score between each sampling frame and each highlight lens in the candidate highlight template;
[0343] Determine a similarity score between each sampling frame and each highlight shot in the candidate highlight template based on the semantic feature vector of each sampling frame and the semantic feature vector of each highlight shot in the candidate highlight template;
[0344] Based on the label score and the similarity score between each sampled frame and each highlight shot in the candidate highlight template, a matching score between each sampled frame and each highlight shot in the candidate highlight template is determined.
[0345] Optionally, the first determining module 802 is specifically configured to:
[0346] For each candidate highlight template, based on the matching score between each sampling frame and each highlight shot in the candidate highlight template and the matching constraint condition of the candidate highlight template, determine whether each highlight shot in the candidate highlight template has a matching sampling frame;
[0347] If each highlight shot in the candidate highlight template has a matching sampling frame, and the number of sampling frames matched by the target highlight shot in the candidate highlight template is multiple, then based on the attribute information of the multiple sampling frames matched by the target highlight lens, the highlight sampling frame corresponding to the target highlight lens is determined, and the candidate highlight template is used as the first highlight template, and the target highlight lens is at least one highlight shot in the candidate highlight template.
[0348] Optionally, the category of each candidate highlight template includes a beginning highlight template, a middle highlight template or an ending highlight template;
[0349] The first determining module 802 is specifically used for:
[0350] Screening the plurality of first highlight templates to obtain a plurality of second highlight templates, wherein highlight sampling frames corresponding to highlight shots in every two second highlight templates do not overlap;
[0351] If there are at least two beginning highlight templates and / or at least two ending highlight templates among the plurality of second highlight templates, selecting the beginning highlight template with the highest priority and the ending highlight template with the highest priority among the plurality of second highlight templates;
[0352] The beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among multiple second highlight templates are determined as target highlight templates, and the highlight sampling frames corresponding to each highlight shot in the beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among multiple second highlight templates are determined as the highlight sampling frames corresponding to each highlight shot in the target highlight template.
[0353] Optionally, the video editing device further includes:
[0354] A second determination module is used to determine target background music from multiple candidate background music based on the attribute information of multiple sampling frames, where the target background music includes at least one music segment, and each music segment corresponds to a number of shots;
[0355] The second sending module is used to send the target background music to the terminal.
[0356] Optionally, the video editing device further includes:
[0357] A second receiving module, configured to receive at least one group of regular sampling frames sent by a terminal;
[0358] A third determination module is used to determine video effect information corresponding to the video film based on at least one group of regular sampling frames and highlight sampling frames corresponding to each highlight shot in the target highlight template;
[0359] The third sending module is used to send the video effect information to the terminal.
[0360] Since the multiple sampling frames are obtained by sampling the multiple editing materials, and the attribute information of the multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can form the highlight paragraphs in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matched by the highlight sampling frames can also be determined, and then the corresponding highlight video paragraphs and regular video paragraphs are generated, and finally the video film is generated. In other words, the video editing method provided by the embodiment of the present application can automatically generate video films without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video film is generated based on the regular video paragraphs and the highlight video paragraphs, the highlight paragraphs are the most exciting video clips in the video film generated according to the target highlight template and the highlight sampling frame, therefore, the video film can also have rich aesthetics and expressiveness. Moreover, the embodiment of the present application can also realize the replacement of the highlight paragraphs and the regular paragraphs, that is, provide more choices for users, so that the finally generated video film can meet the user's preferences and requirements.
[0361] It should be noted that: the video editing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when performing video editing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video editing device provided in the above embodiment and the video editing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0362] Fig. 9 is a schematic diagram of the structure of a video editing device provided by an embodiment of the present application. The video editing device can be implemented as part or all of a terminal by software, hardware, or a combination of both, or the video editing device can be implemented as part or all of a server by software, hardware, or a combination of both. Fig. 9 The device includes: a sampling module 901, a first determination module 902, a second determination module 903, a third determination module 904 and a generation module 905.
[0363] The sampling module 901 is used to sample multiple clip materials to obtain multiple sampling frames, and the clip materials include videos or pictures. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0364] The first determination module 902 is used to determine the attribute information of each sampling frame, where the attribute information indicates the characteristics of the sampling frame. The detailed implementation process refers to the corresponding contents in the above embodiments, which will not be repeated here.
[0365] The second determination module 903 is used to determine, from multiple candidate highlight templates, a target highlight template and a highlight sampling frame corresponding to each highlight lens in the multiple highlight lenses included in the target highlight template based on the attribute information of the multiple sampling frames, wherein the highlight sampling frame is a sampling frame matching the highlight lens in the multiple sampling frames. The detailed implementation process refers to the corresponding contents in the above-mentioned embodiments, which will not be repeated here.
[0366] The third determination module 904 is used to determine the regular video segments and the highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
[0367] The generation module 905 is used to generate a video film obtained by video editing of the multiple clipping materials based on the regular video segments and highlight video segments corresponding to the multiple clipping materials. The detailed implementation process refers to the corresponding content in the above embodiments and will not be repeated here.
[0368] Optionally, each candidate highlight template has a matching constraint;
[0369] The second determining module 903 is specifically used for:
[0370] Based on the attribute information of each sampled frame, determining a matching score between each sampled frame and each highlight shot in each candidate highlight template;
[0371] Based on the matching score between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, determine a plurality of first highlight templates and a highlight sampling frame corresponding to each highlight shot in each first highlight template from a plurality of candidate highlight templates;
[0372] At least one first highlight template is selected from the multiple first highlight templates as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template is used as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
[0373] Optionally, the third determining module 904 is specifically configured to:
[0374] Based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, determining at least one group of regular sampling frames from the sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots;
[0375] Determine, from a plurality of editing materials, a regular shot corresponding to each regular sampling frame in at least one group of regular sampling frames;
[0376] Combining regular shots corresponding to regular sampling frames in the same group of at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment;
[0377] At least one highlight video segment is determined from a plurality of clip materials based on a target highlight template and a highlight sampling frame corresponding to each highlight shot.
[0378] Optionally, the third determining module 904 is specifically configured to:
[0379] Based on the attribute information of the plurality of sampling frames, a plurality of candidate sampling frames are determined from the sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots, and the image contents of the plurality of candidate sampling frames are different;
[0380] Based on the attribute information of the plurality of candidate sampling frames, clustering the plurality of candidate sampling frames to obtain at least one cluster set, each cluster set including at least one candidate sampling frame;
[0381] Determine the number of regular shots included in the video based on the number of at least one cluster set and the number of multiple highlight shots;
[0382] Based on the number of regular shots and the number of candidate sampling frames in each cluster set, a group of regular sampling frames is selected from each cluster set to obtain at least one group of regular sampling frames.
[0383] Optionally, the device further comprises:
[0384] A fourth determination module is used to determine target background music from multiple candidate background music based on the attribute information of multiple sampling frames, where the target background music includes at least one music segment, and each music segment corresponds to a number of shots;
[0385] The third determining module 904 is specifically used for:
[0386] Determining a target music segment from at least one music segment based on the number of at least one cluster set, the number of the plurality of highlight shots, and the number of shots corresponding to each music segment;
[0387] The number of shots corresponding to the target music segment is determined as the number of shots of the completed video;
[0388] The difference between the number of shots in the finished video and the number of the plurality of highlight shots is determined as the number of regular shots included in the finished video.
[0389] Since the multiple sampling frames are obtained by sampling the multiple editing materials, and the attribute information of the multiple sampling frames can indicate the characteristics of the corresponding sampling frames, therefore, based on the attribute information, the wonderful sampling frames that can form the highlight paragraphs in the editing materials, that is, the highlight sampling frames, can be determined from the multiple sampling frames, and the highlight templates matched by the highlight sampling frames can also be determined, and then the corresponding highlight video paragraphs and regular video paragraphs are generated, and finally the video film is generated. In other words, the video editing method provided by the embodiment of the present application can automatically generate video films without the user having to master editing knowledge, thereby effectively improving the efficiency of video editing. Moreover, since the video film is generated based on the regular video paragraphs and the highlight video paragraphs, the highlight paragraphs are the most exciting video clips in the video film generated according to the target highlight template and the highlight sampling frame, therefore, the video film can also have rich aesthetics and expressiveness. Moreover, the embodiment of the present application can also realize the replacement of the highlight paragraphs and the regular paragraphs, that is, provide more choices for users, so that the finally generated video film can meet the user's preferences and requirements. In addition, in the embodiment of the present application, the above-mentioned video editing method can be implemented by only one device without the need for interaction between the terminal and the server, thereby saving communication time and greatly improving the efficiency of video editing.
[0390] It should be noted that: the video editing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when performing video editing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video editing device provided in the above embodiment and the video editing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0391] Fig.10 1 is a block diagram of a terminal 1000 provided in an embodiment of the present application. The terminal 1000 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 1000 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.
[0392] Typically, the terminal 1000 includes a processor 1001 and a memory 1002 .
[0393] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0394] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction, which is used to be executed by the processor 1001 to implement the control method of the smart device provided in the method embodiment of the present application.
[0395] In some embodiments, the terminal 1000 may also optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1003 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004, a touch display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008 and a power supply 1009.
[0396] The peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.
[0397] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may also include circuits related to NFC (Near Field Communication), which is not limited in the embodiments of the present application.
[0398] The display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, and the front panel of the terminal 1000 is set; in other embodiments, the display screen 1005 can be at least two, which are respectively set on different surfaces of the terminal 1000 or are folded; in some other embodiments, the display screen 1005 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1000. Even, the display screen 1005 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0399] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0400] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1001 for processing, or input them into the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1000. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 may also include a headphone jack.
[0401] The positioning component 1008 is used to locate the current geographical location of the terminal 1000 to implement navigation or LBS (Location Based Service). The positioning component 1008 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or Russia's Galileo system.
[0402] The power supply 1009 is used to power various components in the terminal 1000. The power supply 1009 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.
[0403] In some embodiments, the terminal 1000 further includes one or more sensors 1010 , including but not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , a fingerprint sensor 1014 , an optical sensor 1015 , and a proximity sensor 1016 .
[0404] The acceleration sensor 1011 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1000. For example, the acceleration sensor 1011 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1001 can control the touch display screen 1005 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used for collecting game or user motion data.
[0405] The gyro sensor 1012 can detect the body direction and rotation angle of the terminal 1000, and the gyro sensor 1012 can cooperate with the acceleration sensor 1011 to collect the user's 3D actions on the terminal 1000. The processor 1001 can implement the following functions based on the data collected by the gyro sensor 1012: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0406] The pressure sensor 1013 can be set on the side frame of the terminal 1000 and / or the lower layer of the touch display screen 1005. When the pressure sensor 1013 is set on the side frame of the terminal 1000, the user's holding signal of the terminal 1000 can be detected, and the processor 1001 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is set on the lower layer of the touch display screen 1005, the processor 1001 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1005. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0407] The fingerprint sensor 1014 is used to collect the user's fingerprint, and the processor 1001 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1001 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1014 can be set on the front, back, or side of the terminal 1000. When a physical button or a manufacturer logo is set on the terminal 1000, the fingerprint sensor 1014 can be integrated with the physical button or the manufacturer logo.
[0408] The optical sensor 1015 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the touch display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1005 is reduced. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1015.
[0409] The proximity sensor 1016, also called a distance sensor, is usually arranged on the front panel of the terminal 1000. The proximity sensor 1016 is used to collect the distance between the user and the front of the terminal 1000. In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually decreasing, the processor 1001 controls the touch display screen 1005 to switch from the screen-on state to the screen-off state; when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually increasing, the processor 1001 controls the touch display screen 1005 to switch from the screen-off state to the screen-on state.
[0410] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal 1000, and the terminal 1000 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.
[0411] Fig.11 1 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The server 1100 also includes a basic input / output system (I / O system) 1106 that helps transmit information between various devices in the computer, and a large-capacity storage device 1107 for storing an operating system 1113, an application program 1114, and other program modules 1115.
[0412] The basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109 such as a mouse and a keyboard for user inputting information. The display 1108 and the input device 1109 are connected to the central processing unit 1101 through an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include an input / output controller 1110 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, a printer, or other types of output devices.
[0413] The mass storage device 1107 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable media provide non-volatile storage for the server 1100. That is, the mass storage device 1107 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.
[0414] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1104 and mass storage device 1107 can be collectively referred to as memory.
[0415] According to various embodiments of the present application, the server 1100 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1100 can be connected to the network 1112 through the network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 can be used to connect to other types of networks or remote computer systems (not shown).
[0416] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.
[0417] The embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal, the terminal executes the steps of the video editing method described in the above embodiment.
[0418] The embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores instructions, and when the instructions are executed on a server, the server executes the steps of the video editing method described in the above embodiment.
[0419] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a terminal, enables the terminal to execute the steps of the video editing method described in the above embodiment. In other words, a computer program is provided, which, when executed on a terminal, enables the terminal to execute the steps of the video editing method described in the above embodiment.
[0420] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a server, enables the server to execute the steps of the video editing method described in the above embodiment. In other words, a computer program is provided, which, when executed on a server, enables the server to execute the steps of the video editing method described in the above embodiment.
[0421] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.
[0422] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.
[0423] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the multiple clip materials involved in the embodiments of this application are all obtained with full authorization.
[0424] The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A video editing method, characterized in that: Applied to a terminal, the method comprises: Sampling a plurality of clipping materials to obtain a plurality of sampling frames, wherein the clipping materials include videos or pictures; Determining attribute information of each of the sampling frames, wherein the attribute information indicates a feature of the sampling frame; Sending the attribute information of the multiple sampling frames to a server, so that the server determines a target highlight template and a highlight sampling frame corresponding to each of the multiple highlight shots included in the target highlight template, wherein the highlight sampling frame is a sampling frame matching the highlight shot in the multiple sampling frames; receiving the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots sent by the server; Determine the regular video segments and the highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template, and the highlight sampling frames corresponding to the multiple highlight shots; Based on the regular video segments and the highlight video segments, a video film is generated after the multiple clip materials are clipped.
2. The method according to claim 1, characterized in that The determining, based on the multiple clip materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots, the regular video segments and the highlight video segments corresponding to the multiple clip materials comprises: Based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, determining at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots; Determine, from the plurality of editing materials, a regular shot corresponding to each regular sampling frame in the at least one group of regular sampling frames; Combining regular shots corresponding to regular sampling frames in the same group of the at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment; Based on the target highlight template and the highlight sampling frame corresponding to each of the highlight shots, at least one of the highlight video segments is determined from the multiple clip materials.
3. The method according to claim 2, characterized in that The determining, based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots comprises: Based on the attribute information of the multiple sampling frames, determining multiple candidate sampling frames from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots, wherein the image contents of the multiple candidate sampling frames are different; Based on the attribute information of the multiple candidate sampling frames, clustering the multiple candidate sampling frames to obtain at least one cluster set, each of the cluster sets including at least one candidate sampling frame; Determine the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots; Based on the number of the regular shots and the number of candidate sampling frames in each of the cluster sets, a group of regular sampling frames is selected from each of the cluster sets to obtain the at least one group of regular sampling frames.
4. The method according to claim 3, characterized in that After sending the attribute information of the plurality of sampling frames to the server, the method further includes: receiving target background music sent by the server, wherein the target background music includes at least one music segment, and each of the music segments corresponds to a number of shots; The determining the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots comprises: Determine a target music segment from the at least one music segment based on the number of the at least one cluster set, the number of the plurality of highlight shots, and the number of shots corresponding to each of the music segments; Determine the number of shots corresponding to the target music segment as the number of shots of the completed video; The difference between the number of shots in the completed video and the number of the plurality of highlight shots is determined as the number of regular shots included in the completed video.
5. The method according to claim 2, characterized in that After determining at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots based on the attribute information of the multiple sampling frames and the number of the multiple highlight shots, the method further includes: Sending the at least one group of regular sampling frames to the server, so that the service determines the video effect information corresponding to the video film based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each of the highlight shots; Receiving the video effect information sent by the server; The step of generating a video film obtained by performing video editing on the plurality of clipping materials based on the regular video segment and the highlight video segment comprises: The video film is generated based on the regular video segments, the highlight video segments and the video effect information.
6. The method according to any one of claims 1 to 5, characterized in that: The step of generating a video film obtained by performing video editing on the plurality of clipping materials based on the regular video segment and the highlight video segment comprises: Displaying a video preview interface, wherein the video preview interface includes the regular video segment and the highlight video segment; In response to an editing operation of a target video segment, performing the editing operation on the target video segment, wherein the target video segment is any one of the regular video segment and the highlight video segment; The video film is generated based on the regular video segments and the highlight video segments obtained after editing.
7. A video editing method, characterized in that: Applied to a server, the method comprises: receiving attribute information of each sampling frame among a plurality of sampling frames sent by a terminal, wherein the plurality of sampling frames are obtained by sampling a plurality of clip materials, and the attribute information indicates a feature of the sampling frame; Based on the attribute information of the multiple sampling frames, determine a target highlight template from multiple candidate highlight templates and a highlight sampling frame corresponding to each highlight shot in the multiple highlight shots included in the target highlight template, the highlight sampling frame being a sampling frame in the multiple sampling frames that matches the highlight shot; The target highlight template and the highlight sampling frame corresponding to each of the highlight shots are sent to the terminal, so that the terminal determines the regular video segments and the highlight video segments corresponding to the multiple editing materials, and based on the regular video segments and the highlight video segments, generates a video film obtained after video editing of the multiple editing materials.
8. The method according to claim 7, characterized in that Each of the candidate highlight templates has a matching constraint condition; The determining, based on the attribute information of the multiple sampling frames, from multiple candidate highlight templates, a target highlight template and a highlight sampling frame corresponding to each highlight shot of the multiple highlight shots included in the target highlight template, comprises: Based on the attribute information of each of the sampling frames, determining a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates; Based on the matching scores between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, determining a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each of the first highlight templates from the plurality of candidate highlight templates; At least one first highlight template is selected from the multiple first highlight templates as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template is used as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
9. The method according to claim 8, characterized in that The attribute information of each of the sampled frames includes a feature label and a semantic feature vector of the sampled frame, and each highlight shot in each of the candidate highlight templates has a label requirement and a semantic feature vector; The determining, based on the attribute information of each of the sampling frames, a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates comprises: For each of the candidate highlight templates, based on the feature labels of each of the sampling frames and the label requirements of each highlight shot in the candidate highlight template, determine a label score between each of the sampling frames and each highlight shot in the candidate highlight template; Determine a similarity score between each of the sampling frames and each of the highlight shots in the candidate highlight template based on the semantic feature vector of each of the sampling frames and the semantic feature vector of each of the highlight shots in the candidate highlight template; Based on the label score and the similarity score between each of the sampled frames and each of the highlight shots in the candidate highlight template, a matching score between each of the sampled frames and each of the highlight shots in the candidate highlight template is determined.
10. The method according to claim 8, characterized in that The determining, based on the matching score between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each of the first highlight templates from the plurality of candidate highlight templates comprises: For each of the candidate highlight templates, determining whether each highlight shot in the candidate highlight template has a matching sampling frame based on a matching score between each sampling frame and each highlight shot in the candidate highlight template and a matching constraint condition of the candidate highlight template; If each highlight shot in the candidate highlight template has a matching sampling frame, and the number of sampling frames matched by the target highlight shot in the candidate highlight template is multiple, then based on the attribute information of the multiple sampling frames matched by the target highlight shot, the highlight sampling frame corresponding to the target highlight shot is determined, and the candidate highlight template is used as the first highlight template, and the target highlight shot is at least one highlight shot in the candidate highlight template.
11. The method according to claim 8, characterized in that The category of each candidate highlight template includes a beginning highlight template, a middle highlight template or an ending highlight template; The step of selecting at least one first highlight template from the plurality of first highlight templates as the target highlight template, and using the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template as the highlight sampling frame corresponding to each highlight shot in the target highlight template includes: Screening the plurality of first highlight templates to obtain a plurality of second highlight templates, wherein highlight sampling frames corresponding to highlight shots in every two of the second highlight templates do not overlap; If there are at least two beginning highlight templates and / or at least two ending highlight templates in the plurality of second highlight templates, selecting the beginning highlight template with the highest priority and the ending highlight template with the highest priority in the plurality of second highlight templates; The beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the target highlight templates, and the highlight sampling frames corresponding to each highlight shot in the beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the highlight sampling frames corresponding to each highlight shot in the target highlight template.
12. The method according to claim 7, characterized in that After receiving the attribute information of each sampling frame in the plurality of sampling frames sent by the terminal, the method further includes: Based on the attribute information of the plurality of sampling frames, determining target background music from a plurality of candidate background music, the target background music comprising at least one music segment, each of the music segments corresponding to a number of shots; The target background music is sent to the terminal.
13. The method according to claim 7, characterized in that The method further comprises: Receiving at least one set of regular sampling frames sent by the terminal; Determine video effect information corresponding to the video film based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each highlight shot in the target highlight template; The video effect information is sent to the terminal.
14. A video editing method, characterized in that: The method comprises: Sampling a plurality of clipping materials to obtain a plurality of sampling frames, wherein the clipping materials include videos or pictures; Determining attribute information of each of the sampling frames, wherein the attribute information indicates a feature of the sampling frame; Based on the attribute information of the multiple sampling frames, determine a target highlight template from multiple candidate highlight templates and a highlight sampling frame corresponding to each highlight shot in the multiple highlight shots included in the target highlight template, the highlight sampling frame being a sampling frame in the multiple sampling frames that matches the highlight shot; Determine the regular video segments and the highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template, and the highlight sampling frames corresponding to the multiple highlight shots; Based on the regular video segments and the highlight video segments, a video film is generated after the multiple clip materials are clipped.
15. The method according to claim 14, characterized in that Each of the candidate highlight templates has a matching constraint condition; The determining, based on the attribute information of the multiple sampling frames, from multiple candidate highlight templates, a target highlight template and a highlight sampling frame corresponding to each highlight shot of the multiple highlight shots included in the target highlight template, comprises: Based on the attribute information of each of the sampling frames, determining a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates; Based on the matching scores between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, determining a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each of the first highlight templates from the plurality of candidate highlight templates; At least one first highlight template is selected from the multiple first highlight templates as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template is used as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
16. The method according to claim 14, characterized in that The determining, based on the multiple clip materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots, the regular video segments and the highlight video segments corresponding to the multiple clip materials comprises: Based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, determining at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots; Determine, from the plurality of editing materials, a regular shot corresponding to each regular sampling frame in the at least one group of regular sampling frames; Combining regular shots corresponding to regular sampling frames in the same group of the at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment; Based on the target highlight template and the highlight sampling frame corresponding to each of the highlight shots, at least one of the highlight video segments is determined from the multiple clip materials.
17. The method according to claim 16, characterized in that The determining, based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots comprises: Based on the attribute information of the multiple sampling frames, determining multiple candidate sampling frames from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots, wherein the image contents of the multiple candidate sampling frames are different; Based on the attribute information of the multiple candidate sampling frames, clustering the multiple candidate sampling frames to obtain at least one cluster set, each of the cluster sets including at least one candidate sampling frame; Determine the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots; Based on the number of the regular shots and the number of candidate sampling frames in each of the cluster sets, a group of regular sampling frames is selected from each of the cluster sets to obtain the at least one group of regular sampling frames.
18. The method according to claim 17, characterized in that After determining the attribute information of each of the sampling frames, the method further includes: Based on the attribute information of the plurality of sampling frames, determining target background music from a plurality of candidate background music, the target background music comprising at least one music segment, each of the music segments corresponding to a number of shots; The determining the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots comprises: Determine a target music segment from the at least one music segment based on the number of the at least one cluster set, the number of the plurality of highlight shots, and the number of shots corresponding to each of the music segments; Determine the number of shots corresponding to the target music segment as the number of shots of the completed video; The difference between the number of shots in the completed video and the number of the plurality of highlight shots is determined as the number of regular shots included in the completed video.
19. A video editing device, characterized in that: The video editing device comprises: A sampling module, used to sample a plurality of clip materials to obtain a plurality of sampling frames, wherein the clip materials include videos or pictures; A first determining module, configured to determine attribute information of each of the sampling frames, wherein the attribute information indicates a feature of the sampling frame; A first sending module is used to send the attribute information of the multiple sampling frames to a server, so that the server determines a target highlight template and a highlight sampling frame corresponding to each of the multiple highlight shots included in the target highlight template, and the highlight sampling frame is a sampling frame matching the highlight shot in the multiple sampling frames; A first receiving module, configured to receive the target highlight template and highlight sampling frames corresponding to the multiple highlight shots sent by the server; A second determining module is used to determine the regular video segments and the highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots; A generation module is used to generate a video film obtained after video editing of the multiple editing materials based on the regular video segments and the highlight video segments.
20. The video editing device according to claim 19, characterized in that: The second determining module is specifically used for: Based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, determining at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots; Determine, from the plurality of editing materials, a regular shot corresponding to each regular sampling frame in the at least one group of regular sampling frames; Combining regular shots corresponding to regular sampling frames in the same group of the at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment; Based on the target highlight template and the highlight sampling frame corresponding to each of the highlight shots, at least one of the highlight video segments is determined from the multiple clip materials.
21. The video editing device according to claim 20, characterized in that: The second determining module is specifically used for: Based on the attribute information of the multiple sampling frames, determining multiple candidate sampling frames from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots, wherein the image contents of the multiple candidate sampling frames are different; Based on the attribute information of the multiple candidate sampling frames, clustering the multiple candidate sampling frames to obtain at least one cluster set, each of the cluster sets including at least one candidate sampling frame; Determine the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots; Based on the number of the regular shots and the number of candidate sampling frames in each of the cluster sets, a group of regular sampling frames is selected from each of the cluster sets to obtain the at least one group of regular sampling frames.
22. The video editing device according to claim 21, characterized in that: The video editing device also includes: A second receiving module is used to receive target background music sent by the server, wherein the target background music includes at least one music segment, and each music segment corresponds to a number of shots; The second determining module is specifically used for: Determine a target music segment from the at least one music segment based on the number of the at least one cluster set, the number of the plurality of highlight shots, and the number of shots corresponding to each of the music segments; Determine the number of shots corresponding to the target music segment as the number of shots of the completed video; The difference between the number of shots in the completed video and the number of the plurality of highlight shots is determined as the number of regular shots included in the completed video.
23. The video editing device according to claim 20, characterized in that: The video editing device also includes: A second sending module is used to send the at least one group of regular sampling frames to the server, so that the service determines the video effect information corresponding to the video film based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each of the highlight shots; A third receiving module, used for receiving the video effect information sent by the server; The generation module is specifically used for: The video film is generated based on the regular video segments, the highlight video segments and the video effect information.
24. The video editing device according to any one of claims 19 to 23, characterized in that: The generation module is specifically used for: Displaying a video preview interface, wherein the video preview interface includes the regular video segment and the highlight video segment; In response to an editing operation of a target video segment, performing the editing operation on the target video segment, wherein the target video segment is any one of the regular video segment and the highlight video segment; The video film is generated based on the regular video segments and the highlight video segments obtained after editing.
25. A video editing device, characterized in that: The video editing device comprises: A first receiving module, configured to receive attribute information of each sampling frame among a plurality of sampling frames sent by a terminal, wherein the plurality of sampling frames are obtained by sampling a plurality of clip materials, and the attribute information indicates a feature of the sampling frame; A first determining module is configured to determine, from a plurality of candidate highlight templates, a target highlight template and a highlight sampling frame corresponding to each highlight shot in a plurality of highlight shots included in the target highlight template based on the attribute information of the plurality of sampling frames, wherein the highlight sampling frame is a sampling frame in the plurality of sampling frames that matches the highlight shot; The first sending module is used to send the target highlight template and the highlight sampling frame corresponding to each of the highlight shots to the terminal, so that the terminal determines the regular video segments and the highlight video segments corresponding to the multiple editing materials, and based on the regular video segments and the highlight video segments, generates a video film obtained after the multiple editing materials are edited.
26. The video editing device according to claim 25, characterized in that: Each of the candidate highlight templates has a matching constraint condition; The first determining module is specifically used for: Based on the attribute information of each of the sampling frames, determining a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates; Based on the matching scores between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, determining a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each of the first highlight templates from the plurality of candidate highlight templates; At least one first highlight template is selected from the multiple first highlight templates as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template is used as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
27. The video editing device according to claim 26, characterized in that: The attribute information of each of the sampled frames includes a feature label and a semantic feature vector of the sampled frame, and each highlight shot in each of the candidate highlight templates has a label requirement and a semantic feature vector; The first determining module is specifically used for: For each of the candidate highlight templates, based on the feature labels of each of the sampling frames and the label requirements of each highlight shot in the candidate highlight template, determine a label score between each of the sampling frames and each highlight shot in the candidate highlight template; Determine a similarity score between each of the sampling frames and each of the highlight shots in the candidate highlight template based on the semantic feature vector of each of the sampling frames and the semantic feature vector of each of the highlight shots in the candidate highlight template; Based on the label score and the similarity score between each of the sampled frames and each of the highlight shots in the candidate highlight template, a matching score between each of the sampled frames and each of the highlight shots in the candidate highlight template is determined.
28. The video editing device according to claim 26, characterized in that: The first determining module is specifically used for: For each of the candidate highlight templates, determining whether each highlight shot in the candidate highlight template has a matching sampling frame based on a matching score between each sampling frame and each highlight shot in the candidate highlight template and a matching constraint condition of the candidate highlight template; If each highlight shot in the candidate highlight template has a matching sampling frame, and the number of sampling frames matched by the target highlight shot in the candidate highlight template is multiple, then based on the attribute information of the multiple sampling frames matched by the target highlight shot, the highlight sampling frame corresponding to the target highlight shot is determined, and the candidate highlight template is used as the first highlight template, and the target highlight shot is at least one highlight shot in the candidate highlight template.
29. The video editing device according to claim 26, characterized in that: The category of each candidate highlight template includes a beginning highlight template, a middle highlight template or an ending highlight template; The first determining module is specifically used for: Screening the plurality of first highlight templates to obtain a plurality of second highlight templates, wherein highlight sampling frames corresponding to highlight shots in every two of the second highlight templates do not overlap; If there are at least two beginning highlight templates and / or at least two ending highlight templates in the plurality of second highlight templates, selecting the beginning highlight template with the highest priority and the ending highlight template with the highest priority in the plurality of second highlight templates; The beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the target highlight templates, and the highlight sampling frames corresponding to each highlight shot in the beginning highlight template and the ending highlight template with the highest priority, and the middle highlight template among the multiple second highlight templates are determined as the highlight sampling frames corresponding to each highlight shot in the target highlight template.
30. The video editing device according to claim 25, characterized in that: The video editing device also includes: A second determination module is used to determine target background music from multiple candidate background music based on the attribute information of the multiple sampling frames, wherein the target background music includes at least one music segment, and each of the music segments corresponds to a number of shots; The second sending module is used to send the target background music to the terminal.
31. The video editing device according to claim 25, characterized in that: The video editing device also includes: A second receiving module, configured to receive at least one group of regular sampling frames sent by the terminal; A third determination module is used to determine the video effect information corresponding to the video film based on the at least one group of regular sampling frames and the highlight sampling frames corresponding to each highlight shot in the target highlight template; The third sending module is used to send the video effect information to the terminal.
32. A video editing device, characterized in that: The device comprises: A sampling module, used to sample a plurality of clip materials to obtain a plurality of sampling frames, wherein the clip materials include videos or pictures; A first determining module, configured to determine attribute information of each of the sampling frames, wherein the attribute information indicates a feature of the sampling frame; a second determining module, configured to determine, from a plurality of candidate highlight templates, a target highlight template and a highlight sampling frame corresponding to each highlight shot in a plurality of highlight shots included in the target highlight template based on the attribute information of the plurality of sampling frames, wherein the highlight sampling frame is a sampling frame in the plurality of sampling frames that matches the highlight shot; A third determining module is used to determine the regular video segments and the highlight video segments corresponding to the multiple editing materials based on the multiple editing materials, the attribute information of the multiple sampling frames, the target highlight template and the highlight sampling frames corresponding to the multiple highlight shots; A generation module is used to generate a video film obtained after video editing of the multiple editing materials based on the regular video segments and the highlight video segments.
33. The device according to claim 32, characterized in that Each of the candidate highlight templates has a matching constraint condition; The second determining module is specifically used for: Based on the attribute information of each of the sampling frames, determining a matching score between each of the sampling frames and each highlight shot in each of the candidate highlight templates; Based on the matching scores between each sampling frame and each highlight shot in each candidate highlight template and the matching constraint condition of each candidate highlight template, determining a plurality of first highlight templates and highlight sampling frames corresponding to each highlight shot in each of the first highlight templates from the plurality of candidate highlight templates; At least one first highlight template is selected from the multiple first highlight templates as the target highlight template, and the highlight sampling frame corresponding to each highlight shot in the at least one first highlight template is used as the highlight sampling frame corresponding to each highlight shot in the target highlight template.
34. The device according to claim 32, characterized in that The third determination module is specifically used for: Based on the attribute information of the plurality of sampling frames and the number of the plurality of highlight shots, determining at least one group of regular sampling frames from sampling frames other than the highlight sampling frames corresponding to the plurality of highlight shots; Determine, from the plurality of editing materials, a regular shot corresponding to each regular sampling frame in the at least one group of regular sampling frames; Combining regular shots corresponding to regular sampling frames in the same group of the at least one group of regular sampling frames into a regular video segment to obtain at least one regular video segment; Based on the target highlight template and the highlight sampling frame corresponding to each of the highlight shots, at least one of the highlight video segments is determined from the multiple clip materials.
35. The device according to claim 34, characterized in that The third determination module is specifically used for: Based on the attribute information of the multiple sampling frames, determining multiple candidate sampling frames from sampling frames other than the highlight sampling frames corresponding to the multiple highlight shots, wherein the image contents of the multiple candidate sampling frames are different; Based on the attribute information of the multiple candidate sampling frames, clustering the multiple candidate sampling frames to obtain at least one cluster set, each of the cluster sets including at least one candidate sampling frame; Determine the number of regular shots included in the video film based on the number of the at least one cluster set and the number of the plurality of highlight shots; Based on the number of the regular shots and the number of candidate sampling frames in each of the cluster sets, a group of regular sampling frames is selected from each of the cluster sets to obtain the at least one group of regular sampling frames.
36. The device according to claim 35, characterized in that The device also includes: A fourth determination module, configured to determine target background music from a plurality of candidate background music based on the attribute information of the plurality of sampling frames, wherein the target background music includes at least one music segment, and each of the music segments corresponds to a number of shots; The third determination module is specifically used for: Determine a target music segment from the at least one music segment based on the number of the at least one cluster set, the number of the plurality of highlight shots, and the number of shots corresponding to each of the music segments; Determine the number of shots corresponding to the target music segment as the number of shots of the completed video; The difference between the number of shots in the completed video and the number of the plurality of highlight shots is determined as the number of regular shots included in the completed video.
37. A terminal, characterized in that: The terminal includes a memory and a processor, the memory is used to store a computer program, and the processor is configured to execute the computer program stored in the memory to implement the steps of the method according to any one of claims 1-6 or 14-18.
38. A server, characterized in that: The server includes a memory and a processor, the memory is used to store a computer program, and the processor is configured to execute the computer program stored in the memory to implement the steps of the method according to any one of claims 7-13 or 14-18.
39. A computer-readable storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed on the terminal, the terminal executes the steps of any one of the methods described in claims 1-6 or 14-18.
40. A computer-readable storage medium, characterized in that The storage medium stores instructions, and when the instructions are executed on the server, the server executes the steps of any one of the methods described in claims 7-13 or 14-18.
Citation Information
Patent Citations
Live stream editing method, apparatus and device
CN108833969A
Video editing method and device, server and storage medium
CN110996112A
Mobile terminal short video highlight moment editing method based on key behavior recognition
CN116095363A
Automated video editing system and method
US7248778B1
