Video processing method, video processing device and storage medium

Abstract

The invention discloses a video processing method, a video processing device and a storage medium, and the method comprises the steps: determining a template according to the video information of a to-be-processed video material, and enabling the template to at least comprise a video pit position; according to pit position information of a video pit position in the template, a video clip corresponding to the video pit position is determined from the to-be-processed video material, and the pit position information of the video pit position is at least used for indicating a scene corresponding to a to-be-filled video of the video pit position; and filling the video clip into a corresponding video pit of the template to obtain a recommended video.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, a video processing device, and a storage medium. Background Art

[0002] Currently, video content has become the mainstream of self-media, and users share their daily lives by shooting short videos. In order to obtain short videos with rich content, users can freely combine various video materials shot by shooting devices (drones, handheld gimbals, cameras or mobile phones), and edit and merge multiple video materials into one video to publish it on social networking sites. At present, most video editing solutions still require user participation, and there is a lack of effective solutions for automatic video editing. Summary of the invention

[0003] Based on this, the present application provides a video processing method, a video processing device and a storage medium. The video processing method processes the video material to be processed based on a template, aiming to reduce the workload of users when performing video editing and provide a variety of recommended videos.

[0004] In a first aspect, the present application provides a video processing method, comprising:

[0005] Determine a template according to video information of the video material to be processed, wherein the template includes at least one video pit position;

[0006] According to the pit position information of the video pit in the template, determine the video segment corresponding to the video pit from the video material to be processed, wherein the pit position information of the video pit is at least used to indicate the scene corresponding to the video to be filled in the video pit;

[0007] Fill the video clip into the corresponding video slot of the template to obtain a recommended video.

[0008] In a second aspect, the present application also provides a video processing method, comprising:

[0009] Construct a flow network diagram based on the video clips of the video material to be processed and the video pit positions of the template;

[0010] Determine a matching relationship between the video clip and the video pit position based on the stream network graph;

[0011] Filling the video clip into the corresponding video slot of the template according to the matching relationship to obtain a recommended video;

[0012] The flow network diagram includes a plurality of nodes, each of which corresponds to a matching relationship between a video clip and a video pit.

[0013] In a third aspect, the present application also provides a video processing method, comprising:

[0014] Acquire multiple templates, each template including at least one video slot;

[0015] Matching a video clip to a video position of each template to obtain a matching relationship corresponding to each template, and determining a matching score of the matching relationship corresponding to each template, wherein the video clip is a clip of the video material to be processed;

[0016] determining a recommended template from the plurality of templates according to the matching score;

[0017] The video clip is filled into the corresponding video slot of the recommendation template according to the matching relationship corresponding to the recommendation template to obtain a recommended video.

[0018] In a fourth aspect, the present application also provides a video processing method, comprising:

[0019] According to the video information of the video material to be processed, the video material to be processed is segmented to generate a plurality of video segments;

[0020] Determine, according to the pit position information of the video pit positions of the template, the video segments to be filled into each video pit position in the template, and obtain the matching relationship corresponding to the template;

[0021] The video clip is filled into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video.

[0022] In a fifth aspect, the present application further provides a video processing device, the video processing device comprising a processor and a memory;

[0023] The memory is used to store computer programs;

[0024] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0025] Determine a template according to video information of the video material to be processed, wherein the template includes at least one video pit position;

[0026] Determine a video segment corresponding to the video segment according to the segment information of the video segment in the template, and obtain a matching relationship corresponding to the template, wherein the video segment is a segment in the video material to be processed;

[0027] The video clip is filled into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video.

[0028] In a sixth aspect, the present application further provides a video processing device, the video processing device comprising a processor and a memory;

[0029] The memory is used to store computer programs;

[0030] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0031] Construct a flow network diagram based on the video clips of the video material to be processed and the video pit positions of the template;

[0032] Determine a matching relationship between the video clip and the video pit position based on the stream network graph;

[0033] Filling the video clip into the corresponding video slot of the template according to the matching relationship to obtain a recommended video;

[0034] The flow network diagram includes a plurality of nodes, each of which corresponds to a matching relationship between a video clip and a video pit.

[0035] In a seventh aspect, the present application further provides a video processing device, the video processing device comprising a processor and a memory;

[0036] The memory is used to store computer programs;

[0037] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0038] Acquire multiple templates, each template including at least one video slot;

[0039] Matching a video clip to a video position of each template to obtain a matching relationship corresponding to each template, and determining a matching score of the matching relationship corresponding to each template, wherein the video clip is a clip of the video material to be processed;

[0040] determining a recommended template from the plurality of templates according to the matching score;

[0041] The video clip is filled into the corresponding video slot of the recommendation template according to the matching relationship corresponding to the recommendation template to obtain a recommended video.

[0042] In an eighth aspect, the present application further provides a video processing device, the video processing device comprising a processor and a memory;

[0043] The memory is used to store computer programs;

[0044] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0045] According to the video information of the video material to be processed, the video material to be processed is segmented to generate a plurality of video segments;

[0046] Determine, according to the pit position information of the video pit positions of the template, the video segments to be filled into each video pit position in the template, and obtain the matching relationship corresponding to the template;

[0047] The video clip is filled into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video.

[0048] In a ninth aspect, the present application further provides a terminal device, the terminal device comprising a processor and a memory;

[0049] The memory is used to store computer programs;

[0050] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0051] Determine a template according to video information of the video material to be processed, wherein the template includes at least one video pit position;

[0052] Determine a video segment corresponding to the video segment according to the segment information of the video segment in the template, and obtain a matching relationship corresponding to the template, wherein the video segment is a segment in the video material to be processed;

[0053] The video clip is filled into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video.

[0054] In a tenth aspect, the present application further provides a terminal device, the terminal device comprising a processor and a memory;

[0055] The memory is used to store computer programs;

[0056] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0057] Construct a flow network diagram based on the video clips of the video material to be processed and the video pit positions of the template;

[0058] Determine a matching relationship between the video clip and the video pit position based on the stream network graph;

[0059] Filling the video clip into the corresponding video slot of the template according to the matching relationship to obtain a recommended video;

[0060] The flow network diagram includes a plurality of nodes, each of which corresponds to a matching relationship between a video clip and a video pit.

[0061] In an eleventh aspect, the present application further provides a terminal device, the terminal device comprising a processor and a memory;

[0062] The memory is used to store computer programs;

[0063] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0064] Acquire multiple templates, each template including at least one video slot;

[0065] Matching a video clip to a video position of each template to obtain a matching relationship corresponding to each template, and determining a matching score of the matching relationship corresponding to each template, wherein the video clip is a clip of the video material to be processed;

[0066] determining a recommended template from the plurality of templates according to the matching score;

[0067] The video clip is filled into the corresponding video slot of the recommendation template according to the matching relationship corresponding to the recommendation template to obtain a recommended video.

[0068] In a twelfth aspect, the present application further provides a terminal device, the terminal device comprising a processor and a memory;

[0069] The memory is used to store computer programs;

[0070] The processor is configured to execute the computer program and, when executing the computer program, implement:

[0071] According to the video information of the video material to be processed, the video material to be processed is segmented to generate a plurality of video segments;

[0072] Determine, according to the pit position information of the video pit positions of the template, the video segments to be filled into each video pit position in the template, and obtain the matching relationship corresponding to the template;

[0073] The video clip is filled into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video.

[0074] In a thirteenth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the video processing method as described above.

[0075] The embodiments of the present application provide a video processing method, a video processing apparatus, a terminal device, and a storage medium, which can quickly convert video materials to be processed into films, reduce the workload of video editing, increase the diversity of recommended videos, and improve user experience.

[0076] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1 is a flowchart of the steps of a video processing method provided by an embodiment of the present application;

[0079] Figure 2 yes Figure 1 A sub-step flow chart of a video processing method provided in;

[0080] Figure 3 is a flowchart of the steps of clustering and segmenting the first video clip provided by an embodiment of the present application;

[0081] Figure 4 is a training flow chart of the image feature network model provided in the embodiment of the present application;

[0082] Figure 5 is a schematic diagram of a similarity calculation result provided in an embodiment of the present application;

[0083] Figure 6 is a flowchart of another video processing method provided by an embodiment of the present application;

[0084] Figure 7 is a schematic diagram of a flow network diagram constructed in an embodiment of the present application;

[0085] Figure 8 is a flowchart of another video processing method provided by an embodiment of the present application;

[0086] Fig. 9 is a schematic diagram of the source of the video material to be processed provided in an embodiment of the present application;

[0087] Fig.10 yes Figure 8 A sub-step flow chart of a video processing method provided in;

[0088] Fig.11is a flowchart of steps for determining multiple matching templates according to video tags provided by an embodiment of the present application;

[0089] Fig.12 It is a flowchart of the steps of filling the video clips into the corresponding video pits provided by the embodiment of the present application;

[0090] Fig.13 is a flowchart of another video processing method provided by an embodiment of the present application;

[0091] Fig.14 It is a flowchart of the steps for obtaining a matching relationship for matching a video pit position with a video clip provided in an embodiment of the present application;

[0092] Fig.15 is a flowchart of the steps of determining a recommended template according to a matching score provided in an embodiment of the present application;

[0093] Fig.16 yes Fig.15 A flowchart of sub-steps for determining a recommended template based on a matching score provided in;

[0094] Fig.17 yes Fig.13 A sub-step flow chart of a video processing method provided in;

[0095] Fig.18 is a flowchart of the steps of determining a recommended template according to a template type provided in an embodiment of the present application;

[0096] Fig.19 is a schematic diagram of selecting a recommended template from a template group provided in an embodiment of the present application;

[0097] Fig. 20 is a schematic block diagram of a video processing device provided in an embodiment of the present application;

[0098] Fig.21 It is a schematic block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0100] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0101] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0102] See also Figure 1 , Figure 1 1 is a flowchart of the steps of a video processing method provided in an embodiment of the present application. The video processing method can be applied in a terminal device or a cloud device to synthesize the video material to be processed with a preset template. Among them, the terminal device includes a mobile phone, a tablet, and a laptop computer.

[0103] Specifically, Figure 1 As shown, the video processing method includes steps S101 to S103.

[0104] S101. Segment the video material to be processed to generate a plurality of video segments according to video information of the video material to be processed.

[0105] The video material to be processed is used to be synthesized with a preset template to generate a recommended video for the user, and the preset template includes at least one video slot for filling a video clip. The video material to be processed is segmented according to the video information of the video material to be processed to obtain a plurality of video clips, so that the video clips are filled into the video slots of the template to generate a recommended video.

[0106] In one embodiment, the video material to be processed may include video materials from multiple sources, such as video materials shot by a handheld terminal, video materials shot by a mobile platform, video materials obtained from a cloud server, and video materials obtained from a local server.

[0107] The handheld terminal may be, for example, a mobile phone, a tablet, a sports camera, etc., and the movable platform may be, for example, a drone, etc. The drone may be a rotor drone, such as a quad-rotor drone, a hexacopter drone, an octo-rotor drone, or a fixed-wing drone. The drone is equipped with a camera.

[0108] By aggregating video materials from different sources and editing them together through mixed editing, the video diversity of the recommended videos is increased.

[0109] In one embodiment, the video information includes at least one of the camera direction and scene information. Figure 2 , step S101 includes step S1011 and step S1012.

[0110] S1011. Segment the video material to be processed according to video information of the video material to be processed to obtain a plurality of first video segments.

[0111] When the video material to be processed is initially segmented according to the camera movement direction of the video material to be processed, the camera movement direction in each first video segment obtained is the same or similar, and there is no change in the camera movement direction.

[0112] Specifically, the video material to be processed can be segmented according to the change of the camera movement direction in the video material to be processed to generate multiple first video segments. For example, the video material to be processed includes continuous front-to-back camera movement, and the video material to be processed can be segmented into two first video segments according to the change of the front-to-back camera movement direction, one of the first video segments has a forward camera movement direction, and the other first video segment has a backward camera movement direction, and the camera movement direction in each first video segment is the same. In the specific implementation process, some algorithms for detecting the camera movement direction can be used to determine the change of the camera movement direction in the video material to be processed.

[0113] When the video material to be processed is initially segmented according to the scene information of the video material to be processed, the scenes in each first video segment are similar. For example, if the video material to be processed includes a snow mountain image, the video material to be processed can be segmented into a plurality of first video segments according to whether the video material to be processed includes the snow mountain image and images similar to the snow mountain image.

[0114] In addition, when segmenting the video material to be processed, the video material to be processed may also be segmented according to the same subject in the video material to be processed. The subject may be a target person or a pet, etc. For example, if a cat appears continuously in the video material to be processed, the video material to be processed may be segmented according to whether the cat is in the shot of the video material to be processed, and the video material to be processed may be segmented into a plurality of first video segments.

[0115] S1012: Cluster and segment the first video segment to obtain a plurality of second video segments.

[0116] After obtaining the first video segment, the first video segment can be segmented for a second time to obtain multiple second video segments, and the second video segments are used as video segments to be filled in the video slots of the template. The cluster segmentation includes clustering using similar scenes.

[0117] It should be noted that the first video segment can be selectively clustered and segmented, that is, when the first video segment meets certain conditions, the first video segment is clustered and segmented, otherwise, the first video segment may not be clustered and segmented.

[0118] In one embodiment, before clustering and segmenting the first video segments to obtain multiple second video segments, it is determined whether there is a first video segment with a video duration greater than a preset duration among the multiple first video segments; if there is a first video segment with a video duration greater than the preset duration, the step of clustering and segmenting the first video segments is performed.

[0119] Before clustering and segmenting the first video segments, it is determined whether there is a first video segment whose video duration is greater than a preset duration among the obtained multiple first video segments. The preset duration is an empirical value and can be adjusted according to experience or the duration of the video slot of the template.

[0120] If at least one of the plurality of first video segments has a video duration longer than a preset duration, the first video segments longer than the preset duration are segmented for the second time. For first video segments whose video duration is less than or equal to the preset duration, the second segmentation may not be performed.

[0121] In one embodiment, the step of clustering the first video segment is as follows: Figure 3 , specifically including steps S1012a to S1012c.

[0122] S1012a. Determine the sliding window and cluster center.

[0123] When clustering and segmenting the first video clip, first determine a sliding window and a cluster center, wherein the sliding window is used to determine a current video frame to be processed, and the cluster center is used to determine a video segmentation point of the first video clip. In the process of clustering and segmenting the first video clip, the cluster center includes image features of a first video frame of the first video clip.

[0124] The image features of the first video frame of the first video clip may include image coding features of the first video frame of the first video clip. The image features are obtained according to a pre-trained image feature network model. The image feature network model can output image features of each video frame in the first video clip.

[0125] Specifically, for a first video clip, the larger the sliding window is, the fewer video frames are processed for similar clustering during cluster segmentation, and the faster the segmentation speed is; the smaller the sliding window is, the more video frames are processed for similar clustering during cluster segmentation, and the slower the segmentation speed is. Therefore, the size of the sliding window can be set based on this principle. In one embodiment, the size of the sliding window is equal to 1.

[0126] In one embodiment, the size of the sliding window is related to the duration of the first video segment. When the duration of the first video segment is long, in order to quickly perform cluster segmentation on the first video segment, a larger value may be set for the sliding window to increase the speed of cluster segmentation, such as setting the size of the sliding window to 3. When the duration of the first video segment is short, a smaller value may be set for the sliding window, such as setting the size of the sliding window to 1.

[0127] In one embodiment, the size of the sliding window is related to the desired segmentation speed set by the user. When the user expects to segment the first video segment quickly, a larger value can be set for the sliding window to increase the segmentation speed. When the user expects to segment the first video segment slowly, a smaller value can be set for the sliding window to reduce the segmentation speed.

[0128] In one embodiment, the size of the sliding window is related to the fineness of the segmentation. The smaller the sliding window, the more video frames are processed during cluster segmentation and the finer the segmentation. Therefore, the size of the sliding window can be set according to the demand for the fineness of the segmentation.

[0129] In one embodiment, the image feature network model can be pre-trained. The training flow chart can be Figure 4 As shown, the training process can be:

[0130] A training set is prepared, and the video materials in the training set are labeled, and the video materials of the same scene in the training set are labeled as one category. During training, three video materials are respectively input into three convolutional neural networks, and the convolutional neural network can be, for example, a CNN network, wherein two of the three video materials are of the same category and one video material is of another category.

[0131] The convolutional neural network is used to generate image features of three video materials, and the triplet loss is used to measure whether the image distance between video materials of the same category is smaller than the image distance between video materials of different categories, thereby generating a corresponding loss value. The convolutional neural network is iteratively trained based on the loss value, and the weights of the convolutional neural network are continuously adjusted so that the convolutional neural network can learn discriminative image features.

[0132] After the convolutional neural network training is completed, the trained convolutional neural network is used as an image feature network model to output image features of each video frame in the first video clip.

[0133] S1012b: Based on the cluster center, perform cluster analysis on the video frames of the first video segment according to the sliding window to determine video segmentation points.

[0134] After the cluster center is determined, the current video frame to be processed is determined according to the sliding window, and then the current video frame to be processed and the cluster center are clustered to determine the video segmentation point.

[0135] For example, in the first clustering, the cluster center is set to the image feature of the first video frame, and the first sliding is performed. When the sliding window is 1, the current video frame to be processed in the first sliding is the second video frame, when the sliding window is 2, the current video frame to be processed in the first sliding is the third video frame, and when the sliding window is N, the current video frame to be processed in the first sliding is the N+1th video frame.

[0136] In one embodiment, step S1012b is specifically as follows: determining the current video frame according to the sliding window, and determining the similarity between the image features of the current video frame and the cluster center; if the similarity is less than a preset threshold, using the current video frame as a video segmentation point, and redetermining the cluster center; based on the redetermined cluster center, continuing to determine the video segmentation point until the last video frame of the first video clip.

[0137] Specifically, when clustering the first video clip for the first time, the cluster center is initialized and the first sliding is performed. The cluster center is set to the image feature of the first video frame of the first video clip, which is recorded as C0. The size of the sliding window is recorded as N. At this time, the current video frame to be processed determined in the first sliding is the N+1th video frame of the first video clip, and the current video frame to be processed determined in the mth sliding is the m*N+1th video frame of the first video clip. Wherein, m is the number of sliding times of the sliding window.

[0138] After the first slide, the current video frame to be processed, that is, the N+1th frame of the first video clip, is input into the pre-trained image feature network model to obtain the image feature F of the current frame. N+1 , calculate the image feature C0 of the cluster center and the image feature F of the current frame N+1 The similarity between .

[0139] If the similarity is less than the preset threshold, it is considered that the video content between the first frame of the first video segment and the N+1th frame of the first video segment has a large change, and the N+1th frame of the first video segment can be used as the video segmentation point, and the video frames before the N+1th frame of the first video segment can be cut into a second video segment. Then, the image feature F of the N+1th frame of the first video segment is used as the segmentation point. N+1 As the newly determined cluster center, continue to determine the next video segmentation point until reaching the last video frame of the first video segment.

[0140] The similarity can be the cosine similarity between the image features, and the calculated cosine similarity between the image features is used as the image feature C0 of the cluster center and the image feature F of the current frame. N+1 The preset threshold value may be a preset empirical value.

[0141] In one embodiment, the re-determining the cluster center includes: using the image features of the current video frame as the re-determined cluster center.

[0142] When the similarity is less than a preset threshold, the current video frame is used as a video segmentation point, and the video frame before the current video frame is cut into a second video segment. At this time, when clustering the remaining first video segments, since the current video frame is the first frame of the remaining first video segments, the image features of the current video frame are used as the re-determined cluster center.

[0143] In one embodiment, after determining the similarity between the image features of the current video frame and the cluster center, if the similarity is greater than or equal to a preset threshold, the cluster center is updated; and the similarity between the image features of the current video frame and the updated cluster center is continued to be determined based on the updated cluster center.

[0144] If after the first slide, the image feature C0 of the cluster center and the image feature F of the current frame are calculated N+1 If the similarity between them is greater than a preset threshold, it is considered that the video content between the first frame of the first video clip and the N+1th frame of the first video clip does not change much, then the cluster center is updated, and the similarity between the image features of the current video frame to be processed at this time and the updated cluster center is determined based on the updated cluster center.

[0145] For example, after the first slide, the average value of image features from the first video frame to the N+1th video frame can be calculated based on the image features of the first frame of the first video clip and the image features of the N+1th frame of the first video clip, and the calculated average value is used as the image feature of the updated cluster center.

[0146] In one embodiment, the updating of the cluster center includes: acquiring image features of the current video frame; and determining an updated cluster center according to the image features of the current video frame and the cluster center.

[0147] The image features of the current video frame are obtained, and then the average value of the image features from the first video frame to the current video frame is calculated based on the image features of the cluster center and the image features of the current video frame, and the calculated average value is used as the updated image features of the cluster center.

[0148] Among them, the formula for determining the updated cluster center can be:

[0149]

[0150] Among them, m is the number of clustering, N is the size of the sliding window, and C m is the image feature of the cluster center C after the mth clustering, F m*N+1 is the image feature of the current video frame during the m-th clustering.

[0151] For example, there are 10 video frames in the first video clip. When the value of the sliding window N is 1, when the first video clip is clustered for the first time, the cluster center C is initialized to the first frame, and the image feature of the cluster center is C0, and the first sliding is performed. At this time, the current video frame determined by the first sliding is the second frame, and the image feature of the second frame is F2. The similarity between the image feature C0 of the cluster center and the image feature F2 of the second frame is compared. If the similarity is greater than the preset threshold, the cluster center is re-determined. At this time, the image feature C1 of the cluster center C after the first clustering is (C0*1*1+F2) / 2.

[0152] Then the second clustering is started, and the second sliding is performed. At this time, the current video frame determined by the second sliding is the third frame, and the image feature of the third frame is F3. The similarity between the image feature C1 of the cluster center C after the first clustering and the image feature F3 of the third frame is compared. If the similarity is greater than the preset threshold, the cluster center is re-determined. At this time, the image feature C2 of the cluster center C after the second clustering = (C1*2*1+F3) / 3. This process is repeated until the 10th video frame of the first video clip.

[0153] S1012c. Segment the first video segment according to the video segmentation points.

[0154] After determining the video segmentation point, the first video segment is segmented according to the video segmentation point to obtain multiple second video segments. When segmenting the video, the current video frame is used as the video segmentation point, and the video frame before the current video frame is used as a second video segment for segmentation.

[0155] For example, if Figure 5 The figure shows a schematic diagram of the similarity calculation result. The first video segment has 2300 frames, the 490th frame is a video segmentation point, and the 1100th frame is a video segmentation point. When performing video segmentation, the first video segment is segmented at the 490th frame and the 1100th frame, respectively, to obtain three second video segments, namely, the 1st frame to the 489th frame of the first video segment as a second video segment, the 490th frame to the 1099th frame as a second video segment, and the 1100th frame to the 2300th frame as a second video segment.

[0156] S102: Determine, according to the slot information of the video slots of the template, the video clips to be filled into each video slot in the template, and obtain the matching relationship corresponding to the template.

[0157] The matching relationship corresponding to the template is the corresponding relationship between the video slot in the template and the corresponding video clip.

[0158] After determining the template corresponding to the video material to be processed, since the template includes at least one video position, the video segment corresponding to the video position can be determined according to the position information of the video position in the template, thereby obtaining a matching relationship corresponding to the template.

[0159] Specifically, the pit information includes at least one of pit music and pit labels. In some embodiments, the pit label can be pre-set, and each video pit can be pre-set with a pit label.

[0160] Among them, the pit label includes at least one of the camera direction, scene size, pit theme, video theme of the video clip to be filled in the video pit, target object size and position of a single video frame in the video clip to be filled in the video pit, target object size and position of continuous video frames in the video clip to be filled in the video pit, and similarity of adjacent video frames in the video clip to be filled in the video pit.

[0161] The pit music of a video pit can be a fragment of the template music of the entire template. Since the template includes multiple video pits, multiple video pits are combined into one template in sequence. Therefore, for the template music of the template, the template music can be split according to the order and duration of the video pits to obtain the pit music corresponding to each video pit.

[0162] In one embodiment, the template may be segmented into a plurality of video slots, each of which needs to be filled with a video clip. The template may be segmented according to the template music of the template, or according to the segmented plurality of video clips.

[0163] For example, the template music of the template may be divided into multiple segments according to a certain time interval, each segment of music corresponds to a video position, and the duration of the segment of music is equal to the duration of the video position.

[0164] For another example, the music structure of the template music of the template can be divided into multiple sections according to the main song, chorus, transition sentence, popular sentence, interlude, etc. Each section of music corresponds to a video slot, and the length of the music section is equal to the length of the video slot.

[0165] For another example, the rhythm of the template music of the template can be divided into multiple sections, each section of music corresponds to a video position, and the duration of the section of music is equal to the duration of the video position.

[0166] Specifically, since the durations of multiple video segments obtained by segmenting the video material to be processed may be different, the template can be segmented according to the duration of the video segments so that the duration of the segmented video pits can be exactly equal to the duration of the video segments.

[0167] For example, when segmenting the video material to be processed, the 30-second video material to be processed is segmented into three video clips, and the durations of the three video clips are 5 seconds, 15 seconds, and 10 seconds respectively. At this time, the template can be segmented into corresponding video slots according to the durations of the three video clips, so that the video clips can be filled into the corresponding video slots.

[0168] When the template is segmented according to the duration of the video clip, since there are multiple video clips, the template can be segmented according to the wonderful levels of the video clips.

[0169] For example, a template must be filled with at least one video clip with the highest level of excitement. Therefore, at least one video slot with the same length as the video clip with the highest level of excitement can be segmented out in the template to fill in the video clip with the highest level of excitement.

[0170] In another embodiment, the template may be segmented according to the camera movement, scene or scene of the template.

[0171] In one embodiment, determining the video clip corresponding to the video pit position according to the pit position information of the video pit position in the template includes: determining the video clip corresponding to the video pit position according to the pit position music of the video pit position in the template.

[0172] Since the pit music of the video pit itself has a sense of rhythm, for example, the pit music of the video pit has a soothing rhythm or an explosive rhythm. Therefore, a suitable video clip can be matched to the video pit according to the rhythm of the pit music and the video content of the video clip.

[0173] For example, if the music is explosive, and there are powerful bursts of images such as fountains in the video clip, then the two are a good match. In addition, if the rhythm is slow, it is suitable for slow motion of characters; if it is an epic blockbuster, it is suitable for time-lapse photography of large scenes, etc.

[0174] In one embodiment, determining the video clip corresponding to the video pit position according to the pit position music of the video pit position in the template includes: determining the degree of matching between the pit position music of the video pit position in the template and the video clip; and determining the video clip corresponding to the video pit position in the template according to the matching degree.

[0175] The rhythm of the pit music of the video pit is matched with the video clip to determine the degree of matching between the pit music of the video pit and the video clip, thereby determining the video clip corresponding to the video pit in the template according to the degree of matching.

[0176] Specifically, a threshold of the degree of matching may be set, and when the degree of matching exceeds the threshold, it is determined that the video position matches the video clip; or, based on the degree of matching, the video clip with the highest degree of matching with the video position may be selected as the video clip corresponding to the video position.

[0177] In one embodiment, the degree of matching between the pit music of the video pit in the template and the video clip is obtained by using a pre-trained music matching model, and the music matching model can output the matching score between the pit music of the video pit in the template and the video clip.

[0178] The matching degree between the pit music and the video clip is learned by training a neural network, and then the trained neural network is used as a music matching model to output the matching degree score between the pit music and the video clip of the video pit.

[0179] In one embodiment, after determining the video segment corresponding to the video pit position according to the pit position information of the video pit position in the template, the shooting quality of multiple video segments corresponding to the video pit position in the template is determined; the optimal video segment corresponding to the video pit position in the template is determined according to the shooting quality of the multiple video segments; and the matching relationship corresponding to the template is obtained according to the optimal video segment corresponding to the video pit position in the template.

[0180] The shooting quality of the video clip is determined according to the image content of the video clip and the video clip evaluation. The image content includes lens stability, color saturation, whether there is a main shooting object and the amount of information in the lens, etc. The video clip evaluation includes the aesthetic score of the video clip.

[0181] Specifically, the aesthetic scoring of the video clip can take into account factors such as color, composition, camera movement and scene type to score the video clip aesthetically. The shooting quality of the video clip is scored based on the image content of the video clip and the video clip evaluation. The larger the score, the higher the shooting quality of the video clip.

[0182] According to the shooting qualities of multiple video clips, the video clip with the highest shooting quality is selected from the video clips as the optimal video clip corresponding to the video pit position in the template, so as to obtain the matching relationship corresponding to the template.

[0183] In one embodiment, after determining the video segment corresponding to the video pit according to the pit information of the video pit in the template, determine the matching degree between the video segments corresponding to two adjacent video pits in the template; determine the optimal video segment corresponding to the video pit according to the matching degree; and obtain the matching relationship corresponding to the template according to the optimal video segment corresponding to the video pit.

[0184] The matching degree of the video clips corresponding to the two adjacent video slots is determined according to the continuity of the camera direction of the video clips, the increasing and decreasing relationship of the scene scale, and the matching editing.

[0185] The continuity of the camera direction includes ensuring that the camera directions of the video clips corresponding to two adjacent video slots are in the same direction, and avoiding the video clips with opposite camera directions being connected together. The increasing and decreasing relationship of the scene size includes, for example, from long shot to medium shot to close shot, or from close shot to medium shot to long shot, or directly from long shot to close shot, etc. Matching editing includes matching and connecting two shots through similar actions, graphics, colors, etc. to achieve a coherent and smooth narrative, which is used for the transition between two video clips.

[0186] In one embodiment, determining the video segment corresponding to the video pit according to the pit position information of the video pit in the template includes: determining the video segment corresponding to the video pit according to the pit position label of the video pit in the template.

[0187] Each video slot in the template is set with a slot label. The labels can be matched according to the slot labels of the video slots in the template and the labels of the video clips, so that the successfully matched video clips are used as the video clips corresponding to the video slots.

[0188] In one embodiment, determining the video segment corresponding to the video pit according to the pit label of the video pit in the template includes: determining the video tag of the video segment, and taking the video segment corresponding to the video tag that matches the pit label of the video pit as the video segment to be filled in the video pit.

[0189] Labels are extracted from the video clips to determine the video labels of the video clips, and then the video clips corresponding to the video positions are determined according to the position labels of the video positions of the template.

[0190] S103: Fill the video clip into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video.

[0191] After the matching relationship is obtained, the video clips are respectively filled into the corresponding video slots of the template according to the matching relationship, and the video is synthesized to obtain the recommended video. The recommended video is then recommended to the user.

[0192] In one embodiment, step S103 specifically determines whether the video duration of the video clip is greater than the duration of the video pit; if the video duration of the video clip is greater than the duration of the video pit, extract the video clip to obtain a selected clip.

[0193] When the video clip is filled into the corresponding video slot according to the matching relationship, it is determined whether the video duration of the video clip is greater than the duration of the video slot. The duration of the video slot determines the maximum duration of the video clip that can be filled into the video slot. Therefore, when the video duration of the video clip is greater than the duration of the video slot, the video clip cannot be directly filled into the corresponding video slot, and a clip of the corresponding duration needs to be extracted from the video clip and filled into the corresponding video slot.

[0194] The video duration of the selected segment is less than or equal to the duration of the video slot. In the specific implementation process, in order to fill the selected segment into the corresponding video slot and ensure the completeness of the recommended video, when determining the selected segment, the video duration of the selected segment can be equal to the duration of the video slot.

[0195] In one embodiment, extracting segments from the video segments to obtain selected segments includes: extracting segments from the video segments to obtain selected segments according to video elements of the video segments.

[0196] When extracting segments from video segments, the video segments may be extracted according to video elements of the video segments to obtain selected segments.

[0197] The video elements include at least one of a smiling face picture, a laughing sound audio, a character action, a clear human voice, a picture composition and an aesthetic score. When extracting the selected clips, more exciting clips can be extracted from the video clips according to the video elements as the selected clips, for example, clips including smiling faces or clips with higher aesthetic scores can be selected as the selected clips.

[0198] In one embodiment, step S103 specifically includes filling the video clip into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain an initial video; and optimizing the image of the initial video based on the template requirements of the template to obtain a recommended video.

[0199] After the video clips are filled into the corresponding video slots according to the matching relationship corresponding to the template, an initial video is obtained, and then the image of the initial video can be optimized according to the template requirements, and the image-optimized video is recommended to the user as a recommended video. The template requirements include at least one of transition settings, acceleration and deceleration settings, and texture special effects settings.

[0200] In one embodiment, since the source of the video material to be processed may be an aerial video taken by a drone, and the distance between the camera and the object being photographed is relatively far during aerial photography, the content of the picture changes is small. Therefore, when the aerial video is filled into the corresponding video slot, the speed of picture changes can be automatically identified first, and the aerial video can be automatically speed-adjusted according to the speed of picture changes, and then the aerial video after speed adjustment can be filled into the corresponding video slot. The speed of picture changes can be obtained by analyzing multiple consecutive frames within a preset time.

[0201] In one embodiment, when determining the matching relationship between the video slots of the template and the video clips, for the identified aerial video, the aerial video can be placed in the first few video slots and / or the last few video slots of the template, thereby improving the quality of the recommended video.

[0202] The video processing method provided in the above embodiment divides the video material to be processed to obtain multiple video clips, and then determines the video clips to be filled in each video slot in the template according to the slot information of the video slot of the template, obtains the matching relationship corresponding to the template, and finally fills the video clips into the corresponding video slot according to the matching relationship to obtain the recommended video. The video material to be processed is divided, and the video material to be processed with a longer duration is divided into multiple video clips with shorter duration, so that the video clips can be smoothly filled in the video slots to synthesize the recommended video.

[0203] See also Figure 6 , Figure 6 It is a step flow chart of another video processing method provided in an embodiment of the present application.

[0204] Specifically, Figure 6 As shown, the video processing method includes steps S201 to S203.

[0205] S201, constructing a flow network diagram according to the video clips of the video material to be processed and the video positions of the template.

[0206] The video material to be processed includes multiple video clips, and the template includes at least one video slot, and each video slot needs to be filled with a video clip.

[0207] In one embodiment, the above-mentioned video segmentation method may also be used to segment the video material to be processed into multiple video segments.

[0208] The flow network diagram includes a plurality of nodes, each of which corresponds to a matching relationship between a video clip and a video pit.

[0209] Construct a flow network diagram based on video clips and template pit positions, such as Figure 7 As shown in the figure, it is a schematic diagram of the constructed flow network diagram. m Indicates video clips, with the upper horizontal axis S n Indicates the video position of the template. Add source nodes and endpoints in the flow network diagram, where S nodes are source nodes and T nodes are endpoints. In the flow network diagram, the coordinates (x, y) represent the nodes on the horizontal axis x and the vertical axis y. For example, N(1,1) represents the node on the upper left (S1,C1), and N(n,m) represents the node on the lower right (S n ,C m ) node.

[0210] It should be noted that the number of templates can be one or more. When the number of templates is one, a flow network diagram is constructed, and when the number of templates is more than one, a flow network diagram is constructed for each template.

[0211] S202: Determine a matching relationship between the video clip and the video position based on the flow network graph.

[0212] After drawing the flow network diagram, the matching relationship between the video clips in the template and the corresponding video slots can be determined based on the nodes in the flow network diagram. The source node in the flow network diagram is used as the starting point of the path, and the path passes through a node corresponding to each video slot. The end point of the flow network diagram is used as the end point of the path. The path is the matching relationship between the video clips in the template and the corresponding video slots.

[0213] In one embodiment, determining the matching relationship between the video clip and the video pit position based on the flow network diagram includes: matching a suitable video clip to the video pit position of the template based on a maximum flow algorithm to obtain an optimal path; wherein the corresponding relationship between the video clip and the video pit position in the optimal path is used as the matching relationship between the video pit position of the template and the video clip.

[0214] See also Figure 7 , Figure 7The arrow in represents the path between two adjacent nodes in the flow network diagram. A path from the source node S to the end point T is a matching relationship between the video clip and the video pit position.

[0215] The maximum flow algorithm refers to the process of sequentially placing video slots S1 to S n The n video slots are filled with appropriate video clips to maximize the total energy value of the entire path from the source node S to the end point T. The video slots using the maximum flow algorithm as a template are matched with appropriate video clips, thereby converting the video clip selection problem into the problem of finding the maximum energy from the source node S to the end point T.

[0216] In one embodiment, the maximum flow algorithm is used to match the video pit of the template with a suitable video clip to obtain the optimal path, including: determining the optimal path corresponding to the template according to the energy value between two adjacent nodes in the flow network diagram.

[0217] Among them, V can be used to represent the energy value of any path in the flow network graph, and any path includes the path between two nodes. For example, V((x,y),(x+1,y+k)) represents the energy value from node (x,y) to node (x+1,y+k). By calculating the energy value between every two adjacent nodes in a path from the source node S to the end point T, the route with the largest total energy value is taken as the optimal path, and the optimal path corresponding to the template is determined.

[0218] In one embodiment, the energy value between two adjacent nodes may be determined according to the energy value influence factor of each of the nodes.

[0219] Among them, the energy value influencing factors include at least one of the shooting quality of the video clip corresponding to each of the video pits, the matching degree between each of the video pits and the corresponding video clip, and the matching degree between the video clips corresponding to two adjacent video pits.

[0220] Specifically, the shooting quality of the video clip corresponding to each of the video pits is determined according to the image content of the video clip and the video clip evaluation.

[0221] The image content includes lens stability, color saturation, whether there is a main object being photographed, and the amount of information in the lens, and the video clip evaluation includes the aesthetic scoring of the video clip. The aesthetic scoring of the video clip can take into account factors such as color, composition, camera movement, and scene type to score the video clip aesthetically. The shooting quality of the video clip is scored based on the image content and video clip evaluation of the video clip. The larger the score, the higher the shooting quality of the video clip.

[0222] Specifically, the degree of matching between each of the video pits and the corresponding video clip is determined according to the degree of matching between the pit music of the video pit and the video clip.

[0223] Among them, template music is preset in the template, and the pit music of the video pit can be a fragment of the template music of the entire template. Since the template includes multiple video pits, multiple video pits are combined into one template in sequence. Therefore, for the template music of the template, the template music can be split according to the order and duration of the video pits to obtain the pit music corresponding to each video pit.

[0224] The music itself has a sense of rhythm, and the pit music clip corresponding to each video pit in the template also has a corresponding sense of rhythm, for example, the pit music has a soothing rhythm or an explosive rhythm. Therefore, the template pit and the video clip can be matched according to the rhythm of the pit music and the video content of the video clip.

[0225] For example, if the music is explosive, and there are powerful bursts of images such as fountains in the video clip, then the two are a good match. In addition, if the rhythm is slow, it is suitable for slow motion of characters; if it is an epic blockbuster, it is suitable for time-lapse photography of large scenes, etc.

[0226] In one embodiment, the degree of matching between each of the video pits and the corresponding video clip is obtained using a pre-trained music matching model, and the music matching model can output a matching score between the pit music of the video pit and the video clip.

[0227] Among them, the matching degree between the pit music and the video clip can be learned by training a neural network, and then the trained neural network is used as a music matching model to output the matching degree score between the pit music and the video clip of the video pit.

[0228] The matching degree of the video clips corresponding to the two adjacent video slots is determined according to the continuity of the camera direction of the video clips, the increasing and decreasing relationship of the scene scale, and the matching editing.

[0229] The continuity of the camera direction includes ensuring that the camera directions of the video clips corresponding to two adjacent video slots are in the same direction, and avoiding the video clips with opposite camera directions being connected together. The increasing and decreasing relationship of the scene size includes, for example, from long shot to medium shot to close shot, or from close shot to medium shot to long shot, or directly from long shot to close shot, etc. Matching editing includes matching and connecting two shots through similar actions, graphics, colors, etc. to achieve a coherent and smooth narrative, which is used for the transition between two video clips.

[0230] In one embodiment, the matching degree of the video clips corresponding to the two adjacent video pits is obtained by using a pre-trained clip matching model, and the clip matching model can output the matching degree of the video clips filled in the two adjacent video pits.

[0231] Among them, the matching degree between the video clips corresponding to two adjacent video slots can be learned by training a neural network, and then the trained neural network can be used as a clip matching model to output the matching degree score between the video clips corresponding to two adjacent video slots.

[0232] In one embodiment, determining the energy value between two adjacent nodes based on the energy value influence factor of each node includes: obtaining the evaluation score and preset weight of the energy value influence factor; and determining the energy value between two adjacent nodes based on the evaluation score and preset weight of the energy value influence factor.

[0233] For each node in the flow network diagram, the evaluation score and the corresponding preset weight of the energy value influencing factor of each node are obtained to determine the energy value between two adjacent nodes. The preset weight includes the weights corresponding to different energy value influencing factors, which can be pre-set based on experience values.

[0234] For example, the energy value V in any path in the flow network diagram is calculated as:

[0235] V=a*E clip +b*E template +c*E match

[0236] Among them, E clip represents the score of the shooting quality of the video clip corresponding to the video slot, a represents the preset weight corresponding to the score of the shooting quality of the video clip corresponding to the video slot, and E template represents the matching degree between the video position and the corresponding video clip, b represents the preset weight corresponding to the matching degree between the video position and the corresponding video clip, and E match represents the sum of the matching degrees of the video clips corresponding to two adjacent video slots, and c represents the preset weight corresponding to the matching degrees of the video clips corresponding to two adjacent video slots.

[0237] It should be noted that, when calculating the energy value between two adjacent nodes, if the duration of the video clip is less than the duration of the filled video slot, the energy value of the node is set to 0.

[0238] S203: Fill the video clip into the corresponding video slot of the template according to the matching relationship to obtain a recommended video.

[0239] After determining the matching relationship between the video clip and the video slot, the video clip is filled into the corresponding video slot according to the matching relationship, thereby synthesizing a recommended video, and recommending the recommended video to the user.

[0240] In one embodiment, step S203 specifically determines whether the video duration of the video clip is greater than the duration of the video pit; if the video duration of the video clip is greater than the duration of the video pit, extract the video clip to obtain a selected clip.

[0241] When the video clip is filled into the corresponding video slot according to the matching relationship, it is determined whether the video duration of the video clip is greater than the duration of the video slot. The duration of the video slot determines the maximum duration of the video clip that can be filled into the video slot. Therefore, when the video duration of the video clip is greater than the duration of the video slot, the video clip cannot be directly filled into the corresponding video slot, and a clip of the corresponding duration needs to be extracted from the video clip and filled into the corresponding video slot.

[0242] The video duration of the selected segment is less than or equal to the duration of the video slot. In the specific implementation process, in order to fill the selected segment into the corresponding video slot and ensure the completeness of the recommended video, when determining the selected segment, the video duration of the selected segment can be equal to the duration of the video slot.

[0243] In one embodiment, extracting segments from the video segments to obtain selected segments includes: extracting segments from the video segments to obtain selected segments according to video elements of the video segments.

[0244] When extracting segments from video segments, the video segments may be extracted according to video elements of the video segments to obtain selected segments.

[0245] The video elements include at least one of a smiling face picture, a laughing sound audio, a character action, a clear human voice, a picture composition and an aesthetic score. When extracting the selected clips, more exciting clips can be extracted from the video clips according to the video elements as the selected clips, for example, clips including smiling faces or clips with higher aesthetic scores can be selected as the selected clips.

[0246] In one embodiment, step S203 specifically includes filling the video clip into the corresponding video slot of the template according to the matching relationship to obtain an initial video; and optimizing the image of the initial video based on the template requirements of the template to obtain a recommended video.

[0247] After the video clips are filled into the corresponding video slots according to the matching relationship, an initial video is obtained, and then the image of the initial video can be optimized according to the template requirements, and the image-optimized video is recommended to the user as a recommended video. The template requirements include at least one of transition settings, acceleration and deceleration settings, and texture special effects settings.

[0248] In one embodiment, since the source of the video material to be processed may be an aerial video taken by a drone, and the distance between the camera and the object being photographed is relatively far during aerial photography, the content of the picture changes is small. Therefore, when the aerial video is filled into the corresponding video slot, the speed of picture changes can be automatically identified first, and the aerial video can be automatically speed-adjusted according to the speed of picture changes, and then the aerial video after speed adjustment can be filled into the corresponding video slot. The speed of picture changes can be obtained by analyzing multiple consecutive frames within a preset time.

[0249] In one embodiment, when determining the matching relationship between the video slots of the template and the video clips, for the identified aerial video, the aerial video can be placed in the first few video slots and / or the last few video slots of the template, thereby improving the quality of the recommended video.

[0250] The above embodiment provides a video processing method, which uses video clips and template slots to construct a flow network graph, then determines the matching relationship between the video clips and the video slots based on the flow network graph, and finally fills the video clips into the corresponding video slots of the template according to the matching relationship to obtain recommended videos. The matching relationship between the video clips and the video slots is determined by constructing a flow network graph, and the video clip selection problem is modeled as a maximum flow problem, which improves the accuracy and convenience of the matching relationship between the determined video clips and the video slots.

[0251] See also Figure 8 , Figure 8 It is a step flow chart of another video processing method provided in an embodiment of the present application.

[0252] Specifically, Figure 8 As shown, the video processing method includes steps S301 to S303.

[0253] S301. Determine a template according to video information of a video material to be processed.

[0254] The template includes at least one video slot, the video information of the video material to be processed includes the video content of the video material to be processed, and the corresponding template is determined according to the video content of the video material to be processed. In the specific implementation process, the number of templates can be one or more.

[0255] Specifically, Fig. 9As shown, the video material to be processed may include video materials from various sources, such as video materials shot by a handheld terminal, video materials shot by a mobile platform, video materials obtained from a cloud server, and video materials obtained from a local server.

[0256] The handheld terminal may be, for example, a mobile phone, a tablet, a sports camera, etc., and the movable platform may be, for example, a drone, etc. The drone may be a rotor drone, such as a quad-rotor drone, a hexacopter drone, an octo-rotor drone, or a fixed-wing drone. The drone is equipped with a camera.

[0257] By aggregating video materials from different sources and editing them together through mixed editing, the video diversity of the recommended videos is increased.

[0258] In one embodiment, since the source of the video material to be processed can be an aerial video taken by a drone, when the video material to be processed includes an aerial video, when determining the template based on the video information of the video material to be processed, it is possible to first identify whether the video material to be processed is an aerial material. When it is identified that the video material to be processed is an aerial material, a template of the aerial theme is matched to the video material to be processed, thereby completing the determination of the template.

[0259] In one embodiment, when determining a template based on video information of a video material to be processed, template features of the template and video features of the video material to be processed may be extracted, and a template may be determined for the video material to be processed by matching the template features with the video features.

[0260] Among them, the template features can be the template theme, camera direction, scene size, target object size and position of a single video frame in the video material to be processed that needs to be filled in the template, target object size and position of continuous video frames in the video material to be processed that needs to be filled in the template, and the similarity of adjacent video frames in the video material to be processed that needs to be filled in the template, etc.

[0261] The video features of the video material to be processed may include video theme, camera direction, scene size, target object size and position of a single video frame in the video material to be processed, target object size and position of consecutive video frames in the video material to be processed, and similarity between adjacent video frames in the video material to be processed.

[0262] Exemplarily, a feature matching model may be pre-trained to match template features with video features, and a template may be determined for the video material to be processed based on the matching results output by the feature matching model.

[0263] In one embodiment, see Fig.10 , step S301 includes step S3011 and step S3012.

[0264] S3011. Determine a video tag of the video material to be processed according to the video information of the video material to be processed.

[0265] The video tag of the video material to be processed is extracted, and the video tag of the video material to be processed is determined according to the video information of the video material to be processed.

[0266] Among them, the video tags include at least one of the camera direction, scene size, target size and position of a single video frame in the video material to be processed, target size and position of consecutive video frames in the video material to be processed, and similarity of adjacent video frames in the video material to be processed.

[0267] Specifically, for the camera direction, some algorithms for detecting the camera direction can be used to determine the change in the camera direction in the video material to be processed. The scene varies according to the different video contents in the video material to be processed. For characters, the scene has panoramic, medium, close-up and special scene (i.e., close-up). For objects, the scene has distant and close-up. The size and position of the target object in a single video frame in the video material to be processed are determined using an object detection algorithm or a saliency detection algorithm. The size and position of the target object in consecutive video frames in the video material to be processed are determined based on a pre-trained neural network model.

[0268] S3012: Determine, according to the video tag of the video material to be processed, a plurality of templates matching the video tag.

[0269] The template includes a template tag, which can be pre-set when setting the template. After obtaining the video tag of the video material to be processed, the video tag of the video material to be processed and the template tag can be matched, thereby matching the template for the video material to be processed.

[0270] In one embodiment, see Fig.11 The step of determining a plurality of matching templates according to the video tag includes step S3012a and step S3012b. S3012a, determining a video theme corresponding to the video material to be processed according to the video tag; S3012b, determining a plurality of templates matching the video theme according to the video theme of the video material to be processed.

[0271] The video theme corresponding to the video material to be processed is determined according to the extracted video tags. The video theme can be determined according to the video tags of a single video frame and / or continuous video frames in the video material to be processed. For example, if the target object in the continuous video frames is a tower, then the video theme corresponding to the video material to be processed is determined to be travel.

[0272] The video theme can be a theme category, or it can include subcategories under the theme category. For example, the video theme can be travel, food, parent-child, etc. Under the theme category of travel, the video theme can also be travel-natural scenery, travel-city, travel-cultural monuments, etc.

[0273] The template tag may be a template theme. After determining the theme of the video material to be processed, the theme of the video material to be processed is matched with the template theme, thereby determining a plurality of templates matching the video theme.

[0274] In one embodiment, if the video theme corresponding to the to-be-processed video material cannot be determined, a preset template is selected as the template corresponding to the to-be-processed video material.

[0275] The preset template may be a universal template, that is, a template that can be applied in various scenarios. The inability to determine the video theme corresponding to the video material to be processed may be that no clear video theme is identified, or there is no template corresponding to the video theme of the video material to be processed.

[0276] In one embodiment, determining multiple templates matching the video theme according to the video theme of the video material to be processed includes: determining a template corresponding to the video material to be processed from multiple templates matching the video theme according to a template influence factor of the template.

[0277] After the templates are roughly screened according to the video theme and multiple templates matching the video theme are determined, the templates are screened again according to the template influence factors of the templates to determine the templates corresponding to the video material to be processed. The template corresponding to the video material to be processed determined by the secondary screening can be one or more.

[0278] The template influencing factor includes at least one of music matching degree, template popularity and user preference.

[0279] Specifically, each template is pre-set with template music, and the template music is matched with the video material to be processed to determine the matching score between the template music and the video material to be processed. The music matching degree is obtained according to a pre-trained music recommendation network model, and the music recommendation network model can output the matching scores of the template music of multiple templates matching the video theme and the video material to be processed.

[0280] The template popularity is determined based on the frequency of use and / or number of likes of multiple templates matching the video theme. The frequency of use and / or number of likes of each template by all users who use the template can be obtained, and the template popularity can be determined based on the selection of the template by all users who use the template.

[0281] The user preference is determined based on the user's selection frequency and / or satisfaction score for multiple templates matching the video theme. After multiple uses by the user, the user's usage frequency and / or satisfaction score for each template is obtained based on the user's usage habits, thereby determining the user preference.

[0282] In one embodiment, the method of determining a template corresponding to the video material to be processed from a plurality of templates matching the video theme based on the template influence factor of the template includes: obtaining an evaluation score and a preset weight of the template influence factor; determining a template score of a plurality of templates matching the video theme based on the evaluation score and the preset weight of the template influence factor; and determining a template corresponding to the video material to be processed based on the template score.

[0283] The evaluation score and corresponding preset weight of the template influence factor of each template are obtained to determine the template score of each template, and the template corresponding to the to-be-processed video material is determined from multiple templates according to the template score of each template. The preset weight includes the weights corresponding to different template influence factors, which can be preset according to experience values.

[0284] For example, the calculation formula for the template score of each template is:

[0285] M=A*E music +B*E template +C*E user

[0286] Among them, E music represents the score of music matching, A represents the preset weight corresponding to the score of music matching, E template represents the template heat, B represents the preset weight corresponding to the template heat, and E user represents the user's preference, and C represents the preset weight corresponding to the user's preference.

[0287] In one embodiment, material selection is performed on the video material to be processed.

[0288] Specifically, by selecting materials from the video materials to be processed, materials that need to be edited are selected from the video materials to be processed, and then a template is determined according to the selected materials that need to be edited, and a recommended video is generated.

[0289] In one embodiment, the material selection for the video material to be processed includes: selecting the material according to the material parameters of the video material to be processed; wherein the material parameters include at least one of shooting time, shooting location, and shooting target.

[0290] Specifically, when selecting materials according to material parameters, the material parameters may be set by the user. For example, the user wants to select video materials shot within three days from May 1st to May 3rd from the video materials to be processed, or wants to select video materials shot between 6 pm and 10 pm from the video materials to be processed, or the user wants to select video materials shot at Xishuangbanna from the video materials to be processed, or the user wants to select video materials shot at cats from the video materials to be processed.

[0291] When selecting the video material to be processed according to the shooting time, for example, the shooting time recorded by the handheld terminal or the mobile platform when shooting the video can be read to select the material according to the shooting time. For example, the approximate shooting time period can be determined based on the real-time nature of the video shooting and the content of the shot video material, such as the ambient lighting conditions, whether there are lights or billboards lit, etc., to select the material.

[0292] When selecting video materials to be processed based on the shooting location, for example, when the GPS positioning service is turned on, the GPS information at the time of shooting will be recorded when the video is shot, and the video materials to be processed can be screened based on the GPS information recorded at the time of shooting.

[0293] When selecting video materials to be processed based on material parameters, the user can input time or place and select materials based on the time or place entered by the user. The user can also select one or more video materials to be processed and analyze the correlation between the video materials to be processed selected by the user to select materials based on the analyzed correlation.

[0294] In one embodiment, the material selection of the video material to be processed includes: clustering the video material to be processed according to material parameters of the video material to be processed to achieve material selection; wherein the clustering includes at least one of time clustering, location clustering, and target object clustering.

[0295] When selecting materials according to material parameters of the video materials to be processed, the materials can be selected by clustering. For example, clustering selection can be performed by at least one of time clustering, location clustering, and target object clustering, so that users can quickly select a large number of materials and save time for selecting materials.

[0296] In one implementation, the selecting the video material to be processed includes: selecting the video material to be processed according to a selection operation of a user.

[0297] Among them, the video material that needs to be edited can also be selected according to the user's own self-defined selection. In the specific implementation process, all the materials to be processed can be presented to the user, and the user can customize the selection of the material to be edited from all the materials to be processed.

[0298] In addition, after the video materials to be processed are selected or clustered according to material parameters, the results of the selection or clustering can be presented to the user, and the user can customize the materials to be edited from the results of the selection or clustering.

[0299] In addition, in one embodiment, video materials may be selected or clustered according to the user's historical preferences for selecting video materials, so as to provide the user with a personalized usage experience.

[0300] In one embodiment, the video material to be processed may be segmented to generate multiple video segments according to the video information of the video material to be processed, for example, by using the video segmentation method provided in the above embodiment.

[0301] In one embodiment, the image quality of the video material to be processed may also be obtained; and waste films of the video material to be processed may be removed according to the image quality of the video material to be processed.

[0302] The image quality includes at least one of image jitter, image blur, overexposure, underexposure, no clear scene in the image, or no clear subject in the image. The video material to be processed is obtained, and then the video image of the video material to be processed is tested for quality to determine whether at least one of image jitter, image blur, overexposure, underexposure, no clear scene in the image, or no clear subject in the image occurs in the video material to be processed. If so, the part is considered as waste film, and the video material to be processed is removed.

[0303] It should be noted that, for the video segments in the video material to be processed, the waste segments can also be removed according to the image quality of the video segments.

[0304] In one embodiment, the video material to be processed may also be deduplicated, wherein the deduplication process includes clustering similar materials.

[0305] In the process of shooting video materials, in order to obtain a satisfactory video, the same scene is usually shot multiple times, which results in a lot of repeated materials in the video materials to be processed. Similar materials are clustered to classify similar materials into one category, and the material with the longest video duration is selected from the materials. Or the material with the best image quality is selected from the materials, such as clear images and high color saturation.

[0306] It should be noted that, for the video segments in the video material to be processed, the video segments can also be deduplicated.

[0307] S302: Determine a video segment corresponding to the video position according to the position information of the video position in the template, and obtain a matching relationship corresponding to the template.

[0308] Wherein, the video clip is a clip in the video material to be processed. After determining the template corresponding to the video material to be processed, since the template includes at least one video slot, the video clip corresponding to the video slot can be determined according to the slot information of the video slot in the template, thereby obtaining a matching relationship corresponding to the template.

[0309] Specifically, the pit information includes at least one of pit music and pit labels. In some embodiments, the pit label can be pre-set, and each video pit can be pre-set with a pit label.

[0310] The pit music of a video pit can be a fragment of the template music of the entire template. Since the template includes multiple video pits, multiple video pits are combined into one template in sequence. Therefore, for the template music of the template, the template music can be split according to the order and duration of the video pits to obtain the pit music corresponding to each video pit.

[0311] In one embodiment, determining the video clip corresponding to the video pit position according to the pit position information of the video pit position in the template includes: determining the video clip corresponding to the video pit position according to the pit position music of the video pit position in the template.

[0312] Since the pit music of the video pit itself has a sense of rhythm, for example, the pit music of the video pit has a soothing rhythm or an explosive rhythm, a suitable video clip can be matched to the video pit according to the rhythm of the pit music and the video content of the video clip.

[0313] For example, if the music is explosive, and there are powerful bursts of images such as fountains in the video clip, then the two are a good match. In addition, if the rhythm is slow, it is suitable for slow motion of characters; if it is an epic blockbuster, it is suitable for time-lapse photography of large scenes, etc.

[0314] In one embodiment, determining the video clip corresponding to the video pit position according to the pit position music of the video pit position in the template includes: determining the degree of matching between the pit position music of the video pit position in the template and the video clip; and determining the video clip corresponding to the video pit position in the template according to the matching degree.

[0315] The rhythm of the pit music of the video pit is matched with the video clip to determine the degree of matching between the pit music of the video pit and the video clip, thereby determining the video clip corresponding to the video pit in the template according to the degree of matching.

[0316] Specifically, a threshold of the degree of matching may be set, and when the degree of matching exceeds the threshold, it is determined that the video position matches the video clip; or, based on the degree of matching, the video clip with the highest degree of matching with the video position may be selected as the video clip corresponding to the video position.

[0317] In one embodiment, the degree of matching between the pit music of the video pit in the template and the video clip is obtained by using a pre-trained music matching model, and the music matching model can output the matching score between the pit music of the video pit in the template and the video clip.

[0318] The matching degree between the pit music and the video clip is learned by training a neural network, and then the trained neural network is used as a music matching model to output the matching degree score between the pit music and the video clip of the video pit.

[0319] In one embodiment, after determining the video segment corresponding to the video pit position according to the pit position information of the video pit position in the template, the shooting quality of multiple video segments corresponding to the video pit position in the template is determined; the optimal video segment corresponding to the video pit position in the template is determined according to the shooting quality of the multiple video segments; and the matching relationship corresponding to the template is obtained according to the optimal video segment corresponding to the video pit position in the template.

[0320] The shooting quality of the video clip is determined according to the image content of the video clip and the video clip evaluation. The image content includes lens stability, color saturation, whether there is a main shooting object and the amount of information in the lens, etc. The video clip evaluation includes the aesthetic score of the video clip.

[0321] Specifically, the aesthetic scoring of the video clip can take into account factors such as color, composition, camera movement and scene type to score the video clip aesthetically. The shooting quality of the video clip is scored based on the image content of the video clip and the video clip evaluation. The larger the score, the higher the shooting quality of the video clip.

[0322] According to the shooting qualities of multiple video clips, the video clip with the highest shooting quality is selected from the video clips as the optimal video clip corresponding to the video pit position in the template, so as to obtain the matching relationship corresponding to the template.

[0323] In one embodiment, after determining the video segment corresponding to the video pit according to the pit information of the video pit in the template, determine the matching degree between the video segments corresponding to two adjacent video pits in the template; determine the optimal video segment corresponding to the video pit according to the matching degree; and obtain the matching relationship corresponding to the template according to the optimal video segment corresponding to the video pit.

[0324] The matching degree of the video clips corresponding to the two adjacent video slots is determined according to the continuity of the camera direction of the video clips, the increasing and decreasing relationship of the scene scale, and the matching editing.

[0325] The continuity of the camera direction includes ensuring that the camera directions of the video clips corresponding to two adjacent video slots are in the same direction, and avoiding the video clips with opposite camera directions being connected together. The increasing and decreasing relationship of the scene size includes, for example, from long shot to medium shot to close shot, or from close shot to medium shot to long shot, or directly from long shot to close shot, etc. Matching editing includes matching and connecting two shots through similar actions, graphics, colors, etc. to achieve a coherent and smooth narrative, which is used for the transition between two video clips.

[0326] In one embodiment, determining the video segment corresponding to the video pit according to the pit position information of the video pit in the template includes: determining the video segment corresponding to the video pit according to the pit position label of the video pit in the template.

[0327] Each video slot in the template is set with a slot label. The labels can be matched according to the slot labels of the video slots in the template and the labels of the video clips, so that the successfully matched video clips are used as the video clips corresponding to the video slots.

[0328] In one embodiment, determining the video segment corresponding to the video pit according to the pit label of the video pit in the template includes: determining the video tag of the video segment, and taking the video segment corresponding to the video tag that matches the pit label of the video pit as the video segment to be filled in the video pit.

[0329] Labels are extracted from the video clips to determine the video labels of the video clips, and then the video clips corresponding to the video positions are determined according to the position labels of the video positions of the template.

[0330] S303: Fill the video clip into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video.

[0331] After the matching relationship is obtained, the video clips are respectively filled into the corresponding video slots of the template according to the matching relationship, and the video is synthesized to obtain the recommended video. The recommended video is then recommended to the user.

[0332] In one embodiment, see Fig.12 The step of filling the video clip into the corresponding video pit specifically includes step S3031 to step S3032.

[0333] S3031. Determine whether the video duration of the video clip is greater than the duration of the video slot.

[0334] When the video clip is filled into the corresponding video slot according to the matching relationship, it is determined whether the video duration of the video clip is greater than the duration of the video slot. The duration of the video slot determines the maximum duration of the video clip that can be filled into the video slot. Therefore, when the video duration of the video clip is greater than the duration of the video slot, the video clip cannot be directly filled into the corresponding video slot, and a clip of the corresponding duration needs to be extracted from the video clip and filled into the corresponding video slot.

[0335] S3032: If the video duration of the video clip is greater than the duration of the video slot, extract the video clip to obtain a selected clip.

[0336] The video duration of the selected segment is less than or equal to the duration of the video slot. In the specific implementation process, in order to fill the selected segment into the corresponding video slot and ensure the completeness of the recommended video, when determining the selected segment, the video duration of the selected segment can be equal to the duration of the video slot.

[0337] In one embodiment, extracting segments from the video segments to obtain selected segments includes: extracting segments from the video segments to obtain selected segments according to video elements of the video segments.

[0338] When extracting segments from video segments, the video segments may be extracted according to video elements of the video segments to obtain selected segments.

[0339] The video elements include at least one of a smiling face picture, a laughing sound audio, a character action, a clear human voice, a picture composition and an aesthetic score. When extracting the selected clips, more exciting clips can be extracted from the video clips according to the video elements as the selected clips, for example, clips including smiling faces or clips with higher aesthetic scores can be selected as the selected clips.

[0340] In one embodiment, step S303 includes: filling the video clip into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain an initial video; and optimizing the image of the initial video based on the template requirements of the template to obtain a recommended video.

[0341] After the video clips are filled into the corresponding video slots according to the matching relationship corresponding to the template, an initial video is obtained, and then the image of the initial video can be optimized according to the template requirements, and the image-optimized video is recommended to the user as a recommended video. The template requirements include at least one of transition settings, acceleration and deceleration settings, and texture special effects settings.

[0342] In one embodiment, since the source of the video material to be processed may be an aerial video taken by a drone, and the distance between the camera and the object being photographed is relatively far during aerial photography, the content of the picture changes is small. Therefore, when the aerial video is filled into the corresponding video slot, the speed of picture changes can be automatically identified first, and the aerial video can be automatically speed-adjusted according to the speed of picture changes, and then the aerial video after speed adjustment can be filled into the corresponding video slot. The speed of picture changes can be obtained by analyzing multiple consecutive frames within a preset time.

[0343] In one embodiment, when determining the matching relationship between the video slots of the template and the video clips, for the identified aerial video, the aerial video can be placed in the first few video slots and / or the last few video slots of the template, thereby improving the quality of the recommended video.

[0344] The above embodiment provides a video processing method, which determines a template according to the video information of the video material to be processed, and then determines the video segment corresponding to the video slot according to the slot information of the video slot, obtains the matching relationship corresponding to the template, and finally fills the video segment into the corresponding video slot of the template according to the matching relationship corresponding to the template to obtain a recommended video. The template is determined according to the video information, the diversity of the generated recommended videos is improved, the matching relationship between the video slot and the video segment is determined, and the recommended video is synthesized according to the matching relationship, which reduces the workload of users in editing materials and lowers the threshold for editing.

[0345] See also Fig.13 , Fig.13 It is a step flow chart of another video processing method provided in an embodiment of the present application.

[0346] Specifically, Fig.13 As shown, the video processing method includes steps S401 to S404.

[0347] S401. Acquire multiple templates, where the templates include at least one video slot.

[0348] A plurality of templates are obtained, and the templates are used to be synthesized with the video material to be processed to obtain a recommended video, and each template includes at least one video slot.

[0349] Specifically, multiple templates can be obtained arbitrarily from a preset template library, or according to a user's selection operation on a template in the template library, or according to templates frequently used by the user in synthesizing videos in the past.

[0350] In one embodiment, the material to be processed is segmented to generate a plurality of video segments.

[0351] The material to be processed is segmented to generate multiple video clips, and the generated video clips are used to fill in the video slots in the template to synthesize the recommended video.

[0352] In one embodiment, the step of segmenting the material to be processed to generate multiple video segments includes: segmenting the video material to be processed to generate multiple video segments according to video information of the video material to be processed. For example, the video segmentation method provided in the above embodiment may be used.

[0353] S402: Match a video clip to a video position of each template to obtain a matching relationship corresponding to each template, and determine a matching score of the matching relationship corresponding to each template.

[0354] The video segments are segments of the video material to be processed. In some implementations, the video segments may be video segments obtained by segmenting the video material to be processed.

[0355] For each video position in a template, a video clip is matched for each video position in the template to obtain a matching relationship between the video position and the video clip. The matching relationship between the video position and the video clip is used as the matching relationship corresponding to the template, and the matching score of the matching relationship is calculated.

[0356] In one embodiment, see Fig.14 The step of matching the video pit position with the video clip to obtain a matching relationship specifically includes step S4021 and step S4022.

[0357] S4021. Construct multiple flow network graphs according to the video clips and the video pit positions of each template.

[0358] The flow network diagram includes a plurality of nodes, each of which corresponds to a matching relationship between a video clip and a video slot. A flow network diagram is constructed based on the video slot and video clip of each template, wherein the left vertical axis C in the flow network diagram is m Indicates video clips, with the upper horizontal axis S n Indicates the video slot of the template.

[0359] S4022. Determine a matching relationship between a video slot of each template and the video clip based on the plurality of flow network diagrams.

[0360] Each template determines the matching relationship between its own video pit and video clip based on its own flow network graph.

[0361] In one embodiment, the determining the matching relationship between the video slot of each template and the video clip based on the plurality of flow network graphs includes: matching a suitable video clip for the video slot of each template based on a maximum flow algorithm to obtain an optimal path; wherein the corresponding relationship between the video clip and the video slot in the optimal path is used as the matching relationship between the video slot of each template and the video clip. For example, the video segmentation method provided in the above embodiment can be used.

[0362] In one embodiment, determining the matching score of the matching relationship between the video pit position of each template and the video clip includes: determining the matching score of the matching relationship between the video pit position of each template and the video clip based on the energy value between each two adjacent nodes in the optimal path.

[0363] According to the energy value between each two adjacent nodes in the optimal path, the energy values ​​between the two adjacent nodes in the optimal path are added, and the sum of the energy values ​​in the optimal path is used as the matching score of the matching relationship.

[0364] In one embodiment, matching video clips for the video positions of each template to obtain a matching relationship corresponding to each template includes: classifying the video clips according to the pit position labels of the video pit positions of the template or the template labels of the template to obtain classified video clips; and determining the matching relationship corresponding to the template according to the classified video clips.

[0365] Specifically, each video clip has a corresponding video tag, and the slot tags of each video slot of the template are matched with the video tags corresponding to the video clip, so that the video clips are classified according to the matching degree between the video tag and the slot tag, and the video clips are divided into multiple categories.

[0366] In addition, the template tag of the template may be matched with the video tag corresponding to the video clip, so as to classify the video clip according to the matching degree between the video tag and the template tag, and divide the video clip into multiple categories.

[0367] Then, the video clip corresponding to the video slot is determined according to the category of the video clip, and the matching relationship corresponding to the template is obtained.

[0368] In one embodiment, the classifying the video clips according to the pit position labels of the video pit positions of the template or the template labels of the template includes: classifying the multiple video clips into levels according to the pit position labels of the video pit positions or the template labels of the template to obtain video clips of multiple level categories.

[0369] The video clips of the multiple levels categories include at least video clips of the first category, video clips of the second category and video clips of the third category; wherein the wonderful level of the video clips of the first category is greater than the wonderful level of the video clips of the second category, and the wonderful level of the video clips of the second category is greater than the wonderful level of the video clips of the third category.

[0370] Specifically, when classifying the video clips, the video clips may be classified according to their wonderfulness levels. Thus, when matching video clips for video slots, the most wonderful video clip may be selected as the video clip corresponding to the video slot to obtain a matching relationship of the template.

[0371] The excitement level is determined based on the video content and audio content of the video clip. For example, the video content includes the composition of the picture, whether there is a clear subject, the direction of the camera, and the scene, etc. The audio content includes whether there is clear human voice, laughter, cheers, etc.

[0372] In one embodiment, determining the matching relationship between the video pit position of each template and the video clip according to the classified video clips includes: sorting the classified video clips according to the pit position labels of the video pit positions or the template labels of the templates; and determining the matching relationship between the video pit position of each template and the video clip according to the sorting results.

[0373] Specifically, the classified video segments are sorted according to the slot labels of the video slots and the classification levels of the video segments. When sorting, for each slot label, the video segments can be sorted from high to low according to the classification levels. For each video slot of each template, a video segment with a higher ranking is selected as the video segment to be filled in the video slot, thereby obtaining a matching relationship between the video slot of the template and the video segment.

[0374] For example, there are three types of pit labels, ABC, and the video labels and classification levels of the video clips are A1, A2, A3, B1, B3, C1, C2. The classified video clips are A label categories: A1, A2, A3, B label categories: B1, B3, C1, C2.

[0375] When selecting the video clip to be filled in the video slot, if the slot label of the video slot is A, select video clip A1; if the slot label of the video slot is B, select video clip B1; if the slot label of the video slot is C, select video clip C1.

[0376] Similarly, the classified video clips are sorted according to the template label of the template and the classification level of the video clips. When sorting, for each template label, the video clips can be sorted from high to low according to the classification level. For each video slot of each template, the video clip with the highest ranking is selected as the video clip to be filled in the video slot, so as to obtain the matching relationship between the video slot of the template and the video clip.

[0377] In one embodiment, determining the matching relationship between the video slot of each template and the video clip according to the classified video clips includes: allocating video clips to the video slots of the template according to the sorting results, and determining the matching relationship between the video slot of each template and the video clip.

[0378] For each video slot of each template, a video clip with a higher ranking is selected as the video clip to be filled in the video slot, so as to obtain a matching relationship between the video slot of the template and the video clip.

[0379] S403: Determine a recommended template from the multiple templates according to the matching scores.

[0380] After obtaining the matching scores corresponding to the matching relationships of the templates, a recommended template may be determined from the obtained multiple templates, wherein the recommended template is a template used to synthesize a recommended video.

[0381] In one embodiment, see Fig.15 The step of determining the recommended template according to the matching score specifically includes steps S4031 to S4033.

[0382] S4031. Determine a preset number of templates from the multiple templates according to the matching scores.

[0383] First, a preset number of templates is determined from a plurality of templates according to the matching scores, wherein the number of the preset number of templates may be set according to an empirical value.

[0384] In a specific implementation process, a preset number of templates may be selected from multiple templates according to the matching scores from high to low.

[0385] S4032: Select a target number of templates from the preset number of templates and combine them to obtain multiple template groups, where the template groups include the target number of templates.

[0386] The target number is the number of recommended templates recommended to the user. A target number of templates are selected from the preset number of templates for combination, thereby obtaining multiple template groups, each of which includes the target number of templates.

[0387] In the specific implementation process, the combination number can be used to select the target number of templates and obtain the template group. For example, if the preset number is n and the target number is k, then the number of template groups that can be selected can be:

[0388]

[0389] S4033: Determine a recommended template group from multiple template groups according to the template types of a target number of templates in the template group, and use the target number of templates in the recommended template group as recommended templates.

[0390] The template types of the target number of templates included in each template group are determined respectively, and a recommended template group is determined from the multiple template groups according to the template types. The template richness is considered so that the templates recommended to the user include multiple styles for the user to choose.

[0391] In one embodiment, see Fig.16 , step S4033 includes step S4033a and step S4033b.

[0392] S4033a. Obtain template types of a target number of templates in the multiple template groups, and determine combined scores corresponding to the multiple template groups according to the template types and the matching scores.

[0393] For each template group, the template types of the target number of templates in the template group are obtained, and then the template richness score of the template group is calculated according to the template type, and the matching score of each of the target number of templates in the template group is obtained. The combined score corresponding to each template group is determined according to the matching score and the richness score.

[0394] In one embodiment, determining the combined scores corresponding to the multiple template groups according to the template type and the matching score includes: determining the template richness between a target number of templates in the multiple template groups according to the template type; and determining the combined scores of the multiple template groups according to the sum of the template richness between a target number of templates in the template group and the matching scores of the target number of templates in the template group.

[0395] The template type ID of each template in the template group is obtained. Different template types correspond to different IDs. Based on this, the template richness between the templates in the template group is judged.

[0396] For example, when determining the abundance of a template in a template group, the following formula can be used:

[0397]

[0398] Among them, E1 represents the template richness in the template group, a i Indicates the template type of the i-th template, a j Indicates the template type of the jth template, f(a i ,a j ) indicates whether the template type of the i-th template is the same as the template type of the j-th template.

[0399] When the template type of the i-th template is the same as the template type of the j-th template, f(a i ,a j ) is 0. When the template type of the i-th template is different from the template type of the j-th template, f(a i ,a j ) is 1. The larger the value of E1 is, the richer the template types in the template group are.

[0400] After the template richness in the template group is obtained, the combined score of the template group can be determined according to the matching scores of the individual templates in the template group.

[0401] For example, the following formula can be used:

[0402] E=a*E1+E2

[0403] Among them, E is the combined score of the template group, E1 is the template richness of the template group, a is the preset weight of the template richness, and E2 is the sum of the matching scores of the templates in the template group.

[0404] S4033b: Determine a recommended template group from the multiple template groups according to the combined score.

[0405] After obtaining the combined scores of each template group, a recommended template group may be selected from the multiple template groups according to the combined scores of each template group. The templates in the recommended template group are the recommended templates recommended to the user.

[0406] In a specific implementation process, a template group with the highest combination score may be selected from multiple template groups as a recommended template group according to the combination score.

[0407] In one embodiment, the number of the recommended videos includes multiple, and the multiple recommended videos are obtained according to a target number of templates in the recommendation template group.

[0408] In one embodiment, a plurality of the recommended videos are recommended to the user for the user to select.

[0409] For the target number of templates in the recommendation template group, the video clips are filled into the video slots of the templates according to the matching relationships corresponding to the templates, thereby generating multiple recommended videos, and the multiple recommended videos are jointly recommended to the user, so that the user can select the video to be finally used.

[0410] In one embodiment, see Fig.17 , step S403 includes step S4031' and step S4032'.

[0411] S4031′, determining a preset number of templates from the multiple templates according to the matching scores to form a template group.

[0412] According to the matching scores of the multiple templates, a preset number of templates are determined from the multiple templates, and the preset number of templates are formed into a template group.

[0413] Specifically, a preset number of templates may be selected from high to low according to the matching scores of the templates to form a template group. For example, five templates may be selected from high to low in sequence according to the matching scores of the templates to form a template group.

[0414] S4032′: Determine the template type of the template in the template group, and determine a recommended template from the template group according to the template type.

[0415] The template types of the templates in the template group are determined, thereby determining a recommended template from the template group according to the template types.

[0416] In one embodiment, see Fig.18 The step of determining the recommended template according to the template type specifically includes steps S4032'a to S4032'c.

[0417] S4032'a. Determine whether the number of the template types is greater than a preset type threshold.

[0418] Determine whether the number of template types in the template group is greater than a preset type threshold, wherein the preset type threshold is the number of template types expected to be recommended to the user. The preset type threshold may be preset.

[0419] S4032'b: If the number of the template types is greater than the preset type threshold, determine a recommended template according to the matching scores between the template type and templates of the same template type.

[0420] When the number of template types in the template group is greater than the preset type threshold, the recommended template is determined according to the template type and the matching score of templates of the same template type, that is, the template with the highest matching score is selected from the templates of the same template type as the recommended template.

[0421] In one embodiment, determining a recommended template based on the matching scores of the template type and templates of the same template type includes: classifying the templates in the template group according to the template type to obtain templates of multiple types; determining multiple type-optimal templates according to the matching scores, wherein the type-optimal template is the template with the highest matching score in each template type; and selecting the template with the highest matching score from the multiple type-optimal templates as the recommended template.

[0422] The templates in the template group are divided into types according to the template type to obtain multiple types of templates, and the templates of the corresponding template type are divided into the corresponding types. Then, for each divided template type, a template with the highest matching score is selected from multiple templates corresponding to the template type as the optimal template under this template type, that is, the type-optimal template. For type-optimal templates of multiple template types, the template with the highest matching score is selected from multiple type-optimal templates as the recommended template.

[0423] See also Fig.19 , Fig.19 is a schematic diagram of selecting a recommended template from a template group. A, B, C, and D are four template types in the template group, and the preset type threshold is 3, which means that three types of templates need to be selected from the template group to be recommended to the user.

[0424] like Fig.19 As shown, under the A template type there are A1 to A n There are n templates in total, and the matching scores of these n templates are A1-99 points, A2-98 points, A3-96 points, etc. m There are m templates in total. The matching scores of these m templates are B1-97 points, B2-94 points, B3-89 points, etc. The C template type has C1 to C x There are x templates in total, and the matching scores of these x templates are C1-99 points, C2-95 points, C3-90 points, etc. y There are y templates in total, and the matching scores of these y templates are D1-95 points, D2-94 points, D3-93 points, etc.

[0425] For each template type, the optimal template is selected according to the template matching score, and the optimal template of type A template is A1, the optimal template of type B template is B1, the optimal template of type C template is C1, and the optimal template of type D template is D1.

[0426] Since only three types of templates need to be selected to be recommended to the user, the matching scores of template A1, template B1, template C1 and template D1 are compared, where A1-99 points, B1-97 points, C1-99 points, and D1-95 points. Three templates are selected from template A1, template B1, template C1 and template D1 according to the matching scores, namely template A1, template B1 and template C1, and template A1, template B1 and template C1 are recommended to the user as recommended templates.

[0427] S4032'c: If the number of the template types is less than or equal to the preset type threshold, select the template with the highest matching score from multiple templates of the same template type in the template group as the recommended template.

[0428] If the number of template types is less than the preset type threshold, it means that the template types in the template group do not meet the requirements of the preset type threshold. Therefore, the template of the template type in the template group with the highest matching score can be directly selected as the recommended template.

[0429] For example, if the preset type threshold is 3, and there are only two template types A and B in the template group, and the template with the highest matching score under template type A is template A1, and the template with the highest matching score under template type B is template B1, then template A1 and template B1 can be used as recommended templates.

[0430] If the number of template types is equal to the preset type threshold, it means that the template types in the template group just meet the requirements of the preset type threshold. Therefore, the template of the template type in the template group can be directly selected as the recommended template with the highest matching score.

[0431] For example, if the preset type threshold is 3, and there are two template types in the template group, A, B and C, and the template with the highest matching score under template type A is template A1, the template with the highest matching score under template type B is template B1, and the template with the highest matching score under template type C is template C1, then template A1, template B1 and template C1 can be used as recommended templates.

[0432] In one embodiment, step S403 specifically includes obtaining template types of the multiple templates; and determining a recommended template according to the template types and matching scores of the multiple templates.

[0433] For multiple templates, the template type of each template is obtained, so as to determine a recommended template according to the template types and matching scores of the multiple templates, so that recommended templates of multiple template types can be recommended to the user.

[0434] In one embodiment, determining a recommended template based on the template type and matching scores of the multiple templates includes: dividing the multiple templates into multiple type template groups based on the template type, each type template group including at least one template; determining templates that meet a required number of types from the multiple type template groups based on the matching scores of the templates; and selecting templates from the remaining templates of the multiple templates based on the matching scores of the templates, wherein the number of selected templates meets the required number of templates.

[0435] For multiple templates, when the number of template types that meet the requirements of multiple templates is greater than a preset first threshold, the first threshold number of templates of different types are obtained as the first recommended templates; wherein the first recommended template is the template with the highest matching score in the template type to which it belongs; the first threshold is the number of type requirements.

[0436] According to other templates among the multiple templates except the first recommended template, a second recommended template is obtained according to the matching scores; wherein the sum of the first recommended template and the second recommended template is a second threshold, and the second threshold is the required number of templates.

[0437] The multiple templates are classified according to the template type of each template in the multiple templates to obtain multiple type template groups, each type template group corresponds to a template type, and each type template group includes at least one template.

[0438] Then, templates that meet the required number of types are selected from the type template group according to the matching scores of the templates, and then templates are selected from the remaining templates according to the matching scores of the remaining templates until the number of selected templates meets the required number of templates.

[0439] For example, Fig.19 As shown, if there are 24 templates in total, the required number of templates is 5 and the required number of types is 3.

[0440] The template types of the 24 templates are obtained, and the 24 templates are classified according to the different template types to obtain four type template groups A, B, C, and D. Each type template group corresponds to a template type, and each type template group includes six templates.

[0441] Among them, there are 6 templates from A1 to A6 under the A template type, and the matching scores of these 6 templates are A1-99 points, A2-98 points, A3-96 points, etc. There are 6 templates from B1 to B6 under the B template type, and the matching scores of these 6 templates are B1-97 points, B2-94 points, B3-89 points, etc. There are 6 templates from C1 to C6 under the C template type, and the matching scores of these 6 templates are C1-99 points, C2-95 points, C3-90 points, etc. There are 6 templates from D1 to D6 under the D template type, and the matching scores of these 6 templates are D1-95 points, D2-94 points, D3-93 points, etc.

[0442] The highest matching score of templates under template type A is A1-99 points, the highest matching score of templates under template type B is B1-97 points, the highest matching score of templates under template type C is C1-99 points, and the highest matching score of templates under template type D is D1-95 points.

[0443] According to the highest matching scores of templates under the four template types A, B, C, and D, three templates with the highest matching scores are selected from the four template types, namely template A1, template B1, and template C1, so that the selected three templates meet the required number of types.

[0444] At this time, the number of selected templates is 3, while the required number of templates is 5. Therefore, 2 more templates can be selected according to the matching scores of the remaining 21 templates so that the number of selected templates meets the required number of templates.

[0445] Among them, after selecting the required number of templates of the type, the remaining 21 templates include templates A2 to A6, B2 to B6, C2 to C6 and D1 to D6. According to the matching scores of the templates in these 21 templates, template A2 and template A3 are selected. At this time, the number of selected templates meets the required number of templates, and the selected templates that meet the required number of templates are used as recommended templates, that is, the five templates of template A1, template A2, template A3, template B1 and template C1 are used as recommended templates.

[0446] In addition, when at least two templates have the same matching scores, the diversity of templates may be considered and a template of a different template type from the selected template may be selected.

[0447] In one embodiment, determining a recommended template based on the template type and matching scores of the multiple templates includes: selecting templates from the multiple templates in sequence according to the matching scores of the templates until the selected template type meets the required number of types; selecting templates from the remaining templates from the multiple templates according to the matching scores of the templates until the number of selected templates meets the required number of templates.

[0448] A template is selected from multiple templates according to the matching score of the template, and the type of the selected template is determined, so that the type of the selected template is different each time, until the type of the selected template meets the required number of types. Then, a template is selected from the remaining templates according to the matching score of the remaining templates, until the number of selected templates meets the required number of templates.

[0449] For example, Fig.19 As shown, if there are 24 templates in total, the required number of templates is 5 and the required number of types is 3.

[0450] Among them, there are 6 templates from A1 to A6 under the A template type, and the matching scores of these 6 templates are A1-99 points, A2-98 points, A3-96 points, etc. There are 6 templates from B1 to B6 under the B template type, and the matching scores of these 6 templates are B1-97 points, B2-94 points, B3-89 points, etc. There are 6 templates from C1 to C6 under the C template type, and the matching scores of these 6 templates are C1-99 points, C2-95 points, C3-90 points, etc. There are 6 templates from D1 to D6 under the D template type, and the matching scores of these 6 templates are D1-95 points, D2-94 points, D3-93 points, etc.

[0451] When selecting a template, a template with the highest matching score is selected from the 24 templates. At this time, since the matching scores of template A1 and template C1 are the same and both are the highest scores, one of templates A1 and template C1 can be selected as the template selected for the first time. For example, the template selected for the first time is template A1, and the corresponding template type is A.

[0452] Then, the template with the highest matching score is selected from the remaining 23 templates except template A1. The template selected for the second time is template C1, and the corresponding template type is C. At this time, two template types are selected, which does not meet the required number of types. Therefore, the template with the highest matching score is selected from the remaining 22 templates except template A1 and template C1. At this time, the template with the highest matching score is template A2, and the corresponding template type is A. However, since there is already a template A1 of the same template type in the two selected templates (template A1 and template C1), the template with the smallest difference between the matching score and the matching score of template A2 is selected from the remaining 22 templates except template A1 and template C1. The template is template B1, and the corresponding template type is B. Since the types of template B1 and the selected templates A1 and template C1 are different, template B1 is selected as the template for the third time. At this time, the selected templates are template A1, template C1 and template B1, and the template types are three, which meets the required number of types.

[0453] Then, according to the matching scores of the templates, a template with the highest matching score is selected from the remaining 21 templates except template A1, template C1 and template B1. The selected template is template A2. At this time, the number of selected templates is four, which does not meet the required number of templates.

[0454] Then, the template with the highest matching score is selected from the remaining 20 templates, and the selected template is template A3. At this time, the number of selected templates is five, which meets the required number of templates. The selected templates that meet the required number of templates are used as recommended templates, that is, the five templates of template A1, template A2, template A3, template B1 and template C1 are used as recommended templates.

[0455] S404: Fill the video clip into the corresponding video slot of the recommendation template according to the matching relationship corresponding to the recommendation template to obtain a recommended video.

[0456] According to the matching relationship corresponding to the recommendation template, that is, the matching relationship between the video slot in the recommendation template and the video clip, the video clip is filled into the corresponding video slot to obtain the recommended video.

[0457] In one embodiment, step S404 specifically determines whether the video duration of the video clip is greater than the duration of the video pit; if the video duration of the video clip is greater than the duration of the video pit, extract the video clip to obtain a selected clip.

[0458] When the video clip is filled into the corresponding video slot according to the matching relationship, it is determined whether the video duration of the video clip is greater than the duration of the video slot. The duration of the video slot determines the maximum duration of the video clip that can be filled into the video slot. Therefore, when the video duration of the video clip is greater than the duration of the video slot, the video clip cannot be directly filled into the corresponding video slot, and a clip of the corresponding duration needs to be extracted from the video clip and filled into the corresponding video slot.

[0459] The video duration of the selected segment is less than or equal to the duration of the video slot. In the specific implementation process, in order to fill the selected segment into the corresponding video slot and ensure the completeness of the recommended video, when determining the selected segment, the video duration of the selected segment can be equal to the duration of the video slot.

[0460] In one embodiment, extracting segments from the video segments to obtain selected segments includes: extracting segments from the video segments to obtain selected segments according to video elements of the video segments.

[0461] When extracting segments from video segments, the video segments may be extracted according to video elements of the video segments to obtain selected segments.

[0462] The video elements include at least one of a smiling face picture, a laughing sound audio, a character action, a clear human voice, a picture composition and an aesthetic score. When extracting the selected clips, more exciting clips can be extracted from the video clips according to the video elements as the selected clips, for example, clips including smiling faces or clips with higher aesthetic scores can be selected as the selected clips.

[0463] In one embodiment, step S404 specifically includes filling the video clip into the corresponding video slot of the recommendation template according to the matching relationship corresponding to the recommendation template to obtain the initial video; and optimizing the image of the initial video based on the template requirements of the recommendation template to obtain the recommended video.

[0464] After the video clips are filled into the corresponding video slots according to the matching relationship corresponding to the recommended template, an initial video is obtained, and then the image of the initial video can be optimized according to the template requirements, and the image-optimized video is recommended to the user as a recommended video. The template requirements include at least one of transition settings, acceleration and deceleration settings, and texture special effects settings.

[0465] In one embodiment, since the source of the video material to be processed may be an aerial video taken by a drone, and the distance between the camera and the object being photographed is relatively far during aerial photography, the content of the picture changes is small. Therefore, when the aerial video is filled into the corresponding video slot, the speed of picture changes can be automatically identified first, and the aerial video can be automatically speed-adjusted according to the speed of picture changes, and then the aerial video after speed adjustment can be filled into the corresponding video slot. The speed of picture changes can be obtained by analyzing multiple consecutive frames within a preset time.

[0466] In one embodiment, when determining the matching relationship between the video slots of the template and the video clips, for the identified aerial video, the aerial video can be placed in the first few video slots and / or the last few video slots of the template, thereby improving the quality of the recommended video.

[0467] The video processing method provided in the above embodiment obtains multiple templates and matches the video clips for the video pit positions of each template to obtain the matching relationship corresponding to each template, thereby determining the matching score of the matching relationship corresponding to each template, determining the recommended template from the multiple templates according to the matching score, and finally synthesizing the recommended video according to the matching relationship corresponding to the recommended template. Determining the recommended template according to the matching score and synthesizing the recommended video based on the recommended template can automatically determine the appropriate recommended template for the video clip, reduce the workload of the user when performing video editing, and improve the diversity of the synthesized recommended videos.

[0468] It should be noted that the above embodiments can be executed individually or in combination according to actual needs, and the specific execution order and combination method are not specifically limited; the individual steps can also be executed individually or in combination, and the specific execution order and combination method are not specifically limited.

[0469] See also Fig. 20 , Fig. 20 is a schematic block diagram of a video processing device provided in an embodiment of the present application. Fig. 20 As shown, the video processing device 500 further includes at least one or more processors 501 and a memory 502 .

[0470] The processor 501 may be, for example, a micro-controller unit (MCU), a central processing unit (CPU) or a digital signal processor (DSP).

[0471] The memory 502 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0472] Among them, the memory 502 is used to store computer programs; the processor 501 is used to execute the computer program and, when executing the computer program, execute any one of the video processing methods provided in the embodiments of the present application to reduce the workload of users when performing video editing and provide a variety of recommended videos.

[0473] See also Fig.21 , Fig.21 is a schematic block diagram of a terminal device provided in an embodiment of the present application. Fig.21 As shown, the terminal device 600 also includes at least one or more processors 601 and a memory 602 .

[0474] The terminal devices include mobile phones, remote controls, PCs, tablet computers and other terminals.

[0475] The processor 601 may be, for example, a micro-controller unit (MCU), a central processing unit (CPU), or a digital signal processor (DSP).

[0476] The memory 602 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0477] Among them, the memory 602 is used to store computer programs; the processor 601 is used to execute the computer program and, when executing the computer program, execute any one of the video processing methods provided in the embodiments of the present application to reduce the workload of users when performing video editing and provide a variety of recommended videos.

[0478] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement the steps of any one of the video processing methods provided in the above embodiments.

[0479] The computer-readable storage medium may be an internal storage unit of the terminal device described in any of the above embodiments, such as a storage device or a memory of the terminal device. The computer-readable storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.

[0480] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: include: Determine a template according to video information of the video material to be processed, wherein the template includes at least one video pit position; According to the pit position information of the video pit in the template, determine the video segment corresponding to the video pit from the video material to be processed, wherein the pit position information of the video pit is at least used to indicate the scene corresponding to the video to be filled in the video pit; Fill the video clip into the corresponding video slot of the template to obtain a recommended video.

2. The method according to claim 1, characterized in that The scene types corresponding to the video to be filled in the video slot include at least one of the following: close shot, medium shot, long shot, panoramic shot and special shot.

3. The method according to claim 1 or 2, characterized in that: The scene corresponding to the video to be filled in the video slot includes a scene change relationship corresponding to the video to be filled in the plurality of video slots.

4. The method according to claim 3, characterized in that The scene change relationship includes a scene increasing relationship and a scene decreasing relationship. The scene increasing relationship or the scene decreasing relationship is used to indicate the scene change relationship from a close shot to a distant shot or from a distant shot to a close shot.

5. The method according to claim 1, characterized in that The step of determining a template according to video information of the video material to be processed includes: Determining a video tag of the video material to be processed according to the video information of the video material to be processed; According to the video tag of the video material to be processed, a plurality of templates matching the video tag are determined.

6. The method according to claim 5, characterized in that The step of determining, according to the video tag of the video material to be processed, a plurality of templates matching the video tag comprises: Determining a video theme corresponding to the video material to be processed according to the video tag; According to the video theme of the video material to be processed, a plurality of templates matching the video theme are determined.

7. The method according to claim 5, characterized in that The video tag includes at least one of the camera direction, scene size, target size and position of a single video frame in the video material to be processed, target size and position of consecutive video frames in the video material to be processed, and similarity of adjacent video frames in the video material to be processed.

8. The method according to claim 7, characterized in that The size and position of the target object in a single video frame in the video material to be processed are determined using an object detection algorithm or a saliency detection algorithm, or the size and position of the target object in consecutive video frames in the video material to be processed are determined based on a pre-trained neural network model.

9. The method according to claim 3, characterized in that: The method further comprises: If the video theme corresponding to the to-be-processed video material cannot be determined, a preset template is selected as the template corresponding to the to-be-processed video material.

10. The method according to claim 1, characterized in that Before determining, from the video material to be processed, the video segment corresponding to the video position according to the position information of the video position in the template, the method further includes: The video material to be processed is segmented to generate multiple video segments.

11. A video processing device, characterized in that: The video processing device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program and, when executing the computer program, implement: The video processing method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the video processing method according to any one of claims 1 to 10.

Images