Video processing method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a video processing method, electronic equipment and a computer readable storage medium, and relates to the technical field of terminals. When the first video clip exceeds the maximum duration threshold, the mobile phone determines an audio associated frame in the first video clip; wherein the segment duration of the first video segment is greater than the maximum duration threshold, the first video segment comprises continuous multi-frame images and at least one audio segment, and the audio associated frame is an image frame corresponding to a starting moment or an ending moment of each audio segment in the continuous multi-frame images; splitting the first video clip into at least two second video clips; wherein the first frame or the tail frame of each second video clip is the audio associated frame, and the clip duration of any second video clip is less than or equal to the maximum duration threshold. In this way, when a long video clip is split, the integrity of the audio clip can be reserved as much as possible.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of video processing, and in particular, to a video processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Existing editing software can provide the function of automatically editing to obtain a video set. Mobile phones rely on intelligent algorithms and functions to automatically select and integrate video clips, apply transition effects, and even add background music and subtitles, etc. Through automated processing, these software can help us quickly create professional-level video finished products without spending a lot of time and energy.

[0003] In the process of existing application of finished videos, there are the following ways to process sound. The methods of selecting music replacement, selecting the overlay of original sound information and music, or selecting only retaining the original sound information all ignore the valuable information in the original sound information of the video, resulting in a poor effect of the synthesized video set. Summary of the Invention

[0004] Embodiments of the present application provide a video processing method, an electronic device, and a computer-readable storage medium, which are used to solve the technical problem that the valuable information in the original sound information of the video is ignored, resulting in a poor effect of the synthesized video set.

[0005] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a video processing method is provided, which is used to split a video clip with a clip duration exceeding a maximum duration threshold into video clips with shorter clip durations.

[0007] The mobile phone determines the audio-associated frames in the first video clip; wherein, the clip duration of the first video clip is greater than the maximum duration threshold, the first video clip includes a continuous multi-frame image and at least one audio clip, and the audio-associated frames are the image frames corresponding to the start moment or the end moment of each audio clip in the continuous multi-frame image; the first video clip is split into at least two second video clips; wherein, the first frame or the last frame of each second video clip is the audio-associated frame, and the clip duration of any second video clip is less than or equal to the maximum duration threshold.

[0008] In a specific implementation manner of the first aspect, the step of determining the audio-associated frames in the first video clip includes: determining the start moment and the end moment of the audio clip; selecting the image frames with acquisition moments close to the start moment and the end moment of the audio clip from the continuous multi-frame image as the audio-associated frames.

[0009] In a specific implementation of the first aspect, the step of selecting, from the continuous multi-frame images, the image frames whose acquisition times are close to the start time and the end time of the audio segment as the audio-associated frames includes: searching, from all the continuous multi-frame images, for two first image frames close to the start time of the audio segment; wherein, the acquisition time of one first image frame is before the start time of the audio segment, and the acquisition time of one first image frame is after the start time of the audio segment; taking the image frame among the two first image frames whose acquisition time is before the start time of the audio segment as the audio-associated frame corresponding to the start time of the audio segment.

[0010] In a specific implementation of the first aspect, the step of selecting, from the continuous multi-frame images, the image frames whose acquisition times are close to the start time and the end time of the audio segment as the audio-associated frames includes: searching, from all the continuous multi-frame images, for two second image frames close to the end time of the audio segment; wherein, the acquisition time of one second image frame is before the end time of the audio segment, and the acquisition time of one first image frame is after the end time of the audio segment; taking the image frame among the two second image frames whose acquisition time is after the end time of the audio segment as the audio-associated frame corresponding to the end time of the audio segment.

[0011] In a specific implementation of the first aspect, the step of searching, from all the continuous multi-frame images, for two first image frames close to the start time of the audio segment includes: using the binary search method to search, from all the continuous multi-frame images, for two first image frames close to the start time of the audio segment; and / or, using the binary search method to search, from all the continuous multi-frame images, for two second image frames close to the end time of the audio segment.

[0012] In a specific implementation manner of the first aspect, the step of splitting the first video segment into at least two second video segments includes: determining a first frame sequence from the consecutive multiple frames of images; the first frame sequence sequentially includes the first frame of the first video segment, all audio-associated frames, and the last frame of the first video segment; determining at least two frame groups in sequence according to the first frame sequence; wherein each of the frame groups includes a first starting frame and a first ending frame, and the segment duration between the acquisition time of the first starting frame and the acquisition time of the first ending frame in each frame group is less than or equal to the maximum duration threshold, and at least one of the first starting frame and the first ending frame in each frame group is an audio-associated frame, and the first ending frame of the previous frame group in two adjacent frame groups is the first starting frame of the next frame group; splitting the first video segment into corresponding second video segments of each of the frame groups according to the acquisition time of the first starting frame and the acquisition time of the first ending frame of each of the frame groups; wherein the acquisition time of the first starting frame of the frame group is the starting time of the corresponding second video segment, and the acquisition time of the second ending frame of the frame group is the ending time of the second video segment corresponding to the frame group.

[0013] In a specific implementation manner of the first aspect, the step of determining at least two frame groups in sequence according to the first frame sequence includes: taking the first frame of the first video segment as the first starting frame of the first frame group; according to the first starting frame of each frame group, sequentially searching for the first ending frame of each frame group according to the first frame sequence until the found first ending frame is the last frame of the first video segment.

[0014] In a specific implementation manner of the first aspect, the step of sequentially searching for the first ending frame of each frame group according to the first starting frame of each frame group and according to the first frame sequence includes: determining the first starting frame of the current frame group; searching for a first target frame whose acquisition time is after the acquisition time of the first starting frame of the current frame group according to the first frame sequence as the first ending frame of the current frame group; wherein the segment duration between the acquisition time of the first target frame and the acquisition time of the first starting frame of the current frame group is less than or equal to the maximum duration threshold, and the segment duration between the acquisition time of the next frame of the first target frame in the first frame sequence and the acquisition time of the first starting frame of the current frame group is greater than the maximum duration threshold.

[0015] In a specific implementation of the first aspect, the step of searching for a first target frame whose acquisition time is after the first start frame of the current frame group according to the first frame sequence and using it as the first end frame of the current frame group further includes: determining that the segment duration between the acquisition time of the next frame image of the first start frame of the current frame group in the first frame sequence and the acquisition time of the first start frame of the current frame group is greater than the maximum duration threshold; searching for a second target frame in the consecutive multiple frame images and using it as the first end frame of the current frame group; wherein, the segment duration between the acquisition time of the second target frame and the acquisition time of the first start frame of the current frame group is equal to the maximum duration threshold.

[0016] In a specific implementation of the first aspect, the step of splitting the first video segment into at least two second video segments includes: splitting the consecutive multiple frame images of the first video segment and the audio segment to obtain at least two second video segments.

[0017] In a second aspect, a video processing method is provided. The method includes: determining the basic materials to be processed; at least one of the basic materials being a video material including an audio segment; if the video material includes a first video segment with a segment duration greater than the maximum duration threshold, splitting the first video segment into at least two second video segments according to the audio-related frames in the first video segment; wherein, the audio-related frames are the image frames corresponding to the start time or end time of each audio segment in the consecutive multiple frame images, the first frame or the last frame of each second video segment is the audio-related frame, and the segment duration of any second video segment is less than or equal to the maximum duration threshold; extracting highlight segments from all the basic materials according to the image parameters and audio parameters of all the basic materials; generating a video set from all the highlight segments.

[0018] In a specific implementation of the second aspect, the step of extracting highlight segments from all the basic materials according to the image parameters and audio parameters of all the basic materials includes: respectively calculating a first score for each frame image based on the image parameters of each frame image in the basic materials; respectively calculating a second score for each frame image based on the audio parameters of each frame image in the basic materials; extracting a first highlight segment from the basic materials based on the first score and the second score of each frame image.

[0019] In a specific implementation of the second aspect, the step of respectively calculating a second score for each frame image based on the audio parameters of each frame image in the basic materials includes: calculating a second score for each frame image according to the existence state of the audio segment corresponding to each frame image.

[0020] In a specific implementation of the second aspect, the step of calculating a second score for each frame image according to the existence status of the audio segment corresponding to each frame image includes: adding a fade-in period and a fade-out period to each audio segment to determine the existence period of the audio segment; wherein, the existence period of the audio segment includes a fade-in period, an acquisition period of the valid audio segment, and a fade-out period; if the acquisition moment of the current frame image is within the existence period of any audio segment, calculate a second score for the current frame image.

[0021] In a specific implementation of the second aspect, the step of extracting a highlight segment from the basic material based on the first score and the second score of each frame image includes: calculating a frame score for each frame image according to the first score and the second score of each frame image; extracting a highlight segment from the basic material according to the frame scores of each frame image in all the basic materials.

[0022] In a specific implementation of the second aspect, the step of extracting a highlight segment from the basic material according to the frame scores of each frame image in all the basic materials includes: filtering out the images with frame scores greater than a first threshold from all the images to obtain a first sequence of image frames; if there is an interval duration greater than an interval threshold between adjacent images in the first sequence of image frames, splitting the adjacent images with an interval duration greater than the interval threshold into different video segments to obtain one or more first video segments; filtering out the first video segments with segment durations less than a minimum duration value from all the first video segments, and splitting the first video segments with segment durations greater than a maximum duration threshold to obtain a plurality of the second video segments; obtaining a plurality of highlight segments according to the plurality of second video segments.

[0023] In a third aspect, there is provided an electronic device, including a display, a memory, and a processor; the memory stores a computer program; the processor executes the computer program to implement the video processing method according to any one of the first aspect or the second aspect.

[0024] In a fourth aspect, there is provided a computer-readable storage medium, in which a computer program is stored, and when the computer-readable storage medium runs on a computer, the computer is enabled to execute the video processing method according to any one of the first aspect or the second aspect. Description of the Drawings

[0025] Figure 1 Schematic diagram of the application scenario and interface implementation of the video processing method provided by the embodiment of the present application;

[0026] Figures 2 - 4 Schematic diagram of video segment splitting involved in the video processing method provided by the embodiment of the present application;

[0027] Figure 5The flowchart of the video processing method provided by the embodiments of this application relates to the splitting of video segments;

[0028] Figure 6 The schematic diagram of the basic materials involved in the video processing method provided by the embodiments of this application;

[0029] Figure 7 The flowchart of the method for extracting highlight segments involved in the video processing method provided by the embodiments of this application;

[0030] Figure 8 The schematic diagram of the video material including audio segments involved in the video processing method provided by the embodiments of this application;

[0031] Figure 9 The schematic diagram of adding the same fade-in period and fade-out period to the audio segment involved in the video processing method provided by the embodiments of this application;

[0032] Figure 10 The schematic diagram of adding different fade-in periods and fade-out periods to the audio segment involved in the video processing method provided by the embodiments of this application;

[0033] Figure 11 One schematic diagram of calculating the second score involved in the video processing method provided by the embodiments of this application;

[0034] Figure 12 Another schematic diagram of calculating the second score involved in the video processing method provided by the embodiments of this application;

[0035] Figure 13 Another schematic diagram of calculating the second score involved in the video processing method provided by the embodiments of this application;

[0036] Figure 14 One schematic diagram of screening images according to the frame score involved in the video processing method provided by the embodiments of this application;

[0037] Figure 15 Another schematic diagram of screening images according to the frame score involved in the video processing method provided by the embodiments of this application;

[0038] Figure 16 The schematic diagram of splitting video segments according to the duration interval involved in the video processing method provided by the embodiments of this application;

[0039] Figure 17 The schematic diagram of splitting video segments according to the storyboard points involved in the video processing method provided by the embodiments of this application;

[0040] Figure 18Schematic diagram of splitting a video segment according to the segment duration involved in the video processing method provided by the embodiment of the present application;

[0041] Figure 19 Schematic diagram of determining a target audio segment involved in the video processing method provided by the embodiment of the present application;

[0042] Figure 20 Another schematic diagram of determining a target audio segment involved in the video processing method provided by the embodiment of the present application;

[0043] Figure 21 Another schematic diagram of determining a target audio segment involved in the video processing method provided by the embodiment of the present application;

[0044] Figure 22 Schematic diagram of the software and hardware structure of the electronic device provided by the embodiment of the present application;

[0045] Figure 23 A schematic diagram of a thread corresponding to the video processing method provided by the embodiment of the present application;

[0046] Figure 24 Another schematic diagram of a thread corresponding to the video processing method provided by the embodiment of the present application;

[0047] Figure 25 Schematic diagram of the node flow involved in the video processing method provided by the embodiment of the present application;

[0048] Figure 26 Hardware structure diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0049] The following describes exemplary embodiments of the present application with reference to the accompanying drawings. Various details of the embodiments of the present application are included to assist understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0050] For ease of understanding, some common technical knowledge related to the embodiments of the present application is introduced first.

[0051] A highlight segment, also known as a wonderful segment, refers to a single-frame image or a video segment composed of consecutive multiple-frame images extracted from a video or an image for recording wonderful moments. The wonderful moments can be the moments when a person smiles, wins a championship, or wonderful actions corresponding to an airplane landing occur, etc.

[0052] The one - click video compilation function refers to that when an electronic device responds to a user's selection operation on one or more materials, it automatically analyzes the highlight segments in the materials through an algorithm and combines the highlight segments into a compiled video.

[0053] That is to say, the electronic device can extract multiple highlight segments from one or more materials and synthesize the multiple highlight segments into a video set. Among them, the materials selected by the user can be image materials or video materials.

[0054] Among them, the process of the electronic device selecting highlight segments from a video or an image can include: obtaining scoring parameters such as the image color, image texture features, image quality, frame interpolation with the previous and next frames, and edge change rate value of each frame of the image, and scoring each frame of the image according to these scoring parameters. The electronic device can use a single frame of the image or consecutive multiple frames of the image with a score greater than or equal to a preset score as a highlight segment.

[0055] To sum up, a main operation for the electronic device to implement the one - click video compilation function is to obtain highlight segments. However, in conventional technologies, when analyzing highlight segments, only the image information in the above - mentioned materials is used. In the case where the materials selected by the user include video materials, both the image information and the sound information are very important for the final effect of the synthesized video. Based on this, in the solution provided in the embodiments of the present application, when the electronic device implements the above - mentioned one - click video compilation function, it can analyze and obtain highlight segments by combining the image information and the sound information in the materials, and then synthesize a video based on the highlight segments. By adopting this solution, the wonderful image information and sound information in the materials are retained in the video set synthesized according to the materials, thereby making the information of the obtained video set richer and the playback effect better.

[0056] For ease of understanding, the following Figure 1 , taking the electronic device as a mobile phone as an example, exemplarily introduces the application scenarios and interface implementation for the electronic device to implement the "one - click video compilation" function.

[0057] In one application scenario, the user can use the mobile phone to pre - shoot multiple materials, and the mobile phone stores the multiple materials in the gallery. The mobile phone can also pre - obtain materials transmitted from other devices. As Figure 1 shown in (a) below, an icon of the gallery is displayed on the desktop of the mobile phone. When the user wants to use the mobile phone to generate a compiled video set based on multiple materials, the user can click on the icon of the gallery on the mobile phone desktop. In response to the user's click operation on the icon, the mobile phone displays the gallery interface 101 as shown in Figure 1 (b) below. The gallery interface 101 includes an option of "one - click video compilation". After that, in response to the user's click operation on the Figure 1 "one - click video compilation" option shown in (b) below, it can display as shown in Figure 1The gallery interface 102 shown in (c) thereof. Multiple materials taken recently may be included in the gallery interface 102. The user can select Figure 1 any one or more of the multiple materials shown in (c) thereof as candidate materials to generate a one-click blockbuster. For example, in response to the user's Figure 1 selection operation on some of the materials in (c), the mobile phone can display a gallery interface 103 as shown in (d) of Figure 1 . The gallery interface 103 includes all the materials in the gallery. The gallery interface 103 may further include a video generation option, such as a "√" check option.

[0058] In response to the user's click operation on the "√" check option shown in (d) of Figure 1 , the mobile phone can analyze the 4 materials selected by the user in the gallery interface 103, select highlight segments from each material, and generate a video set according to the selected highlight segments. During this process, the mobile phone can display the material analysis progress so that the user can intuitively view the analysis progress.

[0059] In the one-click blockbuster scenario or other video editing scenarios, the mobile phone is involved in splitting a video segment whose segment duration exceeds the maximum duration threshold into multiple video segments with segment durations not exceeding the maximum duration threshold.

[0060] The conventional video splitting scheme adopted by the mobile phone is to sequentially split the video segment from the starting moment of the video segment according to the maximum duration threshold until all the video segments are split. For the case where the video segment includes an audio segment corresponding to the acquisition period, the scheme of sequentially splitting the video segment according to the maximum duration threshold may cause the corresponding audio segment to be split, and the information of the split shorter audio segments is incomplete or less, and it is easily discarded in the video editing scenario, resulting in the lack of valuable audio segments during video editing and poor video processing effects.

[0061] During the process of the mobile phone splitting the video segment according to the segment duration, the segment duration of the video segment exceeds the maximum duration threshold. The video segment also corresponds to an audio segment. From the starting moment of the video segment, the mobile phone splits the video segment and the audio segment equally according to the maximum duration threshold to obtain multiple video segments. As a result, the audio segment is split indiscriminately, and the information of the split audio segment is incomplete or less.

[0062] Based on this, this embodiment provides a video segment splitting method. When splitting a video segment whose segment duration exceeds the maximum duration threshold into multiple sub-segments, the starting moment and the ending moment of the audio segment are referred to, so that the split video segments neither exceed the maximum duration threshold nor can the audio segment be split as much as possible.

[0063] Such asFigure 2 As shown, the provided video clip splitting method mainly includes the following steps:

[0064] S201: The mobile phone determines the audio-related frames in the first video clip.

[0065] The clip duration of the first video clip is greater than the maximum duration threshold. The first video clip includes a continuous sequence of multiple frames of images and at least one audio clip.

[0066] In one example, as shown in (a) of Figure 3 Take the first video clip including a continuous sequence of 20 frames of images as an example. In a video processing scenario, the video clip selected by the user may include a video clip of at least 1 second, and there are usually 50 frames or 120 frames of images or more in the video clip. Here, it is only for a brief explanation of this solution. For example, the first video clip includes a continuous sequence of 20 frames of images, which is not limited to this.

[0067] As shown in (a) of Figure 3 The first video clip includes a continuous sequence of multiple frames of images f1 - f20, and the corresponding acquisition times are t1 - t20 in sequence. The first frame of the first video clip is f1, the last frame of the first video clip is f20, and the duration of the first video clip is the period between Ts1 and Te1, that is, the period between the acquisition time t1 of the first frame f1 of the first video clip and the acquisition time t20 of the last frame f2 of the first video clip. The clip duration of the first video clip is (Te1 - Ts1), or it can also be (t20 - t1). The video processing method provided in this embodiment mainly splits the video for the case where the clip duration of the first video clip is greater than the maximum duration threshold L, and the split video clips are denoted as second video clips. The first video clip is split into multiple second video clips, and the clip durations of the second video clips are all less than or equal to the maximum duration threshold.

[0068] The first video clip also includes an audio clip. The audio clip mentioned here refers to the audio clip composed of the audio signals extracted by the audio recognition algorithm of the mobile phone from the audio track of the first video clip, and the audio clip is a whole continuous audio clip.

[0069] In this whole continuous audio clip, the audio signals of some clips (between ts1 and te1, or between ts2 and te2) fluctuate greatly and contain more audio information, and are usually valid audio clips. There are also some clips with relatively small fluctuations in audio signals and less contained audio information, which are usually invalid audio clips.

[0070] The mobile phone stores an audio analysis algorithm, which can analyze the valid audio clips in the audio clip included in the first video clip. As shown in Figure 3As shown in (b) of [the figure], the mobile phone can call an audio analysis algorithm to retain valid audio segments from a whole audio segment included in the first video segment, obtaining two valid audio segments, namely audio segment 1 and audio segment 2. The audio segments referred to in the subsequent embodiments of this example all refer to the valid audio segments identified by the mobile phone calling the audio analysis algorithm.

[0071] Among the consecutive multiple frames of images in the first video segment, there are some image frames whose acquisition times are relatively close to or coincide with the start time or end time of the audio segment. These image frames can be recorded as audio-associated frames.

[0072] In a specific implementation, when the mobile phone executes S201 to determine the audio-associated frames in the first video segment, the steps may include:

[0073] The mobile phone determines the start time and end time of the audio segment, and selects, from the consecutive multiple frames of images, the image frames whose acquisition times are close to the start time and the end time of the audio segment as the audio-associated frames.

[0074] Furthermore, the mobile phone searches in all the consecutive multiple frames of images for two first image frames close to the start time of the audio segment; among them, the acquisition time of one first image frame is before the start time of the audio segment, and the acquisition time of one first image frame is after the start time of the audio segment;

[0075] The image frame among the two first image frames whose acquisition time is before the start time of the audio segment is used as the audio-associated frame corresponding to the start time of the audio segment.

[0076] For example, the binary search method is used to search in all the consecutive multiple frames of images for two first image frames close to the start time of the audio segment.

[0077] The binary search method here may include a binary search left boundary algorithm and a binary search right boundary algorithm. For example, the audio-associated frame on the left side of the start time of the audio segment can be searched through the binary search left boundary algorithm. For another example, the audio-associated frame on the right side of the end time of the audio segment can also be searched through the binary search right boundary algorithm.

[0078] In addition, the mobile phone searches in all the consecutive multiple frames of images for two second image frames close to the end time of the audio segment; among them, the acquisition time of one second image frame is before the end time of the audio segment, and the acquisition time of one first image frame is after the end time of the audio segment;

[0079] Among the two second image frames, the image frames whose acquisition times are after the termination time of the audio segment are used as the audio correlation frames corresponding to the termination time of the audio segment.

[0080] For another example, the mobile phone can use the binary search method to find two second image frames close to the termination time of the audio segment from all consecutive multiple frames of images.

[0081] Continuing as shown in (b) of Figure 3 The start time ts1 of audio segment 1. The acquisition time t2 of image frame f2 and the acquisition time t3 of image frame f3 are relatively close to the start time ts1 of audio segment 1. Among them, the acquisition time t2 of image frame f2 is before the start time ts1 of audio segment 1, and the acquisition time t3 of image frame f3 is after the start time ts1 of audio segment 1.

[0082] The termination time te1 of audio segment 1. The acquisition time t5 of image frame f5 and the acquisition time t6 of image frame f6 are relatively close to the termination time te1 of audio segment 1. Among them, the acquisition time t5 of image frame f5 is before the termination time te1 of audio segment 1, and the acquisition time t6 of image frame f6 is after the termination time te1 of audio segment 1.

[0083] The start time ts2 of audio segment 2. The acquisition time t11 of image frame f11 and the acquisition time t3 of image frame f12 are relatively close to the start time ts2 of audio segment 2. Among them, the acquisition time t11 of image frame f11 is before the start time ts2 of audio segment 2, and the acquisition time t12 of image frame f12 is after the start time ts2 of audio segment 2.

[0084] The termination time te2 of audio segment 2. The acquisition time t15 of image frame f15 and the acquisition time t16 of image frame f16 are relatively close to the termination time te2 of audio segment 2. Among them, the acquisition time t15 of image frame f15 is before the termination time te2 of audio segment 2, and the acquisition time t16 of image frame f16 is after the termination time te2 of audio segment 2.

[0085] As shown in (c) of Figure 3 The mobile phone can record f2, f6, f11, and f16 as audio correlation frames.

[0086] S202: The mobile phone splits the first video segment into at least two second video segments; among them, the first frame or the last frame of each second video segment is the audio correlation frame, and the segment duration of any second video segment is less than the maximum duration threshold.

[0087] The mobile phone refers to the audio-associated frames corresponding to the start time and end time of each audio segment in the first video segment to perform video splitting, and multiple second video segments can be obtained.

[0088] Splitting according to the video processing method provided in this embodiment can obtain a second video segment as shown in (d) of Figure 3 .

[0089] In a specific implementation manner, the step in which the mobile phone executes S202 to split the first video segment into at least two second video segments may specifically include:

[0090] The mobile phone determines a first frame sequence from the consecutive multiple frames of images; the first frame sequence sequentially includes the first frame of the first video segment, all audio-associated frames, and the last frame of the first video segment;

[0091] According to the first frame sequence, at least two frame groups are sequentially determined; wherein, each frame group includes a first start frame and a first end frame, the segment duration between the acquisition time of the first start frame and the acquisition time of the first end frame in each frame group is less than or equal to the maximum duration threshold, and at least one of the first start frame and the first end frame in each frame group is an audio-associated frame, and the first end frame of the previous frame group in two adjacent frame groups is the first start frame of the subsequent frame group;

[0092] According to the acquisition time of the first start frame and the acquisition time of the first end frame of each frame group, the first video segment is split into corresponding second video segments of each frame group; wherein, the acquisition time of the first start frame of the frame group is the start time of the corresponding second video segment, and the acquisition time of the second end frame of the frame group is the end time of the second video segment corresponding to the frame group.

[0093] In a specific implementation manner, the step in which the mobile phone determines a frame group according to the first frame sequence may include:

[0094] The mobile phone uses the first frame of the first video segment as the first start frame of the first frame group, and according to the first start frame of each frame group, sequentially searches for the first end frame of each frame group in the first frame sequence until the found first end frame is the last frame of the first video segment.

[0095] Further, the mobile phone first determines the first starting frame of the current frame group, and then searches for the first target frame whose acquisition time is after the first starting frame of the current frame group according to the first frame sequence, and uses it as the first termination frame of the current frame group. Wherein, the segment duration between the acquisition time of the first target frame and the acquisition time of the first starting frame of the current frame group is less than or equal to the maximum duration threshold, and the segment duration between the acquisition time of the next frame of the first target frame in the first frame sequence and the acquisition time of the first starting frame of the current frame group is greater than the maximum duration threshold.

[0096] As Figure 3 shown in (c) of Figure 3 , the mobile phone determines the audio-associated frames f2, f6, f11, and f16 of the first video segment. Combining the first frame f1 and the last frame f20 of the first video segment, the first frame sequence [f1, f2, f6, f11, f16, f20] can be obtained, as

[0097] shown in (e) of Figure 4 . As shown in (a) of

[0098] The mobile phone determines that the first starting frame of the first frame group is f1, and the acquisition time is t1 according to the first frame sequence [f1, f2, f6, f11, f16, f20].

[0099] The process by which the mobile phone determines the first frame group is as follows:

[0100] The mobile phone determines that the next frame of f1 in the first frame sequence is f2, and the next frame of f2 in the first frame sequence is f6.

[0101] The segment duration L21 (t2 - t1) between f1 and f2 ≤ L, and the segment duration L61 (t6 - t1) between f1 and f6 > L.

[0102] The mobile phone uses f2 as the first termination frame of the first frame group to obtain the first frame group [f1, f2].

[0103] The mobile phone can split the video segment between t1 and t2 in the first video segment into 1 second video segment V1 according to the acquisition time t1 of f1 and the acquisition time t2 of f2, as Figure 4 shown in (b) of

[0104] The process by which the mobile phone determines the second frame group is as follows:

[0105] The mobile phone determines that the first starting frame of the second frame group is the first termination frame of the first frame group, that is, f2. The mobile phone determines that the next frame of f1 in the first frame sequence is f6, and the next frame of f6 in the first frame sequence is f11.

[0106] The duration L62 (t6 - t2) of the segment between f2 and f6 is ≤ L, and the duration L112 (t11 - t2) of the segment between f2 and f11 is > L.

[0107] The mobile phone uses f6 as the first termination frame of the second frame group, obtaining the second frame group [f2, f6].

[0108] The mobile phone can, according to the acquisition time t2 of f2 and the acquisition time t6 of f6, split the video segment between t2 and t6 in the first video segment into 1 second video segment V2. This video splitting operation does not involve the audio segment, and the complete segment of audio segment 2 is retained in the corresponding second video segment.

[0109] The process by which the mobile phone determines the third frame group is as follows:

[0110] The mobile phone determines the first starting frame of the third frame group as the first termination frame of the second frame group, i.e., f6. The mobile phone determines the next frame of f6 in the first frame sequence as f11, and the next frame of f11 in the first frame sequence as f20.

[0111] The duration L116 (t11 - t6) of the segment between f6 and f11 is > L. In this case, it is necessary to determine a certain frame before f11 as the first termination frame to ensure that the segment duration of the split second video segment ≤ the maximum duration threshold.

[0112] The previous frame of f11, i.e., f10, and the duration L106 (t10 - t6) of the segment between f6 and f10 is ≤ L. The mobile phone can use f10 as the first termination frame of the third frame group.

[0113] The mobile phone uses f10 as the first termination frame of the third frame group, obtaining the third frame group [f6, f10]. The mobile phone can, according to the acquisition time t6 of f6 and the acquisition time t10 of f10, split the video segment between t6 and t10 in the first video segment into 1 second video segment V3. This video splitting operation does not involve the splitting of the audio segment.

[0114] The process by which the mobile phone determines the fourth frame group is as follows:

[0115] The mobile phone determines the first starting frame of the fourth frame group as the first termination frame of the third frame group, i.e., f10. The mobile phone determines the next frame of f10 in the first frame sequence as f16, and the next frame of f16 in the first frame sequence as f20.

[0116] The duration L1610 (t16 - t10) of the segment between f10 and f16 is > L. In this case, it is necessary to determine a certain frame before f16 as the first termination frame to ensure that the segment duration of the split second video segment ≤ the maximum duration threshold.

[0117] The previous frame of f16 is f15, and the segment duration L1510 (t15 - t10) between f10 and f15 is > L. The mobile phone needs to further determine the previous frame of f15, which is f14.

[0118] The segment duration L1410 (t14 - t10) between f10 and f14 is ≤ L, and the mobile phone can use f14 as the first termination frame of the fourth frame group.

[0119] The mobile phone uses f14 as the first termination frame of the third frame group, obtaining the third frame group [f10, f14]. The mobile phone can split the video segment between t10 and t14 in the first video segment into 1 second video segment V4 according to the acquisition time t10 of f10 and the acquisition time t14 of f14. This video splitting operation involves splitting the audio segment into two audio segments, such as Figure 4 Audio2 and Audio3 shown in (b) of

[0120] The process by which the mobile phone determines the fifth frame group is as follows:

[0121] The mobile phone determines that the first start frame of the fifth frame group is the first termination frame of the fourth frame group, which is f14. The mobile phone determines that the next frame of f14 in the first frame sequence is f20, which is the termination frame of the first video segment.

[0122] The segment duration L2014 (t20 - t14) between f14 and f20 is > L. In this case, it is necessary to determine a certain frame before f20 as the first termination frame to ensure that the segment duration of the split second video segment ≤ the maximum duration threshold.

[0123] The previous frame of F20 is f19, and the segment duration L1914 (t19 - t14) between f19 and f14 is > L. The mobile phone needs to further determine the previous frame of f19, which is f18.

[0124] The segment duration L1814 (t18 - t14) between f14 and f18 is ≤ L, and the mobile phone can use f18 as the first termination frame of the fifth frame group.

[0125] The mobile phone uses f18 as the first termination frame of the fifth frame group, obtaining the fifth frame group [f14, f18]. The mobile phone can split the video segment between t14 and t18 in the first video segment into 1 second video segment V5 according to the acquisition time t14 of f14 and the acquisition time t18 of f18. This video splitting operation involves splitting the audio segment, including part of the audio segment Audio3 after splitting audio segment 2.

[0126] The process by which the mobile phone determines the sixth frame group is as follows:

[0127] The mobile phone determines that the first starting frame of the sixth frame group is the first ending frame of the fourth frame group, i.e., f18. The mobile phone determines that the next frame of f18 in the first frame sequence is f20, which is the ending frame of the first video segment.

[0128] The segment duration L2018 (t20 - t18) between F20 and f18 ≤ L, and the mobile phone can use f20 as the first ending frame of the fifth frame group.

[0129] The mobile phone uses f20 as the first ending frame of the fifth frame group to obtain the fifth frame group [f18, f20]. The mobile phone can split the video segment between t18 and t20 in the first video segment into 1 second video segment V6 according to the acquisition time t18 of f18 and the acquisition time t20 of f20. This video splitting operation does not involve the splitting of the audio segment.

[0130] As Figure 4 shown in (b), when the mobile phone splits the first video segment into multiple second video segments according to each frame group, although the same image frame is respectively located in two adjacent frame groups before and after, this image frame can be divided into the second video segment corresponding to the previous frame group.

[0131] For example, the first frame group [f1, f2], the second frame group [f2, f6]. The image frame f2 is divided into the second video segment V1 corresponding to the first frame group, and the next frame after f2, i.e., from f3 to f6, is divided into the second video segment V2 corresponding to the second frame group. Another example, the second frame group [f2, f6], the third frame group [f6, f10]. The image frame f6 is divided into the second video segment V2 corresponding to the second frame group, and the next frame after f6, i.e., from f7 to f10, is divided into the second video segment V3 corresponding to the third frame group.

[0132] In other embodiments, the mobile phone may not use the first ending frame of the previous frame group as the first starting frame of the next frame group.

[0133] For example Figure 4 in, when determining the third frame group, the segment duration between the first starting frame f6 of the third frame group and the next audio-associated frame f11 is greater than the maximum duration threshold. The binary search method can also be directly used to start from f6 and search for the corresponding first ending frame according to the maximum duration threshold. For example, if the first ending frame is f10, then f6 - f10 can be split into 1 video segment, and f10 - f11 can be split into 1 video segment. According to this method, there may be 1 second video segment or multiple second video segments split between f6 - f11, which is not limited.

[0134] Then when determining the fourth frame group, f11 can be used as the first starting frame of the fourth frame group to search for the corresponding first ending frame.

[0135] As shown Figure 5 in the figure, it is a schematic diagram of the algorithm of the display control method provided in this embodiment. First, the concepts involved in Figure 5 are explained.

[0136] Split serial number array: Use the array L to store the serial numbers of the frames, with a length of n.

[0137] Video segment array: Use the array T to store the timestamps of the frames, with a length of m.

[0138] Maximum duration limit: Use the constant a to represent the maximum duration limit of the highlight segment.

[0139] Highlight segment array: Use the array R to store the set of highlight segments. The highlight segment can be created by the start and end frame serial numbers.

[0140] Head pointer, tail pointer: Use i and j to represent the head pointer and the tail pointer respectively, which point to the split serial number array.

[0141] As shown Figure 5 in the figure, it mainly includes the following steps:

[0142] S501, calculate the length n of the frame serial number array L.

[0143] The mobile device constructs the split serial number array L.

[0144] In the entire video segment, use the binary search left boundary algorithm and the binary search right boundary algorithm to find the video frames on the left side of the start time point of each original sound segment and the video frames on the right side of the end time point, and add the first frame and the last frame of the video segment. These frame serial numbers together form the array L, with a length of n. Since the set of original sound segments is ordered, the frame serial numbers are arranged from small to large.

[0145] S502, set the head and tail pointers, j = 0, i = 0;

[0146] The purpose of setting the head and tail pointers is to traverse the entire array L within linear time. The head pointer and the tail pointer are both initialized to the start position of the array, j = 0, i = 0.

[0147] S503, judge whether j < n.

[0148] Judge whether the tail pointer exceeds the array boundary.

[0149] If j ≥ n, then end.

[0150] If j < n, execute S504, judge whether T[L[j]] - T[L[i]] > a.

[0151] If the tail pointer does not exceed the array boundary, determine whether the duration corresponding to the frame numbers of the head and tail pointers exceeds the duration upper limit in the video interval. If it does not exceed, execute S506; if it exceeds, execute S505.

[0152] If T[L[j]] - T[L[i]] ≤ a, then jump to execute S509.

[0153] If T[L[j]] - T[L[i]] ≤ a, execute S505, and determine that L[j - 1] ≠ L[i].

[0154] If the tail pointer exceeds the array boundary, determine whether the tail pointer is larger than the head pointer and the difference is 1.

[0155] If the tail pointer is larger than the head pointer and the difference is 1, it means it is a sub - segment and this segment is over - long. There may be original sound in this segment, or there may be an interval without original sound between two original sounds. This over - long segment needs to be split into several short segments that do not exceed the maximum duration upper limit. According to the known timestamp information and the maximum duration upper limit, use the binary search left - boundary algorithm to search for the frame numbers in turn, and use these frame numbers to form new sub - segments. The sub - segment consists of two frame numbers. If the duration of the sub - segment exceeds the upper limit, continue to split.

[0156] If L[j - 1] ≠ L[i], execute S506 and save the segments L[i], L[j - 1].

[0157] If L[j - 1] == L[i], execute S507, crop the segment and save L[i], L[j - 1].

[0158] S508, assign j - 1 to i, and assign j - 1 to j.

[0159] S509, determine whether j == n - 1 and i ≠ j.

[0160] If so, execute S510 and save the segments L[i], L[j - 1].

[0161] If not, execute S511, assign j + 1 to i, and jump to execute S503.

[0162] If the tail pointer is larger than the head pointer but the difference is not 1, it means that the video segment interval composed of the head and tail pointers consists of several sub - segments, and the segment after the current tail pointer is over - long. Therefore, add the video segment interval [i, j - 1] to the highlight segment array.

[0163] Determine whether the tail pointer reaches the array boundary and whether the head pointer is the same as the tail pointer.

[0164] If the tail pointer reaches the array boundary and the head pointer is different from the tail pointer, it indicates that the video segment interval composed of the head and tail pointers is the last interval. Therefore, add the interval [i, j] to the highlight segment array and continue to execute S507. Output the highlight segment array and the program ends.

[0165] In addition, the video processing method provided in this embodiment can also reduce the overall complexity.

[0166] Time complexity: Assume that a video segment contains k original sound segments. The time complexity of a single binary search is O(log m), and the time complexity of constructing the segmentation sequence number array L is O(2k log m). In the best case, the time complexity of traversing the array L using two pointers is O(n). In the worst case, the time complexity is O(k log m), where k and log m are negatively correlated and cannot increase simultaneously. In fact, m >> n and m >> k. Therefore, the overall time complexity is O(k log m).

[0167] Space complexity: The space complexity of the segmentation sequence number array is O(2k + 2), and the space complexity of the video segment array is O(n). In this method, only the sequence number array is newly added and m >> k. Therefore, the space complexity is relatively low.

[0168] The above video processing method provided in this embodiment can also be applied to the one-key blockbuster scenario.

[0169] During the process of the mobile phone executing the one-key video creation operation, according to multiple materials selected by the user, analyze the images or videos of each material and extract the highlight segments therefrom. When the mobile phone analyzes the video material, the highlight segments obtained according to the analysis of the video material include image information and sound information, and synthesize the highlight segments containing image information and sound information to obtain a video set. In this way, the obtained video set contains wonderful image information and audio segments, and the playback effect of the synthesized video is better.

[0170] The video processing method provided by another embodiment of this application can be applied to an electronic device with image processing capabilities. The applicable electronic devices may include a personal computer (PC), a laptop computer, a portable computer, a mobile phone, a smart TV, a wearable device, a tablet (Pad), a computer with wireless transceiver function, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, or a wireless terminal in smart home, etc. Embodiments of this application do not limit the specific technologies and specific device forms adopted by the electronic device.

[0171] When a mobile phone executes the video processing method provided in this embodiment, it mainly includes the following steps:

[0172] The mobile phone determines the basic materials to be processed; at least one basic material is a video material including an audio segment.

[0173] The mobile phone extracts highlight segments from all the basic materials according to the image parameters and audio parameters of all the basic materials.

[0174] The mobile phone generates a video set according to all the highlight segments.

[0175] The mobile phone receives the user's selection operation and starts to execute the video processing operation. In response to the received selection operation, the mobile phone determines one or more basic materials to be processed.

[0176] In one example, as shown in (b) of Figure 1 , on the gallery interface 101, the mobile phone receives the click operation of the user on the "one-click blockbuster" option. As shown in (c) of Figure 1 , on the gallery interface 102, the mobile phone can also receive the selection operation on various materials and determine the materials receiving the selection operation as the basic materials to be processed. Or on the gallery interface, the mobile phone can receive the click operation on various materials and determine the materials receiving the click operation as the basic materials to be processed.

[0177] In another example, on the gallery interface, the mobile phone may also receive a click operation on each album, and determine all materials contained in the album receiving the click operation as basic materials to be processed.

[0178] In some embodiments, the basic material may include video material 1, which includes multiple frames of images and audio clips. In other embodiments, the basic material includes not only the video material 1, but also an image method. In other embodiments, the basic material includes not only the video material 1, but also an image method and video material 2. The video material 2 includes multiple frames of images and may not include audio clips.

[0179] Exemplarily, the above basic materials may include Figure 6 P1, P2, P3 and P4 shown in (a).

[0180] like Figure 6 As shown in (b) in FIG. 1 , the basic material P2 is an image method, and the image method only includes a single frame image. Figure 6 As shown in (c) in FIG. 1 , the basic material P1 is the video material 2, and the video material may include a plurality of continuous frames of images. Figure 6 As shown in (d) in FIG. 1 , the basic material P3 is the video material 1, which may include a continuous multi-frame image and an audio clip. Figure 6 As shown in (e) in FIG. 1 , the basic material P4 is the video material 1, which may include a continuous multi-frame image and an audio clip. Figure 6 (d) and Figure 6 As shown in (e), the audio segment included in each video material is a continuous audio segment. The audio segment recognized by the mobile phone refers to the audio segment obtained by the mobile phone by extracting the audio signal from the audio track data of the video material, and the time period of the audio segment corresponds to the time period of the continuous multiple frames of images.

[0181] In a specific implementation manner, if among all the basic materials determined by the mobile phone, there is at least one video material including an audio clip, the video processing method provided in this embodiment can be used.

[0182] If all the basic materials determined by the mobile phone do not contain at least one video material including an audio frequency band, for example, all the basic materials are image materials, a conventional video processing method can be used to synthesize multiple image materials to obtain a video set.

[0183] Alternatively, if all the basic materials determined by the mobile phone include video materials, but the included video materials do not include audio, the mobile phone can use a conventional video processing method to synthesize multiple basic materials to obtain a video set. The mobile phone can also use the video processing method provided in this embodiment to skip the relevant operation process involving audio.

[0184] The mobile phone selects the basic materials to be processed, receives the user's operations, and starts to execute the video processing operations on the basic materials. The video processing operations executed by the mobile phone mainly include two steps. First, extract the highlight segments from all the basic materials, and then synthesize all the highlight segments to obtain a video set.

[0185] As Figure 1 shown in the gallery interface 103 shown in (d) in

[0186] The basic materials selected by the mobile phone include video materials, and the video materials include multiple consecutive frames of images and audio segments. The mobile phone extracts one or more highlight segments from all the basic materials. The highlight segments determined by the mobile phone may include only a single frame of image, may include multiple consecutive frames of images, or may include multiple consecutive frames of images and the corresponding audio segments.

[0187] In a specific embodiment, the mobile phone can extract the highlight segments only according to the image information of each basic material. The scheme for the mobile phone to extract the highlight segments from each basic material can include: the mobile phone combines the image parameters of single-frame or multiple-frame images in each basic material, calculates the frame scores of each frame of image in each basic material, and then obtains the highlight segments according to some images with higher frame scores.

[0188] In another specific embodiment, the mobile phone can combine the image parameters and audio parameters of each basic material to extract the highlight segments. As Figure 7 shown, the steps for the mobile phone to execute the foregoing S502 to extract the highlight segments from all the basic materials can include:

[0189] S701: The mobile phone calculates the first score of each frame of image respectively based on the image parameters of each frame of image in the basic material.

[0190] S702: The mobile phone calculates the second score of each frame of image respectively based on the audio parameters of each frame of image in the basic material.

[0191] S703: The mobile phone extracts the first highlight segment from the basic material based on the first score and the second score of each frame of image.

[0192] The basic materials selected by the mobile phone include video materials, and the video materials include image information and audio information. The image information can include image parameters such as the clarity, scene, and brightness of each frame of image, and these image parameters can reflect the quality of the image. The mobile phone can score each frame of image according to the image parameters of each frame of image, and record it as the first score.

[0193] The video materials selected by the mobile phone include multiple consecutive frames of images and one or more audio segments, and the multiple consecutive frames of images coincide with one or more audio segments on the time axis. The audio information of different audio segments is different, and the audio information may include parameters such as the acquisition period, loudness, and signal-to-noise ratio of the audio segment.

[0194] On the time axis, for some images, there are corresponding audio segments at the acquisition moments, while for some images, there are no corresponding audio segments at the acquisition moments. That is to say, the existence states and audio information of the audio segments corresponding to different images are different.

[0195] The mobile phone scores each frame of image according to the existence state and audio information of the audio segment corresponding to the different image, which is recorded as the second score.

[0196] Exemplarily, in this embodiment, continue to combine Figure 6 the basic materials P1, P2, P3, and P4 shown in (a) in, and exemplarily illustrate the specific implementation of S701 - S703. In this example, for the convenience of distinguishing and describing, take the frame rate in each video material as 1fps / s. As Figure 7 shown, assume that the basic material p1 is a 20 - second video material, including 20 consecutive frames of images, namely f1 - f20; the basic material p2 is an image material, including a single - frame image f21; the basic material p3 is a video material, including 20 consecutive frames of images, namely f22 - f41; the basic material p4 is a video material, including 24 consecutive frames of images, namely f42 - 65. It should be noted that the video frame rate obtained by the mobile phone can be 20fps, 30fps, 60fps or even higher, and the number of image frames included in each video material is even higher. In this embodiment, only for the convenience of explaining the technical solution, the situation of the multiple consecutive frames of images included in each video material is exemplarily simplified, which is not a limitation to this solution.

[0197] In one example, as Figure 7 shown, the highlight segments extracted by the mobile phone from all the basic materials may include: G1, G2, G3, G4, and G5. Among them, G1 is extracted from P1 and includes 4 consecutive frames of images, namely f2 - f5. G2 is extracted from P1 and includes 5 consecutive frames of images, namely f11 - f15. G3 is extracted from P3 and includes 3 consecutive frames of images, namely f22 - f25. G4 is extracted from P4 and includes 7 consecutive frames of images, namely f45 - f51. G5 is extracted from P5 and includes 6 consecutive frames of images, namely f55 - f60.

[0198] Next, the solutions for the mobile phone to calculate the first score and the second score, and to obtain the highlight segments will be explained respectively.

[0199] The mobile phone executes S701 to calculate the first score of each frame of image.

[0200] In a specific implementation, the solution for the mobile phone to calculate the first score of each frame of image in combination with image parameters may specifically include: The mobile phone analyzes all the images included in all the basic materials to obtain the image parameters of each frame of image. The mobile phone calculates the scores corresponding to various types of image parameters for each frame of image according to the pre-determined frame scoring rules based on the image parameters of each frame of image. The mobile phone can sum or calculate the weighted sum of the scores corresponding to various types of images to obtain the first score of each frame of image.

[0201] The mobile phone can select the types of image parameters to be analyzed according to the factors affecting the display effect of the video set. The factors affecting the display effect of the video set may include image quality, image aesthetics, human objects, and high-value actions, etc. Among them, the image parameters associated with image quality may include sharpness, brightness, jitter, shot segmentation, etc. The image parameters associated with image aesthetics may include aesthetics, human body, scene, etc. The image parameters associated with human objects may include face, human body, expression, action, etc. The image parameters associated with high-value actions may include action, human body, scene, etc.

[0202] Based on the above analysis, the mobile phone can use parameters such as sharpness, brightness, aesthetics, jitter, face, shot segmentation, human body, expression, scene, and action of the image as the image parameters for calculating the scores of each frame of image. Among them, sharpness, brightness, aesthetics, etc. are all positively correlated with the scores.

[0203] In a specific example, the mobile phone can score the current frame of image respectively for the basic quality, aesthetic score, and high-value scene of the current frame of image, and then sum these three types of scores to obtain the first score. Here, the current frame refers to the frame of image for which the mobile phone is currently performing the scoring operation, and the current frame of image can be any frame of image in the basic materials.

[0204] Among them, the mobile phone scoring the basic quality of the image means that the mobile phone scores according to the sharpness and brightness of the current frame of image. The mobile phone can also set the minimum threshold for sharpness and the minimum threshold for brightness. If the sharpness of the current frame of image is lower than the minimum threshold for sharpness, the sharpness score of the current frame of image can be directly recorded as 0 points. Or, if the sharpness of the current frame of image is lower than the minimum threshold for sharpness, the mobile phone can also directly record the first score of the current frame of image as 0 points.

[0205] The mobile phone's aesthetic scoring of an image means that the mobile phone scores according to aesthetic parameters such as the tone and composition of the current frame image. The mobile phone's high-value scene scoring of an image means that the mobile phone analyzes the scene of the current frame image and the actions of the people in it to determine whether there is a high-value scene. For example, if the mobile phone analyzes that the image scene of the current frame image includes a sunrise scene, a sunset scene, a fireworks blooming scene, a flower blooming scene, or if the mobile phone analyzes that the actions of the people in the current frame image include a person skydiving, a person dancing, a person shooting a basketball, etc., it can increase the high-value scene score of the current frame image.

[0206] In one case, the mobile phone can separately perform basic quality scoring, aesthetic scoring, and high-value scene scoring on the current frame image, and then sum up the three types of scores to obtain the first score.

[0207] In another case, the mobile phone can also preset a basic quality score threshold. The mobile phone first performs basic quality scoring on the current frame image. If the basic quality score exceeds the basic quality score threshold, the mobile phone then performs aesthetic scoring and high-value scene scoring on the current frame image. If the basic quality score does not exceed the basic quality score threshold, the mobile phone can stop performing aesthetic scoring and high-value scene scoring on the current frame image to save unnecessary calculation operations.

[0208] In one case, the mobile phone can configure fixed weights for the basic quality score, aesthetic score, and high-value scene score, and perform weighted summation on the three types of scores according to the fixed weights to obtain the first score of the current frame image.

[0209] In another case, the mobile phone can also flexibly adjust the weights according to various scores, and calculate the weighted sum according to the adjusted weights to obtain the first score of the current frame image. For example, if the aesthetic score of the current frame image is low, the mobile phone can reduce the weight of the aesthetic score. Or, if the high-value scene score of the current frame image is high, the mobile phone can increase the weight of the high-value scene score.

[0210] The mobile phone executes S702 to calculate the second score of each frame image.

[0211] The operation of the mobile phone to calculate the second score of each frame image can be for multiple frame images whose basic material is a video material including audio segments.

[0212] The basic material obtained by the mobile phone is a video material. The mobile phone uses a pre-stored audio recognition algorithm to recognize one or more continuous audio signals from the basic material, and superimposes the one or more continuous audio signals on the corresponding audio track along the time axis to obtain an audio segment for the corresponding continuous multiple frame images.

[0213] For example Figure 8As shown in (a) in it, continue to take the basic material P3 as an example. The basic material P3 includes a continuous multi-frame image f22 - f41 and an audio segment. The acquisition times corresponding to the continuous multi-frame image f22 - f41 are t1, t2, t3... t20 in sequence. The continuous time period of the multi-frame image is from Ts1 to Te1, that is, the time period between the acquisition time t1 of the first frame f22 and the acquisition time t20 of the last frame f41. The continuous time period of the audio segment also corresponds to Ts1 to Te1.

[0214] Within an audio segment recognized by the mobile phone, there may be a valid audio segment and an invalid audio segment. A valid audio segment is an audio signal segment containing more audio features, and the audio signal fluctuates greatly.

[0215] Such as Figure 8 in (a), the audio signal between ts1 and te1 fluctuates greatly, and this part of the audio segment can be recorded as a valid audio segment. The audio signal between ts2 and te2 fluctuates greatly, and this part of the audio segment can also be recorded as a valid audio segment. The audio signal between Ts1 and ts1, between te1 and ts2, and between te2 and Te1 fluctuates little, and this part of the audio segment can be regarded as an invalid audio segment.

[0216] The mobile phone can pre-store a valid audio recognition algorithm to recognize the valid audio segments in the audio segments corresponding to each video material. The mobile phone can obtain one or more valid audio segments included in each video material, and record the start time and end time corresponding to each valid audio segment as the audio parameters for reference in extracting the highlight segments. It should be noted that the audio segments described in the subsequent solutions refer to the valid audio segments recognized by the mobile phone using the valid audio recognition algorithm, and no longer distinguish and describe the original audio segments included in the video material and the valid audio segments recognized by the algorithm.

[0217] Such as Figure 8 As shown in (b), the mobile phone can call the valid audio recognition algorithm to recognize the valid audio segments in the audio frequency band of the basic material P3 and retain them (indicated by "√"), and recognize and delete the invalid audio segments in the audio segment (indicated by "×"). In this way, two audio segments Audio1 and Audio2 of the basic material P3 can be obtained, as Figure 8 shown in (c).

[0218] The audio segments in the video material are usually original sound segments, which may include exciting contents such as cheers and people's voices, or features with retention value. The value of the images collected during the start and end time periods of the audio segment may also be higher. The mobile phone can determine the acquisition time of each frame of the image and the start and end time periods of each audio segment.

[0219] In one example, the mobile phone can calculate a second score for each frame image according to the existence status of the audio segment corresponding to each frame image.

[0220] For example, if the mobile phone determines that the existence status of the audio segment corresponding to the current frame image is that there is an audio segment corresponding to the acquisition moment, or in other words, it determines that the acquisition moment of the current frame image is within the start and end periods of any audio segment, then it can increase the audio score for the current frame image, denoted as the second score.

[0221] For another example, if the mobile phone determines that the existence status of the audio segment corresponding to the current frame image is that there is no audio segment corresponding to the acquisition moment, or in other words, it determines that the acquisition moment of the current frame image is not within the start and end periods of any audio segment, then it can record the second score of the current frame image as 0 points.

[0222] In another example, the mobile phone can also calculate the second score for each frame image according to the images acquired within the start and end moments of each audio segment.

[0223] In one example, as Figure 8 shown in (c) of, the acquisition moments of images f24, f25, and f26 are within the start and end period (ts1, te1) of audio segment Audio1, and they are the image frames corresponding to Audio1. The acquisition moments of images f33, f34, f35, and f36 are within the start and end period (ts2, te2) of audio segment Audio2, and they are the image frames corresponding to Audio2.

[0224] The start and end periods of audio segments Audio1 and Audio2 obtained by the mobile phone are obtained by calling an internal algorithm to analyze the audio track data of the video material. Considering that there may also be fade-in and fade-out periods for the audio segments, in order to obtain more valuable features, the mobile phone can also add the fade-in duration and fade-out duration to each audio segment.

[0225] In one example, the mobile phone can add a fade-in period and a fade-out period to the audio segment. The fade-in duration corresponding to the fade-in period and the fade-out duration corresponding to the fade-out period are the same, both being △t.

[0226] As Figure 9As shown in the figure, the mobile phone advances the start time ts1 of the audio clip Audio1 by the fade-in duration Δt to (ts1 - Δt), and moves the fade-out duration te1 at the end time backward by the fade-out duration Δt to (te1 + Δt). The clip duration L1 of the audio clip determined by the mobile phone is increased by the fade-in duration L2 of the fade-in segment forward and the fade-in duration L3 of the fade-in segment backward, and the total clip duration corresponding to Audio1 is determined to be L4. Within the total clip duration L4 corresponding to Audio1, there are images f23, f24, f25, f26, and f27, and the mobile phone can calculate the second score for the images f23, f24, f25, f26, and f27.

[0227] Correspondingly, the mobile phone advances the start time ts2 of the audio clip Audio2 by the fade-in duration Δt to (ts2 - Δt), and moves the fade-out duration te2 at the end time backward by the fade-out duration Δt to (te2 + Δt). The clip duration L5 of the audio clip determined by the mobile phone is increased by the fade-in duration L6 of the fade-in segment forward and the fade-in duration L7 of the fade-in segment backward, and the total clip duration corresponding to Audio1 is determined to be L8. Within the total clip duration L8 corresponding to Audio1, there are images f32, f33, f34, f35, f36, and f37, and the mobile phone can calculate the second score for the images f32, f33, f34, f35, f36, and f37.

[0228] For the solution provided in this example, the images selected by the mobile phone after adding equal-duration fade-in and fade-out periods to the audio clip, compared to Figure 8 the images selected in (c) without adding fade-in and fade-out periods, have f23, f32, and f37 more, and calculate the second score for more images.

[0229] In another example, the mobile phone can add a fade-in period and a fade-out period to the audio clip, and the fade-in duration corresponding to the fade-in period and the fade-out duration corresponding to the fade-out period are different.

[0230] As Figure 10 shown in the figure, the mobile phone advances the start time ts1 of the audio clip Audio1 by the fade-in duration Δt1 to (ts1 - Δt1), and moves the fade-out duration te1 at the end time backward by the fade-out duration Δt2 to (te1 + Δt2). Within the total clip duration after the mobile phone determines that the audio clip Audio1 increases the fade-in duration Δt1 and the fade-out duration Δt2, there are images f23, f24, f25, and f26, and the mobile phone can calculate the second score for the images f23, f24, f25, and f26.

[0231] Correspondingly, the mobile phone advances the start time ts2 of the audio segment Audio2 by the fade-in duration Δt1 to (ts2 - Δt1), and moves the fade-out duration te2 at the end time backward by the fade-out duration Δt2 to (te2 + Δt2). The mobile phone determines that within the total segment duration after the audio segment Audio2 increases the fade-in duration Δt1 and the fade-out duration Δt2, there are images f32, f33, f34, f35, and f36, and the mobile phone can calculate the second score for the images f32, f33, f34, f35, and f36.

[0232] For the solution provided in this example, the images selected by the mobile phone after adding unequal-duration fade-in periods and fade-out periods to the audio segment have two more images, f23 and f32, compared to Figure 8 the images selected in (c) without adding fade-in periods and fade-out periods, and the second score can also be calculated for more images.

[0233] In other examples, the mobile phone can also set different fade-in durations and fade-out durations for different audio segments. For example, the mobile phone can set a longer fade-in duration and a longer fade-out duration for an audio segment with a longer segment duration, and set a shorter fade-in duration and a shorter fade-out duration for an audio segment with a shorter segment duration. There can be other implementation manners for the mobile phone to set different fade-in durations and fade-out durations for different audio segments, which are not limited.

[0234] The foregoing Figures 8 - 10 example shows a specific implementation solution for the mobile phone to determine each frame image whose acquisition time is within the start and end periods of the audio segment. The subsequent Figures 11 - 13 example shows a specific implementation solution for the mobile phone to calculate the second score for each frame image whose acquisition time is within the start and end periods of the audio segment. For the convenience of description, the mobile phone can record the images whose acquisition time is within the start and end periods of the audio segment among all the images included in all the basic materials as the first images.

[0235] Among the consecutive multiple frame images collected by the mobile phone within the start and end periods of an audio segment, the earlier the acquisition time of an image, the higher its value may be, and the later the acquisition time of an image, the lower its value may be, or it is more likely to be discarded due to the limitation of the total duration of the video set. Based on this, the mobile phone can determine the second type of score for each frame image based on the sequence of the acquisition times of each frame image. Among them, the earlier the acquisition time, the higher the score.

[0236] In one example, the mobile phone can calculate the second score for each frame image in an irregular scoring manner according to the sequence of the acquisition times.

[0237] For example, the mobile phone can increase a relatively high second score, such as 10 points, for f23 with the earliest acquisition time, and sequentially increase relatively lower second scores, such as 8, 6, and 5, for f24, f25, and f26 with relatively later acquisition times, and increase the lowest second score, such as 3 points or 2 points, for f27 with the latest acquisition time. The irregular scoring method provided by the mobile phone can be a method randomly selected by the mobile phone within a preset bonus range, or a method of scoring in response to the user's custom operation, without limitation.

[0238] In another example, the mobile phone can calculate the second score in a linear scoring manner according to the sequence of the acquisition times of each frame image within the start and end periods of the audio segment.

[0239] Continue with the Figure 8 first image determined by the example shown in (c) in Figure 11 to explain the scheme for calculating the second score. As

[0240] shown, the mobile phone sets the highest score of the second score as a and the lowest score as 0. The second scores of each first image linearly decrease from a to 0 according to the sequence of the acquisition times. The acquisition time of each first image is t, and the second score is Score(t).

[0241] If the acquisition time of the first image is within the start and end periods of the audio segment Audio1, the second score Score(t) = a * (te1 - t) / (te1 - ts1).

[0242] If the acquisition time of the current frame image is not within the start and end periods of any audio segment, Score(t) = 0.

[0243] In another example, the mobile phone can calculate the second score in a linear scoring manner according to the sequence of the acquisition times of each frame image within the start and end periods of the audio segment re - determined based on the fade - in duration and fade - out duration.

[0244] Continue with the Figure 9 first image determined by the example shown in Figure 12 to explain the scheme for calculating the second score. As

[0245] If the acquisition time of the first image is within the start and end period of the audio segment Audio1, the second score Score(t) = a * (te1 - t + Δt) / (te1 - ts1 + 2Δt).

[0246] If the acquisition time of the first image is within the start and end period of the audio segment Audio2, the second score Score(t) = a * (te2 - t + Δt) / (te2 - ts2 + 2Δt).

[0247] In another example, the mobile phone can also add a first threshold corresponding to the lowest score of the second score. Based on the first threshold, the second score is calculated in a linear scoring manner according to the order of the acquisition times of each frame of the image.

[0248] As Figure 13 shown, the mobile phone sets the highest score of the second score as a, the first threshold is the lowest score b, and the second scores of each first image linearly decrease from a to b in the order of the acquisition times, and the first threshold is maintained during the fade-out period. The acquisition time of each first image is t, and the second score is Score(t). The formula for the mobile phone to calculate the second score for each frame of the image is as follows:

[0249]

[0250] In one example, Δt = 1 second, b = 0.11 points, and a = 0.21 points.

[0251] This embodiment adds the lowest score, which is beneficial to retaining more valuable images and audio segments. In addition, adding the fade-in period and the fade-out period to participate in the calculation of linear scoring can try to retain the fade-in and fade-out intervals of the audio segment for subsequent fade-in and fade-out processing of the audio segment.

[0252] The mobile phone combines the image parameters to obtain the first score of each frame of the image. And, the mobile phone combines the audio parameters to obtain the second score of each frame of the image. The order of these two steps can be not limited. For example, the mobile phone can first calculate the first score of each frame of the image and then calculate the second score. Or, the mobile phone can first calculate the second score of each frame of the image and then calculate the first score. Or, the mobile phone can also calculate the first score and the second score in parallel.

[0253] The mobile phone executes S703, combines the first score and the second score of each frame of the image, and extracts the highlight segment from all the basic materials.

[0254] The mobile phone obtains the first score and the second score of each frame of the image, and can obtain the frame score of each frame of the image according to the first score and the second score of each frame of the image. The mobile phone then extracts the highlight segment from the basic materials based on the frame score of each frame of the image.

[0255] The mobile phone combines the first score and the second score to extract highlight segments from all the basic materials. The mobile phone can obtain one or more highlight segments from one basic material, or it may not obtain a highlight segment from one basic material. If the mobile phone can obtain multiple highlight segments from all the basic materials, a video set can be synthesized.

[0256] In one implementation, for each frame image, the mobile phone can calculate the sum of the first score and the second score of each frame image to obtain the frame score corresponding to each frame image. In another implementation, the mobile phone pre-sets a first weight for the first score and a second weight for the second score. For the current frame image, the mobile phone can calculate the first product of the first score of the current frame image and the first weight, and the second product of the second score and the second weight respectively; then calculate the sum of the first product and the second product to obtain the frame score of the current frame image.

[0257] The mobile phone obtains the frame scores of each frame image, and extracts multiple highlight segments from all the basic materials according to the frame scores of each frame image. The mobile phone can directly obtain the highlight segments by extracting partial images with higher frame scores. The mobile phone can also screen all the images according to the frame scores to indirectly obtain the frame images.

[0258] In one embodiment, the mobile phone can select the first M frame images as the highlight segments in the order of the frame scores of each frame image in all the basic materials from high to low. M≥2 and M is an integer.

[0259] In one implementation, the mobile phone can select the first M frame images as the highlight segments in the basic material in the order of the frame scores of each frame image in each basic material from high to low. M≥2 and M is an integer.

[0260] As Figure 14 shown, it is a schematic diagram of the frame scores of each frame image in the basic material P3. According to the frame scores of all 20 frame images, the highest frame score is ScoreA. The mobile phone selects the first M frame images with the highest frame scores, such as 4 frame images, which are f22 - f25 respectively. The images f22 - f25 are 4 consecutive frame images, and the mobile phone can use these 4 consecutive frame images f22 - f25 as a highlight segment G3.

[0261] In another implementation, the mobile phone can select partial images with frame scores greater than or equal to the frame threshold from the basic material to obtain the highlight segments.

[0262] As Figure 15As shown, it is a schematic diagram of the frame scores of each frame image in the basic material P4. The mobile phone calculates the frame scores of all 24 frame images, and the highest frame score is ScoreB. The mobile phone selects images with frame scores higher than 70 points and obtains 13 images, namely f45 - f51 and f55 - f60. The images f45 - f51 are 7 consecutive images, so these 7 consecutive images f45 - f51 can be used as a highlight segment G4. The images f55 - f60 are 6 consecutive images, so these 6 consecutive images f55 - f60 can be used as a highlight segment G5.

[0263] In another implementation, if the number of images selected by the mobile phone from the basic material based on the frame threshold does not meet the image number threshold, adjustment operations of increasing or decreasing the selected images can be performed to obtain images that meet the image number threshold.

[0264] The mobile phone can set the minimum value LN of the number of images and the maximum value HN of the number of images.

[0265] If the number of images with frame scores higher than the frame threshold exceeds the maximum value HN of the number of images, the mobile phone can select the first HN frame images in descending order of frame scores.

[0266] If the number of images with frame scores higher than the frame threshold does not exceed the minimum value LN of the number of images, the mobile phone can select the first LN frame images in descending order of frame scores.

[0267] If the number of images with frame scores higher than the frame threshold exceeds the minimum value LN of the number of images and does not exceed the second threshold HN corresponding to the maximum number of images, the mobile phone can directly select all images that exceed the frame threshold.

[0268] In another embodiment, the mobile phone can screen all the images in the basic material according to the frame scores of each frame image in the basic material to obtain multiple second images, and then extract the first highlight segment from the second images.

[0269] The process of the mobile phone screening out the second images from all the images included in all the basic materials can mainly include four screening operations, namely threshold screening, length screening, same-shot filtering, and quantity screening.

[0270] For the first screening operation, the mobile phone screens out some images according to the frame score threshold.

[0271] The mobile phone pre-determines the minimum value of the frame score of the images that can be retained, denoted as the second threshold. The mobile phone retains the images with frame scores higher than the second threshold according to the frame scores of all the images in all the basic materials, and denotes them as the third images.

[0272] In one example, such as Figure 16As shown, it is a schematic diagram of the frame sequence of the basic material P4 and the corresponding frame scores. The mobile phone can set the second threshold corresponding to the lowest frame score to 20 points. As Figure 16 shown in (a) of , among all 24 consecutive frames of the basic material P4, the frame scores of the images f44, f52, f53, and f62 are lower than 20 points. Filter out these images and retain the third images with frame scores higher than the second threshold to obtain Figure 16 the frame sequence shown in (b) of .

[0273] It should be noted that in order to retain audio segments as much as possible, the second threshold corresponding to the lowest frame score set by the mobile phone is less than the first threshold corresponding to the lowest additional score of each frame image within the audio segment set by the mobile phone.

[0274] For the second screening operation, the mobile phone screens out video segments whose segment duration meets the duration requirement according to the length of the video segment.

[0275] The mobile phone realizes the "one-key blockbuster" function and synthesizes a video set from multiple basic materials. The total duration of the video set is usually less than the sum of the durations of all basic materials. In order to retain more highlight segments within a limited duration, the mobile phone will split a single long-duration video material in the basic materials into multiple short-duration video segments, and then select highlight segments from multiple short-duration video segments to synthesize the video set. The video processing method executed by the mobile phone also involves splitting a long-duration video segment composed of multiple consecutive frames of images into multiple relatively short-duration video segments.

[0276] The mobile phone calculates the frame scores of each frame image in all basic materials and splits the video materials according to the frame scores of each frame image to obtain multiple video segments. There can be various schemes for the mobile phone to split the video materials into video segments. For example, it can be split according to the duration interval between adjacent images, or according to the shot.

[0277] In a specific implementation manner, the mobile phone splits the video material into video segments according to the duration interval between adjacent images.

[0278] During the process of the mobile phone first screening out some third images according to the frame score threshold, some images in the video material are filtered out, resulting in an increase in the duration interval between some adjacent images. The mobile phone can perform video splitting between adjacent images with a large duration interval and split these two adjacent images with a large duration interval into two different video segments.

[0279] The mobile phone can preset a third threshold corresponding to the lowest duration interval. There is a video splitting node between adjacent images whose duration interval is greater than or equal to the third threshold, and the splitting operation of splitting the long-duration video material into video segments can be performed.

[0280] As Figure 16 shown in (a) below, before the mobile phone filters out images based on frame scores for the first time, the time intervals between adjacent frames are all fixed frame intervals. As Figure 16 shown in (b) below, after the mobile phone filters out images based on frame scores for the first time, the time intervals between adjacent frames are not necessarily fixed frame intervals. For example, the time interval between f45 and f43 is (t4 - t2), the time interval between f54 and f51 is (t13 - t10), and the time interval between f63 and f61 is (t22 - t20).

[0281] The mobile phone can set the third threshold corresponding to the minimum time interval as T0. For example, T0 can be 1 second. The mobile phone can adjust the specific duration of the third threshold according to factors such as the fixed frame interval of the video material and the upper limit of the video set duration, without limitation.

[0282] In a specific example, as Figure 16 shown in (c) below, the time interval between f45 and f43 is (t4 - t2) ≥ the third threshold T0, the time interval between f54 and f51 is (t13 - t10) ≥ the third threshold T0, and the time interval between f63 and f61 is (t22 - t20) ≥ the third threshold T0. That is to say, there are 3 video splitting nodes. In this way, the basic material P4 can be split into 4 video segments, namely V1(f42, f43), V2(f45 - f51), V3(f54 - f61), and V4(f63 - f65).

[0283] In another example, as Figure 16 shown in (d) below, the time interval between f45 and f43 is (t4 - t2) < the third threshold T0, the time interval between f54 and f51 is (t13 - t10) ≥ the third threshold T0, and the time interval between f63 and f61 is (t22 - t20) < the third threshold T0. That is to say, there is 1 video splitting node. In this way, the basic material P4 can be split into 2 video segments, namely V5(f42, f43, f45 - f51), V6(f54 - f61, f63 - f65).

[0284] In another example, as Figure 16As shown in (e), the time interval (t4 - t2) between f45 and f43 < the third threshold T0, the time interval (t13 - t10) between f54 and f51 < the third threshold T0, and the time interval (t22 - t20) between f63 and f61 < the third threshold T0. That is to say, there is no video splitting node. In this way, the basic material P4 is used as 1 video segment, that is, V7 (f42, f43, f45 - f51, f54 - f61, f63 - f65).

[0285] In another specific implementation, the mobile phone splits the video material according to the shooting scenes to obtain video segments.

[0286] In the video material, there is also an image parameter as a shooting scene point. The shooting scene point usually corresponds to the conversion node of the video content. The images before the acquisition moment of the shooting scene point image and the images after the acquisition moment of the shooting scene point image are not continuously acquired by the same lens. Based on this, the mobile phone can split the video material into multiple video segments according to whether there is a shooting scene point in the image, or split the video segments into more video segments.

[0287] In one example, as Figure 17 shown in (a), there are two video segments V5 and V6, where f48 and f60 are shooting scene points. Figure 17 Corresponding to Figure 16 in (d).

[0288] As Figure 17 shown in (b), the mobile phone can retain the image corresponding to the shooting scene point and divide it forward into the previous video segment for splitting. The mobile phone splits V5 (f42, f43, f45 - f51) into V8 (f42, f43, f45 - f48) and V9 (f49 - f51), and splits V6 (f54 - f61, f63 - f65) into V10 (f54 - f60) and V11 (f61, f63 - f65).

[0289] As Figure 17 shown in (c), the mobile phone can retain the image corresponding to the shooting scene point and divide it backward into the next video segment for splitting. The mobile phone splits V5 (f42, f43, f45 - f51) into V12 (f42, f43, f45 - f47) and V13 (f48 - f51), and splits V6 (f54 - f61, f63 - f65) into V14 (f54 - f59) and V15 (f60, f61, f63 - f65).

[0290] As Figure 17As shown in (d) thereof, the mobile phone can filter out the images corresponding to the storyboard points and split them forward and backward. The mobile phone splits V5(f42, f43, f45 - f51) into V16(f42, f43, f45 - f47) and V17(f49 - f51), and splits V6(f54 - f61, f63 - f65) into V18(f54 - f59) and V19(f61, f63 - f65).

[0291] As Figure 17 As shown in (e) thereof, the mobile phone can retain the images corresponding to the storyboard points and split them forward and backward. The mobile phone splits V5(f42, f43, f45 - f51) into V20(f42, f43, f45 - f48) and V21(f48 - f51), and splits V6(f54 - f61, f63 - f65) into V22(f54 - f60) and V23(f61, f63 - f60).

[0292] It should be noted that the mobile phone can only perform the operation of splitting video segments according to the time interval between adjacent images in the foregoing implementation manner. The mobile phone can also only perform the operation of splitting video segments according to storyboards in this implementation manner. In this case, the object of video splitting by the mobile phone is the video material in the basic material, and the mobile phone splits the video material to obtain video segments. In other cases, the mobile phone can also first split the video material according to the time interval between adjacent images to obtain the first video segment, and then split the first video segment according to the storyboard points to obtain the second video segment.

[0293] That is to say, for the video segment splitting scheme provided in this implementation manner, the splitting object can be the video segment in the video material or the video segment obtained after other splitting operations.

[0294] After splitting the video, there may still be cases where the duration of individual video segments does not meet the duration requirements. In this case, the mobile phone can also perform the second screening operation, and the mobile phone screens out the video segments whose segment durations meet the duration requirements.

[0295] The segment duration of a video segment can be the duration between the acquisition time of the last frame image and the acquisition time of the first frame image among all the images of the video segment.

[0296] In one example, the mobile phone can set a fourth threshold corresponding to the minimum length of the video segment, filter out the video segments whose segment durations are less than or equal to the fourth threshold from all the video segments, and retain the video segments whose segment durations are greater than the fourth threshold.

[0297] For example, the mobile phone can set the fourth threshold to 1.5 seconds. The mobile phone filters out video segments with a segment duration less than or equal to 1.5 seconds in all video segments and retains video segments with a segment duration greater than 1.5 seconds.

[0298] In another example, the mobile phone can also set a fifth threshold corresponding to the maximum duration of a video segment and split video segments with a segment duration greater than or equal to the fifth threshold.

[0299] For example, the mobile phone can set the fifth threshold to 8 seconds. The mobile phone splits video segments with a segment duration greater than or equal to 8 seconds in all video segments.

[0300] As Figure 18 shown, it is a schematic diagram of the mobile phone splitting video segments according to the segment duration. As Figure 18 shown in (a), the segment duration of the video segment exceeds the fifth threshold. The video segment also corresponds to an audio segment.

[0301] In one example, the mobile phone can split the video segment according to the fifth threshold.

[0302] As Figure 18 shown in (b), starting from the start time of the video segment, the video segment and the audio segment are equally spaced and split according to the fifth threshold to obtain multiple video segments and multiple audio segments.

[0303] In one example, the mobile phone can also split the video segment with unequal durations with reference to the fifth threshold and the acquisition period of the audio segment. As Figure 18 shown in (c), referring to the start time and end time of each audio segment, starting from the start time of the video segment, the video segment is first split into 3 video segments, and then the video segment that still exceeds the maximum duration is split again to obtain multiple video segments and corresponding audio segments.

[0304] The mobile phone performs a second screening operation and retains the images included in the video segments that meet the duration requirements, denoted as the fourth images.

[0305] In the third screening operation, the mobile phone screens out the fifth images from the fourth images retained in the previous two screenings according to whether they are in the same shot segment.

[0306] The mobile phone obtains multiple video segments based on the aforementioned two screening operations. Each video segment includes at least 2 frames of images, and each frame of image has a corresponding frame score. The mobile phone can take the image with the highest frame score among all the images included in each video segment as the center frame of the video segment.

[0307] Different video clips may be in the same shot or in different shots. For adjacent video clips in the same shot, if the interval duration between the two adjacent video clips is short, the content of these two adjacent video clips may be relatively close. The mobile phone can select and retain one of the two adjacent video clips that are in the same shot and have a short interval duration.

[0308] It should be noted that the interval duration between adjacent video clips can be the duration between the acquisition time of the first frame of the subsequent video clip and the acquisition time of the last frame of the previous video clip.

[0309] In one example, the frame scores of the central frames of two adjacent video clips in the same shot are different. The mobile phone can retain the video clip with a higher frame score of the central frame and filter out the video clip with a lower frame score of the central frame. Further, the mobile phone can also set a sixth threshold corresponding to the higher frame score and retain the video clip whose frame score of the central frame is greater than or equal to the sixth threshold.

[0310] In another example, the frame scores of the central frames of two adjacent video clips in the same shot are the same. The mobile phone can retain the video clip with a highlight action. Or, the mobile phone can retain the video clip that has an overlapping period with the audio clip.

[0311] For example, for two adjacent video clips V1 and V2 in the same shot, where V1 has a highlight action and V2 does not, the mobile phone can retain V1. Another example, for two adjacent video clips V3 and V4 in the same shot, where there is a corresponding audio clip during the acquisition period of V3 and there is no corresponding audio clip during the acquisition period of V4, the mobile phone can retain V3.

[0312] In the fourth screening operation, the mobile phone screens the video clips that meet the quantity requirements.

[0313] The mobile phone can set a seventh threshold corresponding to the minimum quantity of video clips and an eighth threshold corresponding to the maximum quantity. For example, the mobile phone can set the seventh threshold to 2 and the eighth threshold to 5. The mobile phone obtains all the video clips corresponding to all the basic materials based on the foregoing multiple filtering operations.

[0314] If the quantity of all the video clips is less than the seventh threshold, the mobile phone can further expand and select some video clips from all the images of all the basic materials so that the quantity of all the video clips is greater than or equal to the seventh threshold. For example, the mobile phone can first select the video clip with the highest frame score of the central frame and use the adjacent clips of this video clip as the expanded selected video clips.

[0315] If the number of all video segments is greater than the eighth threshold, the mobile phone can delete some video segments so that the number of all video segments is less than the seventh threshold. For example, the mobile phone can filter out several video segments with relatively low frame scores of the central frames.

[0316] So far, based on the foregoing steps, the mobile phone obtains multiple video segments according to all the images included in all the basic materials, which are recorded as target video segments. The target video segments obtained by the mobile phone may include single-frame images or may include consecutive multiple-frame images.

[0317] When the foregoing content focuses on the splitting of video segments, it mainly describes the splitting of continuous image sequences. When the mobile phone actually splits video segments, it will also split and screen the audio segments corresponding to the video segments on the time axis according to the video splitting and screening scheme and the acquisition time corresponding to the video splitting nodes.

[0318] The mobile phone executes step S703 to obtain all the highlight segments corresponding to all the basic materials. The obtained highlight segments may include images or may include images and corresponding audio segments. For example, for the target video 1 and target video 2 obtained by the mobile phone, the target video segment 1 includes a single-frame image and there is no corresponding audio segment on the time axis, and the target video segment 2 includes consecutive multiple-frame images and there is a corresponding audio segment on the time axis.

[0319] The mobile phone executes the foregoing S502 to extract multiple highlight segments from all the basic materials, and some of the highlight segments have corresponding audio segments. Considering that some audio segments may contain wonderful content such as cheers and birthday blessings, the mobile phone can retain these audio segments. Some audio segments may not contain wonderful content, and the mobile phone can discard these audio segments.

[0320] For example, the highlight segments obtained by the mobile phone include highlight segment 1, highlight segment 2, and highlight segment 3, where both highlight segment 1 and highlight segment 2 have corresponding audio segments, and highlight segment 3 does not have a corresponding audio segment. The mobile phone can only retain the audio segment of highlight segment 1 and discard the audio segment corresponding to highlight segment 2.

[0321] In a specific implementation, the mobile phone selects whether to retain the audio segment corresponding to the highlight segment according to the sequence of the acquisition time of the video segment and the acquisition time of the audio segment.

[0322] As Figure 19 shown, it is a schematic diagram for the mobile phone to determine the target audio segment. As Figure 19 shown, the highlight segments include V16, V17, V18, and V19, corresponding to the video segments shown in (d) of the foregoing Figure 17 .

[0323] As Figure 19As shown in (a) therein, the high-brightness segment V16 corresponds to an audio segment Audio1, the high-brightness segment V18 corresponds to an audio segment Audio2, and the high-brightness segment V19 corresponds to an audio segment Audio3.

[0324] In one example, the mobile phone can directly use the audio segment within the time period of the video segment as the audio segment corresponding to the video segment according to the acquisition time of the video segment and the acquisition time of the audio segment.

[0325] As Figure 19 shown in (a) therein, the start time ts1 of the audio segment Audio1 is later than the start time t1 of the high-brightness segment V16 and within the time period of the high-brightness segment V16. The end time te1 of the audio segment Audio1 is later than the end time t6 of the high-brightness segment V16 and not within the time period of the high-brightness segment V16. That is to say, the time period of the audio segment Audio1 does not completely overlap with the time period of the high-brightness segment V16.

[0326] The mobile phone can truncate the audio segment Audio1 at the end time t6 of the high-brightness segment V16 to obtain the audio segment Audio1' as shown in (b) therein. Figure 19 shown in (b) therein.

[0327] Continuing as Figure 19 shown in (a) therein, the start time ts2 of the audio segment Audio2 is earlier than the start time t13 of the high-brightness segment V18 and not within the time period of the high-brightness segment V18. The end time te1 of the audio segment Audio2 is earlier than the end time t8 of the high-brightness segment V18 and within the time period of the high-brightness segment V18. That is to say, the time period of the audio segment Audio2 does not completely overlap with the time period of the high-brightness segment V18.

[0328] The mobile phone can truncate the audio segment Audio2 at the start time t13 of the high-brightness segment V18 to obtain the audio segment Audio2' as shown in (b) therein. Figure 19 shown in (b) therein.

[0329] The audio segment Audio3 is within the time period of the high-brightness segment V19 and can be completely retained.

[0330] As Figure 19 shown in (b) therein, the mobile phone can obtain three target audio segments, namely: the target audio segment Audio1' corresponding to the high-brightness segment V16, the target audio segment Audio2' corresponding to the high-brightness segment V18, and the target audio segment Audio3 corresponding to the high-brightness segment V19.

[0331] In another example, the mobile phone can also screen the audio segments according to the start and end times of the audio segments.

[0332] In order for the mobile phone to obtain a relatively complete audio segment, it can select whether to retain the audio segment according to the coincidence degree between the start time and end time of the audio segment and the time period of the highlight segment.

[0333] As Figure 20 shown in (a) of , both the start time and end time of the audio segment Audio3 are within the time period of the highlight segment V19, and it can be completely retained.

[0334] The start time ts2 of the audio segment Audio2 is earlier than the start time t13 of the highlight segment V18 and is not within the time period of the highlight segment V18. As Figure 20 shown in (b) of , the mobile phone can directly discard the audio segment Audio2.

[0335] The end time te1 of the audio segment Audio1 is later than the end time t6 of the highlight segment V16 and is not within the time period of the highlight segment V16. The mobile phone can choose to retain the complete segment of the audio segment Audio1, or only retain a partial segment of the audio segment Audio1.

[0336] In one example, the mobile phone can choose to retain the complete segment of the audio segment Audio1. In this way, the mobile phone can obtain two target audio segments, namely: the target audio segment Audio1 corresponding to the highlight segment V16, and the target audio segment Audio3 corresponding to the highlight segment V19.

[0337] In another example, the mobile phone can only retain a partial segment of the audio segment Audio1.

[0338] As Figure 20 shown in (b) of , the mobile phone can discard the part of the audio segment that exceeds the end time of the video segment to obtain the corresponding Audio1”.

[0339] Alternatively, the mobile phone can also preset a ninth threshold corresponding to the maximum duration of an audio segment, such as 3.5 seconds. The mobile phone can choose to retain the audio segment that exceeds 3.5 seconds.

[0340] If the segment duration of the audio segment Audio1” is less than or equal to the ninth threshold, the mobile phone can discard the audio segment Audio1”.

[0341] If the segment duration of the audio segment Audio1” is greater than the ninth threshold, as Figure 20 shown in (c) of , the mobile phone can retain the audio segment Audio1”. As Figure 20 shown in (c) of , the mobile phone can obtain two target audio segments, namely: the target audio segment Audio1” corresponding to the highlight segment V16, and the target audio segment Audio3 corresponding to the highlight segment V19.

[0342] In another example, the mobile phone compares the start and end times of the audio clip with those of the first type of high-light clip, and can also extend the clip duration of the high-light clip.

[0343] As Figure 21 shown in (a) of Figure 20 , in the high-light clip, the time stamps (circles) corresponding to the start and end times of each frame image are discrete on the time axis, and the time stamps (diamonds) corresponding to the start and end times of the audio clip are relatively continuous. If the mobile phone performs the audio clip screening operation shown above, even if the start time ts2 of the audio clip Audio2 is only slightly earlier than the start time t13 of the high-light clip V18, the audio clip Audio2 may be discarded.

[0344] Based on this, the mobile phone can adjust the clip duration of the high-light clip compared with the audio clip. As Figure 21 shown in (a) of Figure 19 and Figure 20 , advance the start time of each high-light clip by e seconds and delay the end time by e seconds. For example, e can be 0.15 seconds. When comparing

[0345] As Figure 21 shown in (b) of

[0346] Figure 21 , the mobile phone determines that the start time ts1 of the audio clip Audio1 is later than the start time (Ts16 - e) of the high-light clip V16 and is within the time period of the high-light clip V16. The end time te1 of the audio clip Audio1 is earlier than the end time (Te16 + e) of the high-light clip V16 and is within the time period of the high-light clip V16. That is to say, the time period of the audio clip Audio1 is completely within the time period of the high-light clip V16 and can be retained as the corresponding target audio clip.

[0347] The audio clip Audio3 is within the time period of the high-light clip V19 and can be completely retained.

[0348] Figure 21 ​​As shown in (c) of [the above], the mobile phone can obtain three target audio segments, namely: the target audio segment Audio1 corresponding to the highlight segment V16, the target audio segment Audio2 corresponding to the highlight segment V18, and the target audio segment Audio3 corresponding to the highlight segment V19.

[0349] So far, the mobile phone has completed S502, extracted the first highlight segment from the basic materials, and determined the target audio segments corresponding to the retained highlight segments.

[0350] Based on the determined all highlight segments and the corresponding audio segments, the mobile phone executes S503 to synthesize all the highlight segments to obtain a video set.

[0351] The mobile phone can perform operations such as automatically selecting and integrating highlight segments and the corresponding target audio segments, applying transition effects, and adding background music according to the conventional video synthesis method to obtain a video set.

[0352] The above embodiments explain the specific implementation manners of the video processing method provided in this embodiment from the perspective of the mobile phone. Next, the specific implementation manners of the video processing method provided in this embodiment will be explained in combination with the software and hardware architecture of the mobile phone.

[0353] As Figure 22 shown, it is a schematic diagram of the software and hardware structure of the mobile phone. The mobile phone has a layered architecture, including an application layer, a framework layer, and an algorithm layer in sequence.

[0354] The application layer includes a clip application, which can be a third-party application installed in the system, or a plug-in or applet for implementing the clip function in the gallery application. The application layer mainly completes functions such as the clip application interface, video and picture file options, and final result presentation.

[0355] The framework layer includes a media middle platform framework, which includes an initialization interface, a channel interface, an image highlight segment analysis interface, a theme summary interface, a highlight segment selection strategy interface, an analysis performance query interface, and a strategy monitoring module.

[0356] The media middle platform framework layer is used to complete tasks such as decoding video and picture files, data format conversion, remaining time monitoring and strategy adjustment, data sending, algorithm control, obtaining results and returning them to the application layer. Among them, the channel interface provides capabilities such as data processing, node management, and status management, and is mainly used for data process handling.

[0357] The algorithm layer includes algorithm modules, and the algorithm modules include algorithm call interfaces. The algorithm call interfaces can call video analysis algorithms, image analysis algorithms, and audio analysis algorithms to perform algorithm analysis operations on corresponding materials. The algorithm layer is used to provide algorithm services, receive commands and parameters from the media middle platform framework layer, calculate results, and return them to the media middle platform framework layer. The final results of the algorithms are collected, processed by the media middle platform framework layer, and returned to the application layer.

[0358] The editing application can receive the user's first operation to determine the basic materials to be processed. The basic materials can include image materials and video materials, and the video materials include video clips and audio clips.

[0359] The editing application sends the basic materials to the initialization interface, and the media middle platform framework performs video processing operations on the basic materials. During the process of the media middle platform framework performing video processing operations, it can pass through the algorithm call interface. The media middle platform framework calls the image analysis algorithm to analyze the image materials and obtains the image analysis results. The media middle platform framework calls the video analysis algorithm to analyze the video clips and obtains the video analysis results. The media middle platform framework can also call the audio analysis algorithm to analyze the audio clips and obtains the audio analysis results.

[0360] The media middle platform framework obtains the highlight clips and the corresponding audio clips according to the image analysis results, video analysis results, and audio analysis results returned by the algorithm layer. The media middle platform framework sends the highlight clips and the corresponding audio clips to the editing application, and the editing application synthesizes and displays the video set.

[0361] Continue to combine Figure 22 the software architecture shown in Figure 23 to introduce the specific process of how each module in the mobile phone software architecture interacts to implement the above video processing method. As

[0362] S2301, the editing application loads the image method and video materials.

[0363] The application layer receives the user's operation of enabling the one-click video creation function and starts the editing application. For example, this operation can specifically be the click operation on the "One-Click Blockbuster" option as shown in (b) in Figure 1 .

[0364] The editing application receives the user's operation of selecting materials and selects one or more basic materials. The basic materials can include the image method and video materials.

[0365] The operation of the editing application receiving the user's selection of multiple image methods and video materials can be, as shown in (c) of Figure 1 , the click operation on photos and videos. The operation of the user input to confirm the execution of the one-click video generation function can be, as shown in (d) of Figure 1 , the click operation on the "√" tick option to confirm the basic materials for one-click blockbuster.

[0366] The editing application receives the image methods and video materials confirmed by the user and loads the image methods and video materials.

[0367] S2302, the editing application calls the initialization interface of the media middle platform framework.

[0368] The media middle platform framework provides relevant interfaces for obtaining highlight segments, including analysis interfaces and channel interfaces. The analysis interfaces and channel interfaces also include interfaces with various subdivision functions, such as initialization interfaces, highlight segment strategy selection interfaces, etc. The media middle platform framework can call relevant algorithms in the algorithm layer to perform data analysis operations and obtain data analysis results.

[0369] The algorithm layer can include image analysis algorithms, video analysis algorithms, and audio analysis algorithms. The image analysis algorithm is used to process image data to obtain image analysis results, and the image analysis results can include image parameters such as clarity and brightness. The video analysis algorithm is used to process video data to obtain video analysis results, and the video analysis results can include frame sequences and corresponding timestamps, clarity, aesthetics, highlight actions, and other relevant image parameters of each frame image. The audio analysis algorithm can be used to process audio data to obtain audio analysis results, and the audio analysis results can include the start and end times of audio segments, or the start and end times of valid audio segments in the audio segments, loudness, signal-to-noise ratio, and other parameters.

[0370] If the editing application determines that the basic materials include image materials, the algorithms to be called by the media middle platform framework include image analysis algorithms. If the editing application determines that the basic materials include video materials, the algorithms to be called by the media middle platform framework include video analysis algorithms and audio analysis algorithms. The editing application determines the relevant algorithms to be called, calls the initialization interface of the media middle platform framework, and initializes the relevant algorithms to be called by the algorithm layer through the initialization interface. In addition, the relevant algorithms also include other algorithms involved in the one-click video generation function, such as highlight segment algorithms, face detection algorithms, video acceleration algorithms, and image super-resolution algorithms, etc., which are not limited.

[0371] For example Figure 1 , as shown in (d) of

[0372] S2303. The analysis interface layer sends relevant parameters to the channel interface layer.

[0373] In the algorithm layer, one algorithm corresponds to one algorithm interface. The media middle platform framework also includes multiple service interfaces, and one service interface corresponds to one algorithm interface in the algorithm layer. Each service interface of the media middle platform framework plays a role in data pass-through between the algorithm interface in the algorithm layer and the channel interface in the media middle platform framework layer.

[0374] The initialization parameters involved in different algorithms are different. Therefore, the algorithm initialization parameters sent to the algorithm interface through each service interface may be different. The initialization parameters may include parameters such as the number of algorithms and the algorithm specifications. The editing application determines the relevant algorithms to be called and notifies the analysis interface layer to instruct the initialization of the relevant algorithms. The analysis interface layer sends the relevant parameters corresponding to each algorithm to the channel interface.

[0375] S2304. The channel interface layer notifies the algorithm layer to initialize the algorithm.

[0376] The channel interface layer interfaces with the relevant parameters of each relevant algorithm and notifies the algorithm layer through the service interface to initialize the relevant algorithm layer using the algorithm parameters.

[0377] S2305. The algorithm layer returns algorithm-related parameters to the channel interface layer.

[0378] S2306. The channel interface layer returns algorithm-related parameters to the analysis interface layer.

[0379] S2307. The analysis interface layer notifies the editing application that the initialization is successful.

[0380] The algorithm layer completes the initialization operation of the relevant algorithm according to the relevant parameters of the relevant algorithm and returns algorithm-related parameters to the channel interface layer, such as algorithm performance parameters, algorithm interface identifiers, storage addresses, and other information. The channel interface layer then returns the algorithm-related parameters of the relevant algorithm sent by the algorithm layer to the analysis interface layer, and the analysis interface layer notifies the editing application that the algorithm initialization is successful.

[0381] S2308. The editing application queries the analysis speed of images and videos from the analysis interface layer.

[0382] The algorithm-related parameters received by the analysis interface layer may also include algorithm performance parameters, such as the algorithm analysis speed of various algorithms.

[0383] Among them, the algorithm analysis speed can characterize the number of image frames in a picture / video analyzed by the image signal processor per unit time; or, the algorithm analysis speed can also characterize the duration for the image signal processor to analyze a picture / a single image frame in a video (single-frame analysis duration). Therefore, based on the chip analysis speed, the single-frame analysis duration of the image signal processor and the processing duration of the image signal processor for a picture can be calculated. The duration of the image signal processor for a picture and the single-frame analysis duration in a video may be different. Exemplarily, the processing duration of a picture can be 400 ms, and the single-frame analysis duration can be 200 ms.

[0384] It should be understood that since the performances of different image signal processors are different, the chip analysis speeds corresponding to different image signal processors may be different. For an electronic device put on the market, the image signal processor is fixed, so the chip analysis speed corresponding to this image signal processor is also fixed.

[0385] The editing application can query the analysis speeds of all video materials from the media middle platform framework layer and calculate the analysis speed and total duration of the video materials.

[0386] S2309. The analysis interface layer returns the analysis speeds and analysis durations of images and videos to the editing application.

[0387] The analysis interface calculates the analysis speed and total duration of the video materials and returns them to the editing application. The selected basic materials by the user are multiple, including image methods and video materials. The analysis duration of a single image method is less than that of a single video material. The analysis interface can determine the analysis speeds and analysis durations of each image material and each video material according to the quantity of all image materials, the quantity of video materials, and a preset maximum analysis duration, and return them to the editing application.

[0388] For example Figure 1 As shown in (d) of [reference], all the selected image materials by the user include 1 image material and 3 video materials, and the maximum analysis duration is 5 seconds. The analysis interface layer can directly divide the analysis duration equally for each image material or video according to the total number of materials 1 + 3 = 4, with an analysis duration of 1.25 seconds. For another example, if the image analysis speed is 0.5 second / frame and the video analysis speed is 1.5 seconds / unit, 0.5 second of image analysis duration can be allocated to the image material, and 4.5 seconds of video analysis duration can be allocated to the video materials, and the total analysis duration is 5 seconds.

[0389] S2310. The editing application calls the analysis interface.

[0390] The editing application calls each analysis interface in the analysis interface layer to analyze the image materials and video materials. Since the data volume of pictures is smaller than that of videos, the image methods can be directly analyzed and processed. The image materials are processed first, and then the video materials are processed.

[0391] The editing application also sends the target total analysis duration and the file descriptors of all the basic materials to be analyzed to the analysis interface layer, so that the analysis interface layer can find the data of each basic material according to the file descriptors of each basic material and perform preliminary parsing operations. The file descriptors can include fd1, fd2, etc., which are used to refer to the data bodies of materials such as image material 1 and image material 2.

[0392] S2311, the analysis interface layer parses the image materials.

[0393] The analysis interface layer finds the corresponding material data according to the file descriptors of each basic material sent by the editing application and performs parsing operations.

[0394] The operations of the analysis interface layer for parsing image materials mainly include: picture decoding, resolution reduction, transcoding, etc., and the parsed image data is stored, and the data address of the parsed image data is recorded.

[0395] S2312, the analysis interface layer sends the data address of the image to the channel interface layer.

[0396] S2313, the channel interface sends the data address of the image to the algorithm layer.

[0397] The analysis interface layer sends the data address of the parsed image data to the algorithm layer via the channel interface layer. The algorithm layer obtains the image data from the data address of the image and passes it into the algorithm module for calculation to obtain the image analysis result.

[0398] S2314, the algorithm layer returns the image analysis result to the channel interface layer.

[0399] S2315, the channel interface layer returns the image analysis result to the analysis interface layer.

[0400] The algorithm layer sequentially obtains the corresponding image data according to the data addresses of each image material, inputs the images into the algorithm module to obtain the image analysis results of each image material, and then returns them to the analysis interface layer via the channel interface layer.

[0401] Until all the image methods are analyzed, the image analysis process is completed, and the video analysis process begins.

[0402] S2316, the analysis interface layer calculates the allocation duration.

[0403] The media middle platform framework enters the video analysis process. There are multiple video materials to be analyzed. The analysis interface layer calculates the allocation duration for each video material according to the video analysis speed, the number of video materials, and the video analysis duration. The allocation duration refers to the duration allocated for analyzing the video material. The actual analysis duration is usually less than or equal to the allocation duration.

[0404] For example, there are 3 video materials, the video analysis speed is 1.5 seconds per piece, and the video allocation duration is 4.5 seconds. Then, the analysis interface layer can calculate the allocation duration for each video material as 1.5 seconds.

[0405] It should be noted that the video processing process provided in this example involves the video processing process and the audio processing process. The video processing process and the audio processing process can be processed in parallel. The video processing process of a single video material is usually longer than the audio processing process, so the audio analysis speed does not need to be considered separately. That is to say, although the audio processing process is added for each video material, there is no need to add an audio allocation duration in addition to the video allocation duration.

[0406] S2317, the analysis interface layer sets a dynamic analysis strategy according to the allocation duration.

[0407] The analysis interface layer also includes a policy monitoring module. The policy monitoring module is used to provide different analysis strategies, mainly setting dynamic analysis strategies according to the allocation duration of each video material and the recommended value of the total duration upper limit.

[0408] Among them, if the video allocation duration is less than or equal to the recommended value of the total analysis duration upper limit, the dynamic analysis strategy can be to analyze all video materials. For example, if the total analysis duration is 4400ms and the recommended value of the total analysis duration upper limit is 10000ms, the material analysis strategy can be to analyze all the video materials selected by the user.

[0409] If the total analysis duration is greater than the recommended value of the total analysis duration upper limit, the dynamic analysis strategy can be a random sampling strategy. For example, if the total analysis duration is 4400ms and the recommended value of the total analysis duration upper limit is 3000ms, the dynamic analysis strategy can be a random sampling strategy. Among them, the random sampling strategy refers to randomly selecting some video materials from all the video materials selected by the user for analysis. For example, according to the principle that the material value of the video material containing an audio segment is higher than that of the video material without an audio segment, all or part of the video materials containing an audio segment can be selected. Another example is that according to the principle that the material value of the video material with a longer duration is lower than that of the video material with a shorter duration, all or part of the video materials with a duration less than the duration threshold can be selected. There can be other implementation schemes for the scheme in which the analysis interface layer sets a dynamic analysis strategy according to the allocation duration, which is not limited.

[0410] S2318, the analysis interface layer sends the data address of the video material to the channel interface layer.

[0411] The analysis interface layer allocates corresponding allocation durations for each video material according to all the video materials and the total allocation duration. The analysis interface layer executes the video analysis process for each video material in sequence according to the allocation duration of each video material. The video analysis process for each video material includes a video analysis process and an audio analysis process, and the video analysis process and the audio analysis process can be executed in parallel.

[0412] The analysis process for each video material mainly includes two steps. First, the channel interface layer performs preliminary video parsing and preliminary audio parsing on the video material, and stores the preliminarily parsed video data and audio data; then the algorithm layer performs algorithm analysis on the preliminarily parsed video data and audio data to obtain video analysis results and audio analysis results.

[0413] The analysis interface layer sends the data address of the current video material to the channel interface layer, and the data address stores the video data and audio data of the current video material.

[0414] S2319, the channel interface layer parses the video segment.

[0415] S2320, the channel interface layer parses the audio segment.

[0416] The channel interface layer obtains the video segment and audio segment of the video material from the data address of the video material sent by the analysis interface layer, and the channel analysis interface preliminarily parses the video segment and audio segment. The preliminary parsing operations for the video segment and audio segment can be executed in parallel.

[0417] Among them, the main operations for the channel interface layer to parse the audio segment include: audio decoding and audio analysis.

[0418] The operations for the channel interface layer to parse the video segment mainly include: video decoding, resolution reduction, format conversion, video analysis, etc.

[0419] For the audio segment in the video material, decoding is also required. After decoding, resampling processing is required and the data is stored to unify the audio data format.

[0420] The audio decoding node is responsible for selecting the audio track data in the video, completing the decoding of data in different coding formats within the start and end time intervals according to the scheme, decoding the audio data in different coding formats into PCM data, and then resampling the decoded PCM data. The purpose of resampling is to normalize the format of the audio data. During resampling, the sampling rate is fixed at 8 kHz, the channel layout is fixed as a single channel, and the PCM data bit depth is fixed at 16 bits. Finally, the resampled data is saved and the data address is sent to the voice recognition node.

[0421] The voice recognition node is responsible for analyzing the voice interval of the audio data. The channel interface sends the processed data address to the algorithm module. The algorithm module finds the audio data according to the data address and calls the voice analysis algorithm for voice analysis. The voice analysis algorithm has trained an AI model, which can output the proportion of corresponding human voice and noise according to the input audio frame by frame, then set a proportion threshold to determine whether it is a human voice, and finally output the voice interval. After the algorithm module finishes the analysis, the result is sent to the highlight segment selection strategy interface module.

[0422] Since the data volume of the audio is much smaller than that of the video, the audio data processing flow takes less time than the video processing flow. At the same time, it is set to process the audio processing flow and the video processing flow in parallel. Under the modern multi-threaded processor process, the audio data can be processed during the video processing without adding extra processing time.

[0423] S2321, the channel interface layer sends the data addresses of the video segment and the audio segment to the algorithm layer.

[0424] S2322, the algorithm layer returns the video analysis result and the audio analysis result to the channel interface layer.

[0425] The channel interface layer stores the preliminarily parsed video data and audio data, and sends the data address after preliminary parsing to the algorithm layer. The algorithm layer obtains the image data of the video segment and the audio data of the audio segment from the data address, calls the relevant algorithms initialized in the previous steps, and starts to execute the video processing flow provided by the embodiment shown above. Figure 7 shown in the embodiment.

[0426] The algorithm layer calls the image analysis algorithm and the video analysis algorithm to analyze the video segment, and obtains the image parameters of the video segment, including the clarity, scene, brightness and other image parameters of each frame of the image. These image parameters can reflect the quality of the image.

[0427] The algorithm layer can also call the audio analysis algorithm to analyze the audio segment, and obtain the audio parameters such as the acquisition period, loudness, and signal-to-noise ratio of the audio segment. The acquisition period of the audio segment obtained by the algorithm layer analysis here refers to the acquisition period of the effective audio segment.

[0428] S2323. The channel interface layer obtains high-light segments according to the image analysis result, video analysis result, and audio analysis result.

[0429] S2324. The channel interface layer returns the high-light segments to the analysis interface layer.

[0430] The analysis interface layer obtains multiple high-light segments according to the image analysis result, video analysis result, and audio analysis result provided by the algorithm layer.

[0431] For example Figure 7 As shown, the mobile phone calculates the first score of each frame image based on the image parameters of each frame image in the basic material. Also, the mobile phone calculates the second score of each frame image based on the audio parameters of each frame image in the basic material. Then, the mobile phone extracts high-light segments from all the basic materials based on the first score and the second score of each frame image.

[0432] In S701, the process of the mobile phone calculating the first score of each frame image based on the image parameters of each frame image in the basic material may specifically include:

[0433] The mobile phone can score the current frame image respectively for the basic quality, aesthetic score, and high-value scene of the current frame image, and then sum up these three types of scores to obtain the first score.

[0434] In S702, the process of the mobile phone calculating the second score of each frame image based on the audio parameters of each frame image in the basic material may specifically include:

[0435] As Figure 8 shown in (c), the analysis interface layer determines the acquisition time period ts1 to te1 of the effective audio segment.

[0436] As Figure 9 shown in (a), the analysis interface layer adds a fade-in time period and a fade-out time period to the audio segment.

[0437] The analysis interface layer moves the start time ts1 of the audio segment Audio1 forward by the fade-in duration △t to (ts1 - △t), and moves the fade-out duration te1 of the end time backward by the fade-out duration △t to (te1 + △t). The segment duration L1 of the audio segment determined by the analysis interface layer is increased by the fade-in duration L2 of the fade-in segment forward and the fade-in duration L3 of the fade-in segment backward, and the total segment duration L4 corresponding to Audio1 is determined. Within the total segment duration L4 corresponding to Audio1, there are images f23, f24, f25, f26, and f27, and the second score can be calculated for the images f23, f24, f25, f26, and f27.

[0438] As Figure 13As shown, the analysis interface layer calculates the second score for each frame and pushes it to the computing layer.

[0439] The mobile phone sets the maximum score of the second score as a, the first threshold as b. The second scores of each first image linearly decrease from a to b in the order of the acquisition time, and the first threshold is maintained during the fade-out period. The acquisition time of each first image is t, and the second score is Score(t). The formula for the mobile phone to calculate the second score for each frame image is as follows:

[0440]

[0441] In one example, △t = 1 second, b = 0.11 points, a = 0.21 points.

[0442] The analysis interface layer executes S703 to extract the highlight segments according to the first score and the second score of each frame image.

[0443] The analysis interface layer obtains the first score and the second score of each frame image, and can obtain the frame score of each frame image according to the first score and the second score of each frame image. The analysis interface layer then extracts the highlight segments from the basic material based on the frame score of each frame image.

[0444] As Figure 16 shown, the analysis interface layer filters all the images based on the frame score.

[0445] As Figure 16 shown in (a) of, among all 24 consecutive frame images of the basic material P4, the frame scores of images f44, f52, f53, and f62 are lower than 20 points. These images are filtered out, and the third images with frame scores higher than the second threshold are retained to obtain Figure 16 the frame sequence shown in (b) of.

[0446] As Figure 16 shown in (b), (c), and (d) of, the analysis interface layer splits the video segment according to whether the duration interval between adjacent pictures is greater than the third threshold.

[0447] As Figure 17 shown in (a) of, the analysis interface layer also determines the storyboard point image according to the video analysis result returned by the algorithm layer. As Figure 17 shown in (d) of, the analysis interface layer retains the storyboard point image according to the storyboard point image, splits forward to a video segment, and splits backward to a video segment to obtain multiple video segments.

[0448] As Figure 18 shown in (a) of, there are also some video segments with lengths greater than the maximum duration L of the video segment. The analysis interface layer can also split the video segment with reference to the start and end times of the audio segment to obtain as Figure 18The video segment shown in (c) thereof.

[0449] In Figure 18 In the video segment splitting operation in, in order to retain the audio segment as much as possible, the reference period of the video segment can also be extended. As Figure 21 Shown in (b) thereof, the mobile phone determines that the start time ts1 of the audio segment Audio1 is later than the start time (Ts16 - e) of the highlight segment V16, and within the period of the highlight segment V16. The end time te1 of the audio segment Audio1 is earlier than the end time (Te16 + e) of the highlight segment V16, and within the period of the highlight segment V16. That is to say, the period of the audio segment Audio1 is completely within the period of the highlight segment V16 and can be retained as the corresponding target audio segment.

[0450] Continuing as Figure 21 Shown in (b) thereof, the start time ts2 of the audio segment Audio2 is later than the start time (Ts18 - e) of the highlight segment V18, and within the period of the highlight segment V18. The end time te1 of the audio segment Audio2 is earlier than the end time (t8 + e) of the highlight segment V18, and within the period of the highlight segment V18. That is to say, the period of the audio segment Audio2 is completely within the highlight segment V18 and can be retained as the corresponding target audio segment.

[0451] The audio segment Audio3 can be completely retained within the period of the highlight segment V19.

[0452] As Figure 21 Shown in (c) thereof, the mobile phone can obtain three target audio segments, namely: the target audio segment Audio1 corresponding to the highlight segment V16, the target audio segment Audio2 corresponding to the highlight segment V18, and the target audio segment Audio3 corresponding to the highlight segment V19.

[0453] So far, the analysis interface layer has completed S703 and extracted all the highlight segments from the basic materials.

[0454] S2325, the analysis interface layer returns the highlight segments to the editing application.

[0455] S2326, the editing application generates a video set according to the highlight segments.

[0456] The analysis interface layer returns the extracted highlight segments to the editing application, and the editing application determines operations such as the theme and editing to generate a video set.

[0457] As Figure 24As shown, it is another schematic diagram of the thread for the mobile phone to execute the video processing method. In this embodiment, the application layer is subdivided into a service layer, an application function layer, and a basic capability layer. The media middle platform framework is subdivided into an image highlight segment analysis interface, a theme summary interface, an initialization interface, an analysis performance query interface, a policy monitoring interface, a channel interface, and a highlight segment selection policy interface.

[0458] The data flow between various interfaces is as Figure 25 shown, and is divided into a video processing flow and an audio processing flow.

[0459] In the video processing flow, the channel interface module builds a video decoding node, a frame rate reduction node, a format conversion node, a resolution conversion node, a format conversion node, and the algorithm module builds algorithm processing nodes such as a visual middle platform node, a jitter detection node, a brightness detection node, and a scene segmentation detection node. When a video data address is sent to the channel interface, the frame data in the video is decoded, frame rate reduced, format converted, resolution converted, and format converted at one time, and the processed data is then sent to each algorithm node, including the visual middle platform node, the jitter detection algorithm, the brightness detection algorithm, and the scene segmentation detection algorithm. After the algorithm module finishes processing, it sends the processing result to the highlight segment processing interface.

[0460] In the audio processing flow, a new audio decoding node and an audio recognition node are added to analyze the audio segment in the video material, and the obtained audio analysis result is sent to the highlight segment processing interface.

[0461] As Figure 24 and Figure 25 shown, the process of the mobile phone executing the video processing method mainly includes:

[0462] S2401, the service layer receives the operation of the user enabling the one-click video creation function.

[0463] In this embodiment, the service layer refers to the one-click video creation module in the service layer. That is, the one-click video creation module in the service layer receives the operation of the user enabling the one-click video creation function. For example, this operation can specifically be the click operation on the "one-click blockbuster" option as shown in (b) of Figure 1 .

[0464] S2402, the service layer loads and displays candidate pictures and candidate videos.

[0465] S2403, the service layer notifies the application function layer to receive the user's selection of video material and image material, and confirm the operation of executing one-click video creation.

[0466] For example, the operation of the user selecting multiple pictures and videos can be the click operation on the photos and videos as shown in (c) of Figure 1 , and the operation of the user inputting to confirm the execution of the one-click video creation function can be such asFigure 1 The business layer receives the image method and video material confirmed by the user, and loads the image method and video material.

[0467] S2404, the application function layer calls the initialization interface of the media middle platform framework layer.

[0468] Referring to the operation of S2302 described above, the application function layer calls the initialization interface to instruct the initialization of related algorithms.

[0469] S2405, the initialization interface sends initialization parameters to the algorithm module.

[0470] S2406, the algorithm module initializes each algorithm.

[0471] S2407, the algorithm layer returns the algorithm analysis speed to the initialization interface

[0472] S2408, the initialization interface indicates to the business layer that the initialization is successful, and carries the algorithm analysis speed.

[0473] Referring to the aforementioned operations of S2303 - S2306 , the relevant algorithms are initialized and the algorithm analysis speed is obtained.

[0474] S2409, the business layer queries the analysis speed of each video in turn.

[0475] The steps for the business layer to query the analysis speed of each video include:

[0476] S24091, the business layer queries the analysis performance query interface for the analysis speed of video 1 (the file descriptor is fd1).

[0477] S24092, analyzing the performance query interface to obtain the resolution of video 1 according to fd1 of video 1.

[0478] S24093, the analysis performance query interface calculates the analysis speed of video 1 according to the resolution of video 1 and the algorithm analysis speed.

[0479] S24094, the analysis performance query interface returns the analysis speed of video 1 to the business layer.

[0480] S2410: The business layer calculates the total target analysis duration based on the analysis speeds of all returned videos.

[0481] S2411, the business layer sends the total target analysis time and the fd of all material files to be analyzed to the policy monitoring module.

[0482] Referring to the foregoing S2308 - 2309, the analysis performance query interface of the interface layer sequentially queries the resolution and algorithm analysis speed of each video, determines the analysis speed corresponding to each video material, and calculates the allocation duration.

[0483] The service layer sequentially queries the analysis speed of each video until the analysis speeds of all videos are queried.

[0484] Among them, the query message includes the file descriptor fd1 of video 1 selected by the user and the chip analysis speed. Among them, the file descriptor can be used as the unique identifier of the video.

[0485] Based on the total target analysis duration and the fds of all material files to be analyzed, the policy monitoring module first analyzes the image materials and then the video materials.

[0486] S2412. The policy monitoring module analyzes each frame of image material.

[0487] The steps for the policy monitoring module to analyze the image material mainly include:

[0488] S24121. The policy monitoring module instructs the channel interface to analyze Picture 1.

[0489] S24122. The channel interface decodes, reduces the resolution, converts the format, etc. of Picture 1, and stores the processed Picture 1.

[0490] S24123. The channel interface sends the frame data address of Picture 1 to the algorithm layer.

[0491] S24124. The algorithm layer finds the processed Picture 1 according to the address, and obtains the analysis result (such as a score) after algorithm analysis.

[0492] S24125. The algorithm layer returns the analysis result of Picture 1 to the channel interface.

[0493] The policy monitoring module executes the operations of S24121 - S24125 for each frame of image material until all image materials are analyzed. For the specific analysis process, reference can be made to the implementation manners of the foregoing S2312 - S2315, which will not be elaborated herein.

[0494] S2413. The policy monitoring module analyzes each video material.

[0495] S2414. The policy monitoring module updates the available analysis duration of each video and confirms the analysis policy.

[0496] S2415. The policy monitoring module sends the file descriptor of video 1 and the analysis policy to the channel interface.

[0497] Referring to the foregoing S2316 - S2317, the policy monitoring module in the interface layer sets dynamic analysis policies based on the allocated duration and the recommended value of the total duration limit of each video material.

[0498] In S2416, the channel interface decodes the video segments of Video 1, performs processing such as format conversion and resolution reduction, and stores the processed Video Segment 1.

[0499] In S2417, the channel interface decodes the audio segments of Video 1, performs operations such as resampling, and stores the processed Audio Segment 1.

[0500] In S2418, after analyzing the audio segment, the channel interface sends the audio data address of Audio Segment 1 to the algorithm layer.

[0501] In S2419, after analyzing the video segment, the channel interface sends the video data address of Video Segment 1 to the algorithm layer.

[0502] Referring to the foregoing S2318 - S2321, the channel interface respectively performs preliminary parsing on the video segments and audio segments of each video material, obtains the data corresponding to the video segments and the data corresponding to the audio segments and stores them. The channel interface sends the storage data addresses of the data corresponding to the video segments and the data corresponding to the audio segments to the algorithm layer so that the algorithm layer can obtain the data corresponding to the video segments and the data corresponding to the audio segments.

[0503] In S2420, the video analysis algorithm analyzes each frame of the image sent by Video 1 to obtain video analysis results.

[0504] Referring to the foregoing S322, the algorithm layer calls the image analysis algorithm and the video analysis algorithm to analyze the video segment, and obtains image parameters of the video segment, including image parameters such as the clarity, scene, and brightness of each frame of the image, and these image parameters can reflect the quality level of the image.

[0505] In S2421, the audio analysis algorithm analyzes the audio data of Audio Segment 1 to obtain audio analysis results.

[0506] Referring to the foregoing S322, the algorithm layer can also call the audio analysis algorithm to analyze the audio segment and obtain audio parameters such as the acquisition period, loudness, and signal - to - noise ratio of the audio segment. Here, the acquisition period of the audio segment analyzed by the algorithm layer refers to the acquisition period of the valid audio segment.

[0507] In S2422, after completing the audio analysis, the algorithm layer returns the audio analysis results and the video analysis results to the highlight segment selection policy interface.

[0508] In S2423, the highlight segment selection policy interface generates highlight segments based on the audio analysis results and the video analysis results.

[0509] S2424. The high - light segment selection strategy interface returns high - light segments to the application function layer.

[0510] The high - light segment selection strategy interface extracts high - light segments from all the basic materials according to the video analysis results and audio analysis results provided by the algorithm layer. Specifically, reference can be made to the specific implementation solution for obtaining high - light segments according to the image analysis results, video analysis results and audio analysis results in the aforementioned S2324.

[0511] S2425. The application function layer clips and filters the user - selected material files according to the analysis results.

[0512] S2426. The application function layer notifies the theme summary interface to obtain the theme.

[0513] In some embodiments, the terminal device may be provided with multiple theme templates (style templates). The theme algorithm can recommend theme templates that match the picture scene based on the picture scene of the high - light segments, such as people, scenery, food, children, pets, sports or travel.

[0514] S2427. The theme summary interface determines the theme that matches the content.

[0515] S2428. The theme summary interface returns the theme to the application function layer.

[0516] S2429. The application function layer sends the theme, video segments and filtered pictures to the basic capability layer.

[0517] S2430. The basic capability layer splices the clipped video and the filtered pictures, and applies the theme template to obtain the target video.

[0518] S2431. The basic capability layer notifies the service layer to generate a video set.

[0519] S2432. The service layer displays the video set.

[0520] In summary, the video processing method provided by the embodiments of this application adds an audio processing data flow, fuses video information and original sound information to obtain high - light segments. After the video and audio data processing is completed, the algorithm layer of the media middle - platform framework layer sends the frame data address of the video and the audio processing data to the algorithm layer. This solution comprehensively analyzes the analysis results of video data and audio data, and decides the position of the high - light segments of the generated video.

[0521] In addition, the embodiments of this application also provide an electronic device, including a display, a memory and a processor;

[0522] The memory stores computer - executable instructions;

[0523] The processor executes computer-executable instructions stored in the memory, causing the electronic device to execute the video processing method provided in the above embodiments. In addition to these main components, the electronic device also includes components for implementing basic functions, which will be specifically described below in conjunction with Figure 26 for specific illustration.

[0524] As Figure 26 shown is a schematic structural diagram of an electronic device 2600 provided by an embodiment of the present application. Among them, the electronic device 2600 may include a processor 2610, a memory 2620, a USB interface 2630, a display 2650, a charging management module 2640, a power management module 2641, a battery chip 2642, a motor, an antenna, a communication module 2660, an audio module 2670, a speaker 2670A, a receiver 2670B, a microphone 2670C, a headphone interface 2670D, a camera 2680, a button 2690, etc.

[0525] The structure schematically shown in the embodiments of the present invention does not limit the electronic device 2600. It may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0526] The processor 2610 may include one or more processing units. For example, the processor 2610 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and / or a Neural-Network Processing Unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0527] The above-mentioned controller may be the decision maker that commands each component of the electronic device 2600 to work in coordination according to instructions. It is the nerve center and command center of the electronic device 2600. The controller generates operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0528] A memory can also be set in the processor 2610 for storing instructions and data. In some embodiments, the memory in the processor 2610 is a cache memory that can hold the instructions or data just used or recycled by the processor 2610. If the processor 2610 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 2610, and thus improves the efficiency of the system.

[0529] In some embodiments, the processor 2610 may include interfaces. The interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0530] The interface connection relationships between the modules illustrated in the embodiments of the present invention are only illustrative and do not constitute a structural limitation on the electronic device 2600. The electronic device 2600 may adopt different interface connection methods in the embodiments of the present invention, or a combination of multiple interface connection methods.

[0531] The charging management module 2640 can be a rechargeable battery or a primary battery. When it is a rechargeable battery, it can receive charging input through a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 2640 can receive the charging input from the wired charger through the USB interface 2630. In some embodiments of wireless charging, the charging management module 2640 can receive the wireless charging input through the wireless charging coil of the electronic device 2600. While charging the battery chip 2642, the charging management module 2640 can also supply power to the electronic device 2600 through the power management module 2641.

[0532] The power management module 2641 is used to connect the battery chip 2642, the charging management module 2640, and the processor 2610. The power management module 2641 receives the inputs from the battery chip 2642 and / or the charging management module 2640, and powers the processor 2610, the memory 2620, the display 2650, the camera 2680, the communication module 2660, etc. The power management module 2641 can also be used to monitor parameters such as the capacity of the charging management module, the number of charging cycles of the charging management module, and the health status (leakage, impedance) of the charging management module. In some embodiments, the power management module 2641 can also be disposed in the processor 2610. In some embodiments, the power management module 2641 and the battery chip 2642 can also be disposed in the same device.

[0533] The wireless communication function of the electronic device 2600 can be implemented by an antenna, a radio frequency module, the communication module 2660, a modem, a baseband processor, etc.

[0534] The antenna is used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 2600 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: The cellular network antenna can be multiplexed as the wireless local area network diversity antenna. In some embodiments, the antenna can be used in combination with a tuning switch. The radio frequency module can provide a communication processing module for wireless communication solutions including 2G / 3G / 4G / 5G, etc. applied to the electronic device 2600.

[0535] The communication module 2660 can provide a communication processing module for wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (bT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the electronic device 2600. The communication module 2660 can be one or more devices integrating at least one communication processing module. The communication module 2660 receives electromagnetic waves via the antenna, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 2610. The communication module 2660 can also receive the signals to be sent from the processor 2610, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna for radiation.

[0536] In some embodiments, the antenna of the electronic device 2600 is coupled to the radio frequency module, and the antenna is coupled to the communication module 2660, so that the electronic device 2600 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), bT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS may include Satellite Based Augmentation Systems (SBAS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0537] The electronic device 2600 implements the display function through the GPU, the display 2650, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display 2650 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 2610 may include one or more GPUs, which execute program instructions to generate or change display information.

[0538] The display 2650 is used to display images, videos, etc. The display 2650 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 2600 may include one or N displays 2650, where N is a positive integer greater than 1.

[0539] The electronic device 2600 can implement the shooting function through an ISP, a camera 2680, a video codec, a GPU, a display screen, and an application processor, etc.

[0540] The electronic device 2600 can implement the audio function through an audio module 2670, a speaker 2670A, a receiver 2670B, a microphone 2670C, a headphone jack 2670D, and an application processor, etc. Such as music playback, recording, etc.

[0541] The keys 2690 include a power-on key, a lock screen key, a volume key, etc. The keys 2690 can be mechanical keys or touch keys. The electronic device 2600 receives the input of the keys 2690 and generates a key signal input related to the user settings and function control of the electronic device 2600.

[0542] The motor can generate a vibration prompt. The motor can be used for incoming call vibration prompts and can also be used for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Touch operations on different areas of the display 2650 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0543] The video processing methods in the foregoing embodiments can all be implemented in the electronic device 2600 with the above hardware structure.

[0544] Based on the above embodiments, an embodiment of the present application further provides a video processing device, which includes a processor for executing the video processing method provided in the above embodiments.

[0545] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which when running on a computer, causes the computer to execute the video processing method provided in the above embodiments.

[0546] An embodiment of the present application also provides a computer program product including instructions, which when running on a computer, enables the computer to execute the video processing method provided in the above embodiments.

[0547] For the specific implementation manners and the technical effects brought by the electronic device, video processing device, computer-readable storage medium, and computer program product including instructions provided in the embodiments of the present application, reference may be made to the specific implementation process and the technical effects brought by the video processing method provided in the foregoing embodiments, which will not be elaborated herein.

[0548] In some embodiments, through the description of the above implementation manners, those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0549] In each embodiment of the embodiments of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0550] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media that can store program codes such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk, or optical disc.

[0551] As described above, it is only the specific implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A video processing method, characterized in that, The method includes: Determining audio-associated frames in a first video segment; wherein, the duration of the first video segment is greater than a maximum duration threshold, the first video segment includes a continuous plurality of image frames and at least one audio segment, and the audio-associated frames are the image frames corresponding to the start time or the end time of each of the audio segments among the continuous plurality of image frames; Splitting the first video segment into at least two second video segments; wherein, the first frame or the last frame of each of the second video segments is the audio-associated frame, and the duration of any one of the second video segments is less than or equal to the maximum duration threshold.

2. The video processing method according to claim 1, wherein The step of determining the audio-associated frames in the first video segment includes: Determining the start time and the end time of the audio segment; Selecting, from the continuous plurality of image frames, the image frames whose acquisition times are close to the start time and the end time of the audio segment as the audio-associated frames.

3. The video processing method according to claim 2, wherein The step of selecting, from the continuous plurality of image frames, the image frames whose acquisition times are close to the start time and the end time of the audio segment as the audio-associated frames includes: Searching, from all the continuous plurality of image frames, for two first image frames close to the start time of the audio segment; wherein, the acquisition time of one first image frame is before the start time of the audio segment, and the acquisition time of one first image frame is after the start time of the audio segment; Taking the image frame whose acquisition time is before the start time of the audio segment among the two first image frames as the audio-associated frame corresponding to the start time of the audio segment.

4. The video processing method according to claim 3, wherein The step of selecting, from the continuous plurality of image frames, the image frames whose acquisition times are close to the start time and the end time of the audio segment as the audio-associated frames includes: Searching, from all the continuous plurality of image frames, for two second image frames close to the end time of the audio segment; wherein, the acquisition time of one second image frame is before the end time of the audio segment, and the acquisition time of one first image frame is after the end time of the audio segment; Taking the image frame whose acquisition time is after the end time of the audio segment among the two second image frames as the audio-associated frame corresponding to the end time of the audio segment.

5. The video processing method according to claim 4, wherein The step of searching, from all the continuous plurality of image frames, for two first image frames close to the start time of the audio segment includes: Using the binary search method to search, from all the continuous plurality of image frames, for two first image frames close to the start time of the audio segment; and / or, Using the binary search method to search, from all the continuous plurality of image frames, for two second image frames close to the end time of the audio segment.

6. The video processing method according to any one of claims 1 to 5, characterized in that The step of splitting the first video segment into at least two second video segments includes: Determining a first frame sequence from the continuous plurality of image frames; the first frame sequence sequentially includes the first frame of the first video segment, all the audio-associated frames, and the last frame of the first video segment; According to the first frame sequence, at least two frame groups are sequentially determined; wherein, each of the frame groups includes a first start frame and a first end frame, and the segment duration between the acquisition time of the first start frame and the acquisition time of the first end frame in each of the frame groups is less than or equal to the maximum duration threshold, and at least one of the first start frame and the first end frame in each of the frame groups is an audio-associated frame, and the first end frame of the previous frame group in two adjacent frame groups is the first start frame of the subsequent frame group; According to the acquisition time of the first start frame and the acquisition time of the first end frame of each of the frame groups, the first video segment is split into corresponding second video segments of each of the frame groups; wherein, the acquisition time of the first start frame of the frame group is the start time of the corresponding second video segment, and the acquisition time of the second end frame of the frame group is the end time of the second video segment corresponding to the frame group.

7. The video processing method according to claim 6, wherein The step of sequentially determining at least two frame groups according to the first frame sequence includes: Regarding the first frame of the first video segment as the first start frame of the first frame group; According to the first start frame of each frame group, sequentially search for the first end frame of each frame group according to the first frame sequence until the found first end frame is the last frame of the first video segment.

8. The video processing method according to claim 7, wherein The step of sequentially searching for the first end frame of each frame group according to the first start frame of each frame group includes: Determine the first start frame of the current frame group; According to the first frame sequence, search for a first target frame whose acquisition time is after the first start frame of the current frame group as the first end frame of the current frame group; wherein, the segment duration between the acquisition time of the first target frame and the acquisition time of the first start frame of the current frame group is less than or equal to the maximum duration threshold, and the segment duration between the acquisition time of the next frame of the first target frame in the first frame sequence and the acquisition time of the first start frame of the current frame group is greater than the maximum duration threshold.

9. The video processing method according to claim 8, wherein The step of searching for a first target frame whose acquisition time is after the first start frame of the current frame group as the first end frame of the current frame group according to the first frame sequence further includes: Determine that the segment duration between the acquisition time of the next frame image of the first start frame of the current frame group in the first frame sequence and the acquisition time of the first start frame of the current frame group is greater than the maximum duration threshold; Search for a second target frame in the continuous multi-frame images as the first end frame of the current frame group; wherein, the segment duration between the acquisition time of the second target frame and the acquisition time of the first start frame of the current frame group is equal to the maximum duration threshold.

10. The video processing method according to any one of claims 1 to 9, characterized in that The step of splitting the first video segment into at least two second video segments includes: Split the continuous multi-frame images of the first video segment and the audio segment to obtain at least two of the second video segments.

11. A video processing method, characterized in that, The method includes: Determine the basic material to be processed; at least one basic material is a video material including an audio segment; If the video material includes a first video segment with a segment duration greater than the maximum duration threshold, split the first video segment into at least two second video segments according to the audio-associated frames in the first video segment; wherein, the audio-associated frames are the image frames corresponding to the start or end moments of each audio segment among the consecutive multiple frames of images, the first frame or the last frame of each second video segment is the audio-associated frame, and the segment duration of any second video segment is less than or equal to the maximum duration threshold; Extract highlight segments from all the basic materials according to the image parameters and audio parameters of all the basic materials; Generate a video set according to all the highlight segments.

12. The video processing method according to claim 11, wherein The step of extracting highlight segments from all the basic materials according to the image parameters and audio parameters of all the basic materials includes: Based on the image parameters of each frame of image in the basic materials, calculate the first score of each frame of image respectively; Based on the audio parameters of each frame of image in the basic materials, calculate the second score of each frame of image respectively; Extract the first highlight segments from the basic materials based on the first scores and second scores of each frame of image.

13. The video processing method according to claim 12, wherein The step of calculating the second score of each frame of image respectively based on the audio parameters of each frame of image in the basic materials includes: Calculate the second score for each frame of image according to the existence state of the audio segment corresponding to each frame of image.

14. The video processing method according to claim 13, wherein The step of calculating the second score for each frame of image according to the existence state of the audio segment corresponding to each frame of image includes: Add a fade-in period and a fade-out period to each audio segment to determine the existence period of the audio segment; wherein, the existence period of the audio segment includes the fade-in period, the acquisition period of the valid audio segment, and the fade-out period; If the acquisition moment of the current frame of image is within the existence period of any audio segment, calculate the second score for the current frame of image.

15. The video processing method according to any one of claims 12 to 14, characterized in that, The step of extracting highlight segments from the basic materials based on the first scores and second scores of each frame of image includes: Calculate the frame score of each frame of image according to the first score and second score of each frame of image; Extract highlight segments from the basic materials according to the frame scores of each frame of image in all the basic materials.

16. The video processing method according to claim 15, wherein The step of extracting highlight segments from the basic materials according to the frame scores of each frame of image in all the basic materials includes: Filter out the images with frame scores greater than the first threshold from all the images to obtain a first sequence of image frames; If there is an interval duration greater than the interval threshold between adjacent images in the first sequence of image frames, split the adjacent images with an interval duration greater than the interval threshold into different video segments to obtain one or more first video segments; Filter out the first video segments with segment durations less than the minimum duration from all the first video segments, and split the first video segments with segment durations greater than the maximum duration threshold to obtain multiple second video segments; Obtain multiple highlight segments according to the multiple second video segments.

17. An electronic device, characterized in that, Including a display, a memory, and a processor; The memory stores a computer program; The processor executes the computer program to implement the video processing method according to any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium. When the computer-readable storage medium runs on a computer, the computer is caused to execute the video processing method according to any one of claims 1 to 16.

Images