Video processing method, electronic device, and storage medium

Abstract

The application discloses a video processing method, an electronic device and a storage medium, and relates to the technical field of video data, and comprises the following steps: performing highlight segment analysis on a plurality of image materials selected by a user by the electronic device; after obtaining the highlight segments of each video in the plurality of image materials, if the sum of the lengths of the highlight segments is less than a preset recommended total length of the highlight segments, the electronic device can supplement the highlight segments from all the videos until the sum of the lengths of the highlight segments is equal to or greater than the recommended total length of the highlight segments, or there is no image frame that can be supplemented in the video. Wherein, the highlight segments of each video in the plurality of image materials are used to splice to obtain a target video set. In the scheme, the length of the output highlight segment meets the length requirement, and the problem of missing highlight segments is avoided, so that the target video set output has high video quality.

Description

Technical Field

[0001] This application relates to the field of video data technology, and in particular to a video processing method, electronic device, and storage medium. Background Technology

[0002] With the development of image and video processing technologies, users can trigger electronic devices to further process photos and videos in their albums. For example, electronic devices can stitch together multiple image materials (including multiple pictures and / or multiple videos) to obtain a complete stitched video.

[0003] For example, electronic devices or third-party video processing software within them can have a one-click video creation function (or a one-click video creation service). This function automatically analyzes and extracts highlight segments from multiple image clips selected by the user using algorithms; then, it automatically generates an edited video based on these extracted highlight segments. These highlight segments, also known as "highlight clips," refer to single-frame images or video clips composed of multiple consecutive frames extracted from the aforementioned materials to record exciting moments. These highlights could be moments of remarkable action, such as a person smiling, a moment of victory, or an airplane landing.

[0004] However, when extracting highlight segments from a video using existing technologies, it is easy to cause the omission of other highlight segments in the video. Summary of the Invention

[0005] This application provides a video processing method, electronic device, and storage medium. When the sum of the highlight segments in a video is less than the preset total duration of the highlight segments, the highlight segments can be supplemented to avoid missing highlight segments in the video. At the same time, the duration and quality of the target video set obtained by splicing the highlight segments are guaranteed.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions.

[0007] Firstly, a video processing method is provided, the method comprising:

[0008] The electronic device receives a user's selection of multiple image resources from its gallery. These multiple image resources may include videos; alternatively, they may include both videos and pictures.

[0009] The electronic device responds to the selection operation by analyzing each video to obtain the highlight segments of each video.

[0010] If the sum of the durations of all highlight clips in multiple image materials is less than the recommended total duration of the highlight clips, highlight clips are selected from the image frames covered by the non-highlight clips of all videos until the output conditions for highlight clips are met.

[0011] Among them, the output conditions for highlight clips include the sum of the durations of all highlight clips being equal to or greater than the recommended total duration of the highlight clips, or there being no image frames in the video that can be selected as replacements.

[0012] Highlight clips from each video in multiple image materials are stitched together to obtain the target video set; the suggested total duration of the highlight clips is the suggested value of the sum of the durations of the highlight clips from all videos in the multiple image materials.

[0013] In this application, an electronic device performs highlight segment analysis on multiple image materials selected by the user. After obtaining highlight segments from each video in the multiple image materials, if the sum of the durations of the highlight segments is less than the preset suggested total duration of highlight segments, the electronic device can supplement highlight segments from all videos, satisfying the condition that the sum of the durations of the highlight segments is equal to or greater than the suggested total duration of the highlight segments, or that there are no image frames in the video that can be supplemented. The highlight segments from each video in the multiple image materials are used to stitch together a target video set. This solution effectively avoids the problem of missing other highlight segments in the video, while ensuring the duration and quality of the target video set obtained by stitching together highlight segments.

[0014] In one possible implementation of the first aspect, the highlight segments of each video are obtained by analyzing each video, including:

[0015] A preliminary analysis of a preset number of image frames is performed on each video to obtain a score for each image frame. The score is an aesthetic score for the corresponding image frame.

[0016] Based on the score of each image frame in the video, the highlight segments of the corresponding video are determined.

[0017] The highlight segment includes the first image frame, as well as image frames within a first preset time period before and after the first image frame; the first image frame is the image frame with the highest score in the overview analysis.

[0018] In this application, the highlight segments in each video are determined by the first image frame with the highest score, which ensures that each video has at least one highlight segment with the highest value. In this way, the output video set including highlight segments has better effect and quality.

[0019] In another possible implementation of the first aspect, highlight segments are supplemented from image frames covered by non-highlight segments of all videos until the output conditions for highlight segments are met, including:

[0020] Highlight segments are selected from the second image frames of all videos until the output conditions for highlight segments are met. Specifically, the score of the second image frame is greater than a first threshold, and the second image frame is not covered by any already selected highlight segments.

[0021] In this application, highlight segments are supplemented from the second image frame with a higher score. The effect of the supplemented highlight segments is better. After the highlight segments are supplemented based on the second image frame, when the highlight segments meet the output conditions, the duration and quality of the output highlight segments are guaranteed, and the problem of missing highlight segments is avoided.

[0022] In another possible implementation of the first aspect, highlight segments are supplemented from the second image frames of all videos until the output conditions for highlight segments are met, including:

[0023] According to the score of each second image frame from high to low, the highlight segment corresponding to each second image frame is obtained in sequence; the highlight segment corresponding to the second image frame includes the second image frame, as well as the image frames before and after the second image frame within a second preset time period.

[0024] If the sum of the durations of all highlight segments in the video after being supplemented from the second image frame is greater than or equal to the suggested total duration of the highlight segments, the highlight segment supplementation operation ends.

[0025] If the sum of the durations of all highlight segments in the video after being supplemented from the second image frame is less than the suggested total duration of the highlight segments, and if the sum of the durations of the highlight segments is greater than or equal to the suggested total duration of the highlight segments at the first magnification, then there is no second image frame in the video available for supplementing the highlight segments, and the highlight segment supplementing operation ends.

[0026] If the sum of the durations of the highlight clips in all videos after being supplemented from the second image frame is less than the suggested total duration of the highlight clips at the first magnification, highlight clips are supplemented from the third image frame of all videos until the output conditions for highlight clips are met.

[0027] The third image frame is an image frame whose score is greater than or equal to the second threshold and is not covered by a highlight segment; the second threshold is less than the first threshold.

[0028] In this application, there is no second image frame available for replacement in the video. When the sum of the durations of the highlight segments is less than the recommended total duration of the highlight segments at the first magnification, highlight segments are selected from the third image frame to obtain highlight segments that meet the output conditions, thus ensuring the duration and quality of the output highlight segments.

[0029] In another possible implementation of the first aspect, the method further includes:

[0030] If the duration of the highlight segment corresponding to the second image frame is greater than the preset maximum duration of the highlight segment, starting from the left and right boundary positions of the highlight segment, the preset distance is reduced towards the midpoint position of the highlight segment until the duration of the reduced highlight segment is equal to or less than the maximum duration of the highlight segment.

[0031] The maximum duration of the preset highlight clip is the maximum allowed duration of a highlight clip to ensure the complete highlight effect of the video.

[0032] In this application, the duration of the highlight segment is verified. If the duration exceeds the maximum duration of the highlight segment, the duration of the highlight segment is reduced to control the duration of the highlight segment so that the duration of the highlight segment is less than the maximum duration of the highlight segment, thereby avoiding an increase in the amount of analysis caused by the longer duration of the highlight segment.

[0033] In another possible implementation of the first aspect, the method further includes:

[0034] If the duration of the highlight segment corresponding to the second image frame is less than the preset minimum duration of the highlight segment, then the second image frame is discarded.

[0035] The preset minimum length of the highlight clip is the minimum duration required for a highlight clip to ensure the complete highlight effect of the video.

[0036] In this application, the duration of the highlight fragment is verified. If the duration is less than the minimum duration of the highlight fragment, it is considered that the highlight fragment cannot represent the highlight moment well and the value of the highlight fragment is relatively low. In order to avoid invalid analysis of such highlight fragments, the image frame corresponding to the highlight fragment can be discarded, and subsequent selection of highlight fragments will not be based on the image frame.

[0037] In another possible implementation of the first aspect, highlight segments are supplemented from the third image frames of all videos until the output conditions for highlight segments are met, including:

[0038] According to the score of each third image frame from high to low, the highlight segment corresponding to each third image frame is obtained in sequence; the highlight segment corresponding to the third image frame includes the third image frame, as well as the image frames before and after the third preset time period.

[0039] If the sum of the durations of all highlight segments in the video after being supplemented from the third image frame is greater than or equal to the suggested total duration of the highlight segments, the highlight segment supplementation operation ends.

[0040] If the sum of the durations of all highlight segments in the video after being supplemented from the third image frame is less than the suggested total duration of the highlight segments, and if the sum of the durations of the highlight segments is greater than or equal to the suggested total duration of the highlight segments at the second magnification, then there are no third image frames in the video available for supplementing the highlight segments, and the highlight segment supplementing operation ends.

[0041] If the sum of the durations of all highlight clips in the video after being supplemented from the third image frame is less than the suggested total duration of the highlight clips at the second magnification, the duration of the highlight clips is extended until the output conditions for the highlight clips are met.

[0042] In this application, there are no optional second or third image frames in the video. When the sum of the durations of the highlight segments is less than the recommended total duration of the highlight segments at the second magnification, the duration of the first target highlight segment is extended, thereby obtaining highlight segments that meet the output conditions, ensuring the duration and quality of the output highlight segments, and avoiding the problem of missing highlight segments.

[0043] In another possible implementation of the first aspect, the method further includes:

[0044] If the duration of the highlight segment corresponding to the third image frame is greater than the preset maximum duration of the highlight segment, the second preset duration is reduced from the left and right boundary positions of the highlight segment to the midpoint position of the highlight segment, until the duration of the highlight segment after the reduction is equal to or less than the maximum duration of the highlight segment.

[0045] The maximum duration of the preset highlight clip is the maximum allowed duration of a highlight clip to ensure the complete highlight effect of the video.

[0046] In this application, the duration of the highlight segment is verified. If the duration exceeds the maximum duration of the highlight segment, the duration of the highlight segment is reduced to control the duration of the highlight segment so that the duration of the highlight segment is less than the maximum duration of the highlight segment, thereby avoiding an increase in the amount of analysis caused by the longer duration of the highlight segment.

[0047] In another possible implementation of the first aspect, the method further includes:

[0048] If the duration of the highlight segment corresponding to the third image frame is less than the preset minimum duration of the highlight segment, then the third image frame is discarded.

[0049] The preset minimum length of the highlight clip is the minimum duration required for a highlight clip to ensure the complete highlight effect of the video.

[0050] In this application, the duration of the highlight fragment is verified. If the duration is less than the minimum duration of the highlight fragment, it is considered that the highlight fragment cannot represent the highlight moment well and the value of the highlight fragment is relatively low. In order to avoid invalid analysis of such highlight fragments, the image frame corresponding to the highlight fragment can be discarded, and subsequent selection of highlight fragments will not be based on the image frame.

[0051] Another possible implementation of the first aspect includes:

[0052] Select the first target highlight segment from the highlight segments, and move the left and right boundary positions of the first target highlight segment to both sides to obtain the second target highlight segment after duration expansion; the second magnification is less than the first magnification.

[0053] The first target highlight segment is a highlight segment whose duration is less than or equal to the maximum duration of a preset highlight segment, and there are expandable areas before and after the target highlight segment, and the distance between the left and right boundary positions of the target highlight segment and the boundary positions of the adjacent segments is greater than a preset distance.

[0054] The expandable region is the connected region in the video excluding highlight segments and regions with scores below the third threshold. Connected regions are regions with the same score and continuous time, and there are no overlapping regions between connected regions.

[0055] The preset maximum duration of the highlight clip is the maximum allowed duration of a highlight clip to ensure the complete highlight effect of the video.

[0056] In this application, there are no optional second or third image frames in the video. When the sum of the durations of the highlight segments is less than the recommended total duration of the highlight segments at the second magnification, the duration of the first target highlight segment is extended, thereby obtaining highlight segments that meet the output conditions, ensuring the duration and quality of the output highlight segments, and avoiding the problem of missing highlight segments.

[0057] In another possible implementation of the first aspect, the left and right boundary positions of the first target highlight fragment are moved to both sides respectively to obtain the second target highlight fragment after duration expansion, including:

[0058] The difference between the suggested total duration of the highlight clip at the second magnification and the sum of the durations of the highlight clips in all videos after being supplemented from the third image frame is taken as the required extended duration of the highlight clip.

[0059] Based on the required extension duration of the highlight fragment, an extension quota duration is allocated to each first target highlight fragment; the extension quota duration is the duration during which the first target highlight fragment is allowed to extend in the corresponding scalable region.

[0060] Within the expandable region of the target highlight fragment, the left and right boundary positions of the first target highlight fragment are moved to both sides by 1 / 2 of the expansion quota duration to obtain the second target highlight fragment.

[0061] In this application, the duration of the first target highlight segment is extended based on the extended quota duration. This ensures that the duration of the extended second target highlight segment is not too long, and the resulting highlight segment meets the output conditions. This guarantees the duration and quality of the output highlight segment and avoids the problem of missing highlight segments.

[0062] In another possible implementation of the first aspect, the allocation of extended quota duration for each first target highlight segment is based on the required extended duration of the highlight segment, including:

[0063] The extension quota duration for each target highlight segment is determined based on the required extension duration of the highlight segment and the duration of each first target highlight segment.

[0064] If the sum of the durations of all highlight clips and the sum of the extended quota durations is greater than the preset maximum total duration of highlight clips, then the value of the extended quota duration will be updated to the difference between the maximum total duration of highlight clips and the sum of the durations of all highlight clips.

[0065] If the sum of the durations of all highlight clips and the sum of the extended quota durations are less than or equal to the preset maximum total duration of highlight clips, the value of the extended quota duration remains unchanged.

[0066] The maximum total duration of the preset highlight clips is the maximum allowed duration of highlight clips in all videos.

[0067] In this application, the extended quota duration of the first target highlight segment is determined based on the sum of the durations of all highlight segments and the cumulative sum of the extended quota duration. This ensures that the duration of the second target highlight segment after the extended duration does not exceed the preset maximum total duration of the highlight segments. Under the limited time consumption and limited performance of electronic devices, the duration and quality of the output highlight segments can be guaranteed.

[0068] In another possible implementation of the first aspect, the method further includes:

[0069] After extending the duration of the highlight clips, if the sum of the durations of all highlight clips is greater than or equal to the suggested total duration of the highlight clips at the second magnification, there are no third image frames available for replacement in the video, and the highlight clips in the video cannot be extended further, the highlight clip replacement operation ends.

[0070] In this application, the highlight segment selection operation ends after there are no second or third image frames available for replacement in the video, and after duration extension of the first target highlight segment. The resulting highlight segments meet the output conditions, ensuring the duration and quality of the output highlight segments, and avoiding the problem of missing highlight segments.

[0071] In a second aspect, an electronic device is provided, comprising a memory, a display screen, and one or more processors; the memory, the display screen, and the processors are coupled together; the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any one of the first aspects above.

[0072] Thirdly, a computer-readable storage medium is provided that stores instructions which, when executed on an electronic device, cause the electronic device to perform any of the methods described in the first aspect.

[0073] Fourthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to perform the method described in any one of the first aspects above.

[0074] Fifthly, embodiments of this application provide a chip, the chip including a processor, the processor being configured to invoke a computer program in memory to perform the method as described in the first aspect.

[0075] It is understood that the beneficial effects of the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect can be referred to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0076] Figure 1 This is a schematic diagram illustrating an application scenario of a video processing method provided in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram illustrating an application scenario of another video processing method provided in an embodiment of this application.

[0078] Figure 3 This is a schematic diagram illustrating an application scenario of another video processing method provided in an embodiment of this application.

[0079] Figure 4 This is a schematic diagram illustrating an application scenario of another video processing method provided in an embodiment of this application.

[0080] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0081] Figure 6 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0082] Figure 7 A flowchart illustrating a video processing method provided in an embodiment of this application;

[0083] Figure 8 A schematic diagram illustrating the process of analyzing video to obtain highlight segments in a video processing method provided in an embodiment of this application;

[0084] Figure 9A schematic diagram of a target area provided in an embodiment of this application;

[0085] Figure 10 A schematic diagram of another target area provided in an embodiment of this application;

[0086] Figure 11 A schematic diagram of a highlight segment provided in an embodiment of this application;

[0087] Figure 12 A schematic diagram of another highlight segment provided in an embodiment of this application;

[0088] Figure 13 A schematic flowchart illustrating the process of selecting highlight segments in a video processing method provided in this application embodiment;

[0089] Figure 14 A schematic diagram illustrating the process of selecting highlight segments in video 1, as provided in an embodiment of this application;

[0090] Figure 15 A schematic diagram illustrating another method for selecting highlight segments in video 1, provided as an embodiment of this application;

[0091] Figure 16 A schematic diagram illustrating another method for selecting highlight segments in video 1, provided as an embodiment of this application;

[0092] Figure 17 A schematic diagram of an expandable region 1 of a highlight segment 1 in a video 1 provided in an embodiment of this application;

[0093] Figure 18 A schematic diagram illustrating the allocation of required extended duration provided in an embodiment of this application;

[0094] Figure 19 This is a schematic diagram illustrating another possible allocation of the required extended duration, provided in an embodiment of this application.

[0095] Figure 20 This is a schematic diagram illustrating the expansion of highlight fragment 2 according to an embodiment of this application. Detailed Implementation

[0096] In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0097] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0098] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0099] First, some of the terms or concepts used in this application will be explained.

[0100] Highlight clips, also known as montage clips, are single-frame images or video clips composed of multiple consecutive frames extracted from videos or images to record exciting moments. Highlight moments can include moments of great action such as a person smiling, winning a competition, jumping, an airplane landing, or a goal scored in a ball game.

[0101] One-click video editing refers to an electronic device's ability to automatically analyze highlight segments from one or more image clips selected by the user, and then combine these highlight segments into a pre-edited video. In other words, the electronic device can extract multiple highlight segments from one or more image clips and combine them into a single video. The image clips selected by the user can be pictures or videos; alternatively, the image clips can include both pictures and videos.

[0102] The process by which an electronic device selects highlight segments from a video or image may include: acquiring aesthetic scoring parameters for each frame, such as image color, image texture features, image quality, frame interpolation with preceding and following frames, and edge change rate values; and then scoring each frame aesthetically based on these parameters. The electronic device can then use the highest-scoring single frame or multiple consecutive frames as a single highlight segment.

[0103] Currently, the one-click video editing feature supports editing of images and videos. However, when implementing this feature, users may select a large number of images or a long video. In such cases, the electronic device may randomly select some images from the user's chosen materials to extract highlight segments, potentially causing other highlight segments in the video to be missed.

[0104] In view of the above problems, this application provides a video processing method. An electronic device performs highlight segment analysis on multiple image materials selected by the user, obtaining highlight segments from each video in the multiple image materials. If the sum of the durations of the highlight segments is less than the preset suggested total duration of highlight segments, the electronic device can supplement highlight segments from all videos until the sum of the durations of the highlight segments is equal to or greater than the suggested total duration of the highlight segments, or there are no image frames in the video that can be supplemented. The highlight segments from each video in the multiple image materials are used to stitch together a target video set. This solution effectively avoids the problem of missing other highlight segments in the video, while ensuring the duration and quality of the target video set obtained by stitching together highlight segments.

[0105] The video processing method provided in this application can be applied to electronic devices with image processing capabilities. It should be noted that the image materials used for one-click video creation in this application embodiment can include both images and videos. Users can use the one-click video creation function to generate a video set from highlight clips of multiple images, or from highlight clips of multiple videos, or from highlight clips of multiple images and videos.

[0106] The aforementioned electronic devices can also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, etc. The embodiments of this application do not limit the specific technologies or device forms used in the electronic devices.

[0107] The following combination Figures 1-4 Taking mobile phones as an example, this paper introduces the application scenarios and interface implementation of one-click video creation on electronic devices.

[0108] In one application scenario, a user can pre-capture multiple images using their mobile phone, which are then stored in the phone's gallery. The phone can also pre-acquire images transmitted from other devices. For example... Figure 1 As shown in (a), a gallery icon is displayed on the phone's home screen. When a user wants to create an edited video clip based on multiple image assets using the phone, the user can tap the gallery icon on the home screen. In response to the user's tap on the icon, the phone displays the following... Figure 1 The gallery interface 101 is shown in (b) above. This gallery interface 101 includes a "One-Click Movies" option. The phone then responds to the user's request... Figure 1 Clicking the "One-Click Blockbuster" option shown in (b) will display the following: Figure 1 The gallery interface 102 is shown in (c). This gallery interface 102 can include multiple recently captured images. Users can select... Figure 1 One or more image assets from the plurality of image assets shown in (c) are used as candidate image assets for one-click image generation. For example, the mobile phone responds to the user's request. Figure 1 The selection operation of some image materials in (c) can display as follows: Figure 1The gallery interface 103 is shown in (d). This gallery interface 103 includes all the image assets in the gallery. The gallery interface 103 may also include video generation options, such as a checkmark option.

[0109] The phone responds to the user's Figure 1 Clicking the checkmark option (d) indicates that the process analyzes the five image materials selected by the user in the gallery interface 103, extracts highlight segments from each image material, and generates a video set based on the selected highlight segments. During this process, the phone can display... Figure 2 The image library interface 201 is shown. This image library interface 201 includes the analysis material progress so that users can intuitively view the analysis progress.

[0110] In one example, after the phone generates a video set, it can display... Figure 3 The gallery interface 301 is shown in (a) above. The generated video set can be displayed in the gallery interface 301. The mobile phone can automatically play this video set in the gallery interface 301. Furthermore, as... Figure 3 As shown in (a), the gallery interface 301 may also include a video export option 302 for supporting the export of generated video sets. In response to a user's click on the video export option 302, the phone can save the video set in the gallery, allowing the user to view it. In response to a user's click on the video export option 302, the phone may also display... Figure 3 The video export interface 303 is shown in (b) of the diagram.

[0111] In one example, as the user selects image materials, the phone can provide prompts to help the user determine the appropriate number of image materials to choose. For example... Figure 1 As shown in (d), the image library interface 103 displays the prompt message "6 or more image materials will produce better results", so that users can know at least how many image materials to select to generate a better video set.

[0112] In one example, as the user selects image assets, the phone can provide prompts to let the user know the maximum number of image assets they can select. For example... Figure 4 As shown, the gallery interface 401 displays the message "A maximum of 30 image materials can be selected", so that users can know how many image materials they can select.

[0113] In one example, after generating the video set, the phone can also display an interface showing other function options, allowing users to edit, add effects, analyze, and perform other operations on the generated video set based on these options. Figure 3As shown in 301, other function options may include, but are not limited to, templates, music, clips, sharing, etc.

[0114] The following example uses a mobile phone as an electronic device, combined with... Figure 5 The hardware structure of electronic devices will be introduced.

[0115] Figure 5 A schematic diagram of the hardware structure of the electronic device 100 provided in an embodiment of this application is shown. Figure 5 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a camera 193, a display screen 194, etc.

[0116] The processor 110 may include one or more processing units, such as a controller, application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). The controller may serve as the central nervous system and command center of the mobile phone 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor 110 may also include memory for storing instructions and data.

[0117] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0118] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0119] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering, such as rendering images... Figures 1-4 The diagram shows the user interface.

[0120] The display screen 194 is used to display the operation interface of the screen mirroring app, the mirrored image, and the mirrored video. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), or a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0121] In this embodiment, the display screen 194 can be used to display, for example, Figure 1 Gallery interfaces 101, 102, and 103; display screen 194 can be used to display, for example, Figure 2 The gallery interface 201; the display screen 194 can be used to display, for example, Figure 3 The image gallery interface 301 and video export interface 303 are shown in the image gallery interface 301; the display screen 194 can be used to display images such as... Figure 4 The image gallery interface in the middle is 401, etc.

[0122] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0123] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0124] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, mobile phone 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0125] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0126] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0127] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0128] The external memory interface 120 can be used to connect an external memory card, thereby expanding the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage.

[0129] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application (APP) required for a function (e.g., camera APP, gallery APP, and third-party video editing software, etc.). The data storage area may store data created during the use of the mobile phone 100 (e.g., photos or videos taken, screenshots, screen recordings, images downloaded from other devices, and video sets generated using the one-click video creation function, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, and universal flash storage (UFS, etc.).

[0130] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0131] For example, after a video set is generated using the one-click video creation function, the audio module 170 decodes the audio signal of the video set, and then the speaker 170A, also known as a "loudspeaker," converts the audio electrical signal into a sound signal. In this way, the user can hear background music synchronized with the video in the highlight clips, as well as added video background music.

[0132] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0133] The following is combined Figure 6 The software architecture of electronic devices will be introduced.

[0134] Figure 6 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application.

[0135] like Figure 6As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom as follows: application (APP) layer, media platform framework layer, application framework (FWK) layer, and hardware abstraction layer (HAL).

[0136] The APP layer, or application layer for short, can include a series of application packages, such as camera, gallery, third-party video editing software, calendar, map, and navigation. When these application packages are run, they can access the various service modules provided by the media platform framework layer and the application framework layer through the application programming interface (API) and execute corresponding intelligent business logic.

[0137] In some embodiments, the camera is used to capture photos, videos, slow-motion images, and panoramic images in response to user actions. After these images are captured by the camera, or after the user triggers a screenshot, or after the user triggers screen recording, or after the electronic device downloads images from other devices, the electronic device can save these images to a gallery, allowing the user to perform video editing operations on the images in the gallery, such as one-click video editing.

[0138] In this embodiment of the application, the image library is divided into three layers from top to bottom: business layer, application function layer, and basic function layer.

[0139] The business layer, also known as the video editing business layer, provides various services (or functions) such as automatic multi-camera recording and editing, AI music videos, one-click video creation, and highlight moments. These services are presented as controls in the gallery's user interface (UI). Users can trigger corresponding video processing actions in the gallery by interacting with these controls. For example, after a user selects image materials (including pictures and / or videos), in response to the user's click on the one-click video creation control in the gallery, the gallery can call the underlying module to automatically analyze and extract highlight segments from the pictures and / or videos using algorithms, and then combine the highlight segments into a pre-edited video set.

[0140] The application functionality layer includes an automatic editing framework. Various business functions in the business layer can call this framework to provide automatic editing services for images and videos. For example, the automatic editing framework may include functional modules such as segment selection, storyline organization, layout splicing, and special effects enhancement. Segment selection is used to call the light segment analysis interface and strategy monitoring interface in the high-media platform framework layer to extract highlight segments from images and / or videos. Storyline organization is used to sequentially splice multiple images and / or videos in the form of a storyline based on their content. Layout splicing is used to adjust the interface layout of images and / or videos. Special effects enhancement is used to adjust the enhancement effects of images and / or videos, such as adjusting screen brightness and beautifying facial features.

[0141] The basic functionality layer is used to perform basic processing on the edited image and / or video clips after the automatic editing framework has edited multiple images and / or videos. For example, the basic functionality layer may include basic functional modules such as video splicing, compositing and saving, video effect rendering, and audio effect processing. Specifically, video splicing is used to splice multiple extracted highlight clips (where highlight clips include images and / or videos). Compositing and saving is used to store the spliced ​​video set. Video effect rendering is used to add video effects to the spliced ​​video set, such as adding style filters and themes. Audio effect processing is used to add sound effects to the spliced ​​video set, such as adding background music.

[0142] The media middleware framework layer is a software layer positioned between the application layer and the application framework. This layer can include an analysis performance query interface, a highlight clip analysis interface, a policy monitoring interface, a pipeline interface, a theme summary interface, and an initialization interface. The analysis performance query interface calculates the total duration of all videos based on the video analysis speed. The highlight clip analysis interface calls the policy monitoring interface to extract highlight clips. The theme summary interface calls underlying algorithms to analyze the content of highlight clips to determine the corresponding theme. The initialization interface initializes the highlight clip algorithm, face detection algorithm, video acceleration algorithm, and image super-resolution algorithm in the HAL layer.

[0143] The policy monitoring interface is used to configure the expected analysis duration for each video based on the total duration of all videos, and to set the analysis strategy for each video based on parameters such as the expected analysis duration and the duration of each video. The channel interface is used to down-resolution the video files based on the file descriptors and analysis strategies issued by the policy monitoring interface, and then forward the data addresses of the down-resolution video files to the hardware abstraction layer through the application framework layer. Finally, the analysis results of the image frames returned by the hardware abstraction layer are reported to the policy monitoring interface.

[0144] The theme summary interface is used to call the underlying algorithms to analyze the content of highlight segments and determine the theme corresponding to the content of the highlight segments. The initialization interface is used to initialize the highlight segment algorithm, face detection algorithm, video acceleration algorithm, and image super-resolution algorithm of the HAL layer.

[0145] It should be noted that this application uses the one-click image creation function provided by an image library as an example for illustration, and does not limit the embodiments of this application. In actual implementation, third-party video editing software can use the video processing method provided in the embodiments of this application to combine multiple images and videos selected by the user into a single video set with one click.

[0146] The FWK layer, or framework layer for short, supports the operation of various modules within the media middleware framework layer. For example, the framework layer may include a one-click video creation interface, a parameter management interface, an image data transmission interface, a theme analysis interface, and a performance analysis interface.

[0147] The Hardware Abstraction Layer (HAL) is a wrapper around Linux kernel drivers, providing interfaces to higher-level systems. It hides the hardware interface details of specific platforms, providing the operating system with a virtual hardware platform that is hardware-independent and portable across multiple platforms. For example, the HAL can include chip analysis speed interfaces, highlight fragment algorithms, face detection algorithms, video acceleration algorithms, and image super-resolution algorithms. The highlight fragment algorithm is an image processing algorithm provided by the image signal processor. This algorithm performs an aesthetic score on each frame of an image based on factors such as image color, texture features, image quality, frame interpolation with preceding and following frames, and edge change rate. The aesthetic score serves as the basis for evaluating whether a frame contains a highlight fragment.

[0148] In this embodiment, the highlight segment algorithm analyzes each image frame in the video to obtain a score for each frame. For example, if the score of an image frame is greater than a preset scoring threshold, then that image frame and the image frames within its adjacent preset duration can be considered as highlight segments of the source video. For instance, the preset scoring threshold can be 90. If the score of an image frame in the video is greater than 90, then that image frame and the image frames within its adjacent preset duration can be considered as a highlight segment of the video. Alternatively, if the video includes scores for N image frames, the image frame with the highest score and the image frames before and after that image frame within a preset duration are considered as highlight segments of the video.

[0149] In some other feasible embodiments, assuming the score range for an image frame is 0-100, the score can be divided into different levels based on the different score ranges. For example, a first threshold of 80: if the score of an image frame is greater than or equal to the first threshold (score range 80-100), the score is considered "high". A second threshold of 50: if the score of an image frame is greater than or equal to the second threshold (score range 50-79), the score is considered "relatively high". A third threshold of 20: if the score of an image frame is greater than or equal to the third threshold (score range 20-49), the score is considered "medium". A score less than the third threshold (score range 0-19) is considered "low". For example, in one implementation, highlight segments of the video can be extracted from image frames whose scores are greater than or equal to a first threshold / second threshold / third threshold. In some other implementations, the image frame with the highest score can be used as the optional image frame for highlight segment extraction.

[0150] It should be noted that, Figure 6 The layers and components within each layer of the illustrated software architecture do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more layers than illustrated, such as a system library (FWKLIB) layer and a kernel layer. Each layer may include more or fewer components than illustrated. Furthermore, the aforementioned functional modules may be combined into a single functional module, and the layers may be combined into a single layer; for example, highlight segment analysis may include policy monitoring, and a media middleware framework layer may be located within an application framework layer.

[0151] It is understood that, in order to implement the video processing method in the embodiments of this application, the electronic device includes hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments.

[0152] In this embodiment, after the strategy monitoring module of the media platform framework layer extracts the first highlight segment of each video in the image material, if the sum of the durations of the highlight segments of all videos does not reach the total recommended duration of the highlight segments, it can perform a highlight segment supplementation operation on the analysis results of the analyzed image frames of all videos, so that the duration of the output highlight segments meets the duration requirement and has high video quality, thereby improving the user's experience of using the one-click blockbuster function.

[0153] The strategy monitoring module in the media platform framework layer can analyze video footage to obtain a highlight segment from each video. This module sends each video frame to the HAL (Hardware Algorithm) layer's algorithm module for analysis, obtaining a score for each frame. Based on the highest-scoring frame and the corresponding regions before and after it, each video highlight segment is determined. If the sum of the durations of all video highlight segments does not reach the recommended total duration, the strategy monitoring module can supplement the highlight segments from the analyzed frames, ensuring the final output meets the requirements and has high video quality, thus enhancing the user experience of the "one-click blockbuster" feature.

[0154] The following section takes the execution entity of the audio and video processing method as an example. Figure 6 Using the modules shown in the software structure diagram as examples, the video processing method provided in this application embodiment will be illustrated by way of example.

[0155] Figure 7 This paper presents a flowchart illustrating a method in video processing before the electronic device policy monitoring module acquires analysis parameters. This method can be applied to, for example... Figures 1-4 In the one-click image generation scenario shown, for example... Figure 7 As shown, taking image materials including pictures and videos as an example, the method may include the following steps S01-S16.

[0156] S01, the business layer receives user input to enable the one-click video creation function.

[0157] In this embodiment, the "business layer" refers to the one-click video creation module within the business layer. That is, the one-click video creation module in the business layer receives user input to enable the one-click video creation function. For example, this operation can specifically be as follows: Figure 1 The click operation of the "One-Click Blockbuster" card is shown in (b) in the image.

[0158] S02, the business layer loads and displays candidate images and candidate videos.

[0159] S03, the business layer receives the user's operation of selecting multiple pictures and videos, and receives the user's confirmation to execute the one-click video creation function.

[0160] For example, the user's selection of multiple images and videos can be done as follows: Figure 1 The click operation on photos and videos shown in (c) allows the user to confirm and execute the one-click video creation function, which can be done as follows: Figure 1 (d) shows the click action for the checkmark option.

[0161] S04, the business layer calls the initialization interface of the media middle platform framework layer through the application function layer to initialize the relevant algorithms of the HAL layer.

[0162] In this embodiment, "related algorithms" refers to the algorithms for the functions that the business layer needs to implement. Here, the business function is a one-click video creation function, so the related algorithms are those involved in the one-click video creation function. For example, related algorithms include highlight segment algorithms, face detection algorithms, video acceleration algorithms, and image super-resolution algorithms, etc.

[0163] S05, the initialization interface of the media middle platform framework layer sends initialization parameters to the algorithm module of the HAL layer through the channel interface of the media middle platform framework layer and the service interface of the FWK layer in sequence.

[0164] In the HAL layer, one algorithm corresponds to one algorithm interface. The FWK layer has multiple service interfaces, and one service interface in the FWK layer corresponds to one algorithm interface in the HAL layer. The various service interfaces in the FWK layer play a role in data pass-through between the algorithm interfaces in the HAL layer and the channel interfaces in the media middleware framework layer.

[0165] Different algorithms involve different initialization parameters, so the algorithm initialization parameters sent to the algorithm interface through each service interface may be different.

[0166] S06, the algorithm module of the HAL layer is initialized according to the initialization parameters.

[0167] S07, the algorithm module of the HAL layer returns an initialization success message to the channel interface of the media middleware framework layer through the service interface of the FWK layer.

[0168] S08, the channel interface of the media middleware framework layer calls the analysis speed interface of the HAL layer through the service interface of the FWK layer to obtain the chip analysis speed.

[0169] Among them, the service interface of the FWK layer can be a performance analysis interface.

[0170] S09, the HAL layer's analysis speed interface returns the chip analysis speed to the media middleware framework layer's channel interface through the FWK layer's service interface.

[0171] S10, the channel interface of the media middleware framework layer returns the chip analysis speed to the initialization interface of the media middleware framework layer.

[0172] The chip analysis speed can characterize the number of image frames analyzed by the image signal processor per unit time; alternatively, it can characterize the duration of analysis of a single image frame (single-frame analysis time). Therefore, the single-frame analysis time and the processing time for a single image can be calculated based on the chip analysis speed. The processing time for a single image and the single-frame analysis time for a video may differ. For example, the processing time for a single image could be 400ms, and the single-frame analysis time could be 200ms.

[0173] It should be understood that because different image signal processors have different performance characteristics, the chip analysis speed corresponding to different image signal processors may vary. For a commercially available electronic device, the image signal processor is fixed, and therefore the chip analysis speed corresponding to that image signal processor is also fixed.

[0174] In some embodiments, the channel interface of the media middleware framework layer can also return other relevant performance parameters of various algorithms to the initialization interface of the media middleware framework layer.

[0175] S11, the initialization interface of the media middleware framework layer returns an initialization success message to the business layer through the application function layer.

[0176] The initialization success message can carry performance parameters of various algorithms, such as chip analysis speed.

[0177] After obtaining the chip analysis speed, for each video in the image materials selected by the user, the following steps S12-S15 can be executed to obtain the analysis parameters for the video and all image materials.

[0178] S12, the business layer sends a query message to the analysis performance query interface of the media platform framework layer through the application function layer.

[0179] The query message includes the file descriptor fd1 of the user-selected video 1 and the chip analysis speed. The file descriptor serves as a unique identifier for the video.

[0180] S13, the media middle platform framework layer's analysis performance query interface obtains the estimated analysis duration of video 1 based on video 1's file descriptor fd1.

[0181] The estimated analysis duration can be the time required for the image signal processor to analyze a video. Analyzing a video can refer to analyzing a specified number of image frames within the video. For example, the specified number can be 1, or it can be greater than 1 but less than or equal to the number of image frames contained in the video.

[0182] For example, when the specified quantity is 1, that is, when the estimated analysis time represents the time required for the image signal processor to analyze one image frame in a video, the estimated analysis time can also represent the minimum analysis time corresponding to the analyzed image frames. For example, if the single-frame analysis time of the image signal processor is 200ms, then the estimated analysis time for each video (including video 1) is 200ms.

[0183] In some embodiments, the specified quantity can also be the number of image frames contained in the video. In this case, the estimated analysis time represents the time it takes for the image signal processor to analyze all the image frames in a video. For example, if the single-frame analysis time of the image signal processor is 200ms, and video 1 includes 10 image frames, then the estimated analysis time for video 1 is 10 * 200ms = 2000ms. Similarly, if video 2 includes 12 image frames, then the estimated analysis time for video 2 is 12 * 200ms = 2400ms.

[0184] In some embodiments, the specified quantity can also be 'a', where 'a' is greater than 1 and less than the number of image frames contained in the video, and 'a' is a natural number. Therefore, the estimated analysis duration represents the time it takes for the image signal processor to analyze 'a' image frames in the video. For example, if the single-frame analysis duration of the image signal processor is 200ms, and video 1 includes 10 image frames, then 'a' is 5. Therefore, the estimated analysis duration of video 1 is 5 * 200ms = 1000ms.

[0185] S14, the analysis performance query interface of the media middle platform framework layer returns the estimated analysis duration of video 1 to the business layer through the application function layer.

[0186] After the one-click video generation module in the business layer obtains the estimated analysis duration of video 1, it can continue to return to execute S12-S15 to obtain the estimated analysis duration of the next video among multiple image materials, until the estimated analysis duration of all videos among multiple image materials is obtained.

[0187] S15, the business layer obtains analysis parameters based on the estimated analysis time in the image material.

[0188] The analysis parameters may include total analysis duration, suggested upper limit of total analysis duration, maximum duration of highlight clips, minimum duration of highlight clips, total suggested duration of highlight clips, suggested duration of highlight clips, whether to force each video to output a highlight clip, whether to enable audio analysis, selected highlight clips, etc.

[0189] The option to force each video to output a highlight clip is set to "yes" by default, meaning that in this embodiment, each video needs to output a highlight clip.

[0190] The total analysis time represents the total time required to complete the analysis of all image materials selected by the user (including all pictures and all videos selected by the user). The total analysis time includes the sum of the estimated analysis time of all pictures and the sum of the estimated analysis time of all videos.

[0191] For images in the image source material, the processing time for an image can be determined directly based on the chip speed of the image processor. Therefore, the sum of the estimated analysis times for all images can be determined directly based on the number of images in the image source material.

[0192] For videos in the image material, the estimated analysis duration of each video can be obtained according to S12-S15. The estimated analysis durations of each video are summed up to obtain the sum of the estimated analysis durations of all videos in the image material.

[0193] The total analysis time of the image materials can be obtained by summing the estimated analysis times of all images and all videos.

[0194] For example, suppose the user selects 10 images, video 1, and video 2. The estimated analysis time for one image is 400ms, the estimated analysis time for video 1 is 200ms, and the estimated analysis time for video 2 is 300ms. Then the total analysis time is 10 * 400ms + 200ms + 300ms = 4500ms.

[0195] The suggested upper limit for total analysis time represents the maximum recommended time required to complete the analysis of all images and videos. This upper limit can be determined based on the estimated analysis time for each video and image. Generally, the suggested upper limit for total analysis time is greater than the total analysis time. For example, assuming a minimum of one image frame is analyzed per video, the total analysis time would be 4400ms. However, considering that each video may require analyzing multiple image frames, the suggested upper limit for total analysis time can be significantly larger than the total analysis time; for instance, it could be a preset value of 10000ms.

[0196] In scenarios where analysis parameters are input abnormally, the recommended upper limit for the total analysis time may also be set to be less than the total analysis time. When the recommended upper limit for the total analysis time is less than the total analysis time, that is, when the recommended upper limit for the total analysis time is insufficient to analyze all images and all videos (one image frame), a subset of the selected image materials can be selected for analysis. This part is implemented by the strategy monitoring module of the media platform framework layer, which will be described in detail in the following embodiments and will not be elaborated here.

[0197] The maximum duration of a highlight clip represents the maximum allowed duration of a highlight clip in a video. The minimum duration of a highlight clip represents the minimum allowed duration of a highlight clip in a video. Both the maximum and minimum durations of a highlight clip can be preset values. For example, the maximum duration of a highlight clip can be 3000ms, and the minimum duration of a highlight clip can be 1000ms.

[0198] The total suggested duration of highlight clips represents the suggested value of the sum of the suggested durations of all highlight clips across multiple video clips. The total suggested duration of highlight clips can be determined based on the number of videos, the maximum duration of a highlight clip, and the minimum duration of a highlight clip. For example, the maximum duration of a highlight clip can be 3000ms, the minimum duration of a highlight clip can be 1000ms, and when there are 5 videos, the total suggested duration of highlight clips can range from 5000ms to 15000ms; for instance, the total suggested duration of highlight clips could be 8000ms.

[0199] The suggested duration for a highlight clip refers to a recommended value for the duration of a highlight clip in a video. Highlight clips of this suggested duration can effectively showcase the highlight effect. The suggested duration for a highlight clip can be determined based on the maximum and minimum duration of the highlight clip. For example, the maximum duration of a highlight clip can be 3000ms, the minimum duration can be 1000ms, and the suggested duration can range from 1000ms to 3000ms. For instance, the suggested duration for a highlight clip could be 2000ms.

[0200] It is important to understand that the maximum duration of highlight clips, the minimum duration of highlight clips, the total recommended duration of highlight clips, and the recommended duration of highlight clips can all be set according to the actual situation.

[0201] In some embodiments, the analysis parameters may also include the actual duration of each video.

[0202] S16, the business layer sends the file descriptors (fd) and analysis parameters of all materials to be analyzed to the image highlight segment analysis interface of the media platform framework layer through the application function layer.

[0203] After receiving the analysis parameters, the strategy monitoring module of the media middle platform framework layer can determine the material analysis strategy based on the suggested upper limit of the total analysis time and the total analysis time in the analysis parameters. The material analysis strategy refers to analyzing all image materials selected by the user, or selecting a portion of the selected image materials for analysis. In the following embodiments, the image materials include all videos and all images selected by the user.

[0204] After obtaining the total analysis time of the image materials, the strategy monitoring module can determine the actual number of images and videos to be analyzed based on the total analysis time and the suggested upper limit value of the total analysis time in the analysis parameters. Figure 7 After S16, execute:

[0205] S17, the strategy monitoring module of the media middle platform framework layer determines the material analysis strategy based on the total analysis time and the upper limit of the total analysis time suggested in the analysis parameters.

[0206] Specifically, if the total analysis time is less than or equal to the recommended upper limit for total analysis time, the content analysis strategy can be to analyze all images and all videos. For example, if the total analysis time is 4400ms and the recommended upper limit for total analysis time is 10000ms, the content analysis strategy can be to analyze all images and all videos in the image content selected by the user.

[0207] If the total analysis time exceeds the recommended upper limit, the content analysis strategy can be a random sampling strategy. For example, if the total analysis time is 4400ms and the recommended upper limit is 3000ms, then the content analysis strategy can be a random sampling strategy. Here, random sampling means randomly selecting a portion of the image content from the user-selected image content for analysis.

[0208] Assuming the analytical value of a single image is greater than that of a single frame from a video, the random sampling strategy could be: For every N images selected, M videos can be selected, until the total analysis time of the selected image materials reaches the suggested upper limit or all images in the selected image materials have been extracted. Here, M < N; for example, N can be a natural number such as 3, 4, or 5, and M can be a natural number less than N such as 1, 2, or 3. The specific values ​​of M and N can be determined based on the number of image materials selected by the user. For example, it could be that for every 4 images selected, 1 video can be selected, until the total analysis time reaches 3000ms; or, all images in the selected image materials have been extracted.

[0209] For example, the total analysis time for 10 images and 2 videos is 4400ms, which exceeds the recommended maximum analysis time of 3000ms. Following a random sampling strategy of allowing 1 video to be selected for every 4 images, selecting 4 images and 1 video results in a total analysis time of 1800ms, which is still below the recommended maximum of 3000ms. Continuing to select image materials, when the third image is selected in this round, the total analysis time reaches 3000ms, at which point image material selection stops. Therefore, the selected image materials are 7 images and 1 video. The remaining 3 images are not analyzed.

[0210] In another embodiment, assuming the analytical value of an image is less than the analytical value of one image frame from a video, the random sampling strategy could be to allow the selection of Q images for every P videos selected, until the total analysis time of the selected image materials reaches the suggested upper limit of the analysis time or all videos in the selected image materials have been selected. Here, Q < P. For example, P can be a natural number such as 3, 4, or 5, and Q can be a natural number less than P such as 1, 2, or 3. The specific values ​​of P and Q can be determined based on the number of image materials selected by the user. For example, it could be that one image is allowed for every three videos selected, until the total analysis time reaches 3000ms; or, until all videos in the selected image materials have been selected.

[0211] For example, the total analysis time for 10 videos and 3 images is 3200ms, which exceeds the recommended maximum analysis time of 3000ms. Following a random sampling strategy of allowing one image to be selected for every three videos, selecting 3 videos and 1 image results in a total analysis time of 1000ms, which is still below the recommended maximum. Continuing to select image materials, after three rounds of selecting 3 videos and 1 image, a total of 9 videos and 3 images have been selected, with a total analysis time of 3000ms. At this point, image material selection stops. Therefore, the selected image materials are 9 videos and 3 images. The remaining 1 video is not analyzed.

[0212] In some embodiments, the electronic device determines analyzable image materials from the user-selected materials according to the random sampling strategy described above (e.g., a mobile phone). Specifically, images and videos can be randomly selected from the user-selected image materials in the order they were chosen. Alternatively, a random number less than the number of materials can be generated using Java's Random class, and the video or image corresponding to that random number can be selected.

[0213] In some embodiments, the image material includes only pictures, and the total analysis time is the sum of the estimated analysis times of all pictures. When the total analysis time exceeds the suggested upper limit of the analysis time, the number of analyzable pictures is calculated based on the suggested upper limit of the analysis time and the estimated analysis time for analyzing one picture, and a corresponding number of pictures are randomly selected from all pictures for analysis.

[0214] In some embodiments, the image material includes only videos, and the total analysis time is the sum of the estimated analysis times of all videos. When the total analysis time exceeds a suggested upper limit for analysis time, the number of analyzable videos is calculated based on the suggested upper limit, and a corresponding number of videos are randomly selected from all videos for analysis. The time required to reach the number of analyzable videos is less than or equal to the total analysis time.

[0215] After selecting the images and / or videos that can be analyzed within the suggested maximum analysis time from the image materials, the remaining images and videos in the user's selected image materials will not be analyzed.

[0216] The image material includes pictures. After selecting the images that can be analyzed within the recommended maximum analysis time, the electronic device executes the process. Figure 7 S18-S22, as shown, perform highlight segment analysis on the analyzable image.

[0217] The following example illustrates the process of performing highlight segment analysis on image 1 using steps S18-S22. Image 1 is one of the images that can be analyzed within the recommended upper limit of the analysis time.

[0218] S18, the strategy monitoring module of the media middle platform framework layer sends an instruction message to the channel interface of the media middle platform framework layer. The instruction message includes the file descriptor of Figure 1.

[0219] S19, the channel interface of the media middle platform framework layer performs decoding, resolution reduction and format conversion on image 1 according to the file descriptor of image 1, and stores the processed image 1.

[0220] S20, the channel interface of the media middleware framework layer sends the frame data address of image 1 to the highlight fragment algorithm interface of the HAL layer through the service interface of the FWK layer.

[0221] Among them, the service interface can be a one-click production interface.

[0222] S21, the HAL layer's specular highlight algorithm interface retrieves image 1 based on its frame data address, then performs an aesthetic score on image 1 using a preset specular highlight algorithm, yielding the analysis result. The aesthetic score is represented within the analysis result.

[0223] S22, the HAL layer's highlight fragment algorithm interface sequentially returns the analysis results of image 1 to the media platform framework layer's strategy monitoring module through the FWK layer's service interface and the media platform framework layer's channel interface.

[0224] After the strategy monitoring module of the media middle platform framework layer obtains the analysis results of image 1, if there are other images, such as image 2, the electronic device can continue to execute S18-S22 above to obtain the analysis results of the other images.

[0225] Among them, the strategy monitoring module of the media middle platform framework layer can determine the highlight segments of multiple image materials based on the analysis results of each image. The images with scores greater than or equal to the first threshold / second threshold are selected as highlight segments.

[0226] After obtaining the analysis results of all images, if the image material includes video, the electronic device can use steps S23-S38 as described below to obtain the analysis results of each video. If the image material does not include video, the electronic device outputs a target video set consisting of images of highlight segments.

[0227] If the image material includes multiple images and videos, since the computational load on images is smaller and takes less time, electronic devices typically perform highlight segment analysis on each image individually first. After completing the highlight segment analysis for all images, they then perform highlight segment analysis on each video individually. In other words, after completing the analysis for all images, the electronic device will proceed with the next step. Figure 7 As shown in S18-S22, continue to execute as follows: Figure 8 S23-S38 are shown.

[0228] If the image material only includes multiple videos, then after executing S17, the electronic device will execute as follows: Figure 8 As shown in S23-S38, there is no need to execute as follows: Figure 7 S18-S22 are shown.

[0229] This embodiment illustrates the case where the image material includes both pictures and videos. The videos mentioned below refer to those that can be analyzed within the recommended upper limit of the total analysis time.

[0230] refer to Figure 8 , Figure 8 A flowchart illustrating the process of analyzing video to extract highlight segments in a video processing method is provided. This includes:

[0231] S23, the strategy monitoring module determines the video analysis strategy based on the remaining analysis time.

[0232] The analysis strategies can include image frame analysis strategies, keyframe analysis strategies, and analysis strategies that combine overview analysis and frame-by-frame analysis, etc.

[0233] The remaining analysis time refers to the remaining available time for analyzing the video after S01-S22.

[0234] In some embodiments, the image frame analysis strategy refers to extracting and analyzing all or a predetermined number of image frames from the video to obtain the analysis results of the video's image frames; and determining the highlight segments of the video based on the image frame with the highest score. The keyframe analysis strategy refers to extracting and analyzing all or a predetermined number of keyframes from the video to obtain the analysis results of the video's keyframes; and determining the highlight segments of the video based on the keyframe with the highest score. The analysis strategy combining overview analysis and frame-by-frame analysis includes two analysis stages. In the first analysis stage, a predetermined number of image frames from the video are extracted and analyzed to obtain the analysis results of the image frames, and the target region of the video is determined based on the image frame with the highest score. In the second analysis stage, all or a predetermined number of image frames in the target region are analyzed to obtain the analysis results of the image frames in the target region, and the highlight segments of the video are determined based on the image frame with the highest score. Here, image frames can be keyframes, ordinary image frames, etc.

[0235] If the remaining analysis time is sufficient to analyze all image frames of all videos or a preset number of image frames, then the image frame analysis strategy is adopted; if the remaining analysis time is sufficient to analyze all keyframes of all videos or a preset number of keyframes, then the keyframe analysis strategy is adopted; if the remaining analysis time is sufficient to perform a combination of overview analysis and frame-by-frame analysis, then the analysis strategy combining overview analysis and frame-by-frame analysis is adopted.

[0236] Based on the analysis strategy determined by the strategy monitoring module, the following steps are performed to obtain the first highlight segment of each video.

[0237] Taking video 1 as an example, and using an analysis strategy that combines overview analysis and frame-by-frame analysis (analyzing the image frames of the video) as an example, the process of analyzing the highlight segments of the video in steps S24-S38 is illustrated. S24-S30 constitute the first analysis stage, and S31-S38 constitute the second analysis stage.

[0238] S24, the strategy monitoring module sends the file descriptor fd1 of video 1 and the first position of the image frame of video 1 to the channel interface of the media middle platform framework layer.

[0239] The positions of the first number of image frames in the video can be evenly distributed. Therefore, the first position can be the position of the first image frame starting from the beginning of the video.

[0240] In some embodiments, the positions of the first number of image frames in the video can also be distributed according to the video's shot division points or similar scene areas. The first position can be the position of any image frame in video 1. For example, the first position can be the position of the first image frame in video 1; or, the first position can also be other specified positions in video 1.

[0241] S25, the channel interface performs decoding, resolution reduction and format conversion on video 1 according to the file descriptor of video 1, and stores the processed video 1.

[0242] S26, the channel interface of the media middle platform framework layer sends the frame data address of video 1 and the corresponding first position image frame in video 1 to the highlight segment algorithm interface of the HAL layer through the service interface of the FWK layer (such as the one-click video creation interface).

[0243] S27, the HAL layer's highlight fragment algorithm interface obtains the image frame of the first position in video 1 based on the frame data address of video 1 and the corresponding first position in video 1, and then analyzes the image frame of the first position based on the preset highlight fragment algorithm to obtain the analysis result of the image frame of the first position.

[0244] S28, the HAL layer's highlight fragment algorithm interface sequentially returns the analysis results of the first image frame to the strategy monitoring module through the FWK layer's service interface and the media middleware framework layer's channel interface.

[0245] The analysis results can represent the aesthetic score of the image frame at the first position.

[0246] S29, the policy monitoring module obtains the second position and returns to execute S24 until the number of analyzed image frames meets the first number allocated to the video.

[0247] The first number can be determined based on the actual duration of each video and the remaining analysis time. For example, based on the single-frame analysis time, the total number of image frames in all videos that can be analyzed within the remaining analysis time can be calculated. The videos are then sorted from longest to shortest duration, and the total number of image frames is allocated to each video until the total number is exhausted, resulting in the first number allocated to each video. This first number of image frames can be evenly distributed across the various locations within the video.

[0248] Update the remaining analysis time after analyzing each image frame of a video.

[0249] S30, the strategy monitoring module obtains the analysis results of the first number of image frames of all videos.

[0250] After obtaining the analysis results of the first number of image frames from all videos, the strategy monitoring module can determine the target region for each video based on the analysis results of the image frames in each video. Then, it performs frame-by-frame analysis on the target region of each video to obtain the first highlight segment of each video.

[0251] S31, the strategy monitoring module determines the target area for each video based on the analysis results of the first number of image frames of all videos.

[0252] In this embodiment, for each video, after obtaining the analysis results of a first number of image frames, the strategy monitoring module determines the image frame with the highest score based on the scores of each image frame. The image frame with the highest score may include at least one image frame. If the image frame with the highest score includes only one image frame, the image frames before and after that image frame within a preset duration of 1 can be directly determined as the target area of ​​video 1. For example, such as... Figure 9 (a) provides a schematic diagram of a target region. The highest-scoring image frames in video 1 include image frame 1 (e.g., ...). Figure 9 In (a) 1), the segment covered by the preset duration 1 before and after image frame 1 is the target area.

[0253] Alternatively, if the highest-scoring image frame comprises only one image frame, based on the suggested duration of the highlight segment, segments of half the suggested duration of the highlight segment before and after the highest-scoring image frame can be identified as candidate highlight segments. Image frames within a preset duration of 2 before and after the candidate highlight segments are identified as the target region of video 1. The duration of the target region can be b times the length of the candidate highlight segment. For example, b can be a number greater than 1 and less than 2.

[0254] For example, such as Figure 9 (b) provides a schematic diagram of another target region. Among them, the highest-scoring image frames in video 1 include image frame 1 (e.g., ...). Figure 9 In (b) 1), the segments with a suggested duration of 1 / 2 of the highlight segments before and after image frame 1 are candidate highlight segments, and the segments covered by the preset duration 2 before and after the candidate highlight segments are the target areas.

[0255] If the highest-scoring image frame includes multiple consecutive image frames, the segment covered by the preset duration of 3 before the first image frame of the multiple consecutive first image frames and the preset duration of 3 after the last image frame of the multiple consecutive second image frames is determined as a candidate highlight segment, based on the suggested duration of the highlight segment.

[0256] For example, such as Figure 10 , Figure 10 A schematic diagram of another target region is given. The highest-scoring image frames in video 1 include consecutive image frames 1 (e.g., ...). Figure 10 1) Image frame 2 (e.g.) Figure 10 2) Image frame 3 (e.g.) Figure 10 (3) Therefore, the segment covered by the preset duration 3 before image frame 1 and the preset duration 3 after image frame 3 is the candidate highlight segment of video 1. The segments covered by the preset duration 4 before and after the candidate highlight segment are the target area.

[0257] After determining the candidate highlight segments for each video, if the sum of the durations of the candidate highlight segments for all videos is greater than Y times the total recommended duration of the highlight segments, the strategy monitoring module needs to adjust the duration of the candidate highlight segments for each video so that the sum of the durations of the candidate highlight segments for all videos is less than or equal to the second multiple of the total recommended duration of the highlight segments.

[0258] Where Y times can be a number greater than 1, for example, Y times can be 1.05. For instance, the duration of candidate highlight segments can be reduced based on the length by which the duration of all candidate highlight segments exceeds Y times the total suggested duration of highlight segments (the duration to be adjusted), according to the actual duration of each video. For instance, the shorter the video, the more seconds its candidate highlight segments will be reduced.

[0259] Assume we have two input videos: Video 1 has an actual duration of 10 seconds, and Video 2 has an actual duration of 20 seconds. The suggested duration for highlight clips is 6 seconds, and the total suggested duration for highlight clips is 10 seconds. The duration of candidate highlight clips for both Video 1 and Video 2 is 6 seconds; the duration of target regions for both Video 1 and Video 2 is 12 seconds.

[0260] The combined duration of the candidate highlight segments in Videos 1 and 2 is 12 seconds, exceeding the recommended total duration of highlight segments (10 seconds) by 1.05 times (10.5 seconds). The strategy monitoring module needs to adjust the duration of the candidate highlight segments for each video. The required adjustment is 12 seconds - 10 seconds * 1.05 = 1.5 seconds. The strategy monitoring module uses a weighted allocation based on the reciprocal of the video duration, reducing the duration of the candidate highlight segment in Video 1 (10 seconds) by 1 second; updating the duration of the candidate highlight segment in Video 1 from 6 seconds to 5 seconds; reducing the duration of the candidate highlight segment in Video 2 (20 seconds) by 0.5 seconds; and updating the duration of the candidate highlight segment in Video 2 from 6 seconds to 5.5 seconds. Therefore, the combined duration of the candidate highlight segments in Videos 1 and 2, 10.5 seconds, does not exceed 10 seconds * 1.05, and no further adjustment is needed.

[0261] Therefore, the duration of the candidate highlight segment in video 1 is determined to be 5s, and the duration of the target area corresponding to the candidate highlight segment in video 1 is 10s; the duration of the candidate highlight segment in video 2 is 5.5s, and the duration of the target area corresponding to the candidate highlight segment in video 2 is 11s.

[0262] The target region for each video can be determined using the method provided in S31. After determining the target region for each video, S31-S36 can be executed to analyze the image frames of the target region.

[0263] After identifying the target region for each video, the second analysis phase begins:

[0264] The strategy monitoring module in the media middle platform framework layer performs frame-by-frame analysis of the target area. Taking video 1 as an example, the process of performing frame-by-frame analysis of the target area of ​​video 1 may include:

[0265] S32, the strategy monitoring module sends the file descriptor fd1 of video 1 and the third position of the image frame of the target area of ​​video 1 to the channel interface of the media middle platform framework layer.

[0266] The position of the third image frame can be any image frame in the target area of ​​video 1.

[0267] S33, the channel interface performs decoding, resolution reduction, and format conversion on video 1 according to the file descriptor of video 1, and stores the processed video 1.

[0268] S34, the channel interface sends the frame data address of video 1 and the third position of the image frame to the highlight segment algorithm interface of the HAL layer through the service interface of the FWK layer (such as the one-click video creation interface).

[0269] S35, the HAL layer's highlight fragment algorithm interface obtains the third image frame of video 1 based on the frame data address and third position of video 1, and then analyzes the image frame at the third position based on the preset highlight fragment algorithm to obtain the analysis result of the image frame at the third position.

[0270] The S36 and HAL layer's highlight fragment algorithm interface sequentially returns the analysis results of the image frame at the third position to the strategy monitoring module of the media platform framework layer through the service interface of the FWK layer and the channel interface of the media platform framework layer.

[0271] The strategy monitoring module of the media middle platform framework layer continues to send the fourth position of the target area of ​​video 1 to the channel interface of the media middle platform framework layer for analysis, until the analysis of the second number of image frames in the target area is completed.

[0272] The fourth position can be the position of the next adjacent image frame to the third position in the target area of ​​video 1. The second number of image frames can be evenly distributed across the target area.

[0273] S37, the strategy monitoring module determines the highlight segment of video 1 from the analysis results of the image frames of the target area of ​​video 1.

[0274] In this embodiment, the strategy monitoring module obtains the highlight segment of video 1 based on the suggested duration of the highlight segment of video 1 and the image frame with the highest score in the target area.

[0275] In some embodiments, the policy monitoring module obtains the image frame with the highest score based on the image frame's score. The image frame with the highest score may include at least one image frame. If the image frame with the highest score includes only one image frame, the image frames before and after the image frame with the highest score within a first preset time period can be directly identified as the highlight segments of Video 1.

[0276] For example, such as Figure 11 (a) provides a schematic diagram of a highlight segment. The highest-scoring image frames in video 1 include image frame 1 (e.g., ...). Figure 11 In (a) 1), the segments covered by a preset duration of 5 before and after image frame 1 are the highlight segments of video 1. The duration of the highlight segment is less than or equal to the suggested duration of the highlight segment.

[0277] Alternatively, if the highest-scoring image frame comprises only one image frame, based on the suggested duration of the highlight segment, the segment preceding and following the highest-scoring image frame, which is half the suggested duration of the highlight segment, can be determined as the highlight segment of Video 1. The duration of the highlight segment is equal to the suggested duration of the highlight segment.

[0278] For example, such as Figure 11 (b) provides a schematic diagram of another highlight segment. The highest-scoring image frames in video 1 include image frame 1 (e.g., ...). Figure 11 In (b) 1), the recommended duration of the 1 / 2 highlight segment before and after image frame 1 is the highlight segment of video 1.

[0279] If the highest-scoring image frame includes multiple consecutive image frames, the segment covered by the preset duration of 6 before the first image frame of the multiple consecutive highest-scoring image frames and the preset duration of 6 after the last image frame of the multiple consecutive second image frames is determined as the highlight segment of Video 1, based on the suggested duration of the highlight segment.

[0280] For example, such as Figure 12 , Figure 12 Another schematic diagram of a highlight segment is given. The highest-scoring image frames in video 1 include consecutive image frames 1 (such as...). Figure 12 1) Image frame 2 (e.g.) Figure 12 2) Image frame 3 (e.g.) Figure 12 (3) Therefore, the segment covered by the preset duration 6 before image frame 1 and the preset duration 6 after image frame 3 is the highlight segment of video 1.

[0281] In some embodiments, the policy monitoring module updates the remaining analysis time after each video is analyzed.

[0282] S38, the strategy monitoring module obtains the analysis results of the highlight segments of all videos.

[0283] After the above steps, a highlight segment can be obtained for each video. To avoid missing other highlight segments in the video footage, the policy monitoring module can further verify the duration of highlight segments in all videos. If the sum of the durations of all highlight segments does not meet the total recommended duration of highlight segments, additional highlight segments are selected until the output conditions for highlight segments are met. These conditions include: the sum of the durations of all highlight segments is equal to or greater than the recommended total duration of highlight segments; or, there are no image frames in the video that can be selected as additional highlight segments.

[0284] refer to Figure 13 In this embodiment Figure 13 A flowchart illustrating the highlight clip selection process in a video processing method is provided. After executing S38, the electronic device performs the highlight clip selection operation:

[0285] S39, the strategy monitoring module calculates the first duration of the highlight segments of all videos.

[0286] The first duration refers to the sum of the durations of a highlight segment in each video from the image materials obtained after S38. A highlight segment in each video obtained in S38 is determined by the first image frame with a score greater than or equal to the first threshold. That is, the highlight segments in S38 are determined by the first image frame with a score of "high".

[0287] S40, if the first duration does not reach the total recommended duration of the highlight segment, the strategy monitoring module selects the highlight segment from the second image frame of all videos.

[0288] The second image frame is an image frame with a score greater than or equal to the second threshold that is not covered by the selected highlight fragments. That is, the highlight fragments selected from the second image frame are fragments selected from the fragments corresponding to the image frames with scores of "high" and "relatively high".

[0289] During the selection of highlight segments from the second image frame, the strategy monitoring module can directly filter the second image frame based on the analysis results of the analyzed image frames obtained in S30 for each video. During the highlight segment selection process, the strategy monitoring module no longer performs the image frame analysis. It can be understood that if S30 obtains the analysis results of the keyframes of each video, then the highlight segment selection is based on the keyframes; if S30 obtains the analysis results of the ordinary image frames of each video, then the highlight segment selection is based on the ordinary image frames.

[0290] During the highlight segment selection process, to avoid the selected segments overlapping with the target area or a highlight segment in the video, resulting in invalid selection, image frames covered by highlight segments in each video are not included in the selection operation. That is, the policy monitoring module performs highlight segment selection from the second image frame of the non-highlight segments in each video.

[0291] In some embodiments, the policy monitoring module can sort the image frames of non-highlight segments of all videos according to their scores from highest to lowest, and process each second image frame sequentially from a score of "high" to a score of "relatively high".

[0292] When processing each second image frame, the corresponding highlight segment in the video is obtained. The highlight segment corresponding to the second image frame includes the second image frame itself, as well as image frames within a second preset time period before and after the second image frame.

[0293] In some embodiments, the conditions for determining the highlight segment of the second image frame include that the start and end positions of the highlight segment of the second image frame do not exceed the start and end positions of the video itself, and that the highlight segment does not contain segments (or parts of segments) of the target region, segments (or parts of segments) corresponding to image frames with scores less than the third threshold, and segments (or parts of segments) that have been selected as highlight segments.

[0294] The distance between the start or end position of the highlight segment and the position of the second image frame should be greater than or equal to a preset distance, such as 500ms, 800ms, 1000ms, etc.

[0295] In this context, the segment corresponding to an image frame with a score less than the third threshold can be understood as the segment from 500ms before the image frame with a score less than the third threshold to 500ms after that image frame. The segment corresponding to an image frame with a score less than the third threshold has a score of 0, and can also be called a 0-score region or 0-score segment.

[0296] If the duration of a highlight fragment is less than or equal to the suggested duration of a highlight fragment, then the highlight fragment is added as a highlight fragment, and the processing of the current second image frame ends. Processing of the next second image frame continues according to the scoring and sorting of the second image frames.

[0297] If the duration of a highlight clip exceeds the recommended duration, the duration of the highlight clip needs to be reduced. The reduction step size can be determined based on the difference between the highlight clip's duration and its recommended duration. For example, the first reduction step size can be half of the difference. For instance, if the highlight clip's duration is 15 seconds and the recommended duration is 5 seconds, then the duration is reduced by 5 seconds ((15-5) / 2 seconds) from both the left and right sides of the second image frame towards the midpoint of the highlight clip.

[0298] If one side is reduced to a preset distance from the second image frame, for example, if the right side of the highlight fragment in the second image frame has been reduced to 500ms from the position of the second image frame, and if the duration of the highlight fragment is still greater than the recommended duration of the highlight fragment, then the left side of the highlight fragment is reduced, and the right side is no longer reduced, until the duration of the highlight fragment is reduced to less than or equal to the recommended duration of the highlight fragment.

[0299] For example, refer to Figure 14 , Figure 14 A schematic diagram of selecting highlight segments in video 1 is given. Assume there is only one video (Video 1) in the image material. Video 1 is 20 seconds long and includes a 4-second highlight segment 1 (from 1 second to 5 seconds) obtained through step S37. Video 1 includes image frame 1 with a score greater than a second threshold. Image frame 1 has a score of 88, a score result of "high," and is located at position 15 seconds in the video. The explanation uses an example where the suggested duration of the highlight segment is 5 seconds, and the distance between the start or end position of the effective segment and the position of the image frame should be greater than or equal to 500 ms.

[0300] The strategy monitoring module acquires image frame 1 and performs highlight segment replacement. The available segments in image frame 1 include segment 2 from video 1 (5s-20s). Segment 2 is 15s long, exceeding the recommended highlight segment length (5s). Therefore, segment 2 needs to be shortened.

[0301] For example, with a reduction step of 5 seconds, starting from the beginning and end position of segment 2, image frame 1 is reduced by 5 seconds on both the left and right sides. After the left side of image frame 1 is reduced by 5 seconds, the starting position of segment 2 is updated from the 5-second position of video 1 to the 10-second position; after the right side of image frame 1 is reduced by 5 seconds, the ending position of segment 2 overlaps with the position of image frame 1. According to the requirement that the distance between the ending position of an effective segment and the position of the image frame should be greater than or equal to 500 ms, the ending position of segment 2 is determined to be 500 ms to the right of image frame 1, that is, the 15.5-second position of video 1. At this time, the duration of segment 2 is 5.5 seconds (10 seconds - 15.5 seconds), which is still greater than the recommended duration of the highlight segment. Since the right side of image frame 1 cannot be reduced further, the left side of image frame 2 is reduced by 0.5 seconds, so that the duration of segment 2 is less than or equal to the recommended duration of the highlight segment.

[0302] Finally, highlight segments were selected based on image frame 1 in video 1, resulting in highlight segment 2 (10.5s-15.5s).

[0303] refer to Figure 15 , Figure 15 Another schematic diagram of supplementing highlight segments in Video 1 is provided. Combined with... Figure 14 In the provided example, video 1 also includes image frame 2, which has a score of 40, resulting in a "low" rating. Image frame 2 is located at the 7s mark in the video. The corresponding segment for image frame 2 is segment 3 (6.5s-7.5s), consisting of 500ms segments to its left and right. The usable segments for image frame 1 should not include the segments corresponding to image frames with scores below the third threshold (segment 3). Therefore, the usable segment for image frame 1 is segment 4 (7.5s-20s) in video 1. This segment 4 has a duration of 12.5s, exceeding the recommended duration for highlight segments. Therefore, segment 4 still needs to be shortened.

[0304] For example, taking a reduction step of 3 seconds, the starting and ending positions of segment 4 are each reduced by 3 seconds. After reducing the left side of image frame 1 by 3 seconds, the starting position of segment 4 is updated from the 7.5-second position of video 1 to the 10.5-second position; after reducing the right side of image frame 1 by 3 seconds, the ending position of segment 4 is updated from the ending position of video 1 (20-second position) to the 17-second position. At this time, the duration of segment 4 is 6.5 seconds (10.5 seconds - 17 seconds), which is still longer than the recommended duration of the highlight segment. Since the sides of image frame 2 can still be reduced, each side of image frame 2 is reduced by another 1 second. The starting position of segment 4 is updated from the 10.5-second position of video 1 to the 11.5-second position, and the ending position of segment 4 is updated from the 17-second position of video 1 to the 16-second position. At this time, the duration of segment 4 is 4.5 seconds (11.5 seconds - 16 seconds), which is shorter than the recommended duration of the highlight segment.

[0305] Finally, highlight segments were selected based on image frame 1 in video 1, resulting in highlight segment 4 (11.5 seconds - 16 seconds).

[0306] refer to Figure 16 , Figure 16 Another schematic diagram of supplementing highlight segments in Video 1 is provided. Combined with... Figure 14 In the provided example, video 1 also includes image frame 3, which has a score of 75, resulting in a "high" score. Image frame 3 is located at 17 seconds into the video. The highlight segment corresponding to image frame 1 is segment 2 (10.5 seconds - 15.5 seconds). Therefore, the usable segment for image frame 3 is segment 5 from 15.5 seconds to 20 seconds into video 1. This segment 5 has a duration of 4.5 seconds, which does not exceed the recommended duration for highlight segments. Therefore, segment 5 is determined as the highlight segment corresponding to image frame 3. Thus, when the image material includes video 1, and video 1 includes image frames 1 and 2 with scores greater than the second threshold, the highlight segments of video 1 are supplemented, resulting in highlight segments including segment 2 corresponding to image frame 1 and segment 5 corresponding to image frame 3.

[0307] In some embodiments, if the duration of the highlight segment of the final obtained image frame 1 is less than the minimum duration of a highlight segment, or if the duration of the highlight segment of image frame 1 is less than a preset duration threshold (e.g., 500ms), then image frame 1 is discarded, and the determination of the highlight segment of image frame 1 is no longer performed. In some embodiments, the discarded image frame 1 can be marked to avoid repeatedly judging the highlight segment of image frame 1 during subsequent highlight segment replacement, thus avoiding computational redundancy.

[0308] After each image frame is processed, the selected highlight fragments will no longer participate in subsequent selection processes, and the image frames covered by the highlight fragments will not participate in the subsequent selection process. The newly selected highlight fragments should not overlap with the previously selected highlight fragments.

[0309] When a new highlight segment is obtained through supplementary selection, the strategy monitoring module calculates and updates the sum of the durations of all highlight segments in all videos (the second duration) based on the newly obtained highlight segment.

[0310] When the total recommended duration of the highlight segment is reached in the second duration, the strategy monitoring module will no longer perform the highlight segment supplementation operation and can directly execute S43.

[0311] If the second duration does not reach the total recommended duration of the highlight segment, and all second image frames with a score greater than the second threshold in all videos of the image material have been processed, the policy monitoring module will proceed to the next step of judgment.

[0312] Among them, the cases in which the second image frame with a score greater than the second threshold has been processed include: the second image frame has been supplemented to a highlight segment or the segment corresponding to the second image frame is discarded because its duration is less than the minimum duration of the highlight segment, or the segment corresponding to the second image frame is covered by an existing highlight segment in the video, etc.

[0313] The strategy monitoring module of the media middle platform framework layer makes further judgments:

[0314] S41, if the second duration does not reach the total recommended duration of the highlight segment at the first magnification, the strategy monitoring module supplements the highlight segment from the third image frame of all videos.

[0315] The first magnification factor can be 0.7, 0.8, 0.9, or any factor greater than 0.6 and less than 1. The third image frame is the image frame with a score greater than or equal to the third threshold.

[0316] Assuming the total suggested duration of the highlight clip is T, and the first multiplier is 0.8, if the second duration is less than T*0.8, the strategy monitoring module will supplement the highlight clip from the third image frame of all videos whose score is greater than the third threshold.

[0317] After passing through S40, the video footage of the image material may also include second image frames with scores greater than the second threshold, or it may not include second image frames with scores greater than the second threshold. The policy monitoring module acquires the third image frames in the video footage with scores greater than the third threshold, sorts them from highest to lowest score, and processes the third image frames with scores greater than the third threshold.

[0318] Similar to the method for obtaining highlight fragments from the second image frame, when processing each image frame:

[0319] Obtain the highlight segment corresponding to the third image frame in the video. The conditions for determining the highlight segment include that the start and end positions of the highlight segment do not exceed the start and end positions of the video itself, and that the highlight segment does not contain segments of the target area, segments corresponding to image frames with a score less than the third threshold, or segments that have already been selected as highlight segments.

[0320] The distance between the start or end position of the highlight segment and the position of the image frame should be greater than or equal to a preset distance, such as 500ms, 800ms, 1000ms, etc.

[0321] The segment corresponding to the image frame with a score less than the third threshold can be understood as the segment from 500ms before the image frame with a score less than the third threshold to 500ms after the image frame.

[0322] If the duration of a highlight fragment is less than or equal to the suggested duration of a highlight fragment, then the highlight fragment is added as a highlight fragment, and the processing of the current third image frame ends. Processing of the next third image frame continues according to the scoring and sorting of the third image frames.

[0323] If the duration of a highlight segment exceeds the recommended duration, the duration of that highlight segment needs to be reduced. The reduction method can refer to the reduction method involved in the embodiment of supplementing highlight segments from the second image frame, and will not be elaborated here. In some embodiments, if the duration of the highlight segment of the final obtained image frame 1 is less than the minimum duration of a highlight segment, or if the duration of the highlight segment of image frame 1 is less than a preset duration threshold (e.g., 500ms), then image frame 1 is discarded, and the determination of the highlight segment of image frame 1 is no longer performed. In some embodiments, the discarded image frame 1 can be marked to avoid repeatedly judging the highlight segment of image frame 1 during subsequent highlight segment supplementation, thus avoiding computational redundancy.

[0324] Similarly, after each third image frame is processed, the selected highlight fragments will no longer participate in the subsequent selection process, and the third image frames covered by the highlight fragments will not participate in the subsequent selection judgment. The newly selected highlight fragments should not overlap with the previously selected highlight fragments.

[0325] When a new highlight segment is obtained through supplementary selection, the strategy monitoring module calculates and updates the sum of the durations of all highlight segments in the videos (the third duration) based on the newly obtained highlight segment.

[0326] When the total recommended duration of the highlight segment is reached in the third duration, the strategy monitoring module will no longer perform the highlight segment supplementation operation and can directly execute S43.

[0327] If the third duration does not reach the total recommended duration of the highlight segment, and all third image frames in the video of the image material with a score greater than the third threshold have been processed, the policy monitoring module will proceed to the next step of judgment.

[0328] Among them, the cases where the third image frame with a score greater than the third threshold has been processed include: the third image frame has been supplemented to the third highlight segment or the segment corresponding to the third image frame has been discarded because its duration is less than the minimum duration of the highlight segment; or the segment corresponding to the image frame has been covered by the highlight segment of the video, etc.

[0329] The strategy monitoring module of the media middle platform framework layer makes further judgments:

[0330] S42, if the third duration does not reach the total recommended duration of the highlight clip at the second magnification, the strategy monitoring module extends the duration of the highlight clip until there are no expandable highlight clips or the sum of the durations of the highlight clips reaches the total recommended duration of the highlight clip at the second magnification.

[0331] The second magnification factor is less than the first magnification factor. For example, the second magnification factor can be a value greater than 0 and less than the first magnification factor, such as 0.4, 0.5, or 0.6. Assuming the total suggested duration of the highlight clip is T, and the second magnification factor is 0.6, if the third duration is less than T*0.6, the strategy monitoring module extends the duration of the highlight clip.

[0332] After steps S40 and S41, the selection of highlight segments from the second and third image frames in the video has been completed. There are no more second or third image frames available for selection in the video. However, the sum of the durations of all highlight segments in the video is still too short, not meeting the second multiple of the recommended total duration of highlight segments. To ensure the duration and quality of the output target video set, the duration of existing highlight segments can be extended to obtain highlight segments whose duration meets the second multiple of the recommended total duration of highlight segments.

[0333] Understandably, when expanding highlight segments, the region containing the highlight segment and the segment region corresponding to image frames with scores below the third threshold are not considered expandable regions. In other words, expanding existing highlight segments cannot overlap with other highlight segment regions or segment regions with low scores.

[0334] In some embodiments, if the duration of a highlight segment reaches the maximum duration of a highlight segment; or, there is no available expansion area on either side of the highlight segment; or, the distance from the edge on either side of the highlight segment is less than a preset duration threshold (e.g., 2 seconds), then these highlight segments will not be expanded. (See reference) Figure 17 , Figure 17 A schematic diagram of an expandable region 1 in a highlight segment 1 of video 1 is provided. Video 1 also includes a target region 2 of the first highlight segment 2 and a region 3 with a low score. The expandable region of highlight segment 1 does not overlap with or only partially overlaps with other regions. The boundary of the expandable region 1 is 2s from the adjacent edge.

[0335] In some embodiments, before extending the duration of highlight clips, the required extension duration for all videos can be calculated first. The required extension duration for all videos can be the difference between the total suggested duration T*0.6 of the highlight clips and the third duration.

[0336] In some embodiments, the strategy monitoring module can sort all highlight segments of the video footage according to their scores. The score of a highlight segment can be calculated as the average score of the image frames it covers. Based on the scores, different weight values ​​are assigned to highlight segments with different scores to allocate the required extension duration. For example, when sorting all highlight segments of the video footage according to their scores, the top 25% of highlight segments have a weight of 3, the middle 50% have a weight of 1.5, and the bottom 25% have a weight of 1.

[0337] In one example, suppose all videos include 5 highlight clips, the sum of the durations of the 5 highlight clips is 20 seconds, and the total suggested duration T of the highlight clips is 45 seconds. Then, after calculation, the required extended duration is 45 seconds * 60% - 20 seconds = 7 seconds. 7 seconds is the total extended duration required for all 5 videos.

[0338] Assuming that highlight segment 1 has reached its maximum duration, then highlight segment 1 will not undergo duration extension. The remaining four highlight segments will be extended.

[0339] Among them, reference Figure 18 , Figure 18 A schematic diagram illustrating the allocation of required extended duration is provided. Assume highlight segment 2 scores 90 points, highlight segment 3 scores 80 points, highlight segment 4 scores 70 points, and highlight segment 5 scores 60 points. Sorted from highest to lowest score, the top 25% of the four highlight segments include highlight segment 2, assigned a weight of 3; the middle 50% include highlight segments 3 and 4, assigned a weight of 1.5; and the bottom 25% include highlight segment 5, assigned a weight of 1.

[0340] Based on the weights assigned to the highlight segments and the required extension duration of 7s, the extension quota duration for highlight segment 2 is calculated to be 3s, the extension quota duration for highlight segment 3 is 1.5s, the extension quota duration for highlight segment 4 is 1.5s, and the extension quota duration for highlight segment 5 is 1s.

[0341] In one example, refer to Figure 19 , Figure 19This provides another diagram illustrating the allocation of required extension duration. Assume all videos contain 12 highlight segments, all of which can be extended. Sorted from highest to lowest score, the top 25% of the 12 highlight segments include 3 highlight segments, each with a weight of 3; the middle 50% include 6 highlight segments, each with a weight of 1.5; and the bottom 25% include 3 highlight segments, each with a weight of 1.

[0342] It is understandable that the allocation ratio after sorting, and the specific values ​​of the weights corresponding to the highlight segments in different ratios, can be determined according to the actual situation. The weight allocation principle is that the higher the score of the highlight segment and the earlier the highlight segment is ranked, the greater the weight value corresponding to the highlight segment.

[0343] In some embodiments, after calculating the extended quota duration of each highlight segment, the extended quota duration of the highlight segment can be further verified to ensure that the duration of the extended highlight segment does not exceed the maximum duration of the highlight segment in the analysis parameters.

[0344] Specifically, when performing the expansion operation for each highlight segment, the fourth duration of the highlight segment's duration and the allocated expansion quota duration is obtained. If the fourth duration is less than the maximum duration of the highlight segment, the difference between the fourth duration and the highlight segment's duration is taken as the expansion quota duration of the highlight segment; if the fourth duration is greater than the maximum duration of the highlight segment, the difference between the maximum duration and the highlight segment's duration is taken as the expansion quota duration of the highlight segment.

[0345] When expanding each highlight segment, the expansion quota time is halved on both the left and right sides of the highlight segment. If one side expands to a preset distance (e.g., 1s, 2s) from the edge of the expandable region, expansion on that side stops; if there is still remaining expansion time, expansion begins from the other side. If neither side can be expanded further, or if the expansion quota time on both sides has been reached, the expansion operation on that highlight segment stops. "Neither side can be expanded further" can include the already expanded boundaries on both sides reaching a preset distance from the edge of the expandable region.

[0346] In one example, reference Figure 20 , Figure 20 A schematic diagram of expanding highlight segment 2 is provided. Assuming that the duration of video 1 is 25s, video 1 includes highlight segment 1 (1s-7s) and highlight segment 2 (13s-16s), and video 1 also includes a region 0 (19s-20s) with a low score.

[0347] Assume the maximum duration of a highlight clip is 6 seconds. Highlight clip 1 has already reached its maximum duration, therefore, no extension operation is performed on highlight clip 1. Highlight clip 2 has a duration of 3 seconds, and the allocated extension quota for highlight clip 2 is 4 seconds. The fourth duration corresponding to highlight clip 2 is greater than the maximum duration of the highlight clip. Therefore, the difference between the maximum duration of the highlight clip (6 seconds) and the duration of highlight clip 2 (3 seconds) is used as the extension quota duration (3 seconds).

[0348] When expanding highlight clip 2, both the left and right sides of highlight clip 2 are expanded by 1.5s. The left side can be expanded by 1.5s to 11.5s, while the right side can only be expanded by 1s to 17s. The distance from the edge of the expandable area (the boundary of area 0) has reached the preset distance (2s). At this point, the expanded duration is 2.5s, which has not reached the expansion quota duration, with 0.5s remaining.

[0349] The highlight fragment 2 was further extended. The right side of highlight fragment 2 could no longer be extended, but the left side could be extended by 0.5s to 11s. The actual extension time reached the extension quota time of 3s, and the extension operation of highlight fragment 2 ended.

[0350] Highlight fragments are expanded one by one according to their scores, from highest to lowest. After each highlight fragment is expanded, the remaining expandable highlight fragments and the remaining expansion quota duration in the image material are updated. Then, based on the updated remaining expandable highlight fragments and the remaining expansion quota duration, the expansion quota duration is reallocated, and a new round of highlight fragment expansion processing is performed.

[0351] The process of extending highlight segments continues until there is no remaining extension quota time or no more extendable highlight segments, at which point the extension operation of highlight segments is stopped, resulting in the final result of all highlight segments in the image material.

[0352] After obtaining the final result of all highlight segments in the image material, you can perform the following steps: Figure 17 The provided video processing method includes:

[0353] S43, the strategy monitoring module of the media middle platform framework layer obtains the analysis results of all videos based on the highlight segments of each video after the extended duration.

[0354] The analysis results for all videos can include the locations of highlight segments in all videos. The analysis results for all images and videos can be reported using the following S45 method.

[0355] S44, the strategy monitoring module of the media middle platform framework layer reports the analysis results of all images and videos to the application function layer through the image highlight segment analysis interface of the media middle platform framework layer.

[0356] S45, the application function layer, based on the analysis results of all images and videos, edits and filters the user-selected materials to obtain all highlight segments.

[0357] S46, the application layer's application function layer calls the media middle platform framework's theme summary interface to request and obtain the theme template.

[0358] The request is transmitted to the HAL layer through the FKW layer via the theme summary interface and channel interface of the media middle platform framework.

[0359] S47, the HAL layer determines a theme template that matches the scene based on the scene in the highlight clip.

[0360] In some embodiments, electronic devices may be configured with multiple theme templates (style templates). The theme algorithm may recommend theme templates that match the scene based on highlight clips, such as people, landscapes, food, children, pets, sports, or travel.

[0361] S48, the HAL layer returns the determined theme template to the theme summary interface of the media middle platform framework layer, and the theme summary interface returns the theme template to the application function layer of the application layer.

[0362] For example, assuming that the highlights are mostly scenes of parents and children, then the theme template that matches the scene can be identified as the parent-child theme class.

[0363] S49, the application function layer sends the obtained subject and all highlight fragments to the basic capability layer.

[0364] The application functionality layer will send the theme obtained from S49 and all highlight fragments obtained from S45 to the basic capability layer.

[0365] S50, the basic capability layer, generates a target video set based on the theme and all highlight clips.

[0366] That is, the target video set is a set of videos that conforms to the recommended theme, generated from all the selected highlight clips.

[0367] S51, the basic capability layer sends an instruction message to the video editing business layer to display the target video set.

[0368] S52, the video editing business layer displays the target video set in the gallery interface.

[0369] In this embodiment, the media platform framework layer is used for tasks such as decoding video and image files, converting data formats to a unified format, monitoring remaining time and adjusting computational strategies, sending data, controlling algorithm execution and termination, obtaining results, and returning them to the application layer. The FKW layer is used to package data and provide data and program execution services. After receiving commands from the media platform framework layer, the HAL layer performs highlight analysis according to the commands and returns the parameter calculation results of the highlight analysis to the media platform framework layer. The final algorithm results are collected and organized by the media platform framework layer before being sent to the application layer for processing. The application layer can present the editing application interface, video and image file options, and the final algorithm results.

[0370] After the user activates the one-click video creation function, they select the video and image files to be edited (for example, a maximum of 30 files are supported). After waiting for a moment, the "one-click video creation" application automatically edits the highlight segments of the video and combines the highlight segments and images together according to the algorithm results to generate the edited short video, which can then be previewed and played.

[0371] The video processing method provided in this application involves an electronic device analyzing highlight segments from multiple image materials selected by a user. After obtaining highlight segments from each video in the multiple image materials, if the sum of the durations of the highlight segments is less than the preset suggested total duration of highlight segments, the electronic device can supplement highlight segments from all videos until the sum of the durations of the highlight segments in all videos is equal to or greater than the suggested total duration of highlight segments. The highlight segments from each video in the multiple image materials are used to stitch together a target video set. This solution effectively avoids the problem of missing other highlight segments in the video, while ensuring the duration and quality of the target video set obtained by stitching together highlight segments.

[0372] It should also be noted that in the embodiments of this application, "greater than" can be replaced with "greater than or equal to", "less than or equal to" can be replaced with "less than", or "greater than or equal to" can be replaced with "greater than", and "less than" can be replaced with "less than or equal to".

[0373] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0374] It is understood that the methods and operations implemented by electronic devices in the above-described method embodiments can also be implemented by components (such as chips or circuits) that can be used in electronic devices.

[0375] It should be noted that the personal information used in the technical solution of this application is limited to information for which separate consent has been obtained, including but not limited to notifying and reminding users to read the relevant user agreement (notification) and sign the agreement (authorization) which includes authorization of relevant user information before users use the function. Personal information includes images, videos, and other information stored by users.

[0376] The technical solutions disclosed in this application involve the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, all of which comply with relevant laws and regulations and do not violate public order and good morals.

[0377] The method embodiments provided in this application have been described above. The apparatus embodiments provided in this application will be described below. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, any content not described in detail can be referred to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0378] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to achieve the above functions, the electronic device implementing this method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.

[0379] This application embodiment can divide an electronic device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other feasible division methods may exist. The following description uses the division of functional modules according to each function as an example.

[0380] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0381] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, causing the methods in the various embodiments to be performed.

[0382] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device in the above method embodiments.

[0383] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the electronic device in the above method embodiments. For example, the computer may be the aforementioned electronic device.

[0384] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0385] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0386] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0387] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0388] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of video processing, the method comprising: The method comprises: receiving a selection operation of a user on a plurality of image materials of a gallery; wherein the plurality of image materials comprises videos; in response to the selection operation, analyzing each video to obtain a highlight segment of each video; if a sum of time lengths of the highlight segments of all videos in the plurality of image materials is less than a recommended total time length of the highlight segments, supplementing the highlight segments from image frames of the all videos that are not covered by the highlight segments until an output condition of the highlight segments is met; wherein the output condition of the highlight segments comprises that the sum of time lengths of all highlight segments is equal to or greater than the recommended total time length of the highlight segments, or there is no image frame that can be supplemented in the videos; the highlight segments of each video in the plurality of image materials are used to splice to obtain a target video set; the recommended total time length of the highlight segments is a recommended value of the sum of time lengths of the highlight segments of all videos in the plurality of image materials, and the recommended total time length of the highlight segments is determined based on a maximum time length of the highlight segments of all videos in the plurality of image materials, a minimum time length of the highlight segments of all videos in the plurality of image materials, and a number of the all videos.

2. The method of claim 1, wherein, The analysis of each video to obtain a highlight segment of each video comprises: performing overview analysis on a preset number of image frames of each video to obtain a score of each image frame; the score is an aesthetic score of the corresponding image frame; determining the highlight segment of the corresponding video based on the scores of the image frames in each video; wherein the highlight segment comprises a first image frame and image frames within a first preset time length before and after the first image frame; the first image frame is the image frame with the highest score in the overview analysis.

3. The method according to claim 1 or 2, characterized in that, The supplementing of the highlight segments from the image frames of the all videos that are not covered by the highlight segments until the output condition of the highlight segments is met comprises: supplementing the highlight segments from second image frames of the all videos until the output condition of the highlight segments is met; wherein the score of the second image frame is greater than a first threshold value, and the second image frame is not covered by the selected highlight segment.

4. The method of claim 3, wherein, The supplementing of the highlight segments from the second image frames of the all videos until the output condition of the highlight segments is met comprises: obtaining the highlight segment corresponding to each second image frame in turn according to the order from high to low of the scores of the second image frames; the highlight segment corresponding to the second image frame comprises the second image frame and image frames within a second preset time length before and after the second image frame; if the sum of time lengths of the highlight segments of all videos after the supplement from the second image frames is greater than or equal to the recommended total time length of the highlight segments, ending the supplement operation of the highlight segments; if the sum of time lengths of the highlight segments of all videos after the supplement from the second image frames is less than the recommended total time length of the highlight segments, and if the sum of time lengths of the highlight segments is greater than or equal to a first multiple of the recommended total time length of the highlight segments, there is no second image frame that can be supplemented in the videos to perform the supplement operation of the highlight segments, and the supplement operation of the highlight segments is ended. If the sum of the time lengths of the high-light segments of all the videos after the supplement from the second image frame is less than the recommended total time length of the high-light segments at the first multiple, supplement high-light segments from third image frames of all the videos until the output condition of the high-light segments is met; Wherein, the third image frame is an image frame whose score is greater than or equal to a second threshold value, and the image frame is not covered by the high-light segment; the second threshold value is less than the first threshold value.

5. The method of claim 4, wherein, The method further comprises: If the time length of the high-light segment corresponding to the second image frame is greater than the preset maximum time length of the high-light segment, respectively reduce a preset distance from the left and right boundary positions of the high-light segment to the midpoint position of the high-light segment until the time length of the reduced high-light segment is equal to or less than the maximum time length of the high-light segment; Wherein, the preset maximum time length of the high-light segment is the maximum time length allowed for a high-light segment to ensure the complete highlight effect of the video.

6. The method according to claim 4 or 5, characterized in that, The method further comprises: If the time length of the high-light segment corresponding to the second image frame is less than the preset minimum time length of the high-light segment, discard the second image frame; Wherein, the preset minimum time length of the high-light segment is the minimum time length required for a high-light segment to ensure the complete highlight effect of the video.

7. The method of claim 4, wherein, The method further comprises: In the order of the scores of each third image frame from high to low, sequentially obtain the high-light segment corresponding to each third image frame; the high-light segment corresponding to the third image frame includes the third image frame and the image frames within a third preset time length before and after the third image frame; If the sum of the time lengths of the high-light segments of all the videos after the supplement from the third image frame is greater than or equal to the recommended total time length of the high-light segments, end the supplement operation of the high-light segments; If the sum of the time lengths of the high-light segments of all the videos after the supplement from the third image frame is less than the recommended total time length of the high-light segments, and if the sum of the time lengths of the high-light segments is greater than or equal to the recommended total time length of the high-light segments at a second multiple, there is no third image frame available for the supplement operation of the high-light segments in the video, end the supplement operation of the high-light segments; If the sum of the time lengths of the high-light segments of all the videos after the supplement from the third image frame is less than the recommended total time length of the high-light segments at the second multiple, expand the time length of the high-light segment until the output condition of the high-light segment is met.

8. The method of claim 7, wherein, The method further comprises: If the time length of the high-light segment corresponding to the third image frame is greater than the preset maximum time length of the high-light segment, respectively reduce a second preset time length from the left and right boundary positions of the high-light segment to the midpoint position of the high-light segment until the time length of the reduced high-light segment is equal to or less than the maximum time length of the high-light segment; Wherein, the preset maximum time length of the high-light segment is the maximum time length allowed for a high-light segment to ensure the complete highlight effect of the video.

9. The method according to claim 7 or 8, characterized in that, The method further comprises: If the time length of the high-light segment corresponding to the third image frame is less than the preset minimum time length of the high-light segment, discard the third image frame; The preset minimum length of the highlight segment is a minimum length required by a highlight segment to ensure a complete highlight effect of the video.

10. The method of claim 7, wherein, The length of the highlight segment is extended, including: A first target highlight segment is selected from the highlight segment, and the left and right boundary positions of the first target highlight segment are moved to both sides respectively to obtain a second target highlight segment after the length is extended; the second ratio is less than the first ratio. The first target highlight segment is a highlight segment in the highlight segment, and the length of the first target highlight segment is less than or equal to a preset maximum length of the highlight segment, and there is an expandable area before and after the target highlight segment, and the distance between the left and right boundary positions of the target highlight segment and the boundary positions of the adjacent segments is greater than a preset distance. The expandable area is a connected area in the video except the highlight segment and the area with a score less than a third threshold, and the connected area is an area with the same score and time continuity, and there is no overlapping area between the connected areas. The preset maximum length of the highlight segment is a maximum length allowed by a highlight segment to ensure a complete highlight effect of the video.

11. The method of claim 10, wherein, The left and right boundary positions of the first target highlight segment are moved to both sides respectively to obtain a second target highlight segment after the length is extended, including: The difference between the recommended total length of the highlight segment with the second ratio and the sum of the lengths of all highlight segments of the video selected from the third image frame is taken as the required expansion length of the highlight segment; According to the required expansion length of the highlight segment, an expansion quota length is allocated to each first target highlight segment; the expansion quota length is the length allowed to be expanded in the corresponding expandable area of the first target highlight segment. The left and right boundary positions of the first target highlight segment are moved to both sides respectively by 1 / 2 of the expansion quota length in the expandable area of the target highlight segment to obtain the second target highlight segment.

12. The method of claim 11, wherein, According to the required expansion length of the highlight segment and the length of each first target highlight segment, the expansion quota length of each target highlight segment is determined; If the sum of the lengths of all highlight segments and the cumulative sum of the expansion quota lengths are greater than a preset maximum total length of the highlight segment, the value of the expansion quota length is updated to the difference between the maximum total length of the highlight segment and the sum of the lengths of all highlight segments; If the sum of the lengths of all highlight segments and the cumulative sum of the expansion quota lengths are less than or equal to the preset maximum total length of the highlight segment, the value of the expansion quota length is unchanged. The preset maximum total length of the highlight segment is a maximum length allowed by all highlight segments of the video. The method further includes:

13. The method of claim 7, wherein, ​ After the duration of the high-light segment is extended, if the sum of the durations of all high-light segments is greater than or equal to the recommended total duration of the high-light segment at the second ratio, there is no third image frame in the video that can be added to the high-light segment, and the duration of the high-light segment in the video cannot be continuously extended, the adding operation of the high-light segment is ended.

14. An electronic device, comprising: The electronic device includes a display screen, a memory, and one or more processors; the display screen, the memory, and the processors are coupled; the memory stores computer program code, including computer instructions, which, when executed by the processors, causes the electronic device to perform the method of any one of claims 1-13.

15. A computer-readable storage medium, characterized in that, Computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-13.

16. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the method of any one of claims 1-13.

Images