Video processing method, electronic device and storage medium

Abstract

The present application provides a video processing method, electronic device, and storage medium, which relate to the field of video data processing technology. In a scenario where multiple video material files of different lengths are selected for one-click filming, an expected total analysis time is preliminarily determined based on the video decoding speed and the chip analysis speed. Then, based on the expected total analysis time, the actual analysis time of each video material file is reasonably allocated according to the time allocation algorithm, thereby quickly completing the video analysis, extracting highlight clips or wonderful moments in the video, and using them for video clip editing and splicing, completing one-click filming, and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of video data technology, and in particular to a video processing method, electronic equipment, and storage medium. Background Art

[0002] Currently, camera functions are common services provided by electronic devices, allowing users to take photos or videos. If users need to process the captured photos or videos, such as stitching together multiple photos or combining photos and videos, they typically need to use electronic devices or third-party image processing software to perform the stitching.

[0003] For example, the one-click video editing service provided by electronic devices can automatically analyze multiple videos and / or images selected by the user through an algorithm, extracting highlights (i.e., exciting moments such as a character's smile or a championship moment) from the original video, and automatically generate a pre-edited video. However, if the user inputs multiple video files of varying lengths, the video analysis time will be too long, affecting the user experience. Therefore, it is necessary to properly allocate the analysis time for each video file. Summary of the Invention

[0004] The present application provides a video processing method and electronic device. In a scenario where multiple video material files with different video lengths are selected for one-click film generation, the method can reasonably allocate the actual analysis time of each video material file according to a time allocation algorithm, thereby quickly completing video analysis and extracting highlight clips or wonderful moments in the video, completing one-click film generation, and improving the user experience.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a video processing method, the method comprising:

[0007] In response to the user's first operation, loading and displaying videos and pictures in the gallery;

[0008] In response to a user selecting an image material in the gallery and confirming an operation to perform image stitching, obtaining a first parameter based on the image material; wherein the image material includes a video; the first parameter includes the total number of videos in the image material, the duration of each video, the sum of the durations of all videos, the analysis speed of each video, the sum of the analysis speeds of all videos, and the sum of the expected analysis durations of all videos;

[0009] Perform the following operations on each video in the image material: determine, based on the first parameter, an actual analysis time allocated to a first video; wherein the first video is any video in the image material;

[0010] After determining the actual analysis duration of all videos, performing the following operations on each video in the image material: using the actual analysis duration allocated to the first video, analyzing highlight segments in the first video;

[0011] After analyzing all videos, extracting all highlight segments from the image material, and splicing all the highlight segments to obtain a second video;

[0012] The actual analysis duration allocated to each video includes a first duration and a second duration, wherein the first duration is obtained by averaging a portion of the sum of the expected analysis durations of all videos, and the second duration is determined based on a ratio between the duration of each video in the image material and the sum of the durations of all videos, and a ratio between the analysis speed of each video in the image material and the sum of the analysis speeds of all videos;

[0013] The sum of the actual analysis durations allocated to all videos in the image material is equal to the sum of the expected analysis durations of all videos.

[0014] The video processing method provided in the embodiment of the present application preliminarily determines an expected total analysis time based on the video decoding speed and the chip analysis speed in a scenario where multiple video material files with different video lengths are selected for one-click film formation. Then, based on the expected total analysis time, the actual analysis time of each video material file is reasonably allocated according to the time allocation algorithm, thereby quickly completing the video analysis, extracting highlight clips or wonderful moments in the video for use in video clip editing and splicing, completing one-click film formation, and improving the user experience.

[0015] The first operation may be a user clicking on a "one-click to create a movie" option (or "one-click to create a movie"). The "one-click to create a movie" option is set in the gallery interface.

[0016] In some embodiments, determining the actual analysis time allocated to the first video based on the first parameter includes: calculating the actual analysis time allocated to the first video using the following equation: n =T n1 +T n2 ; Among them, T n1 Indicates the first duration, T n2 represents the second duration;

[0017] The first duration is calculated using the following equation: n1=T V ×δ / n; where T v represents the sum of the expected analysis durations of all videos in the image material, δ represents the proportion coefficient, n represents the total number of all videos in the image material, and δ is greater than 0 and less than 1;

[0018] The second duration is calculated using the following equation:

[0019] T n2 =T V ×(1-δ)×(W n / W A );

[0020] W n =log2(1.5+L n / L A ) / log2(1.5+S n / S A );

[0021] W A =W1+…+W n ;

[0022] Wherein, W1 represents the weight value of the first video in the image material, W n Indicates the weight value of the nth video in the image material, W A Represents the sum of the weight values ​​of all videos in the image material, L n / L A Indicates the length L of the nth video in the image material n The sum of the duration of all videos L A The ratio between n / S A Indicates the analysis speed S of the nth video in the image material n The sum of the analysis speeds of all videos S A The ratio between them.

[0023] In some embodiments, δ is 0.5. The value of δ can be determined according to actual use requirements and is not limited in this embodiment of the present application.

[0024] In some embodiments, before acquiring the first parameter according to the image material, the method further includes:

[0025] The analysis speed of the first video is determined according to the video decoding speed and the chip analysis speed of the first video using the following equation: S n =min(V1, V2)×α; where S nrepresents the analysis speed of the first video, V1 represents the video decoding speed of the first video, V2 represents the chip analysis speed, α represents the tolerance factor, min() is used to obtain the minimum value, and α is greater than 0 and less than 1.

[0026] In some embodiments, α is set to 0.9. The value of α can be determined according to actual use requirements and is not limited in this embodiment of the application.

[0027] In some embodiments, before determining the analysis speed of the first video based on the video decoding speed and the chip analysis speed of the first video, the method further includes:

[0028] The following equation is used to obtain the video decoding speed of the first video according to the resolution and frame rate of the first video: V1 = V d ×(R d / R A )×(F d / F A ); wherein V1 represents the video decoding speed of the first video, V d Indicates the preset video decoding speed, R d Indicates the preset video resolution, F d Indicates the preset video frame rate; R A represents the video resolution of the first video, F A Indicates the video frame rate of the first video.

[0029] In some embodiments, before using the actual analysis time allocated to the first video to analyze the highlight segments in the first video, the method also includes: determining that the strategy for analyzing the first video is a first analysis strategy, and the first analysis strategy is a full analysis strategy, an intensive key segment analysis strategy, a sparse key segment analysis strategy, or a simple analysis strategy based on I frames.

[0030] The step of using the actual analysis time allocated to the first video to analyze the highlight segments in the first video includes: using the actual analysis time allocated to the first video to analyze the highlight segments in the first video according to the first analysis strategy.

[0031] In some embodiments, after determining that the strategy for analyzing the first video is the first analysis strategy, the method further includes: performing a preset process on the first video to obtain a processed first video; wherein the preset process includes at least one of the following: decoding, format conversion, and resolution reduction. According to the first analysis strategy, using the actual analysis time allocated to the first video, analyzing the highlight segments in the first video includes:

[0032] According to the first analysis strategy, the highlight segments in the processed first video are analyzed using the actual analysis duration allocated to the first video.

[0033] In some embodiments, the analyzing of the highlight segment in the first video includes: comparing each frame of the first video with preset image features frame by frame; when multiple consecutive frames of images in the first video contain the preset image features, determining that the first video contains a first highlight segment, and determining the start time and end time of the first highlight segment.

[0034] In some embodiments, extracting all highlight segments from the image material includes: extracting the first highlight segment from the first video according to the start time and end time of the first highlight segment.

[0035] In some embodiments, the image material also includes pictures. In this case, the method further includes: performing the following operations on each picture in the image material: using a preset analysis time, analyzing the first picture, where the first picture is any one of the pictures in the image material.

[0036] In some embodiments, analyzing the first image includes: when the first image contains a preset image feature, determining that the first image is a highlight segment.

[0037] In some embodiments, extracting all highlight segments from the image material and splicing all the highlight segments to obtain a second video includes: when the image material includes videos and pictures, extracting all highlight segments from all videos and all pictures in the image material, splicing all the highlight segments to obtain the second video.

[0038] In some embodiments, after splicing all the highlight segments to obtain a second video, the method further includes: displaying the second video in the gallery.

[0039] In some embodiments, after displaying the second video in the gallery, the method further includes: in response to a second operation of the user, post-processing the second video, the post-processing including adding a theme, adding background music, re-editing, and / or video sharing.

[0040] In a second aspect, the present application provides a video processing device, comprising a unit for executing the method described in the first aspect. The device may be configured to execute the method described in the first aspect. For a description of the units in the device, please refer to the description of the first aspect above and will not be repeated here for the sake of brevity.

[0041] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0042] In a third aspect, the present application provides an electronic device, comprising a processor coupled to a memory, the memory being configured to store computer programs or instructions, and the processor being configured to execute the computer programs or instructions stored in the memory, thereby performing the method of the first aspect. For example, the processor is configured to execute the computer programs or instructions stored in the memory, thereby performing the method of the first aspect.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program (also referred to as instructions or code) for implementing the method in the first aspect. For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.

[0044] In a fifth aspect, the present application provides a chip comprising a processor. The processor is configured to read and execute a computer program stored in a memory to perform the method of the first aspect and any possible implementation thereof. Optionally, the chip further comprises a memory, the memory being connected to the processor via a circuit or wire.

[0045] In a sixth aspect, the present application provides a chip system comprising a processor. The processor is configured to read and execute a computer program stored in a memory to perform the method of the first aspect and any possible implementation thereof. Optionally, the chip system further comprises a memory, the memory being connected to the processor via a circuit or wire.

[0046] In a seventh aspect, the present application provides a computer program product, which includes a computer program (also referred to as instructions or codes), and when the computer program is executed by a computer, enables the computer to implement the method in the first aspect.

[0047] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the software architecture of an electronic device disclosed in an embodiment of the present application;

[0050] Figure 3This is an example of an application scenario of the video processing method disclosed in the embodiment of this application. Figure 1 ;

[0051] Figure 4 This is an example of an application scenario of the video processing method disclosed in the embodiment of this application. Figure 2 ;

[0052] Figure 5 This is an example of an application scenario of the video processing method disclosed in the embodiment of this application. Figure 3 ;

[0053] Figure 6 This is an example of an application scenario of the video processing method disclosed in the embodiment of this application. Figure 4 ;

[0054] Figure 7 A schematic diagram of a duration allocation algorithm in a video processing method disclosed in an embodiment of the present application;

[0055] Figure 8 The timing of a video processing method disclosed in the embodiment of this application Figure 1 ;

[0056] Figure 9 The timing of a video processing method disclosed in the embodiment of this application Figure 2 ;

[0057] Figure 10 A schematic diagram of a duration allocation algorithm in a video processing method disclosed in an embodiment of the present application;

[0058] Figure 11 A schematic diagram of an analysis strategy in a video processing method disclosed in an embodiment of the present application;

[0059] Figure 12 A schematic diagram of highlight analysis of the video processing method disclosed in an embodiment of the present application;

[0060] Figure 13 A schematic diagram of a video processing method disclosed in an embodiment of the present application;

[0061] Figure 14 This is a structural block diagram of a video processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0062] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0064] In the specification and claims herein, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. In the description of the embodiments of this application, unless otherwise specified, the meaning of "plurality" refers to two or more. For example, "multiple processing units" refers to two or more processing units, etc.; "multiple components" refers to two or more components, etc.

[0065] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0066] Currently, camera functions are common services provided by electronic devices, allowing users to take photos or videos. If users need to process the captured photos or videos, such as stitching together multiple photos or combining photos and videos, they typically need to use electronic devices or third-party image processing software to perform the stitching.

[0067] For example, consider the one-click video editing service offered by electronic devices. This service automatically analyzes multiple videos and / or images selected by the user through an algorithm, extracting highlights (i.e., exciting moments, such as a character's smile or a championship moment) from the source videos and automatically generating a single, edited video. However, if a user inputs multiple source files of varying lengths, the video analysis process can be lengthy, causing the user to wait for extended periods and impacting the user experience. Therefore, it is necessary to rationally allocate the actual analysis time for each source video file.

[0068] In response to the above problems, the embodiments of the present application provide a video processing method and electronic device. In the scenario of one-click filming of multiple video material files with different video lengths, it is possible to preliminarily determine an expected total analysis time (equal to the sum of the expected analysis times of all videos) based on the video decoding speed and the chip analysis speed. Then, based on the expected total analysis time, the actual analysis time of each video material file is reasonably allocated according to the time allocation algorithm provided by the present application, thereby quickly completing the video analysis. The embodiments of the present application improve the user experience by making improvements at the bottom layer of the electronic device system.

[0069] The video processing method provided in the embodiment of the present application is applied to electronic devices. Electronic devices include electronic devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The electronic devices can be mobile phones, smart TVs, wearable devices, tablet computers (Pad), computers with wireless transceiver functions, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control (industrial control), wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the electronic devices.

[0070] See also Figure 1, is a structural diagram of an electronic device (taking a mobile phone as an example) provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, and the like.

[0071] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0072] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. For example, the processor 110 is used to execute the ambient light detection method in the embodiment of the present application.

[0073] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0074] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0075] The external memory 120 generally refers to an external memory. In the embodiment of the present application, the external memory refers to a memory other than the memory of the electronic device and the cache of the processor, and the memory is generally a non-volatile memory.

[0076] Internal memory 121, also referred to as "memory," can be used to store computer-executable program code, including instructions. Internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (e.g., sound playback, image playback, etc.).

[0077] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0078] The electronic device 100 also includes various sensors that can convert various physical signals into electrical signals. For example, the pressure sensor 180A is used to sense and convert pressure signals into electrical signals. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall effect sensor. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc. The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. The bone conduction sensor 180M can obtain a vibration signal.

[0079] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0080] For example, in an embodiment of the present application, the touch sensor 180K can detect a user's click operation on an application icon, and pass the detected click operation to the application processor, determine that the click operation is used to start or run the application, and then execute the running operation of the application.

[0081] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0082] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0083] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0084] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0085] The above is a specific description of the embodiments of the present application using the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in the figure may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0086] The electronic device provided in the embodiments of the present application can be user equipment (UE), for example, it can be a mobile terminal (such as a user's mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), etc.

[0087] Furthermore, operating systems run on the above components, such as the iOS operating system developed by Apple, the Android open-source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on these operating systems.

[0088] The operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0089] Figure 2 A schematic diagram of the software structure of the electronic device provided in an embodiment of the present application.

[0090] like Figure 2As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom into: application (APP) layer, media middle platform framework layer, application framework (FWK) layer, and hardware abstraction layer (HAL).

[0091] The application layer, referred to as the application layer, can include a series of application packages, such as camera, gallery, calendar, map, and navigation. When these application packages are running, they can access the various service modules provided by the media platform framework layer and the application framework layer through the application programming interface (API) and execute corresponding intelligent services.

[0092] In some embodiments, the camera is used to capture photos, videos, slow-motion images, and panoramic images in response to user operations. After these images are captured by the camera, or after the user triggers the phone to take a screenshot, or after the user triggers the phone to record the screen, or after the electronic device downloads images from other devices, the electronic device can save these images in a gallery, so that the user can perform video editing operations on the images in the gallery, such as a one-click movie operation.

[0093] The embodiment of the present application divides the gallery into the following layers from top to bottom: business layer, application function layer and basic function layer.

[0094] The service layer offers a variety of services, including automatic multi-camera video generation, AI-generated music videos, one-click video generation, and highlight moments. These services are presented as controls in the gallery's user interface (UI). By operating a control, the user can trigger the camera to perform the corresponding video processing action. For example, after the user selects one or more source videos and clicks the one-click video generation control in the gallery, the gallery automatically analyzes and extracts the highlights from the source videos through an algorithm, and combines these highlights into a single, edited video.

[0095] The application function layer includes an automatic editing framework. Each business in the business layer can call the automatic editing framework to provide automatic editing services for pictures and videos. For example, the automatic editing framework may include functional modules such as segment selection, storyline organization, layout splicing, and special effects beautification. Segment selection is used to call the highlight segment analysis interface and the policy monitoring interface to extract highlight segments from the material video. Storyline organization is used to sequentially splice multiple material videos in the form of a storyline based on the content of the material video. Layout splicing is used to modulate the interface layout of the material video. Special effects beautification is used to adjust the beautification effect of the video, such as adjusting the brightness of the picture and beautifying the face of the character.

[0096] The basic function layer is used to perform basic function processing on the edited video clips after the automatic editing framework edits multiple material videos. Exemplarily, the basic function layer may include basic function modules such as video splicing, synthesis and preservation, video effect rendering, and audio effect processing. Among them, the video splicing user splices multiple extracted highlight clips. Synthesis and preservation are used to store the video set obtained after splicing. Video effect rendering is used to add video effects to the video set obtained after splicing, such as adding style filters and themes to the video. Audio effect processing is used to add sound effects to the video set obtained after splicing, such as adding background music.

[0097] The media middle platform framework layer is a software layer set between the application layer and the application framework. The media middle platform framework layer may include an analysis performance query interface, a highlight segment analysis interface, a policy monitoring interface, a pipeline interface, and a topic summary interface. Among them, the analysis performance query interface is used to calculate the total duration of all material videos based on the video analysis speed. The highlight segment analysis interface is used to call the policy monitoring interface to extract highlight segments. The policy monitoring interface is used to configure the expected analysis duration for each material video based on the total duration of all material videos, and dynamically set the analysis strategy for each material video based on parameters such as the expected analysis duration and the duration of each video. The pipeline interface is used to reduce the resolution of the video file according to the file description issued by the policy monitoring interface in accordance with the analysis strategy, and forward the data address of the video file after the resolution reduction to the hardware abstraction layer through the application framework, and then report the analysis results of the highlight segment returned by the hardware abstraction layer to the policy monitoring interface. The topic summary interface is used to obtain the topic corresponding to the content of the highlight segment.

[0098] The application framework layer, also referred to as the framework layer, supports the operation of various modules in the media platform framework layer. For example, the framework layer may include a one-click video creation interface, a parameter management interface, an image data transmission interface, a theme analysis interface, and a performance analysis interface.

[0099] The hardware abstraction layer (HAL) encapsulates the Linux kernel driver and provides an interface to the upper layer. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable across multiple platforms. For example, the HAL may include a highlight segment analysis algorithm (also known as a highlight segment algorithm), a face detection algorithm, a video acceleration algorithm, and an image super-resolution algorithm. The highlight segment analysis algorithm is an image processing algorithm provided by the image chip. This algorithm scores each image frame based on its color, texture characteristics, image quality, interpolation with previous and next frames, and edge change rate. The scoring result can be used to evaluate whether an image frame is a highlight segment. For example, if the score of an image frame is greater than or equal to 60, the image frame is a highlight segment.

[0100] It should be noted that Figure 2 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more layers than shown, such as a system library (FWK LIB) layer and a kernel layer. In addition, each layer may include more or fewer components than shown, which is not limited in this application.

[0101] The kernel uses file descriptors to access files. File descriptors are non-negative integers. When you open an existing file or create a new file, the kernel returns a file descriptor. Reading and writing files also require a file descriptor to specify the file to be read or written.

[0102] It is understandable that in order to implement the video processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0103] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0104] The following briefly summarizes the video processing method provided in the embodiment of the present application in combination with the above software architecture.

[0105] In an embodiment of the present application, the one-click film-making service is located in the video editing APP business layer and is a sub-business of video editing. Multiple materials to be analyzed are selected at the business layer and the materials to be analyzed are sent to the application function layer.

[0106] At the application layer, the one-click film-making service is implemented based on an automatic editing framework, which includes basic implementation functions such as clip selection, storyline organization, layout splicing, and / or special effects beautification. At the application layer, based on the transmitted material to be analyzed, parameters such as the expected total analysis duration, the minimum interval duration between highlight clips, and the maximum highlight clip duration (for example, 20 seconds) are generated and then sent to the media center framework layer.

[0107] The media middle platform framework layer is mainly responsible for basic data processing for the one-click film-making business, and mainly includes modules such as image highlight segment analysis, theme application, analysis performance query, and policy monitoring. The highlight segment analysis interface and policy monitoring module mainly control the analysis position and analysis duration of the actual highlight segment of the video material, and decode and analyze it according to the duration and position allocated by the policy algorithm. The content protected by this application is: the policy monitoring module will allocate the actual analysis time of each video based on the data sent down from the upper layer and the chip analysis speed (also called algorithm analysis speed) obtained from the HAL layer.

[0108] After calculating the actual analysis duration for each video, the Strategy Monitoring Module assigns different analysis strategies based on the duration. Four main selection strategies are available. When selecting a large number of videos or relatively long clips, performing algorithmic analysis on all videos can lead to resource constraints and excessive analysis time. In these cases, the Strategy Monitoring Module uses intensive and sparse analysis strategies to allocate and select analysis segments.

[0109] The media middle platform framework layer sends the processed video clip data to the HAL layer through the FWK layer, performs data processing at the HAL layer, and processes the data into data that can be recognized and calculated by the chip algorithm end. The data is then subjected to algorithmic analysis. The HAL layer then returns the analysis results to the media middle platform framework layer, and finally returns them to the application layer, which performs video splicing and presentation.

[0110] The execution subject of the video processing method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the video processing method, and the present application solution can be implemented by hardware and / or software. The specific implementation can be determined according to actual use requirements and is not limited by the embodiment of the present application. The video processing method provided in the embodiment of the present application is exemplarily described below with reference to the accompanying drawings using an electronic device as an example.

[0111] The following first introduces the video processing method provided by the embodiment of the present application in conjunction with specific application scenarios.

[0112] In an exemplary application scenario, a user uses a mobile phone to shoot multiple video materials in advance, and the mobile phone stores the multiple video materials in a gallery. Figure 3 As shown in (a) of FIG, a gallery icon is displayed on the mobile phone desktop. When the user wants the mobile phone to generate a video clip based on multiple video materials, the user can click the gallery icon on the mobile phone desktop. In response to the user's click operation on the icon, the mobile phone displays the gallery interface A1, as shown in FIG. Figure 3 As shown in (b) of FIG, the gallery interface A1 provides a “one-click blockbuster” option, such as Figure 3 As shown in (c) in FIG, the user can click on the “One-click blockbuster” option. In response to the user clicking on the “One-click blockbuster” option, the mobile phone displays the gallery interface A2, as shown in FIG. Figure 3 As shown in (d) in FIG, the gallery interface A2 may include multiple video materials that were recently shot, such as multiple video materials shot on November 10, 2022. The user can click on any one of the multiple video materials, such as Figure 3 As shown in (e) in the figure, the user clicks on the first video material among the multiple video materials. In response to the user's click operation on any video material, the mobile phone displays the gallery interface A3, which includes all the materials in the gallery. In this way, the user can select the video material required to generate the video clip in the gallery interface A3, for example, Figure 3 As shown in (f) in FIG, it is assumed that the user has selected 5 video materials. The gallery interface A3 also includes video generation options, such as Figure 3 As shown in (f), the video generation option is checked.

[0113] Refer again Figure 4 In (a), after selecting the video material, the user can click on the video generation option, such as clicking the check mark. Figure 4 As shown in (b) of the figure, in response to the user clicking the check mark, the phone begins analyzing the five video clips selected by the user in Gallery interface A3, selecting highlight clips (i.e., wonderful clips, such as clips of the user's smiling face or the moment of victory) from each video clip, and generates a video clip based on the selected highlight clips. During this process, the phone can display the progress of the analysis in Gallery interface A3 so that the user can intuitively view the analysis progress.

[0114] In one example, if Figure 5 As shown in (a) in FIG, after the video clip is generated, the mobile phone displays the gallery interface A4, which includes the generated video clip, and the mobile phone can automatically play the video clip. In addition, the gallery interface A4 can provide a video export option, such as Figure 5 As shown in (b), the user can click on the video export option. In response to the user's click operation on the video export option, the mobile phone exports the video clip, as shown in FIG. Figure 5 As shown in (c) in the figure, the video clip is stored in the gallery, so that the user can view the video clip from the gallery.

[0115] In another exemplary application scenario, a user uses a mobile phone to pre-shoot multiple video clips and multiple picture clips, and the mobile phone stores the multiple video clips and multiple picture clips in a gallery. In this way, the user can select the video clips and picture clips in the gallery, and the mobile phone generates a video clip based on the video clips and picture clips.

[0116] In one example, if Figure 6 As shown in (a) in the figure, after entering the gallery interface A2, the gallery interface A2 includes multiple video materials and multiple picture materials that have been recently shot. The user can click on any one of the multiple video materials and multiple picture materials, such as the first video material. In response to the user's click operation on any of the materials, the mobile phone displays the gallery interface A3, which includes all the materials in the gallery. In this way, the user can select the materials required to generate the video clip in the gallery interface A3, such as Figure 6 As shown in (b) of FIG, it is assumed that the user has selected 4 video materials and 2 picture materials. The gallery interface A3 also includes video generation options, such as Figure 6 As shown in (b) in the figure, the video generation option is in the form of a check mark. Figure 6 In (c), after selecting the material, the user can click on the video generation option, such as clicking on the check mark option. Figure 6 As shown in (d) in the figure, in response to the user clicking the check mark, the phone begins analyzing the four video clips and two image clips selected by the user in the gallery interface A3, selecting highlight clips from each video clip and highlight images (also known as wonderful images) from the two image clips. During this process, the phone can display the progress of the analysis in the gallery interface A3 so that the user can intuitively view the analysis progress.

[0117] Then, after the video clip is generated, the mobile phone displays a UI interface containing the video clip, in which the user is supported to preview the generated video clip. For example, in the scenario where the video material and the picture material are made into a film with one click, the UI interface for presenting the generated video clip can also refer to Figure 5 .like Figure 5 As shown in (a) in FIG, after the video clip is generated, the mobile phone displays the gallery interface A4, which includes the generated video clip, and the mobile phone can automatically play the video clip. In addition, the gallery interface A4 can provide a video export option, such as Figure 5 As shown in (b), the user can click on the video export option. In response to the user's click operation on the video export option, the mobile phone exports the video clip, as shown in FIG. Figure 5 As shown in (c) in the figure, the video clip is stored in the gallery, so that the user can view the video clip from the gallery.

[0118] In one example, after the video clip is generated, other function options may be provided in the application interface A4 displaying the video clip, so that the user can edit, add special effects, analyze, etc. the generated video clip based on these function options, for example Figure 5 As shown in (a) in the figure, the other function options may include but are not limited to templates, music, clips, sharing, etc. For example, the user can select function options according to needs to further process the generated video clips, such as selecting a theme template to apply to the generated video clips, adding music to the generated video clips, cutting out the generated video clips, or quickly sharing the generated video clips.

[0119] In one example, if the number of materials selected by the user is small, the mobile phone can prompt the user during the process of selecting materials so that the user can know how many materials are appropriate to select. Figure 3 As shown in (f) in the figure, a prompt message “more than 6 materials will produce better results” is displayed in the gallery interface A3, so that the user can know how many materials are needed to generate a better video clip.

[0120] In one example, if the number of materials selected by the user is large, the mobile phone can prompt the user during the process of selecting the materials so that the user can know how many materials can be selected at most. Figure 6 As shown in (b) in FIG, a prompt message “a maximum of 30 materials can be selected” is displayed in the gallery interface A3, so that the user can know the maximum number of materials that can be selected.

[0121] For ease of explanation, the video material and the picture material are collectively referred to as material files or materials to be analyzed. In some cases, the material to be analyzed includes only video material; in some cases, the material to be analyzed includes only picture material; in some cases, the material to be analyzed includes both video material and picture material.

[0122] It should be noted that in the embodiment of the present application, in the scenario where multiple video material files selected by the user are spliced ​​into one video (i.e., applying the one-click film-making function), each video material file needs to be analyzed to select highlight clips (i.e., wonderful clips, such as user smiling face clips, winning moment clips, etc.) from the video material. Since the expected total analysis time is limited, different actual analysis times need to be allocated to different video material files to ensure reasonable allocation.

[0123] The following describes in detail how the present application solution implements time allocation for analyzing multiple video materials and / or multiple image materials during the one-click film-making process in the above-mentioned application scenario.

[0124] In some cases, multiple video files may have different durations, i.e., the video playback times may differ. In an embodiment of the present application, the analysis duration allocation principle may include: for video files with longer video durations, a greater actual analysis duration should be allocated. The rate of increase in analysis duration may be lower than the rate of increase in video duration.

[0125] For example, assume that there are two videos to be analyzed in the process of one-click film formation, one video is 60 seconds long and the other video is 300 seconds long; by calculation, the expected total analysis time (i.e., the video to be allocated) is 30 seconds. It should be noted that this is based on the premise that the two videos to be analyzed have the same resolution and frame rate. According to the analysis time allocation scheme of this application, a possible allocation result is: for a video with a video length of 60 seconds, the actual analysis time allocated is 10 seconds; for a video with a video length of 300 seconds, the actual analysis time allocated is 20 seconds. By comparison, it can be seen that the growth rate of the duration of these two videos is 5 times, while the growth rate of the actual analysis time allocated to these two videos is 2 times. It can be seen that the increase rate of the analysis time is lower than the increase rate of the video length. In addition, it can be seen that for the video material file with a longer video length (300 seconds), the actual analysis time allocated is longer (20 seconds); for the video material file with a shorter video length (60 seconds), the actual analysis time allocated is less (10 seconds). It should be noted that this is a simple example, and the specific calculation process can be found below.

[0126] In other cases, the resolutions and / or frame rates of multiple video material files may be different, which may result in different analysis speeds for the multiple video material files. Specifically, the higher the resolution of the video material file, the higher the analysis speed of the video material file; and the higher the frame rate of the video material file, the higher the analysis speed of the video material file. In an embodiment of the present application, the principle for allocating analysis time may include: for video material files with faster analysis speeds, the actual analysis time allocated should be shorter. Specifically, the rate of decrease in analysis time may be lower than the rate of increase in analysis speed.

[0127] For example, assume that there are two videos to be analyzed in the process of one-click film formation, the resolution of one video to be analyzed is 720p, and the analysis speed queried is 1x speed, and the resolution of the other video to be analyzed is 1080p, and the analysis speed queried is 5x speed; the expected total analysis time (i.e., the video time to be allocated) is calculated to be 30 seconds. It should be noted that the premise here is that the video time of the two videos to be analyzed is the same (for example, both are 60 seconds). According to the analysis time allocation scheme of the present application, a possible allocation result is: for videos with a resolution of 720p, the actual analysis time allocated is 20 seconds; for videos with a resolution of 1080p, the actual analysis time allocated is 10 seconds. By comparison, it can be seen that the reduction rate of the analysis time of the two videos is 0.5 times, while the growth rate of the analysis speed of the two videos is 5 times. It can be seen that the reduction rate of the analysis time is lower than the growth rate of the analysis speed. Furthermore, it can be seen that for a video file with a faster analysis speed (1080p, 5x speed), the actual analysis time allocated is shorter (10 seconds), while for a video file with a slower analysis speed (720p, 1x speed), the actual analysis time allocated is longer (20 seconds). It should be noted that this is a simple example, and the specific calculation process can be found below.

[0128] That is to say, in the embodiment of the present application, the analysis time for multiple video material files is not distributed evenly, but is distributed differently for video material files with different characteristics in a targeted manner, which can improve the effect of video analysis.

[0129] The video processing method provided in the embodiment of the present application can, in a scenario where a plurality of material files with different video lengths are converted into films with one click, preliminarily determine an expected total analysis time (equal to the sum of the expected analysis times of all videos) based on the video decoding speed and the chip analysis speed, and then reasonably allocate the actual analysis time of each video material file based on the expected total analysis time according to the time allocation algorithm, thereby quickly completing the video analysis and improving the user experience.

[0130] The following describes in detail the duration allocation algorithm provided in the embodiment of this application. Figure 7 The input and output parameters of the duration allocation algorithm are explained, followed by the logic of the duration allocation algorithm. Finally, an example is given to illustrate how to allocate the actual analysis duration of each video material based on the duration allocation algorithm.

[0131] refer to Figure 7 As shown, in the embodiment of the present application, for multiple video material files selected by the user, based on the total number n of video material files, the length L of each video n , the sum of all video durations L A , the analysis speed of each video S n , the sum of all video analysis speeds S A , Total target analysis time T A ,The time allocation algorithm is used to calculate and determine the actual analysis time allocated to each video material file.

[0132] First, the main input parameters of the duration allocation algorithm are explained. Figure 7 As shown, the input data of the duration allocation algorithm includes the following six items:

[0133] (1) The total number of video material files (denoted as n).

[0134] (2) The duration of each video (denoted as L n )

[0135] For example, the duration of the first video is recorded as L1, the duration of the second video is recorded as L2, and so on. The duration of the nth video is L n . Among them, by parsing a single video material file, the duration of the video material file can be obtained.

[0136] (3) Total video length (denoted as L) A )

[0137] The total video duration refers to the sum of all video durations, i.e. L A =L1+…+L n .

[0138] (4) The analysis speed of each video (denoted as S n )

[0139] For example, the analysis speed of the first video is S1, the analysis speed of the second video is S2, and so on. The analysis speed of the nth video is S n The analysis speed of a single video can be obtained from the HAL layer. For example, the analysis speed of a single video can be 3 times the speed, or 4 times the speed, or 5 times the speed, or other analysis speeds.

[0140] The speed of video analysis is related to the video's resolution and frame rate. For example, the higher the video resolution, the slower the analysis speed. The higher the video frame rate, the slower the analysis speed. Since the resolution and frame rate of each video file vary, the analysis speed may vary from file to file.

[0141] In the embodiment of the present application, the analysis speed of a single video ultimately depends on the video decoding speed and the chip analysis speed. In other words, the analysis speed of a single video can be determined based on the chip analysis speed and the video decoding speed. The following details how to calculate the analysis speed S of each video. n process.

[0142] In the embodiment of the present application, the analysis speed S of a single video can be calculated using the following equation: n :

[0143] S n =min(V1, V2)×α; (Equation 1-1)

[0144] Where V1 represents the video decoding speed (decodeSpeed), V2 represents the chip analysis speed (chipestSpeed), and α represents the tolerance factor. Here, α is set to 0.9 as an example. min(V1, V2) means taking the smaller value of V1 or V2.

[0145] In the embodiments of the present application, chip analysis speed V2 is a performance parameter obtained from the HAL layer during initialization. For example, chip analysis speed V2 can be 3x, 4x, or 5x, and can be determined based on the actual chip conditions. For ease of explanation, the embodiments of the present application use a chip analysis speed V2 of 5x as an example.

[0146] In the embodiment of the present application, the default video decoding speed (denoted as V d The process of calculating the video decoding speed V1 is described in detail below.

[0147] The following equation can be used to calculate the video decoding speed V1 based on the video resolution and frame rate:

[0148] V1=V d ×(R d / R A )×(F d / F A ); (Equation 1-2)

[0149] Among them, V d Indicates the default video decoding speed, R dIndicates the default video resolution, R A Indicates the actual video resolution. d Indicates the default video frame rate, F A Indicates the actual video frame rate.

[0150] For example, the default video decoding speed V d You can take 4 times the speed, the default video resolution is R d You can choose 1080p (ie 1920×1080), the default video frame rate F d It can take 30 frames per second.

[0151] At the actual video resolution R A For 720p (ie 1280×720), the actual video frame rate F A Taking a 60-frame-per-second video as an example, substitute the above parameters into equation 1-2 to calculate the video decoding speed V1:

[0152] V1=V d ×(R d / R A )×(R d / R A )=4×(1920×1080) / (1280×720)×(30 / 60)=4.5.

[0153] Therefore, the video decoding speed V1 is calculated by the above equation 1-2.

[0154] Next, we use the above equation 1-1 to calculate the analysis speed S of a single video n Here we assume that the chip analysis speed V2 is 5 times faster. Substitute the video decoding speed V1 and the chip analysis speed V2 into equation 1-1 to calculate the single video analysis speed S n :

[0155] S n =min(4.5,5)×0.9=4.5×0.9≈4.

[0156] As can be seen from this, the present application solution determines the analysis speed of a single video based on the chip analysis speed and the video decoding speed; wherein the video decoding speed is determined by the resolution and frame rate of the single video. In other words, the present application solution calculates the analysis speed of a single video based on the chip analysis speed and the resolution and frame rate of the single video.

[0157] (5) The sum of all video analysis speeds (denoted as S A )

[0158] S A It is the sum of the analysis speeds of all videos to be analyzed.

[0159] (6) Total target analysis time (denoted as T A )

[0160] Total target analysis time T A The time it takes to analyze all the material files is issued by the application layer. First, calculate the total target analysis time T A Then, according to the time allocation algorithm provided in the embodiment of the present application, the total target analysis time T A Assigned to all video footage files.

[0161] It should be noted that in the embodiment of the present application, the multiple video material files may all be videos, or may include one or more videos and one or more pictures, or may all be pictures. For videos, query analysis speed is required; for pictures, query analysis speed is not required.

[0162] Scenario 1: Multiple video material files only include video material

[0163] In some embodiments, when the plurality of video material files are all videos, the target analysis total duration T is calculated according to the following equation: A :

[0164] T A =∑T Vn ; (Equation 2-1)

[0165] T Vn =min(L n / S n , β); (Equation 2-2)

[0166] Among them, min(L n / S n ,β) represents taking L n / S n The minimum value among L and β. n Indicates the video length of a single video, S n Indicates the analysis speed of the single video, T Vn Indicates the expected analysis duration of a single video in multiple video material files, which can be calculated based on the video duration and analysis speed. Vn Represents the sum of the expected analysis durations for all videos. Where β is the upper limit of the expected analysis duration for a single video, which ensures that the expected analysis duration for a single video is within the limit. It is understood that in actual implementation, the upper limit of the expected analysis duration for a single video can be set based on actual usage requirements and is not limited in this embodiment of the application.

[0167] For example, β is 20 seconds. The expected analysis duration of a single video is expressed as:

[0168] T Vn =min(L n / S n , 20s); (Equation 2-3)

[0169] For example, the video length L of a video n The video analysis speed is S n is 5 times faster, then L n / S n is 12 seconds. According to the above equations 2-3, the expected analysis time of the video is T Vn =min(12s, 20s)=12s.

[0170] For example, the video length L of a video n is 200s, the analysis speed of the video is S n is 5 times faster, then L n / S n is 40 seconds. According to the above equations 2-3, the expected analysis time of the video is T Vn =min(40s, 20s)=20s.

[0171] That is, when multiple video material files are all videos, the target total analysis time T is calculated based on the sum of the expected analysis times of all videos to be analyzed. A For example, suppose the user selects 5 videos as the video material files for one-click filming, queries the analysis speed of each video, and calculates the expected analysis time of each video based on the analysis speed of each video. Then, the expected analysis time of these 5 videos is summed up to obtain the target total analysis time T. A For example, the total time of target analysis is T A Taking 20 seconds as an example, the embodiment of the present application will allocate 20 seconds to these 5 videos according to the duration allocation algorithm. The specific method of calculating the actual analysis duration allocated to each video will be described in detail below.

[0172] Scenario 2: Multiple video material files including video material and picture material

[0173] In other embodiments, when the plurality of video material files include one or more videos and one or more pictures, the target analysis total duration T is calculated according to the following equation: A :

[0174] T A =∑T Vn +∑T P ; (Equation 3-1)

[0175] TVn =min(L n / S n , β); (Equation 3-2)

[0176] T P =γ; (Equation 3-3)

[0177] Among them, T Vn Indicates the expected analysis duration of a single video in multiple video material files, ∑T Vn represents the sum of the expected analysis duration of all videos, T P Indicates the expected analysis time of a single picture in multiple video material files, ∑T P Indicates the sum of the expected analysis time for all images.

[0178] γ is a preset value. The value of γ can be determined according to actual usage requirements and is not limited in the present embodiment. For example, γ is 1, which means that the expected analysis time of a single image is 1 second. The following description is based on the expected analysis time of a single image being 1 second.

[0179] Assuming that multiple video material files include n videos and m pictures, the sum of the expected analysis time of these m pictures is m. Accordingly, the target total analysis time T can be calculated according to the following equation: A :

[0180] T A =∑min(L n / S n ,β)+m; (Equation 3-4)

[0181] It can be seen that when multiple video material files include videos and pictures, the sum of the expected analysis time of all videos and the expected analysis time of all pictures can be added to obtain the target total analysis time T A .

[0182] It should be noted that in the embodiment of the present application, the expected analysis time of each picture can be a fixed time, that is, the expected analysis time allocated to each picture is fixed. The expected analysis time of each picture can be set according to actual usage requirements, and the embodiment of the present application is not limited thereto. For example, the expected analysis time of each picture is 1 second. In this case, the highlight segment analysis algorithm will take 1 second to perform highlight analysis on each picture to determine whether the picture contains a highlight segment (also called a highlight moment or a wonderful moment).

[0183] In the embodiment of the present application, the total target analysis time T calculated above is AThis is the duration of the analysis of all video material files by the Highlight Clip Analysis Algorithm. The actual analysis duration allocated to each video in all video material files needs to be recalculated according to the duration allocation algorithm. Assuming that the actual analysis duration allocated to a video is 10 seconds, in this case, the Highlight Clip Analysis Algorithm will spend 10 seconds analyzing the highlights of the video to determine the highlights (also called highlights) in the video. The highlights in the video will then be captured as the video clip content for one-click filming.

[0184] Combination of the above Figure 7 The various input parameters of the duration allocation algorithm are explained, and then refer to Figure 7 As shown, the output data of the duration allocation algorithm is: the actual analysis duration allocated to each video material file (denoted as T n For example, the actual analysis time allocated to the first video material file is T1, the actual analysis time allocated to the second video material file is T1, and so on. The actual analysis time allocated to the nth video material file is T n .

[0185] Combination of the above Figure 7 The input parameters and output parameters of the duration allocation algorithm are described. The following details the algorithm rules of the duration allocation algorithm in the embodiment of the present application, as well as how to calculate the actual analysis duration T of each video material file according to the duration allocation algorithm. n process.

[0186] In the embodiment of the present application, the overall idea of ​​the duration allocation algorithm can be: the expected analysis duration of all videos (denoted as T V ) is divided into two parts for allocation. First, one part is evenly distributed to each video, which is called the basic analysis time. The other part is allocated according to the corresponding weight ratio of the video, which is called the weight-based analysis time. In other words, the actual analysis time allocated to a single video file is composed of the following two parts:

[0187] T n =T n1 +T n2 ; (Equation 4-1)

[0188] Among them, T n1 Indicates the basic analysis time, T n2 Indicates the analysis duration based on weight.

[0189] It should be noted that, in the case where multiple video material files are all videos, the video to be allocated time T V The total target analysis time T in the above scenario 1 AIn the case where multiple video material files include videos and pictures, the video to be allocated time T V is the sum of the expected analysis duration of all videos in the above scenario 2 ∑T Vn .

[0190] Optionally, a portion of the video's to-be-allocated duration in a preset proportion can be evenly distributed to each video, and the remaining portion of the video's to-be-allocated duration can be distributed to each video according to the corresponding weight ratio of the video. The preset ratio can be set according to actual usage requirements and is not limited in the embodiments of the present application. For example, the preset ratio is 50%. That is, 50% of the video's to-be-allocated duration is evenly distributed to each video, and then the remaining 50% of the video's to-be-allocated duration is distributed to each video according to the weight ratio.

[0191] The basic analysis time T is calculated by the following equation: n1 :

[0192] T n1 =T V ×0.5 / n; (Equation 4-2)

[0193] Where n represents the number of videos to be analyzed.

[0194] The weighted analysis time T is calculated using the following equation: n2 :

[0195] T n2 =T V ×0.5×(W n / W A ); (Equation 4-3)

[0196] W n =log2(1.5+L n / L A ) / log2(1.5+S n / S A ); (Equation 4-4)

[0197] W A =W1+…+W n ; (Equation 4-5)

[0198] Among them, W n Represents the weight value of a single video, for example, W1 represents the weight of the first video, W2 represents the weight of the second video, and so on. n W represents the weight of the nth video. A It should be noted that 1.5 in Equation 4-4 is a preset value of the algorithm, which can be set according to actual use requirements and is not limited in this embodiment of the application.

[0199] In the embodiment of the present application, the longer the video material file is, the longer the actual analysis time should be allocated. In other words, the video length L of a certain video n The sum of all video durations L A The ratio L n / L A The larger the value, the more actual analysis time is allocated to the video.

[0200] In the embodiment of the present application, the faster the analysis speed of the video material file, the shorter the actual analysis time should be allocated. In other words, the analysis speed S of a video n The sum of all video analysis speeds S A The ratio S n / S A The larger the value, the less actual analysis time is allocated to the video.

[0201] Among them, the weight value W of a single video n Specifically, it is calculated using the above equation 4-4.

[0202] For example, let's take a case where two videos are included in multiple video material files. The sum of all video durations is L A The sum of the video analysis speeds is 200 seconds. A Assume that the video is to be allocated for a length of T V 20 seconds.

[0203] Assuming that the length L1 of the first video is 60 seconds and the analysis speed S1 of the video is 5x, the weight value W1 of the first video can be calculated using the above equation 4-4:

[0204] W1=log2(1.5+L1 / L A ) / log2(1.5+S1 / S A )=log2(1.5+60 / 200) / log2(1.5+5 / 10)=0.85.

[0205] Assume that the length of the second video L2 is 140 seconds and the analysis speed S2 is 5 times the speed. The weight value W2 of the second video can be calculated by the above equation 4-4:

[0206] W2=log2(1.5+L2 / L A ) / log2(1.5+S2 / S A )=log2(1.5+140 / 200) / log2(1.5+5 / 10)=1.14.

[0207] The sum of the weight values ​​of all videos W can be calculated by the above equations 4-5A Equal to 1.99.

[0208] The weighted analysis duration T of the first video is calculated using the above equation 4-3 n2 :

[0209] T n2 =T V ×0.5×(W1 / W A )=20×0.5×(0.85 / 1.99)=4.

[0210] The basic analysis time T of the first video is calculated using the above equation 4-2. n1 :

[0211] T n1 =T V ×0.5 / n=20×0.5 / 2=5.

[0212] Then use the above equation 4-1 to calculate the actual analysis time T1 allocated to the first video:

[0213] T n =T n1 +T n2 =4+5=9.

[0214] It can be understood that the actual analysis time T1 allocated to the first video is 9 seconds. V The actual analysis time allocated to the second video is 20-9 = 11 seconds.

[0215] It should be noted that the example here of allocating the duration of two videos according to the duration allocation algorithm is used for exemplary explanation. It can be understood that in actual implementation, if the user selects three or more videos, this application can allocate the duration of a large number of videos according to the above-mentioned duration allocation algorithm, thereby allocating appropriate actual analysis duration to each video.

[0216] In the process of one-click film formation, in order to analyze the highlight segments at different positions in multiple video files within a limited time, the embodiment of the present application allocates a suitable duration to each video through a duration allocation algorithm, performs video highlight segment analysis, and then completes the one-click film formation by splicing the highlight segments of multiple videos.

[0217] The specific implementation process of the video processing method provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0218] Figure 8 This is a schematic flow chart of a video processing method provided in an embodiment of the present application, which can be executed by an electronic device. The hardware structure diagram of the electronic device can be as follows: Figure 1 As shown, the software structure diagram of the electronic device can be as follows Figure 2 As shown, but the embodiments of the present application are not limited thereto.

[0219] S101, the application editing business layer receives the user's operation on the one-click film option.

[0220] The application editing business layer includes a one-click film creation application. This application is presented as a "One-click Film Creation" option on the interface. When a user clicks the "One-click Film Creation" option, the application editing business layer activates the application.

[0221] S102 , in response to a user operation, the application editing service layer loads and displays pictures and videos.

[0222] For example, a one-touch capture option is displayed in the UI of a camera application of an electronic device. When a user clicks the one-touch capture option, the UI is updated to a gallery interface. The gallery interface displays multiple images and multiple videos, and a confirmation control (e.g., √) for the one-touch capture option is displayed.

[0223] S103: The application editing service layer receives the user's selection of videos and pictures and confirms the execution of the one-click film-making operation.

[0224] Among them, the user first selects one or more videos and one or more pictures in the gallery interface, and then clicks the √ control to confirm the one-click filming.

[0225] Exemplarily, the user selected video materials such as video 1, video 2, and video 3, and also selected picture materials such as picture 1 and picture 2. Based on video 1, video 2, video 3, picture 1, and picture 2, the electronic device will analyze the highlight clips or wonderful moments therein, and then splice the highlight clips or wonderful moments in the videos and pictures, and finally generate a video, completing one-click filming. In the one-click filming process, it is necessary to analyze the highlight clips at different positions of multiple video files within a limited time to avoid taking too much time. In this regard, the improvement of the embodiment of the present application is that a suitable duration is allocated to each video through a duration allocation algorithm, and the video highlight clip analysis is performed, and then the highlight clips of multiple videos are spliced ​​together to complete the one-click filming. The following will step by step explain how the present application solution reasonably allocates duration to each video.

[0226] After the user triggers the one-click slice generation, the modules at different layers interact with each other to complete parameter and algorithm initialization. For the specific initialization process, see the following steps.

[0227] S104, the application editing business layer sends a one-click film-making instruction to the application function layer. In response to the instruction, the application function layer calls the media middle platform framework to initialize each interface.

[0228] The application function layer instructs the media center framework layer to initialize the image highlight segment analysis interface, theme summary interface, and analysis performance query interface. Each interface in the media center framework layer completes initialization separately.

[0229] S105: The performance analysis query interface sends initialization parameters to the one-key slice module of the FWK layer through the channel interface.

[0230] The initialization parameter is used to instruct the initialization of the highlight segment analysis algorithm.

[0231] S106 , the one-key slice module of the FWK layer instructs the algorithm module of the HAL layer to initialize.

[0232] S107 , the algorithm module of the HAL layer initializes the highlight segment analysis algorithm.

[0233] S108, the algorithm module of the HAL layer reports the message of successful initialization to the channel interface in the media middle platform framework layer; wherein, the message of successful initialization is transmitted to the channel interface through the one-key film module of the FWK layer.

[0234] S109, the channel interface in the media middle platform framework layer calls the chip analysis speed interface to obtain the chip analysis speed from the HAL layer.

[0235] S110, the algorithm module of the HAL layer returns the chip analysis speed to the channel interface in the media middle platform framework layer.

[0236] Exemplarily, the chip analysis speed is 5 times the speed. For example, for a 60-second video, if the analysis is performed at a 5-times-speed chip analysis speed, it will take 12 seconds to complete the analysis of the video.

[0237] Among them, the call of the chip analysis speed interface and the return of the chip analysis speed are both transmitted through the one-click chip module of the FWK layer.

[0238] S111, the channel interface in the media middle platform framework layer sends the chip analysis speed to the analysis performance query interface.

[0239] The chip analysis speed obtained from the performance query interface here can be used to calculate the analysis speed of each video in the following steps.

[0240] S112, the analysis performance query interface in the media middle platform framework layer reports the initialization success message to the application layer.

[0241] Therefore, each module of the video editing service layer and the application function layer of the application layer can be informed that the corresponding initialization has been completed.

[0242] After initialization, the application layer will query the analysis performance query interface for the analysis speed of each video in turn.

[0243] S113 , the video editing service layer of the application layer sends query parameters to the analysis performance query interface to query the analysis speed of video 1 .

[0244] The query parameters include the file descriptor of the video, and the file descriptor of video 1 is recorded as fd1.

[0245] The file descriptor of a video can be used to uniquely identify the video. According to the file descriptor of the video, relevant information of the video, such as the resolution and frame rate of the video, can be obtained.

[0246] S114, analyzing the performance query interface to obtain the resolution and frame rate of video 1 according to the file descriptor fd1 of video 1.

[0247] S115 , the analysis performance query interface calculates the analysis speed of video 1 according to the resolution and frame rate of video 1 and the chip analysis speed.

[0248] As mentioned above, the analysis speed of a single video ultimately depends on the video decoding speed and the chip analysis speed. Specifically, the video decoding speed of Video 1 can be calculated based on its resolution and frame rate. It's understandable that since the resolution and frame rate of each video may vary, the video decoding speed may also vary. The specific process for calculating the analysis speed of Video 1 can be found above and will not be further elaborated here.

[0249] S116 , the analysis performance query interface returns the analysis speed of video 1 to the video editing service layer of the application layer.

[0250] After the application layer video editing service obtains the analysis speed of video 1, S113-S116 are executed again to continue to query the analysis speed of the next video. Thus, the application layer video editing service sequentially queries the analysis speeds of video 1, video 2, and video 3.

[0251] It should be noted that the analysis speed of the video is mainly considered here, and the analysis speed of the image is not considered.

[0252] S117: The video editing service layer of the application layer calculates the target total analysis time according to the analysis speed of all returned videos.

[0253] For example, assuming that the video length L1 of video 1 is 120 seconds, the video length L2 of video 2 is 60 seconds, and the video length L3 of video 3 is 25 seconds; the analysis speed S1 of video 1 is 5 times the speed, the analysis speed S2 of video 2 is 5 times the speed, and the analysis speed S3 of video 3 is 4 times the speed.

[0254] First, combined with equation 2-2 above, we can know that:

[0255] The expected analysis duration of Video 1 is min(L1 / S1, 20s), that is, min(120 / 5, 20s)=20s.

[0256] The expected analysis duration of Video 2 is min(L2 / S2, 20s), that is, min(60 / 5, 20s)=12s.

[0257] The expected analysis duration of Video 3 is min(L3 / S3, 20s), that is, min(20 / 4, 20s)=5s.

[0258] Combined with equation 2-1 above, we can see that the total expected analysis time of all videos is 20s+12s+5s=37s.

[0259] As mentioned above, when multiple materials to be analyzed include videos and pictures, the target total analysis time is obtained by adding the sum of the expected analysis times of all videos and the sum of the expected analysis times of all pictures. Among them, the expected analysis time of each picture is a fixed value, such as 1 second. Here, the sum of the expected analysis times of the two pictures, Picture 1 and Picture 2, is 2 seconds. Therefore, the target total analysis time is 39 seconds, which is the sum of the expected analysis times of all videos and the sum of the expected analysis times of all pictures, 2s. It should be noted that the target total analysis time is the expected analysis time preliminarily determined for all materials, and the target total analysis time will be allocated to each video and picture according to actual needs. Among them, the focus of the present application is how to re-divide the sum of the expected analysis times of all videos according to the time allocation algorithm, and allocate a suitable actual analysis time to each video.

[0260] Figure 9 Another schematic flow chart of the video processing method provided by the embodiment of the present application is shown. Figure 8 ,like Figure 9 As shown, after S117, the video processing method further includes the following S118-S139.

[0261] S118, the video editing business layer of the application layer calls the image highlight segment analysis interface, and sends the target analysis total time and the file descriptors fd of all material files to be analyzed to the policy monitoring module of the media middle platform framework layer.

[0262] After S118, all the material files to be analyzed are analyzed. Optionally, each image can be analyzed sequentially first, and after all images are analyzed, each video can be analyzed sequentially. Alternatively, each video can be analyzed sequentially first, and after all videos are analyzed, each image can be analyzed sequentially. For ease of explanation, this example uses the example of analyzing images first and then videos.

[0263] For example, Figure 10 Schematic diagram of analysis pictures and videos provided in the embodiment of the present application is shown. Figure 10 As shown, each image is analyzed in sequence first, and after all images are analyzed, each video is analyzed in sequence. For example, the analysis time of each image is 1 second, and the analysis time of each video is the actual analysis time calculated and allocated by the above-mentioned time allocation algorithm.

[0264] The image analysis process is described below. It involves analyzing multiple images one by one. During each analysis, the image is first processed and then analyzed within a preset analysis duration to determine whether it is a highlight segment.

[0265] S119, the policy monitoring module of the media middle platform framework layer instructs the channel interface to analyze picture 1.

[0266] S120 , the channel interface performs processing such as decoding, reducing resolution, and converting format on the picture 1 , and stores the processed picture 1 .

[0267] In an embodiment of the present application, in order to improve the processing speed of the highlight segment algorithm interface on the image, the channel interface can reduce the resolution of the image before performing highlight analysis on the image. In addition, because the resolution reduction algorithm only supports a certain format (for example, i420 format), it is necessary to first convert all video frames from the current format (for example, NV12 format) to i420 format, then reduce the resolution of the image (for example, from 1080p to 480p), and then convert the reduced resolution image to the original NV12 format and store it in the memory.

[0268] S121: The channel interface sends the frame data address of picture 1 to the HAL layer through the one-key slice module of the FWK layer.

[0269] S122 , the algorithm module of the HAL layer finds the processed picture 1 according to the frame data address, analyzes the picture 1 using the highlight segment analysis algorithm, and obtains an analysis result (such as a score) of the picture 1 .

[0270] The highlight analysis algorithm is an image processing algorithm provided by the image chip. This algorithm scores each frame based on its color, texture, quality, interpolation with previous and next frames, and edge change rate. The score can be used to determine whether a frame represents a highlight or a wonderful moment. For example, if the score is greater than or equal to 60, the frame is considered a highlight or a wonderful moment.

[0271] S123, the algorithm module of the HAL layer transmits the analysis result of picture 1 to the policy monitoring module of the media middle platform framework layer.

[0272] The analysis results of Figure 1 are transmitted through the one-click film-making module of the FWK layer and the channel interface of the media middle platform framework layer.

[0273] S119-S123 illustrate the analysis process for a single image. S119-S123 can be executed in a loop to analyze each image in turn. After analyzing a single image, S119-S123 are executed again to analyze the next image. For example, after analyzing image 1, S119-S123 are executed again to analyze image 2.

[0274] For example, the analysis result of picture 1 is 80 points, that is, picture 1 is analyzed as a wonderful moment.

[0275] For another example, the analysis result of picture 2 is 50 points, that is, picture 2 is analyzed as a non-wonderful moment.

[0276] For example, the actual analysis time for Image 1 and Image 2 is 1 second each, so the image analysis takes 2 seconds.

[0277] It should be noted that the channel interface is used here as an example to illustrate the decoding, resolution reduction, format conversion and other processing of images. It can be understood that in actual implementation, the images can also be decoded, reduced in resolution, format conversion and other processing through other methods, such as processing the images through other interfaces or modules of the media middle platform framework layer.

[0278] The following describes the video analysis process. It involves analyzing multiple videos one by one. During each analysis, an actual analysis duration is assigned to a video (referred to as the first video), an analysis strategy is set for the first video, and the first video is processed. Finally, the first video is analyzed again within the actual analysis duration, based on the analysis strategy, to identify the highlights within the first video.

[0279] S124, the policy monitoring module of the media middle platform framework layer allocates the actual analysis time of each video according to the sum of the expected analysis time of all videos.

[0280] The sum of the expected analysis time for all videos is the difference between the target analysis time and the time required to analyze all images.

[0281] For example, suppose the user selects 3 videos and 2 pictures as the materials to be analyzed for one-click filming. Taking the target analysis time as 39 seconds, the sum of the expected analysis times for these 2 pictures (i.e., the time required to analyze all pictures) is 2 seconds, and the sum of the expected analysis times for these 3 videos is 37 seconds.

[0282] In an embodiment of the present application, the actual analysis time allocated to each video can be determined based on the total number of all videos, the duration of each video, the sum of the durations of all videos, the analysis speed of each video, the sum of the analysis speeds of all videos, and the sum of the expected analysis durations of all videos (these parameters are collectively referred to as first parameters).

[0283] For example, 50% of the total expected analysis duration of all videos (called the video time to be allocated) is evenly distributed to each video, and then the remaining 50% of the video time to be allocated is allocated to each video according to the weight ratio.

[0284] For example, we still take the example of multiple video material files including three videos (video 1, video 2 and video 3) as an example for illustration. The sum of the lengths of all videos L A The sum of the video analysis speeds is 200 seconds. A It is 14 times faster. Assume that the video duration is T V 37 seconds.

[0285] Calculation process of Video 1:

[0286] Assuming that the video length L1 of video 1 is 120 seconds and the analysis speed S1 of video 1 is 5 times the speed, the weight value W1 of video 1 can be calculated by the above equation 4-4:

[0287] W1=log2(1.5+L1 / L A ) / log2(1.5+S1 / S A )=log2(1.5+120 / 200) / log2(1.5+5 / 14)≈1.20.

[0288] Assume that the video length L2 of video 2 is 60 seconds and the analysis speed S2 of video 2 is 5 times the speed. The weight value W2 of video 2 can be calculated by the above equation 4-4:

[0289] W2=log2(1.5+L2 / L A ) / log2(1.5+S2 / S A)=log2(1.5+60 / 200) / log2(1.5+5 / 14)≈0.95.

[0290] Assume that the video length L2 of video 3 is 40 seconds and the analysis speed S2 of video 3 is 4 times the speed. The weight value W3 of video 3 can be calculated by the above equation 4-4:

[0291] W3=log2(1.5+L3 / L A ) / log2(1.5+S3 / S A )=log2(1.5+20 / 200) / log2(1.5+4 / 14)≈0.81.

[0292] The above equation 4-5 can calculate the sum of the weight values ​​​​W of all videos A =1.20+0.95+0.81=2.96.

[0293] The weighted analysis duration T of video 1 is calculated using equation 4-3 above. 12 :

[0294] T 12 =T V ×0.5×(W1 / W A )=37×0.5×(1.2 / 2.96)=7.5.

[0295] The basic analysis time T of video 1 is calculated by the above equation 4-2 11 :

[0296] T 11 =T V ×0.5 / n=37×0.5 / 3≈6.2.

[0297] Then, the actual analysis time T1 allocated for video 1 is calculated using equation 4-1 above:

[0298] T1=T 11 +T 12 =7.5+6.2=13.7.

[0299] It can be understood that, in other words, the actual analysis time T1 allocated to Video 1 is 13.7 seconds.

[0300] Calculation process of Video 2:

[0301] The actual analysis time allocated for Video 2 is calculated in the same manner as above.

[0302] The weighted analysis duration T of video 2 is calculated using equation 4-3 above. 22 :

[0303] T22 =T V ×0.5×(W2 / W A )=37×0.5×(0.95 / 2.96)=5.9.

[0304] The basic analysis time T of video 2 is calculated by the above equation 4-2 21 :

[0305] T 21 =T V ×0.5 / n=37×0.5 / 3≈6.2.

[0306] Then, the actual analysis time T2 allocated for video 2 is calculated using equation 4-1 above:

[0307] T2=T 21 +T 22 =5.9+6.2=12.1.

[0308] It can be understood that, in other words, the actual analysis time T2 allocated to Video 2 is 12.1 seconds.

[0309] Calculation process of Video 3:

[0310] The actual analysis time allocated for Video 3 is calculated in the same manner as above.

[0311] The weighted analysis duration T of video 3 is calculated using equation 4-3 above. 32 :

[0312] T 32 =T V ×0.5×(W3 / W A )=37×0.5×(0.81 / 2.96)≈5.

[0313] The basic analysis time T of video 3 is calculated by the above equation 4-2 31 :

[0314] T 31 =T V ×0.5 / n=37×0.5 / 3≈6.2.

[0315] Then, the actual analysis time T3 allocated to video 3 is calculated using equation 4-1 above:

[0316] T3=T 31 +T 32 =5.0+6.2=11.2.

[0317] It can be understood that, in other words, the actual analysis time T3 allocated to Video 2 is 11.2 seconds.

[0318] Alternatively, when calculating the actual analysis time of the last video, the following method can be used: Since the video time to be allocated is T V The above process shows that 13.7 seconds of timing analysis time is allocated to Video 1 and 12.1 seconds of timing analysis time is allocated to Video 2. Therefore, the actual analysis time allocated to Video 3 is 37-13.7-12.1=11.2 seconds.

[0319] Thus, the actual analysis duration was allocated to each video.

[0320] It should be noted that after allocating the actual analysis duration to each video, each video is analyzed sequentially. Taking Video 1 as an example, the analysis strategy for Video 1 is first set, Video 1 is processed, and then Video 1 is analyzed again according to the analysis strategy. The actual analysis duration is 13.7 seconds to identify the highlights in Video 1. In other words, the subsequent analysis of the highlights in Video 1 will use a duration of 13.7 seconds (referred to as highlight analysis).

[0321] S125, the policy monitoring module dynamically sets the analysis policy.

[0322] Among them, the analysis strategies include full analysis strategy, intensive key segment analysis strategy, sparse key segment analysis strategy, or simple analysis strategy based on I frame.

[0323] The embodiments of the present application provide four analysis strategies: a full analysis strategy, an intensive key segment analysis strategy, a sparse key segment analysis strategy, and an I-frame-based simple analysis strategy. Among them, the full analysis strategy includes every frame of the video in the analysis content, including all highlight segments, but it consumes relatively more time and resources. The I-frame-based simple analysis strategy refers to when the allocated analysis time can only meet the time for analyzing three pictures, analyzing three pictures located at different positions in the video, and taking the one with the highest score among the three pictures as the starting point of the highlight segment result, and returning a video segment of a certain length as the result. The intensive key segment analysis strategy is an algorithm strategy that calculates the number of analysis segments and the analysis segment duration of the material video to obtain a reasonable interval duration between adjacent segments, and then evenly distributes each analysis segment so that the analysis segment covers the highlight segments located at different positions as much as possible. The sparse key segment analysis strategy calculates and analyzes the same number of video segments and durations as the intensive strategy, but differs in that it distributes the durations of adjacent segments unevenly. This algorithm analyzes segments primarily in the beginning and middle of the video, with a moderate distribution at the end. These four analysis strategies will be described in detail in the following examples and will not be detailed here.

[0324] For each material video: If the entire material video can be analyzed within the available analysis time, a full analysis strategy is adopted. If the available analysis time is less than or equal to the analysis time of three key frames, a simple analysis strategy based on I frames is adopted. If the available analysis time is not enough to analyze the entire material video, but is enough to analyze more than 60% of the material, an intensive key segment analysis strategy can be adopted. If the available analysis time is longer than the analysis time of three key frames, but is not enough to analyze more than 60% of the material, a sparse key segment analysis strategy can be adopted. After determining the analysis strategy corresponding to each video, the image signal processor scores the video frames to screen out highlight segments with high scores.

[0325] For example, Figure 11 The following is a schematic diagram showing the time consumption of the four analysis strategies provided in the embodiment of the present application. Figure 11 As shown, the time consumption of the full analysis strategy, the intensive key segment analysis strategy, the sparse key segment analysis strategy, and the simple analysis strategy based on I frames decreases in sequence. Since the full analysis strategy analyzes all frames of the video, it consumes the longest time. The simple analysis strategy based on I frames analyzes up to three frames of video, so it consumes the shortest time. The time consumption of the intensive key segment analysis strategy and the sparse key segment analysis strategy is between the full analysis strategy and the simple analysis strategy based on I frames. In addition, compared with the intensive key segment analysis strategy, the sparse key segment analysis strategy focuses on analyzing the front and middle sections of the video, so it may take less time.

[0326] For example, the analysis strategy set for video 1 is used as a full analysis strategy as an example for explanation.

[0327] S126, the policy monitoring module sends the file descriptor of video 1 and the set analysis policy to the channel interface.

[0328] S127 , the channel interface performs processing such as decoding, format conversion, and resolution reduction on the video 1 , and stores the processed video 1 .

[0329] In the embodiment of the present application, the policy monitoring module sends the file descriptor fd1 of video 1 and the analysis policy (full analysis policy) set for video 1 to the channel interface. Then, the channel interface executes the following steps A1 to A5.

[0330] In step A1, the channel interface decodes all video frames of video 1 based on the file descriptor fd1 of video 1 and the analysis policy (full analysis policy) set for video 1. It will be appreciated that the policy monitoring module, by issuing the analysis policy (full analysis policy) set for video 1, enables the channel interface to determine that the video frames to be decoded are all video frames of video 1 based on the policy.

[0331] Step A2: The channel interface converts all decoded video frames from the first format to the second format.

[0332] Exemplarily, the first format is nv12 format, and the second format is i420 format.

[0333] In step A3, the channel interface reduces all the video frames after the format change from the first resolution to the second resolution.

[0334] Exemplarily, the first resolution is 1080p and the second resolution is 480p.

[0335] In some embodiments, different analysis strategies may correspond to different second resolutions. Therefore, the strategy monitoring module sends the analysis strategy (full analysis strategy) set for video 1, so that the channel interface can determine the second resolution corresponding to the full analysis strategy.

[0336] In step A4, the channel interface converts all video frames with reduced resolution from the second format to the first format.

[0337] In step A5 , the channel interface stores all video frames converted into the first format again in a memory (buffer).

[0338] It should be noted that to increase the processing speed of the highlight segment algorithm interface, the channel interface can reduce the resolution of the video frames before performing highlight analysis. Furthermore, because the resolution reduction algorithm only supports the second format, it is necessary to first convert all video frames from the first format to the second format, then reduce the resolution of the video frames, and then convert the reduced-resolution video frames back to the original first format for storage in the memory.

[0339] S128: The channel interface sends the frame data address of video 1 to the HAL layer through the one-key slice module of the FWK layer.

[0340] Among them, the one-click segmentation module of the FWK layer can complete data packaging and provide data and program operation services.

[0341] S129 , the algorithm module of the HAL layer finds the processed video 1 according to the frame data address, analyzes the video 1 using the highlight segment analysis algorithm, and obtains the analysis result of the video 1 .

[0342] The highlight segment position of video 1 is analyzed by the highlight segment analysis algorithm, and the highlight segment position is specifically represented by the start time point and the end time point of the highlight segment. Figure 12As shown, the duration of video 1 is 60 seconds. Video 1 is analyzed using a highlight segment analysis algorithm, and the segments between the 20th second (i.e., the start time) and the 30th second (i.e., the end time) are determined to be highlight segments (e.g., multiple consecutive frames of images containing a person's smiling face).

[0343] S130, the algorithm module of the HAL layer transmits the analysis results of video 1 to the policy monitoring module of the media middle platform framework layer.

[0344] Among them, the analysis results of Video 1 pass through the one-click film-making module of the FWK layer and the channel interface of the media middle platform framework layer during transmission.

[0345] The analysis result of video 1 includes the position information of the highlight segment, for example, video 1: [20s, 30s].

[0346] S124-S130 illustrate the analysis process of a video.

[0347] In some embodiments, S125-S130 may be executed in a loop to analyze each video in sequence. That is, after the analysis of one video is completed, the process returns to S125-S130 to continue analyzing the next video.

[0348] For example, after the analysis of video 1 is completed, the process returns to S125-S130 to continue analyzing video 2: the analysis strategy for video 2 is set, video 2 is processed, and then highlight analysis is performed on video 2 according to the analysis strategy for video 2. The actual analysis time is 12.1 seconds to determine the highlight segments in video 2.

[0349] Then, after the analysis of video 2 is completed, S125-S130 are executed again to analyze video 3: the analysis strategy for video 3 is set, then video 3 is processed, and then highlight analysis is performed on video 3 according to the analysis strategy for video 3. The actual analysis time takes 11.2 seconds to determine the highlight clips in video 3.

[0350] For example, if Figure 13 As shown in the figure, the analysis result for Video 1 is: [20s, 30s], that is, [20s, 30s] in Video 1 is analyzed as a highlight segment. The analysis results for Video 2 are: [5s, 10s] and [20s, 25s], that is, Video 2 contains two highlight segments. The analysis results for Video 3 are: [1s, 5s] and the 12th second, that is, the frame images of [1s, 5s] and the 12th second in Video 3 are both analyzed as highlight segments.

[0351] In other embodiments, after the analysis of a video is completed, when the actual analysis progress significantly lags behind expectations (for example, the actual analysis progress exceeds 120% of the video's allocated time), a rescheduling of the allocated analysis time can be triggered. That is, after the analysis of a video is completed, when the actual analysis progress significantly lags behind expectations, the process returns to S124-S130, reallocates the analysis time for all videos that have not been analyzed, and then sets an analysis strategy for highlight analysis. In this way, through an adaptive monitoring scheduling algorithm, while ensuring that the analysis is completed within a reasonable time limit, the analysis time allocation is dynamically planned to fully utilize the analysis time to achieve better analysis results.

[0352] S131, the policy monitoring module of the media middle platform framework layer calls the image highlight segment analysis interface to notify the application function layer of the application layer that the analysis of all material files has been completed, and reports the analysis results of all material files to the application function layer of the application layer.

[0353] S132 , the application function layer of the application layer edits and filters the material files selected by the user according to the analysis results of all the material files.

[0354] For example, refer to Figure 13 The analysis results of all material files include: Picture 1 contains a highlight clip (the highlight clip is marked as 1), Picture 2 does not contain a highlight clip, the highlight clip of Video 1 is [20s, 30s] (the highlight clip is marked as 2), the highlight clips of Video 2 are [5s, 10s] (the highlight clip is marked as 3) and [20s, 25s] (the highlight clip is marked as 4), and the highlight clips of Video 3 are [1s, 5s] (the highlight clip is marked as 5) and the 12th second (the highlight clip is marked as 6).

[0355] Then, based on the analysis results of all material files, the material files selected by the user are edited and filtered to retain the highlight clips.

[0356] S133, the application function layer of the application layer calls the theme summary interface of the media middle platform framework to request to obtain the theme template.

[0357] Among them, the request passes through the topic summary interface and channel interface of the media middle platform framework, and is transmitted to the HAL layer through the FKW layer.

[0358] S134: The HAL layer determines a theme template that matches the scene according to the scene in the highlight segment.

[0359] S135, the HAL layer returns the determined theme template to the theme summary interface of the media middle platform framework layer, and the theme summary interface returns the theme template to the application function layer of the application layer.

[0360] For example, assuming that most of the scenes in the highlight clips are parent-child scenes, it can be determined that the theme template that matches the scene is a parent-child theme.

[0361] S136 , the application function layer of the application layer sends the theme template, the edited video clips, and the screened pictures to the basic capability layer.

[0362] S137 , the basic capability layer splices the edited video clips and the screened pictures, and applies a theme template to obtain a second video.

[0363] For example, refer to Figure 13 , you can splice picture 1, [20s, 30s] of video 1, [5s, 10s] and [20s, 25s] of video 2, and [1s, 5s] and the 12th second of video 3, i.e., splice highlight clips 1, 2, 3, 4, 5, and 6. Alternatively, apply a parent-child themed template to the spliced ​​video to obtain a second video.

[0364] S138: The basic capability layer instructs the video editing service layer to display the second video.

[0365] S139, the video editing service layer displays the second video in the gallery interface.

[0366] In an embodiment of the present application, the media middle platform framework layer is used to decode video and image files, convert data formats into a unified format, monitor remaining time and adjust operation strategies, send data, control algorithm operation and termination, obtain results and return to the application layer, etc. The FKW layer is used to complete data packaging and provide data and program operation services. After receiving the command sent by the media middle platform framework layer, the HAL layer performs highlight analysis according to the command, and returns the parameter calculation results of the highlight analysis to the media middle platform framework layer. The final result of the algorithm is collected and organized by the media middle platform framework layer, and then sent to the application layer for processing. The application layer can present the editing application interface, video and image file options, and present the final result of the algorithm.

[0367] After the user starts the one-click film-making function, select the video and picture files that need to be edited (for example, a maximum of 30 files are supported). After waiting for a while, the "one-click film-making" application automatically edits to the highlight clips of the video, and combines the highlight clips and pictures according to the algorithm results to generate the edited short video, which can be previewed and played.

[0368] The video processing method provided in the embodiment of the present application preliminarily determines an expected total analysis time (equal to the sum of the expected analysis times of all videos) based on the video decoding speed and the chip analysis speed in a scenario where multiple video material files with different video lengths are selected for one-click film formation. Then, based on the expected total analysis time, the actual analysis time of each video material file is reasonably allocated according to the time allocation algorithm, thereby quickly completing the video analysis, extracting highlight clips or wonderful moments in the video for video clip editing and splicing, completing one-click film formation, and improving the user experience.

[0369] It should also be noted that, in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0370] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the scope of protection of this application.

[0371] It can be understood that the methods and operations implemented by electronic devices in the above-mentioned various method embodiments can also be implemented by components (such as chips or circuits) that can be used in electronic devices.

[0372] The above describes the method embodiments provided by this application, and the following describes the device embodiments provided by this application. It should be understood that the description of the device embodiments corresponds to the description of the method embodiments. Therefore, for matters not described in detail, reference can be made to the method embodiments above. For the sake of brevity, they will not be repeated here.

[0373] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of this application.

[0374] In the embodiment of the present application, the electronic device can be divided into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other feasible division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0375] Figure 14 This is a schematic block diagram of a video processing device 200 provided in an embodiment of the present application. The device 200 can be used to perform the actions performed by the electronic device in the above method embodiment. The device 200 includes a display unit 210, an acquisition unit 220, and a processing unit 230.

[0376] The display unit 210 is configured to load and display the videos and pictures in the gallery in response to a first operation by the user;

[0377] An acquisition unit 220 is configured to, in response to a user selecting an image material in the image library and confirming an operation to perform image stitching, acquire a first parameter based on the image material; wherein the image material includes only videos or includes videos and pictures; the first parameter includes the total number of videos in the image material, the duration of each video, the sum of the durations of all videos, the analysis speed of each video, the sum of the analysis speeds of all videos, and the sum of the expected analysis durations of all videos;

[0378] The processing unit 230 is configured to perform the following operations on each video in the image material: determining an actual analysis time allocated to a first video according to the first parameter; wherein the first video is any video in the image material;

[0379] The processing unit 230 is further configured to, after determining the actual analysis durations of all videos, perform the following operations on each video in the image material: using the actual analysis duration allocated to the first video, analyzing highlight segments in the first video;

[0380] The processing unit 230 is further configured to extract all highlight segments from the image material after analyzing all videos, and to splice all highlight segments to obtain a second video;

[0381] The actual analysis duration allocated to each video includes a first duration and a second duration, wherein the first duration is obtained by averaging a portion of the sum of the expected analysis durations of all videos, and the second duration is determined based on a ratio between the duration of each video in the image material and the sum of the durations of all videos, and a ratio between the analysis speed of each video in the image material and the sum of the analysis speeds of all videos;

[0382] The sum of the actual analysis durations allocated to all videos in the image material is equal to the sum of the expected analysis durations of all videos.

[0383] In some embodiments, the processing unit 230 is specifically configured to:

[0384] The actual analysis time allocated to the first video is calculated using the following equation: n =T n1 +T n2 ; Among them, T n1 Indicates the first duration, T n2 represents the second duration;

[0385] The first duration is calculated using the following equation: n1 =T V ×δ / n; where T v represents the sum of the expected analysis durations of all videos in the image material, δ represents the proportion coefficient, n represents the total number of all videos in the image material, and δ is greater than 0 and less than 1;

[0386] The second duration is calculated using the following equation:

[0387] T n2 =T V ×(1-δ)×(W n / W A );

[0388] W n =log2(1.5+L n / L A ) / log2(1.5+S n / S A );

[0389] W A =W1+…+W n ;

[0390] Wherein, W1 represents the weight value of the first video in the image material, W n Indicates the weight value of the nth video in the image material, W A Represents the sum of the weight values ​​of all videos in the image material, Ln / L A Indicates the length L of the nth video in the image material n The sum of the duration of all videos L A The ratio between n / S A Indicates the analysis speed S of the nth video in the image material n The sum of the analysis speeds of all videos S A The ratio between them.

[0391] In some embodiments, the processing unit 230 is further configured to determine the analysis speed of the first video according to the video decoding speed and the chip analysis speed of the first video using the following equation: n =min(V1, V2)×α. Where, S n represents the analysis speed of the first video, V1 represents the video decoding speed of the first video, V2 represents the chip analysis speed, α represents the tolerance factor, min() is used to obtain the minimum value, and α is greater than 0 and less than 1.

[0392] In some embodiments, the processing unit 230 is further configured to obtain the video decoding speed of the first video according to the resolution and frame rate of the first video using the following equation: V1=V d ×(R d / R A )×(F d / F A ); wherein V1 represents the video decoding speed of the first video, V d Indicates the preset video decoding speed, R d Indicates the preset video resolution, F d Indicates the preset video frame rate; R A represents the video resolution of the first video, F A Indicates the video frame rate of the first video.

[0393] In some embodiments, after the processing unit 230 stitches all the highlight segments to obtain the second video, the display unit 210 is further used to: display the second video in a gallery.

[0394] The video processing device provided in the embodiment of the present application preliminarily determines an expected total analysis time based on the video decoding speed and the chip analysis speed in a scenario where multiple video material files with different video lengths are selected for one-click film formation. Then, based on the expected total analysis time, the actual analysis time of each video material file is reasonably allocated according to the time allocation algorithm, thereby quickly completing the video analysis, extracting highlight clips or wonderful moments in the video for use in video clip editing and splicing, completing one-click film formation, and improving the user experience.

[0395] According to the embodiment of the present application, the device 200 may correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of the units in the device 200 are respectively for implementing the corresponding processes of the method. For the sake of brevity, they will not be repeated here.

[0396] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0397] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute computer programs or instructions stored in a memory, so that the methods in each embodiment are executed.

[0398] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the video processing method in the above-mentioned embodiment.

[0399] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program code. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the video processing method in the above-mentioned embodiment.

[0400] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the video processing method in the above-mentioned method embodiments.

[0401] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0402] The embodiments of the present application do not specifically limit the specific structure of the execution subject of the method provided in the embodiments of the present application; as long as the program containing the code of the method provided in the embodiments of the present application can be run to perform video processing according to the method provided in the embodiments of the present application, the execution subject of the method provided in the embodiments of the present application can be an electronic device, or a functional module in the electronic device that can call and execute the program.

[0403] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0404] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0405] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0406] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer software product, which is stored in a storage medium. The computer software product includes a number of instructions that enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium may include, but is not limited to: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0407] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video processing method, characterized in that: include: In response to the user's first operation, loading and displaying videos and pictures in the gallery; In response to a user selecting an image material in the gallery and confirming an operation to perform image stitching, obtaining a first parameter based on the image material; wherein the image material includes a video; the first parameter includes the total number of videos in the image material, the duration of each video, the sum of the durations of all videos, the analysis speed of each video, the sum of the analysis speeds of all videos, and the sum of the expected analysis durations of all videos; Perform the following operations on each video in the image material: determine, based on the first parameter, an actual analysis time allocated to a first video; wherein the first video is any video in the image material; After determining the actual analysis duration of all videos, performing the following operations on each video in the image material: using the actual analysis duration allocated to the first video, analyzing highlight segments in the first video; After analyzing all videos, extracting all highlight segments from the image material, and splicing all the highlight segments to obtain a second video; The actual analysis duration allocated to each video includes a first duration and a second duration, wherein the first duration is obtained by averaging a portion of the sum of the expected analysis durations of all videos; and the second duration is determined based on a ratio between the duration of each video in the image material and the sum of the durations of all videos, and a ratio between the analysis speed of each video in the image material and the sum of the analysis speeds of all videos. The sum of the actual analysis durations allocated to all videos in the image material is equal to the sum of the expected analysis durations of all videos.

2. The method according to claim 1, characterized in that The determining, based on the first parameter, an actual analysis duration allocated to the first video includes: The actual analysis time allocated to the first video is calculated using the following equation: n =T n1 +T n2 ; Among them, T n1 represents the first duration, T n2 represents the second duration; The first duration is calculated using the following equation: n1 =T V ×δ / n; Among them, T v represents the sum of the expected analysis durations of all videos in the image material, δ represents the proportion coefficient, n represents the total number of all videos in the image material, and δ is greater than 0 and less than 1; The second duration is calculated using the following equation: T n2 =T V ×(1-δ)×(W n / W A ); W n =log2(1.5+L n / L A ) / log2(1.5+S n / S A ); IN A =W1+…+W n ; Wherein, W1 represents the weight value of the first video in the image material, W n Indicates the weight value of the nth video in the image material, W A Represents the sum of the weight values ​​of all videos in the image material, L n / L A Indicates the length L of the nth video in the image material n The sum of the duration of all videos L A The ratio between n / S A Indicates the analysis speed S of the nth video in the image material n The sum of the analysis speeds of all videos S A The ratio between them.

3. The method according to claim 1, characterized in that Before acquiring the first parameter according to the image material, the method further includes: The analysis speed of the first video is determined based on the video decoding speed and the chip analysis speed of the first video using the following equation: S n =min(V1,V2)×α; Among them, S n represents the analysis speed of the first video, V1 represents the video decoding speed of the first video, V2 represents the chip analysis speed, α represents the tolerance factor, min() is used to obtain the minimum value, and α is greater than 0 and less than 1.

4. The method according to claim 3, characterized in that Before determining the analysis speed of the first video according to the video decoding speed and the chip analysis speed of the first video, the method further includes: The video decoding speed of the first video is obtained according to the resolution and frame rate of the first video using the following equation: V1=V d ×(R d / R A )×(F d / F A ); Wherein, V1 represents the video decoding speed of the first video, V d Indicates the preset video decoding speed, R d Indicates the preset video resolution, F d Indicates the preset video frame rate; R A represents the video resolution of the first video, F A Indicates the video frame rate of the first video.

5. The method according to any one of claims 1 to 4, characterized in that Before analyzing the highlight segments in the first video using the actual analysis duration allocated to the first video, the method further includes: Determine that the strategy for analyzing the first video is the first analysis strategy, which is a full analysis strategy, an intensive key segment analysis strategy, a sparse key segment analysis strategy, or a simple analysis strategy based on I frames; wherein the full analysis strategy includes every frame of the video in the analysis content, including all highlight segments; the intensive key segment analysis strategy is an algorithm strategy that calculates the number of analysis segments and the analysis segment duration of the material video to obtain a reasonable interval duration between adjacent segments, and then evenly distributes each analysis segment so that the analysis segments cover highlight segments at different positions as much as possible; the sparse key segment analysis strategy is a strategy algorithm that makes an unequal distribution of the interval duration of adjacent segments, and the algorithm analyzes segments mainly distributed in the front and middle sections of the video, with appropriate distribution in the tail section; the simple analysis strategy based on I frames means that when the allocated analysis duration can only meet the time for analyzing three pictures, three pictures located at different positions of the video are analyzed, the one with the highest score among the three pictures is used as the starting point of the highlight segment result, and a video segment of a certain length is returned as the result; The step of using the actual analysis time allocated to the first video to analyze the highlight segments in the first video includes: using the actual analysis time allocated to the first video to analyze the highlight segments in the first video according to the first analysis strategy.

6. The method according to claim 5, characterized in that After determining that the strategy for analyzing the first video is the first analysis strategy, the method further includes: Performing a preset process on the first video to obtain a processed first video; wherein the preset process includes at least one of the following: decoding, format conversion, and resolution reduction; According to the first analysis strategy, using the actual analysis time allocated to the first video, analyzing the highlight segments in the first video includes: According to the first analysis strategy, the highlight segments in the processed first video are analyzed using the actual analysis duration allocated to the first video.

7. The method according to any one of claims 1 to 4, characterized in that The analyzing the highlight segment in the first video includes: Comparing each frame of the first video with the preset image features frame by frame; When the continuous multiple frames of images in the first video contain the preset image feature, it is determined that the first video contains a first highlight segment, and the start time and end time of the first highlight segment are determined.

8. The method according to claim 7, characterized in that The step of extracting all highlight segments from the image material includes: Extract the first highlight segment from the first video according to the start time and end time of the first highlight segment.

9. The method according to any one of claims 1 to 4, characterized in that The image material also includes a picture; the method further includes: Perform the following operations on each image in the image material: A preset analysis time is used to analyze a first picture, where the first picture is any one of the image materials.

10. The method according to claim 9, characterized in that The analyzing the first picture includes: When the first picture contains a preset image feature, the first picture is determined to be a highlight segment.

11. The method according to any one of claims 1 to 4, characterized in that The step of extracting all highlight segments from the image material and splicing all highlight segments to obtain a second video includes: In a case where the image material includes videos and pictures, all highlight segments are extracted from all videos and all pictures in the image material, and all the highlight segments are spliced ​​together to obtain the second video.

12. The method according to any one of claims 1 to 4, characterized in that After splicing all the highlight segments to obtain the second video, the method further includes: The second video is displayed in the gallery.

13. The method according to claim 12, characterized in that After displaying the second video in the gallery, the method further includes: In response to the user's second operation, the second video is post-processed, and the post-processing includes adding a theme, adding background music, re-editing, and / or video sharing.

14. The method according to claim 2, characterized in that δ is taken as 0.

5.

15. The method according to claim 3 or 4, characterized in that α is set to 0.

9.

16. An electronic device, characterized in that: The electronic device comprises a processor coupled to a memory, wherein the processor is configured to execute a computer program or instruction stored in the memory, so that the electronic device implements the method according to any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 15.

Images