Video processing monitoring scheduling method, device and storage medium

Abstract

The present application discloses a monitoring and scheduling method, device and storage medium for video processing, which belongs to the field of video data technology. The method includes: in response to the operation of splicing multiple file materials into a target video, performing a video analysis operation on at least one video in the multiple material files; in the process of performing the video analysis operation on at least one video, monitoring the analyzed duration of the first video being analyzed, if it is determined that the first video analysis has timed out based on the analyzed duration of the first video, then stopping the analysis of the first video, and determining the position of the highlight segment of the first video based on the analyzed result of the first video. In this way, the video analysis can be completed quickly, avoiding the video analysis time being too long, ensuring that the video analysis is completed within a reasonable time limit, extracting the highlight segments in the video, and using them for video segment editing and splicing, completing one-click filming, and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of video data technology, and in particular to a monitoring and scheduling method, device, and storage medium for video processing. Background Art

[0002] Currently, camera functions are common services provided by electronic devices, allowing users to take photos or videos. If users need to process the captured photos or videos, such as stitching together multiple photos or combining photos and videos, they typically need to use electronic devices or third-party image processing software to perform the stitching.

[0003] For example, the one-click video editing service provided by electronic devices can automatically analyze multiple videos and / or images selected by the user through an algorithm, extracting highlights (i.e., exciting moments such as a character's smile or a championship moment) from the source videos, and automatically generate a pre-edited video. However, during the analysis of the multiple source files entered by the user, some videos may take too long to analyze, resulting in low efficiency of the one-click video editing service and a poor user experience. Summary of the Invention

[0004] This application provides a monitoring scheduling method, device, and storage medium for video processing, which can avoid excessive video analysis time, ensure that video analysis is completed within a reasonable time limit, and extract highlight clips from the video. The technical solution is as follows:

[0005] In a first aspect, a monitoring and scheduling method for video processing is provided, characterized in that it is applied to an electronic device and includes:

[0006] A first operation is received, where the first operation is used to trigger splicing of multiple file materials into a target video, where the multiple material files include at least one video; in response to the first operation, a video analysis operation is performed on the at least one video, where the video analysis operation is used to analyze highlight segments of each video in the at least one video; while performing the video analysis operation on the at least one video, the analyzed duration of the first video is monitored, where the first video is the video being analyzed in the at least one video; if it is determined that the analysis of the first video has timed out based on the analyzed duration of the first video, the analysis of the first video is stopped, and the position of the highlight segment of the first video is determined based on the analyzed result of the first video.

[0007] Among them, the first operation can be that the user selects multiple file materials in the gallery and confirms the operation of performing image stitching. Before receiving the first operation, the electronic device can also receive the user's second operation, and in response to the second operation, load and display the videos and pictures in the gallery.

[0008] The monitoring and scheduling method for video processing provided in the embodiments of the present application can monitor and schedule the video analysis process using an adaptive monitoring and scheduling algorithm during the process of performing an analysis operation on the video, so as to ensure that the video analysis is completed within a reasonable time limit and improve the efficiency of video analysis. For example, during the process of performing an analysis operation on the video, the analyzed duration of the video can be monitored. If the video analysis timeout is determined based on the analyzed duration of the video, the video analysis is stopped and the location of the highlight segment of the video is determined based on the analyzed result of the video. In this way, the video analysis time can be avoided from being too long, and the video analysis can be completed within a reasonable time limit to extract the highlight segment of the video.

[0009] As an example, determining the highlight segment position of the first video based on the analysis result of the first video includes: if the highlight segment position of the first video exists in the analyzed result of the first video, obtaining the highlight segment position of the first video existing in the analyzed result of the first video; if the highlight segment position of the first video does not exist in the analyzed result of the first video, analyzing the first video using a second analysis strategy different from the first analysis strategy, the first analysis strategy being the analysis strategy used to analyze the first video before stopping the analysis of the first video, and the video analysis time of the second analysis strategy is less than the video analysis time of the first analysis strategy.

[0010] By adopting a second analysis strategy with less complexity to analyze the first video when the highlight segment position of the first video does not exist in the analyzed results of the first video, the analysis time of the first video can be saved and the video analysis efficiency can be improved.

[0011] As an example, determining the first video analysis timeout according to the analyzed duration of the first video includes:

[0012] If the analyzed duration of the first video meets the first preset condition, determining that the first video analysis has timed out;

[0013] The first preset condition includes one or more of the following conditions:

[0014] The analyzed duration of the first video is greater than or equal to the product of the allocated duration of the first video and the first preset ratio, where the allocated duration of the first video is the analysis duration allocated to the first video in advance based on the expected analysis duration of at least one video;

[0015] The analyzed duration of the first video is greater than or equal to the product of the maximum timeout duration tolerance of at least one video and the first value, and the maximum timeout duration tolerance of at least one video is used to indicate the allowed timeout duration of each video in the at least one video.

[0016] As an example, the first value is the product of the number of videos of the at least one video and a first preset value.

[0017] As an example, determining before the first video analysis times out further includes:

[0018] Determining a maximum timeout tolerance for at least one video based on an expected analysis time of the at least one video;

[0019] The maximum tolerance of the timeout duration is the product of the expected analysis duration and the second preset ratio, or the larger value of the first preset duration.

[0020] As an example, an embodiment of the present application can also monitor the analyzed duration of at least one video during the process of performing a video analysis operation on at least one video; after the first video analysis is completed, if it is determined that the overall analysis of at least one video has timed out based on the analyzed duration of at least one video, a third analysis strategy is used to analyze the unanalyzed videos in the at least one video in turn to obtain the highlight segment position of each video in the unanalyzed video.

[0021] Among them, the third analysis strategy is a fixed-length analysis strategy, which refers to an analysis strategy that analyzes a preset position in the video as a highlight segment position.

[0022] By adopting a fixed-length analysis strategy to analyze the remaining unanalyzed videos when the overall video analysis times out, the video analysis speed can be further accelerated and the video analysis efficiency can be improved.

[0023] The first video analysis ends in either of the following two situations:

[0024] If it is determined that the first video analysis has exceeded the time limit according to the video analysis time of the first video, the first video analysis is stopped;

[0025] When it is determined that the analysis of the first video has not exceeded the time limit according to the video analysis time of the first video, the analysis of the first video is completed.

[0026] As an example, determining at least one video overall analysis timeout based on the analyzed duration of at least one video includes:

[0027] If the analyzed duration of at least one video meets a second preset condition, determining that the overall analysis of at least one video has timed out;

[0028] The second preset condition includes one or more of the following conditions:

[0029] The analyzed duration of the at least one video is greater than or equal to the product of the expected analysis duration of the at least one video and a third preset ratio;

[0030] The analyzed duration of the at least one video is greater than or equal to the sum of the expected analysis duration of the at least one video and a second preset duration.

[0031] As an example, before performing the video analysis operation on at least one video, the method further includes:

[0032] Allocating an analysis time for each video in the at least one video according to an expected analysis time of the at least one video, the analysis time allocated to each video being an allocated time for each video;

[0033] determining an analysis strategy for each of the at least one video based on an allocated duration of each of the at least one video;

[0034] Perform video analysis operations on at least one video, including:

[0035] A video analysis operation is performed on the at least one video according to the analysis strategy for each of the at least one video.

[0036] By reasonably allocating the analysis time for each video based on the expected analysis time, and setting the analysis strategy for each video according to the allocated time for each video, you can make full use of the analysis time and analysis strategy to quickly complete video analysis.

[0037] As an example, if it is determined based on the analyzed duration of at least one video that at least one video has not been analyzed as a whole and has timed out, then based on the analyzed duration of the at least one video and the expected analysis duration, it is determined whether the analysis progress of the at least one video is behind schedule; if the analysis progress of the at least one video is behind schedule, the unanalyzed video included in the at least one video is treated as at least one video, and the expected analysis duration of the at least one video is re-planned; based on the re-planned expected analysis duration, the process returns to the step of performing the video analysis operation on the at least one video.

[0038] Among them, according to the re-planned expected analysis time, returning to the step of performing a video analysis operation on at least one video, including: according to the re-planned expected analysis time, re-allocating the analysis time for each video in the at least one video, and the re-allocated analysis time for each video is the re-planned allocated time for each video; according to the re-planned allocated time for each video in the at least one video, re-determining the analysis strategy for each video in the at least one video, and the re-determined analysis strategy for each video is the re-planned analysis strategy; according to the re-planned analysis strategy for each video in the at least one video, performing a video analysis operation on the at least one video.

[0039] In this way, we can ensure that the analysis is completed within a reasonable time limit while dynamically planning the analysis time allocation and analysis strategy allocation, making full use of the analysis time and analysis strategy to achieve better analysis results and improve video analysis efficiency.

[0040] Among them, judging whether the analysis progress of at least one video is behind schedule based on the analyzed time and expected analysis time of at least one video, including: judging whether the analysis progress of at least one video is behind schedule based on the analyzed time, expected analysis time and maximum tolerance of timeout time of at least one video, and the total allocated time of unanalyzed videos in at least one video.

[0041] For example, if the difference between the second value and the expected analysis time of at least one video is greater than or equal to the ratio of the maximum tolerance of the timeout time of at least one video to the second preset value, it is determined that the analysis progress of at least one video is behind schedule, and the second value is the sum of the analyzed time of at least one video and the total allocated time of unanalyzed videos in at least one video.

[0042] As an example, the first video may be subjected to preset processing to obtain a processed first video; wherein the preset processing includes at least one of the following: decoding, format conversion, and resolution reduction; and then the processed first video is analyzed.

[0043] As an example, multiple materials also include at least one picture. Before performing the video analysis operation on at least one video, it also includes: analyzing at least one picture in turn to obtain an analysis result for each picture in at least one picture, and the analysis result of each picture is used to indicate whether the corresponding picture is a highlight clip.

[0044] As an example, the method further includes: when the multiple materials include videos and pictures, extracting all highlight segments from all videos and all pictures in the multiple materials, and splicing all highlight segments to obtain the target video.

[0045] As an example, after all highlight clips are spliced ​​together to obtain a target video, the process further includes: displaying the target video in a gallery.

[0046] As an example, after displaying the target video in the gallery, it also includes: in response to the user's third operation, post-processing the target video, the post-processing includes adding a theme, adding background music, re-editing, and / or video sharing.

[0047] In a second aspect, a video processing monitoring and scheduling device is provided, wherein the video processing monitoring and scheduling device has the function of implementing the video processing monitoring and scheduling method described in the first aspect. The video processing monitoring and scheduling device includes at least one module, and the at least one module is configured to implement the video processing monitoring and scheduling method described in the first aspect.

[0048] In a third aspect, a video processing monitoring and scheduling device is provided, wherein the structure of the video processing monitoring and scheduling device includes a processor and a memory, wherein the memory is used to store a program that supports the video processing monitoring and scheduling device to execute the video processing monitoring and scheduling method provided by the first aspect, and to store data involved in implementing the video processing monitoring and scheduling method described in the first aspect. The processor is configured to execute the program stored in the memory. The video processing monitoring and scheduling device may also include a communication bus, which is used to establish a connection between the processor and the memory.

[0049] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the video processing monitoring and scheduling method described in the first aspect.

[0050] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the video processing monitoring and scheduling method described in the first aspect.

[0051] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application;

[0054] Figure 3 Schematic diagram of the application scenario of the video processing method provided in the embodiment of the present application Figure 1 ;

[0055] Figure 4 Schematic diagram of the application scenario of the video processing method provided in the embodiment of the present application Figure 2 ;

[0056] Figure 5 Schematic diagram of the application scenario of the video processing method provided in the embodiment of the present application Figure 3 ;

[0057] Figure 6 Schematic diagram of the application scenario of the video processing method provided in the embodiment of the present application Figure 4 ;

[0058] Figure 7 This is a flowchart of a monitoring scheduling method for video processing provided by an embodiment of the present application;

[0059] Figure 8 is a schematic diagram of the analysis strategy provided in the embodiment of the present application;

[0060] Figure 9 The timing of a video processing monitoring scheduling method provided by the embodiment of the present application is Figure 1 ;

[0061] Figure 10 The timing of a video processing monitoring scheduling method provided by the embodiment of the present application is Figure 1 ;

[0062] Figure 11 The timing of a video processing monitoring scheduling method provided by the embodiment of the present application is Figure 1 ;

[0063] Figure 12 The timing of a video processing monitoring scheduling method provided by the embodiment of the present application is Figure 1 ;

[0064] Figure 13 Schematic diagram of highlight analysis of the monitoring scheduling method for video processing provided by an embodiment of the present application;

[0065] Figure 14 It is a schematic diagram of the video processing process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0068] In the specification and claims herein, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. In the description of the embodiments of this application, unless otherwise specified, the meaning of "plurality" refers to two or more. For example, "multiple processing units" refers to two or more processing units, etc.; "multiple components" refers to two or more components, etc.

[0069] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0070] Currently, camera functions are common services provided by electronic devices, allowing users to take photos or videos. If users need to process the captured photos or videos, such as stitching together multiple photos or combining photos and videos, they typically need to use electronic devices or third-party image processing software to perform the stitching.

[0071] For example, consider the one-click video editing service offered by electronic devices. This service automatically analyzes multiple videos and / or images selected by the user through an algorithm, extracting highlights (i.e., exciting moments, such as a character's smile or a championship moment) from the source videos and generating a single, edited video. However, during the analysis of the multiple source files entered by the user, some videos may take too long to analyze, resulting in low efficiency and prolonged user waiting, impacting the user experience.

[0072] In response to the above problems, the embodiments of the present application provide a video processing method and electronic device. In the scenario of one-click filming of multiple material files containing videos, an adaptive monitoring and scheduling algorithm can be used to monitor and schedule the video analysis process during the process of performing analysis operations on the video, so as to ensure that the video analysis is completed within a reasonable time limit and improve the efficiency of video analysis. For example, in the process of performing analysis operations on the video, the analyzed time of the video can be monitored. If the video analysis timeout is determined based on the analyzed time of the video, the analysis of the video is stopped, and the position of the highlight segment of the video is determined based on the analyzed results of the video. In this way, the video analysis time can be avoided from being too long, and the video analysis can be ensured to be completed within a reasonable time limit. The embodiments of the present application improve the user experience by making improvements at the bottom layer of the electronic device system.

[0073] The video processing method provided in the embodiment of the present application is applied to electronic devices. Electronic devices include electronic devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The electronic devices can be mobile phones, smart TVs, wearable devices, tablet computers (Pad), computers with wireless transceiver functions, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control (industrial control), wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the electronic devices.

[0074] See also Figure 1 , is a structural diagram of an electronic device (taking a mobile phone as an example) provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, and the like.

[0075] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0076] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. For example, the processor 110 is used to execute the ambient light detection method in the embodiment of the present application.

[0077] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0078] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0079] The external memory 120 generally refers to an external memory. In the embodiment of the present application, the external memory refers to a memory other than the memory of the electronic device and the cache of the processor, and the memory is generally a non-volatile memory.

[0080] Internal memory 121, also referred to as "memory," can be used to store computer-executable program code, including instructions. Internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (e.g., sound playback, image playback, etc.).

[0081] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0082] The electronic device 100 also includes various sensors that can convert various physical signals into electrical signals. For example, the pressure sensor 180A is used to sense and convert pressure signals into electrical signals. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall effect sensor. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc. The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. The bone conduction sensor 180M can obtain a vibration signal.

[0083] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0084] For example, in an embodiment of the present application, the touch sensor 180K can detect a user's click operation on an application icon, and pass the detected click operation to the application processor, determine that the click operation is used to start or run the application, and then execute the running operation of the application.

[0085] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0086] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0087] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0088] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0089] The above is a specific description of the embodiments of the present application using the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in the figure may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0090] The electronic device provided in the embodiments of the present application can be user equipment (UE), for example, it can be a mobile terminal (such as a user's mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), etc.

[0091] Furthermore, operating systems run on the above components, such as the iOS operating system developed by Apple, the Android open-source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on these operating systems.

[0092] The operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0093] Figure 2 Schematic diagram of the software structure of an electronic device 100 provided in an embodiment of the present application.

[0094] like Figure 2 As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom into: application (APP) layer, media middle platform framework layer, application framework (FWK) layer, and hardware abstraction layer (HAL).

[0095] The application layer, referred to as the application layer, can include a series of application packages, such as camera, gallery, calendar, map, and navigation. When these application packages are running, they can access the various service modules provided by the media platform framework layer and the application framework layer through the application programming interface (API) and execute corresponding intelligent services.

[0096] In some embodiments, the camera is used to capture photos, videos, slow-motion images, and panoramic images in response to user operations. After these images are captured by the camera, or after the user triggers the phone to take a screenshot, or after the user triggers the phone to record the screen, or after the electronic device downloads images from other devices, the electronic device can save these images in a gallery, so that the user can perform video editing operations on the images in the gallery, such as a one-click movie operation.

[0097] The embodiment of the present application divides the gallery into the following layers from top to bottom: business layer, application function layer and basic function layer.

[0098] The service layer offers a variety of services, including automatic multi-camera video generation, AI-generated music videos, one-click video generation, and highlight moments. These services are presented as controls in the gallery's user interface (UI). By operating a control, the user can trigger the camera to perform the corresponding video processing action. For example, after the user selects one or more source videos and clicks the one-click video generation control in the gallery, the gallery automatically analyzes and extracts the highlights from the source videos through an algorithm, and combines these highlights into a single, edited video.

[0099] The application function layer includes an automatic editing framework. Each business in the business layer can call the automatic editing framework to provide automatic editing services for pictures and videos. For example, the automatic editing framework may include functional modules such as segment selection, storyline organization, layout splicing, and special effects beautification. Segment selection is used to call the highlight segment analysis interface and the policy monitoring interface to extract highlight segments from the material video. Storyline organization is used to sequentially splice multiple material videos in the form of a storyline based on the content of the material video. Layout splicing is used to modulate the interface layout of the material video. Special effects beautification is used to adjust the beautification effect of the video, such as adjusting the brightness of the picture and beautifying the face of the character.

[0100] The basic function layer is used to perform basic function processing on the edited video clips after the automatic editing framework edits multiple material videos. Exemplarily, the basic function layer may include basic function modules such as video splicing, synthesis and preservation, video effect rendering, and audio effect processing. Among them, the video splicing user splices multiple extracted highlight clips. Synthesis and preservation are used to store the video set obtained after splicing. Video effect rendering is used to add video effects to the video set obtained after splicing, such as adding style filters and themes to the video. Audio effect processing is used to add sound effects to the video set obtained after splicing, such as adding background music.

[0101] The media platform framework layer is a software layer located between the application layer and the application framework. The media platform framework layer may include an analysis performance query interface, a highlight analysis interface, a policy monitoring interface, a pipeline interface, and a topic summary interface. The analysis performance query interface is used to calculate the total duration of all source videos based on the video analysis speed. The highlight analysis interface is used to call the policy monitoring interface to extract highlight clips. The policy monitoring interface is used to configure an expected analysis duration for each source video based on the total duration of all source videos. It also dynamically sets analysis policies for each source video based on parameters such as the expected analysis duration and the duration of each video. The policy monitoring interface also uses an adaptive monitoring scheduling algorithm to monitor and schedule the video analysis process to ensure that video analysis is completed within a reasonable time limit. The pipeline interface is used to downscale video files based on the file description issued by the policy monitoring interface to ensure that it complies with the analysis policy. The data address of the downscaled video file is forwarded to the hardware abstraction layer through the application framework. The analysis results of the highlight clips returned by the hardware abstraction layer are then reported to the policy monitoring interface. The topic summary interface is used to obtain topics corresponding to the content of the highlight clips.

[0102] The application framework layer, also referred to as the framework layer, supports the operation of various modules in the media platform framework layer. For example, the framework layer may include a one-click video creation interface, a parameter management interface, an image data transmission interface, a theme analysis interface, and a performance analysis interface.

[0103] The hardware abstraction layer (HAL) encapsulates the Linux kernel driver and provides an interface to the upper layer. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable across multiple platforms. For example, the HAL may include a highlight segment analysis algorithm (also known as a highlight segment algorithm), a face detection algorithm, a video acceleration algorithm, and an image super-resolution algorithm. The highlight segment analysis algorithm is an image processing algorithm provided by the image chip. This algorithm scores each image frame based on image color, image texture characteristics, image quality, frame interpolation with previous and next frames, and edge change rate. The scoring result can be used to determine whether an image frame is a highlight segment. For example, if the score of an image frame is greater than or equal to 60, the image frame is a highlight segment.

[0104] It should be noted that Figure 2 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more layers than shown, such as a system library (FWK LIB) layer and a kernel layer. In addition, each layer may include more or fewer components than shown, which is not limited in this application.

[0105] The kernel uses file descriptors to access files. File descriptors are non-negative integers. When you open an existing file or create a new file, the kernel returns a file descriptor. Reading and writing files also require a file descriptor to specify the file to be read or written.

[0106] It is understandable that in order to implement the video processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0107] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0108] The following briefly summarizes the monitoring and scheduling method for video processing provided by the embodiment of the present application in combination with the above software architecture.

[0109] In an embodiment of the present application, the one-click film-making service is located in the video editing APP business layer and is a sub-business of video editing. Multiple materials to be analyzed are selected at the business layer and the materials to be analyzed are sent to the application function layer.

[0110] At the application layer, the one-click film-making service is implemented based on an automatic editing framework, which includes basic implementation functions such as clip selection, storyline organization, layout splicing, and / or special effects beautification. At the application layer, based on the transmitted material to be analyzed, parameters such as the expected total analysis duration, the minimum interval duration between highlight clips, and the maximum highlight clip duration (for example, 20 seconds) are generated and then sent to the media center framework layer.

[0111] The media middle platform framework layer is mainly responsible for basic data processing for the one-click film-making business, and mainly includes modules such as image highlight segment analysis, theme application, analysis performance query, and policy monitoring. The highlight segment analysis interface and policy monitoring module mainly control the analysis position and analysis duration of the actual highlight segment of the video material, decode and analyze according to the duration and position allocated by the policy algorithm, and use an adaptive monitoring and scheduling algorithm to monitor and schedule the video analysis process. The content protected by this application is: The policy monitoring module will use an adaptive monitoring and scheduling algorithm to monitor and schedule the video analysis process to ensure that the video analysis is completed within a reasonable time limit and improve the efficiency of video analysis.

[0112] After allocating analysis time for each video, the Strategy Monitoring Module assigns different analysis strategies based on the duration. Four main selection strategies are available. When selecting a large number of videos or relatively long clips, performing algorithmic analysis on all videos can lead to resource constraints and excessive analysis time. In this case, the Strategy Monitoring Module uses intensive and sparse analysis strategies to allocate and select analysis segments.

[0113] The media middle platform framework layer sends the processed video clip data to the HAL layer through the FWK layer, performs data processing at the HAL layer, and processes the data into data that can be recognized and calculated by the chip algorithm end. The data is then subjected to algorithmic analysis. The HAL layer then returns the analysis results to the media middle platform framework layer, and finally returns them to the application layer, which performs video splicing and presentation.

[0114] The execution subject of the video processing method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the video processing method, and the present application solution can be implemented by hardware and / or software. The specific implementation can be determined according to actual use requirements and is not limited by the embodiment of the present application. The video processing method provided in the embodiment of the present application is exemplarily described below with reference to the accompanying drawings using an electronic device as an example.

[0115] The following first introduces the video processing method provided by the embodiment of the present application in conjunction with specific application scenarios.

[0116] In an exemplary application scenario, a user uses a mobile phone to shoot multiple video materials in advance, and the mobile phone stores the multiple video materials in a gallery. Figure 3 As shown in (a) of FIG, a gallery icon is displayed on the mobile phone desktop. When the user wants the mobile phone to generate a video clip based on multiple video materials, the user can click the gallery icon on the mobile phone desktop. In response to the user's click operation on the icon, the mobile phone displays the gallery interface A1, as shown in FIG. Figure 3 As shown in (b) of FIG, the gallery interface A1 provides a "one-click blockbuster" option, such as Figure 3 As shown in (c) in FIG, the user can click on the “One-click blockbuster” option. In response to the user clicking on the “One-click blockbuster” option, the mobile phone displays the gallery interface A2, as shown in FIG. Figure 3 As shown in (d) in FIG, the gallery interface A2 may include multiple video materials that were recently shot, such as multiple video materials shot on November 10, 2022. The user can click on any one of the multiple video materials, such as Figure 3 As shown in (e) in the figure, the user clicks on the first video material among the multiple video materials. In response to the user's click operation on any video material, the mobile phone displays the gallery interface A3, which includes all the materials in the gallery. In this way, the user can select the video material required to generate the video clip in the gallery interface A3, for example, Figure 3 As shown in (f) in FIG, it is assumed that the user has selected 5 video materials. The gallery interface A3 also includes video generation options, such as Figure 3 As shown in (f), the video generation option is checked.

[0117] Refer again Figure 4 In (a), after selecting the video material, the user can click on the video generation option, such as clicking the check mark. Figure 4As shown in (b) of the figure, in response to the user clicking the check mark, the phone begins analyzing the five video clips selected by the user in Gallery interface A3, selecting highlight clips (i.e., wonderful clips, such as clips of the user's smiling face or the moment of victory) from each video clip, and generates a video clip based on the selected highlight clips. During this process, the phone can display the progress of the analysis in Gallery interface A3 so that the user can intuitively view the analysis progress.

[0118] In one example, if Figure 5 As shown in (a) in FIG, after the video clip is generated, the mobile phone displays the gallery interface A4, which includes the generated video clip, and the mobile phone can automatically play the video clip. In addition, the gallery interface A4 can provide a video export option, such as Figure 5 As shown in (b), the user can click on the video export option. In response to the user's click operation on the video export option, the mobile phone exports the video clip, as shown in FIG. Figure 5 As shown in (c) in the figure, the video clip is stored in the gallery, so that the user can view the video clip from the gallery.

[0119] In another exemplary application scenario, a user uses a mobile phone to pre-shoot multiple video clips and multiple picture clips, and the mobile phone stores the multiple video clips and multiple picture clips in a gallery. In this way, the user can select the video clips and picture clips in the gallery, and the mobile phone generates a video clip based on the video clips and picture clips.

[0120] In one example, if Figure 6 As shown in (a) in the figure, after entering the gallery interface A2, the gallery interface A2 includes multiple video materials and multiple picture materials that have been recently shot. The user can click on any one of the multiple video materials and multiple picture materials, such as the first video material. In response to the user's click operation on any of the materials, the mobile phone displays the gallery interface A3, which includes all the materials in the gallery. In this way, the user can select the materials required to generate the video clip in the gallery interface A3, such as Figure 6 As shown in (b) of FIG, it is assumed that the user has selected 4 video materials and 2 picture materials. The gallery interface A3 also includes video generation options, such as Figure 6 As shown in (b) in the figure, the video generation option is in the form of a check mark. Figure 6 In (c), after selecting the material, the user can click on the video generation option, such as clicking on the check mark option. Figure 6As shown in (d) in the figure, in response to the user clicking the check mark, the phone begins analyzing the four video clips and two image clips selected by the user in the gallery interface A3, selecting highlight clips from each video clip and highlight images (also known as wonderful images) from the two image clips. During this process, the phone can display the progress of the analysis in the gallery interface A3 so that the user can intuitively view the analysis progress.

[0121] Then, after the video clip is generated, the mobile phone displays a UI interface containing the video clip, in which the user is supported to preview the generated video clip. For example, in the scenario where the video material and the picture material are made into a film with one click, the UI interface for presenting the generated video clip can also refer to Figure 5 .like Figure 5 As shown in (a) in FIG, after the video clip is generated, the mobile phone displays the gallery interface A4, which includes the generated video clip, and the mobile phone can automatically play the video clip. In addition, the gallery interface A4 can provide a video export option, such as Figure 5 As shown in (b), the user can click on the video export option. In response to the user's click operation on the video export option, the mobile phone exports the video clip, as shown in FIG. Figure 5 As shown in (c) in the figure, the video clip is stored in the gallery, so that the user can view the video clip from the gallery.

[0122] In one example, after the video clip is generated, other function options may be provided in the application interface A4 displaying the video clip, so that the user can edit, add special effects, analyze, etc. the generated video clip based on these function options, for example Figure 5 As shown in (a) in the figure, the other function options may include but are not limited to templates, music, clips, sharing, etc. For example, the user can select function options according to needs to further process the generated video clips, such as selecting a theme template to apply to the generated video clips, adding music to the generated video clips, cutting out the generated video clips, or quickly sharing the generated video clips.

[0123] In one example, if the number of materials selected by the user is small, the mobile phone can prompt the user during the process of selecting materials so that the user can know how many materials are appropriate to select. Figure 3 As shown in (f) in the figure, a prompt message “more than 6 materials will produce better results” is displayed in the gallery interface A3, so that the user can know how many materials are needed to generate a better video clip.

[0124] In one example, if the number of materials selected by the user is large, the mobile phone can prompt the user during the process of selecting the materials so that the user can know how many materials can be selected at most. Figure 6 As shown in (b) in FIG, a prompt message “a maximum of 30 materials can be selected” is displayed in the gallery interface A3, so that the user can know the maximum number of materials that can be selected.

[0125] For ease of explanation, the video material and the picture material are collectively referred to as material files or materials to be analyzed. In some cases, the material to be analyzed includes only video material; in some cases, the material to be analyzed includes only picture material; in some cases, the material to be analyzed includes both video material and picture material.

[0126] It should be noted that in the embodiment of the present application, in the scenario where multiple source files containing videos selected by the user are spliced ​​into one video (i.e., using the one-click video generation function), it is necessary to analyze each video to select highlight clips (i.e., wonderful clips, such as clips of the user's smiling face, clips of the winning moment, etc.) from the video. Because the analysis time of some videos may be too long, which may lead to low efficiency of the one-click video generation service and affect the user experience, it is necessary to reasonably monitor and schedule the video analysis process to ensure that the video analysis is completed within a reasonable time limit and improve the efficiency of video analysis.

[0127] The following describes in detail how the present application solution monitors and schedules the video analysis process in the one-click video generation process of the above application scenario in conjunction with the accompanying drawings.

[0128] Figure 7 This is a flow chart of a video processing monitoring scheduling method provided by an embodiment of the present application, which is applied to electronic devices. Figure 7 , the method comprising:

[0129] Step 701: Determine a target total analysis duration for a plurality of material files, where the plurality of material files include at least one picture and at least one video.

[0130] Among them, multiple material files are files used to splice and generate the target video, and the material files can be pictures or videos. The embodiment of the present application is only described by taking the multiple material files including at least one picture and at least one video as an example. It should be understood that the multiple material files can also include only pictures or only videos.

[0131] Among them, multiple material files can be obtained by user selection, such as by the user selecting from a gallery. It should be understood that multiple material files can also be configured in other ways, such as by default configuration of the electronic device, and this embodiment of the application does not limit this.

[0132] As an example, the electronic device receives a first operation that triggers splicing of multiple file materials into a target video. In response to the first operation, the electronic device determines a target total analysis duration of the multiple material files.

[0133] The first operation may be that the user selects multiple file materials in the gallery and confirms to perform an image stitching operation.

[0134] As an example, the electronic device loads and displays the videos and pictures in the gallery in response to the user's second operation. In response to the user selecting multiple material files in the gallery and confirming the operation of performing image stitching, the target total analysis time of the multiple material files is determined. Among them, the second operation can be the user clicking on the "one-click movie" option (or "one-click blockbuster"). The "one-click movie" option is set in the gallery interface.

[0135] Among them, the total target analysis time T A The time it takes to analyze all source files, as determined by the application layer. The total target analysis time can be calculated based on the analysis speed of all videos included in multiple source files. The analysis speed of each video can be calculated based on its resolution and frame rate, as well as the analysis speed of the electronic device's chip.

[0136] As an example, the target analysis total time T can be calculated by the following formulas (1)-(2): A :

[0137] T A =∑T Vn +∑T P (1)

[0138] T Vn =min(L n / S n , β) (2)

[0139] Among them, T Vn Indicates the expected analysis duration of a single video in at least one video, T P Indicates the preset analysis time of a single picture in at least one picture. The following description takes the preset analysis time of 1 second as an example. min(Ln / Sn, β) means taking the minimum value of Ln / Sn and β. n is the video duration of the nth video. β is the upper limit of the expected analysis duration of a single video, which ensures that the expected analysis duration of a single video is within the limit. In practice, the upper limit of the expected analysis duration of a single video can be set according to actual usage requirements and is not limited in this embodiment of the application.

[0140] Assuming that the multiple material files selected by the user include n videos and m pictures, the sum of the expected analysis time of these m pictures is m. Accordingly, the target total analysis time T can be calculated according to the following formula (3): A :

[0141] T A =∑min(L n / S n ,β)+m (3)

[0142] It can be seen that when the multiple material files selected by the user include at least one video and at least one picture, the target total analysis time T can be obtained by adding the sum of the expected analysis time of all videos and the sum of the preset analysis time of all pictures. A .

[0143] It should be noted that the embodiment of the present application is described by taking the multiple material files selected by the user including videos and pictures as an example. In another example, if the multiple material files selected by the user only include videos, the total target analysis time T is calculated according to the following formulas (4) and (5): A :

[0144] T A =∑T Vn (4)

[0145] T Vn =min(L n / S n , β) (5)

[0146] Step 702: Analyze at least one picture to obtain analysis results for each picture in the at least one picture.

[0147] The analysis results for each image may include a score for each image. The image score can be used as a basis for evaluating whether the image is a highlight clip or a wonderful moment. For example, if the score of a frame is greater than or equal to 70 points, the frame is considered a highlight clip or a wonderful moment.

[0148] As an example, at least one picture may be analyzed sequentially to obtain analysis results for each video in the at least one picture. For example, the pictures may be analyzed sequentially according to their order, and after analysis of one picture is completed, the next picture may be analyzed.

[0149] As an example, when analyzing a picture, it can be determined whether the picture contains a preset image feature. If the picture contains the preset image feature, the picture is determined to be a highlight segment.

[0150] Step 703: Taking at least one video as a video to be processed, and determining an expected analysis duration of the video to be processed according to the target total analysis duration and the image analysis time of at least one image.

[0151] The expected analysis time of the video to be processed refers to the total analysis time expected to be consumed by the video to be processed. For example, the expected analysis time of the video to be processed can be determined as the difference between the target total analysis time and the image analysis time of at least one image.

[0152] After determining the expected analysis time of the video to be processed, an analysis strategy for the video to be processed may be planned based on the expected analysis time of the video to be processed, so as to analyze the video to be processed according to the planned analysis strategy.

[0153] The analysis result of the video includes the position of the highlight segment of the video, which is used to indicate the highlight segment in the video. For example, the position of the highlight segment of the video may include the start time and the end time of the highlight segment in the video.

[0154] As an example, according to the expected analysis duration of the video to be processed, the operation of planning the analysis strategy of the video to be processed can be implemented through the following steps 704 to 705.

[0155] Step 704: Allocate analysis time for each video in the video to be processed according to the expected analysis time of the video to be processed, and the analysis time allocated to each video is the allocated time for each video.

[0156] For example, an analysis duration may be allocated to each video in the video to be processed according to an expected analysis duration of the video to be processed and the video duration of each video in the video to be processed.

[0157] The analysis time allocated to each video is recorded as T n For example, the analysis time allocated to the first video is T1, the analysis time allocated to the second video is T1, and so on, the analysis time allocated to the nth video is T n .

[0158] As an example of the present application, based on the expected analysis time and the video length of each video, the specific implementation of determining the allocated time of each video may include: evenly allocating a portion of the expected analysis time in a preset proportion to each video, and allocating the remaining part of the expected analysis time to each video according to the corresponding weight ratio of the video.

[0159] The preset ratio can be set according to actual use requirements, and is not limited in the present embodiment. For example, the preset ratio is 50%. V 50% is evenly distributed to each video, and then the expected analysis time TV The other 50% is allocated to each video according to the weight ratio. The following is an example of a preset ratio of 50% (ie 0.5).

[0160] Specifically, the basic analysis time T allocated to a single video can be calculated by the following formula (6): n1 :

[0161] T n1 =T V ×0.5 / n (6)

[0162] Where n represents the number of videos in the video to be processed.

[0163] The weighted analysis duration T assigned to a single video is calculated using the following formulas (7)-(9): n2 :

[0164] T n2 =T V ×0.5×(W n / W A ) (7)

[0165] W n =log2(1.5+L n / L A ) / log2(1.5+S n / S A ) (8)

[0166] W A =W1+…+W n (9)

[0167] Among them, W n Represents the weight value of a single video, for example, W1 represents the weight of the first video, W2 represents the weight of the second video, and so on. n Indicates the weight of the nth video. L A Indicates the sum of the video durations of all videos. S A W represents the sum of the analysis speeds of all videos. A It should be noted that 1.5 in formula (8) is a preset value of the algorithm, which can be set according to actual use requirements and is not limited in this embodiment of the application.

[0168] Thus, the analysis time T allocated to a single video is n It consists of two parts, namely T n =T n1 +T n2 .

[0169] In the embodiment of the present application, the longer the video duration, the more analysis time should be allocated to the video. In other words, the video duration L of a video n The sum of the video duration of all videos L A The ratio L n / L A The larger the value, the more analysis time is allocated to the video.

[0170] In the embodiment of the present application, the faster the analysis speed of a video, the shorter the analysis time should be allocated. In other words, the analysis speed S of a video n The sum of the analysis speeds of all video materials S A The ratio S n / S A The larger the value, the less analysis time is allocated to the video material.

[0171] Step 705: Determine an analysis strategy for each video in the videos to be processed based on the allocated duration of each video in the videos to be processed.

[0172] As an example of the present application, multiple analysis strategies are provided for videos, for example, the multiple analysis strategies may include a full analysis strategy, a sparse key segment analysis strategy, a dense key segment analysis strategy, and a simple analysis strategy based on I frames.

[0173] Among them, the full analysis strategy includes every frame of the video in the analysis, including all highlight clips, but it consumes relatively more time and resources. The simple I-frame analysis strategy refers to when the allocated analysis time can only meet the analysis time of three images. It analyzes three images located at different positions in the video, uses the highest-scoring image as the starting point for the highlight clip results, and returns a video clip of a certain length as the result. The intensive key clip analysis strategy is an algorithm strategy that calculates the number of analysis clips and the analysis clip duration of the source video to determine the appropriate interval duration between adjacent clips. It then evenly distributes the analysis clips to ensure that the analysis clips cover the highlight clips located in different positions as much as possible. The sparse key clip analysis strategy calculates the number of analysis clips and the segment duration in the same way as the intensive strategy, but the difference is that the sparse strategy distributes the interval duration between adjacent clips unequally. The analysis clips are mainly distributed in the front and middle of the video, with appropriate distribution at the end.

[0174] For each video: If the entire video can be analyzed within the available analysis time, a full analysis strategy is adopted. If the available analysis time is less than or equal to the analysis time of three key frames, a simple analysis strategy based on I frames is adopted. If the available analysis time is not enough to analyze the entire video, but is enough to analyze more than 60% of the material, an intensive key segment analysis strategy can be adopted. If the available analysis time is greater than the analysis time of three key frames, but is not enough to analyze more than 60% of the material, a sparse key segment analysis strategy can be adopted. After determining the analysis strategy corresponding to each video, the image signal processor scores the video frames to filter out highlight segments with high scores.

[0175] For example, Figure 8 The following is a schematic diagram showing the time consumption of the four analysis strategies provided in the embodiment of the present application. Figure 8 As shown, the time consumption of the full analysis strategy, the intensive key segment analysis strategy, the sparse key segment analysis strategy, and the simple analysis strategy based on I frames decreases in sequence. Since the full analysis strategy analyzes all frames of the video, it consumes the longest time. The simple analysis strategy based on I frames analyzes up to three frames of video, so it consumes the shortest time. The time consumption of the intensive key segment analysis strategy and the sparse key segment analysis strategy is between the full analysis strategy and the simple analysis strategy based on I frames. In addition, compared with the intensive key segment analysis strategy, the sparse key segment analysis strategy focuses on analyzing the front and middle sections of the video, so it may take less time.

[0176] Different analysis strategies can be used for analyzing videos with different allocated durations. As an example of this application, the analysis strategy for each video can be determined based on the allocated duration, video duration, and video analysis rate of each video. The video analysis rate is a performance parameter of the electronic device itself and is usually fixed.

[0177] In one possible case, based on the video duration, allocated duration, and video analysis rate of each video, the percentage of video frames that can be processed within the allocated duration allocated to each video can be determined, and then the analysis strategy for each video can be determined based on the percentage.

[0178] For example, let's say the video duration is 10 seconds, the allocated duration is 1 second, and the video analysis rate is 5. Thus, the analysis time required to analyze all video frames of the video can be calculated based on the video duration and the video analysis rate. For example, the analysis time required to analyze all video frames of a video is the video duration divided by the video analysis rate, which is 10 seconds / 5 = 2 seconds. The allocated analysis time for the video is 1 second. This means that within the allocated analysis time, the percentage of the video that can be analyzed is 1 second / 2 seconds = 50%.

[0179] Optionally, if the percentage of video frames that can be processed within the analysis time allocated to the video is greater than a first preset threshold, the analysis strategy assigned to the video is a full analysis strategy. Exemplarily, the first preset threshold is 100%.

[0180] For example, the analysis time allocated to the video is 2.2s, and the analysis time required to analyze all video frames in the video is 2s. Then, the percentage of video frames in the video that can be processed within the analysis time allocated to the video is 2.2 / 2=110%, which is greater than the first preset threshold of 100%. Therefore, the analysis strategy allocated to the video is the full analysis strategy.

[0181] Optionally, if the percentage of video frames that can be processed within the allocated analysis time is greater than a second preset threshold and less than a first preset threshold, the target analysis strategy assigned to the video is an intensive key segment analysis strategy. The second preset threshold is less than the first preset threshold. For example, the first preset threshold is 100% and the second preset threshold is 70%.

[0182] For example, the analysis time allocated to the video is 1.6s, and the analysis time required to analyze all the video frames of the video is 2s. Then, the percentage of video frames in the video that can be processed within the analysis time allocated to the video is 1.6 / 2=80%, which is less than the first preset threshold of 100% and greater than the second preset threshold of 70%. Therefore, the analysis strategy assigned to the video is an intensive key segment analysis strategy.

[0183] Optionally, if the percentage of video frames in the video that can be processed within the analysis time allocated to the video is greater than a third preset threshold and less than a second preset threshold, the target analysis strategy assigned to the video is a sparse key segment analysis strategy. The third preset threshold is less than the second preset threshold. Exemplarily, the third preset threshold can be a percentage obtained based on the minimum highlight duration, wherein the minimum highlight duration can refer to the shortest duration required to generate a highlight segment, and the third preset threshold can be a percentage obtained by dividing the minimum highlight duration by the video duration of the video. The second preset threshold can be 70%.

[0184] For example, the minimum highlight duration t0 can be obtained by the following formula (10):

[0185] t0=min((t3+t1*1.2) / v, (t3+t2*0.5) / v) (10)

[0186] Where t0 is the minimum highlight duration, t3 is the minimum number of video frames required to analyze a highlight frequency band, t1 is the minimum duration of a highlight segment, and t2 is the maximum duration of a highlight segment. v is the video analysis rate.

[0187] For example, generating a highlight segment usually requires at least 3 video frames, so at least 3 video frames need to be analyzed to obtain a highlight frequency band. In one possible case, the computing performance of the electronic device requires a duration t3 of 0.2s for analyzing 3 video frames. The electronic device usually sets a duration range for the highlight segment, including a minimum and maximum duration of the highlight segment. For example, the duration range of the highlight frequency band set by the electronic device is 1s-2s. That is, the minimum duration t1 of the highlight segment is 1s, and the maximum duration t2 of the highlight segment is 2s. The video analysis rate is 5. Therefore, the minimum highlight duration t0 = min((0.2+1*1.2) / 5, (0.2+2*0.5) / 5) = min(0.44, 0.6) = 0.44s can be determined by the above formula (10). The third preset threshold can be 0.44 / 2 = 22%.

[0188] For example, the analysis time allocated to the video is 1s, and the analysis time required to analyze all the video frames of the video is 2s. Then, the percentage of video frames in the video that can be processed within the analysis time allocated to the video is 1 / 2=50%, which is less than the second preset threshold of 70% and greater than the third preset threshold of 22%. Therefore, the analysis strategy allocated to the video is a sparse key segment analysis strategy.

[0189] Optionally, if the percentage of video frames that can be processed within the analysis time allocated for the video is less than a third preset threshold, the target analysis strategy assigned to the video is a simple I-frame analysis strategy. The third preset threshold may be a percentage obtained based on the minimum highlight duration.

[0190] For example, the analysis time allocated to the video is 0.4s, and the analysis time required to analyze all the video frames of the video is 2s. The percentage of video frames in the video that can be processed within the analysis time allocated to the video is 0.4 / 2=20%, which is less than the third preset threshold of 22%. Therefore, the target analysis strategy assigned to the video is a simple analysis strategy based on I frames.

[0191] In one possible scenario, the analysis strategy for each video may be determined directly based on the analysis duration allocated to the video and the analysis time required to analyze all video frames of the video. The analysis time required to analyze all video frames of the video may be obtained based on the video duration and the video analysis rate.

[0192] For example, if a video has a duration of 10 seconds, an analysis time of 1 second, and a video analysis rate of 5, the analysis time required to analyze all video frames can be calculated by dividing the video duration of 10 seconds by the video analysis rate of 5: 10 / 5 = 2 seconds. The analysis time allocated for the video and the analysis time required to analyze all video frames are then compared to determine the video analysis strategy.

[0193] Optionally, if the analysis duration allocated to a video is longer than the time required to analyze all frames in the video, the target analysis strategy assigned to the video is the full analysis strategy. For example, if the analysis duration allocated to a video is 2.2 seconds, and the time required to analyze all frames in the video is 2 seconds, the target analysis strategy assigned to the video is the full analysis strategy.

[0194] Optionally, when the analysis duration allocated to the video is less than the analysis time required to analyze all video frames of the video, and is greater than the product of the analysis time required to analyze all video frames of the video and a fourth preset threshold, the analysis strategy assigned to the video is an intensive key segment analysis strategy. For example, the analysis duration allocated to the video is 1.6s, the fourth preset threshold is 70%, the analysis time required to analyze all video frames in the video is 2s, the analysis duration of 1.6s allocated to the video is less than the analysis time of 2s required to analyze all video frames in the video, and is greater than the product of the analysis time of 2s required to analyze all video frames in the video and 70% of the fourth preset threshold (2*70%=1.2s), then the analysis strategy assigned to the video is an intensive key segment analysis strategy.

[0195] Optionally, if the analysis duration allocated to the video is less than the product of the analysis time required to analyze all video frames of the video and a fourth preset threshold, and greater than a fifth preset threshold, the analysis strategy assigned to the video is a sparse highlight segment analysis strategy. The fifth preset threshold may be a minimum highlight duration.

[0196] For example, the minimum highlight duration t0 can be obtained based on the following formula (11):

[0197] t0=min((t3+t1*1.2) / v, (t3+t2*0.5) / v) (11)

[0198] Where t0 is the minimum highlight duration, t3 is the minimum number of video frames required to analyze a highlight frequency band, t1 is the minimum duration of a highlight segment, and t2 is the maximum duration of a highlight segment. v can be the video analysis rate.

[0199] For example, generating a highlight segment typically requires at least three video frames, so at least three video frames need to be analyzed to obtain a highlight frequency band. In one possible scenario, the computing performance of the electronic device requires a duration t3 of 0.2 seconds to analyze three video frames. Electronic devices typically set a duration range for highlight segments, including a minimum and maximum duration for the highlight segments. For example, the duration range for the highlight frequency band set by the electronic device is 1 second to 2 seconds. In other words, the minimum duration t1 of the highlight segment is 1 second, and the maximum duration t2 of the highlight segment is 2 seconds. The video analysis rate is 5. Therefore, based on the above formula, the minimum highlight duration t0 can be obtained as min((0.2+1*1.2) / 5,(0.2+2*0.5) / 5)=min(0.44, 0.6)=0.44 seconds. That is, the fifth preset threshold is 0.44 seconds.

[0200] For example, if the analysis time allocated to a video is 1 second, the analysis time required to analyze all video frames in the video is 2 seconds, the fourth preset threshold is 70%, and the fifth preset threshold is 0.44 seconds, then if the analysis time allocated to the video is 1 second less than the product of the analysis time required to analyze all video frames in the video, 2 seconds, and 70% of the fourth preset threshold (2 * 70% = 1.2 seconds), and greater than the fifth preset threshold of 0.44, then the analysis strategy assigned to the video is the simple I-frame analysis strategy.

[0201] Optionally, when the analysis duration allocated to a video is less than a fifth preset threshold, the target analysis strategy assigned to the video is a sparse key segment analysis strategy. The fifth preset threshold may refer to a minimum highlight duration. For example, the analysis duration allocated to a video is 0.4s, the fifth preset threshold is 0.44s, and the analysis duration allocated to the video, 0.4s, is less than the fifth preset threshold 0.44s. Therefore, the analysis strategy assigned to the video is a simple I-frame-based analysis strategy.

[0202] Step 706: Determine whether there is any unanalyzed video in the video to be processed.

[0203] That is, it is determined whether there are any videos that have not been analyzed in the videos to be processed.

[0204] Step 707: If there is an unanalyzed video in the videos to be processed, analyze Video 1 according to the analysis strategy of Video 1 in the unanalyzed videos.

[0205] In the embodiment of the present application, the unanalyzed videos may be analyzed sequentially. Video 1 may be the first unanalyzed video.

[0206] In this embodiment of the present application, analyzing a video refers to analyzing the location of highlight segments in the video. During the video analysis process, each frame of the video can be compared with a preset image feature. When multiple consecutive frames of the video contain the preset image feature, it is determined that the video contains a highlight segment, and the start and end times of the highlight segment are determined.

[0207] Step 708: Determine whether the analysis of the currently analyzed video 1 has timed out.

[0208] In the embodiment of the present application, during the analysis of video 1, the analyzed time of video 1 can be monitored, and then it can be determined whether the currently analyzed video 1 has timed out based on the analyzed time of video 1. The analyzed time of video 1 refers to the time taken to analyze video 1.

[0209] As an example, a timer may be started when analysis of video 1 begins, and the timer duration is used as the analyzed duration of video 1.

[0210] As an example, it can be determined whether the analyzed duration of the first video meets the first preset condition. If so, it is determined that the analysis of video 1 has timed out. If not, it is determined that the analysis of video 1 has not timed out.

[0211] The first preset condition includes one or more of the following conditions:

[0212] 1) The analyzed duration of video 1 is greater than or equal to the product of the allocated duration of video 1 and the first preset ratio.

[0213] The first preset ratio can be set in advance according to actual needs. In addition, the first preset ratio is usually greater than 1, for example, the first preset ratio can be 120% or 130%.

[0214] For example, condition 1) may be: the analyzed duration of video 1 ≥ the allocated duration of video 1 × 120%.

[0215] 2) The analyzed duration of video 1 is greater than or equal to the product of the maximum tolerance of the timeout duration of the video to be processed and the first value.

[0216] The first value may be a preset value or may be determined based on the number of videos to be processed. For example, the first value may be the product of the number of videos to be processed and the first preset value. The first preset value may be pre-set, for example, 0.8 or 0.9.

[0217] For example, condition 2) may be: the analyzed duration of video 1 ≥ the maximum tolerance of the timeout duration of the video to be processed / the number of videos to be processed×0.8.

[0218] The maximum timeout tolerance of the video to be processed is used to indicate the timeout duration allowed for each video in the video to be processed. The maximum timeout tolerance of the video to be processed can be set in advance as needed, or determined according to the expected analysis duration of the video to be processed.

[0219] As an example, the maximum tolerance for the timeout of the video to be processed can be any of the following durations:

[0220] 1) First preset duration: The first preset duration can be set in advance according to needs, for example, the first preset duration is 10 seconds or 15 seconds.

[0221] 2) The product of the expected analysis duration of the video to be processed and a second preset ratio. The second preset ratio can be pre-set. In addition, the second preset ratio is usually less than 1, for example, the second preset ratio is 20% or 30%.

[0222] 3) A larger value between the first preset duration and the specified duration, where the specified duration refers to the product of the expected analysis duration of the video to be processed and the second preset ratio.

[0223] For example, the maximum tolerance for the timeout duration of the video to be processed is: (expected analysis duration of the processed video×20%) and 10 seconds, whichever is greater.

[0224] Step 709: If the analysis of video 1 times out, stop analyzing video 1.

[0225] In addition, if the analysis of video 1 has not timed out, the process may jump to step 712 to determine whether the analysis of the entire video to be processed has timed out.

[0226] Step 710 : Determine whether the analyzed result of video 1 contains the highlight segment position of video 1 .

[0227] The highlight segment position of video 1 is used to indicate the highlight segment in video 1 and may include the start time and end time of the highlight segment in video 1.

[0228] When the analysis of Video 1 is stopped, Video 1 has already been analyzed for a period of time using the pre-planned analysis strategy. During this analysis process, the location of the highlight segment of Video 1 may or may not have been obtained. In this embodiment of the present application, different operations can be performed for different situations. Therefore, it is possible to first determine whether the location of the highlight segment of Video 1 is present in the analyzed results of Video 1.

[0229] Step 711: If the analysis result of video 1 contains the highlight segment position of video 1, determine whether the overall analysis of the video to be processed has timed out.

[0230] If the analysis result of video 1 contains the highlight segment position of video 1 , that is, the highlight segment position of video 1 has been obtained, the highlight segment position of video 1 can be directly obtained as the analysis result of video 1 .

[0231] In an embodiment of the present application, during the analysis of a video to be processed, the analyzed duration of the video to be processed can be monitored, and then, based on the analyzed duration of the video to be processed, a determination can be made as to whether the overall analysis of the video to be processed has timed out. For example, each time analysis of a video is completed, a determination can be made as to whether the overall analysis of the video to be processed has timed out based on the analyzed duration of the video to be processed. The end of video analysis can include both stopped and completed analysis.

[0232] The analyzed time of the video to be processed refers to the total time spent analyzing the video to be processed. For example, the analyzed time of the video to be processed can be the total time spent analyzing the video to be processed.

[0233] As an example, a timer may be started when the analysis operation on the video to be processed begins, and the duration of the timer is used as the analyzed duration of the video to be processed.

[0234] As an example, it can be determined whether the analyzed duration of the video to be processed meets the second preset condition. If so, it is determined that the video to be processed is analyzed as a whole. If not, it is determined that the video to be processed has not been analyzed as a whole and the timeout has occurred.

[0235] As an example, the second preset condition may include one or more of the following conditions:

[0236] 1) The analyzed duration of the video to be processed is greater than or equal to the product of the expected analysis duration of the video to be processed and the third preset ratio.

[0237] The analyzed duration of the video to be processed is the analyzed duration of the video to be processed, for example, the total analyzed duration of the analyzed videos in the video to be processed. The third preset ratio can be pre-set. In addition, the third preset ratio is typically greater than 1, for example, the third preset ratio is 115% or 120%.

[0238] For example, condition 1) is: the analyzed duration of the video to be processed ≥ the expected analysis duration of the video to be processed × 115%.

[0239] 2) The analyzed duration of the video to be processed is greater than or equal to the sum of the expected analysis duration of the video to be processed and the second preset duration.

[0240] The second preset time length may be pre-set, for example, the second preset time length may be 7 seconds or 8 seconds.

[0241] For example, condition 2) is: the analyzed duration of the video to be processed ≥ the expected analysis duration of the video to be processed + 7 seconds.

[0242] For example, suppose five videos need to be analyzed, with an expected analysis duration of 15 seconds. The analysis duration allocated to each video is 1 second, 2 seconds, 3 seconds, 4 seconds, and 5 seconds, for a total of 15 seconds. The maximum timeout tolerance is the larger of (expected analysis duration × 20%) and 10 seconds, i.e., 10 seconds, whichever is greater.

[0243] Assume that the judgment condition for the timeout of single video analysis is: the analyzed video duration ≥ the allocated video duration × 120%, and the analyzed video duration ≥ the maximum timeout duration tolerance / the number of videos × 0.8.

[0244] When analyzing the first video, the analysis time allocated for the video is 1s. If the time consumed in analyzing the video exceeds the allocated time of the video × 120% (i.e. 1s*120%=1.2s) and exceeds the maximum tolerance of the timeout duration / number of videos × 0.8 (i.e. 10s / 5*0.8=1.6s), the video analysis is determined to have timed out and the analysis of the video is stopped.

[0245] In addition, after each video analysis is completed, it can also be determined whether the overall video analysis has timed out. Assume that the judgment conditions for the overall video analysis timeout are: the analyzed time of the video to be processed ≥ the expected analysis time of the video to be processed × 115%, and the analyzed time of the video to be processed ≥ the expected analysis time of the video to be processed + 7 seconds.

[0246] If the overall analysis time of the five videos exceeds 17.25 seconds (15 seconds * 115%) and exceeds 22 seconds (15 seconds + 7 seconds), the overall video analysis is determined to have timed out, and the analysis of the remaining unanalyzed videos according to the configured analysis policy is stopped.

[0247] Step 712: If the highlight segment position of video 1 does not exist in the analyzed results of video 1, the analysis strategy of video 1 is changed to the specified analysis strategy.

[0248] The designated analysis strategy is a relatively simple analysis strategy that takes a short time to analyze the video. The designated analysis strategy can be pre-set, for example, the designated analysis strategy can be a simple analysis strategy based on I frames.

[0249] After the analysis strategy of video 1 is changed to the specified analysis strategy, the process may jump to step 711 to determine whether the overall analysis of the video to be processed has timed out.

[0250] It should be noted that the embodiment of the present application can analyze the videos in the video to be processed in a serial manner. After the analysis strategy of video 1 is changed to a specified analysis strategy, video 1 can be re-used as the next video to be analyzed, so that in the next round of video analysis, the specified analysis strategy is used to analyze video 1.

[0251] In the embodiment of the present application, after the analysis strategy of video 1 is changed to the specified analysis strategy, if the entire analysis of the video to be processed has not timed out, the process can jump to step 715 to determine whether the analysis progress of the video to be processed is behind schedule. If the analysis progress of the video to be processed is not behind schedule, video 1 can be re-selected as the next video to be analyzed and analyzed using the specified analysis strategy.

[0252] Step 713: If the overall analysis of the video to be processed times out, the analysis strategy for the remaining unanalyzed videos in the video to be processed is changed to a fixed-length analysis strategy.

[0253] That is, if it is determined that the overall video analysis has timed out, the analysis of the remaining unanalyzed videos according to the configured analysis strategy is stopped, and the analysis strategy of the remaining unanalyzed videos is changed to a fixed-length analysis strategy.

[0254] The fixed-length analysis strategy uses a fixed position in the video as the location of the highlight segment. The fixed position can be pre-set, such as the first few seconds, middle few seconds, or last few seconds of the video. This embodiment of the application does not limit the fixed position.

[0255] Step 714: Analyze the remaining unanalyzed videos in sequence using a fixed-length analysis strategy to obtain analysis results for each of the remaining unanalyzed videos.

[0256] After obtaining the analysis results for each of the remaining unanalyzed videos, the analysis of the multiple source files is completed. Based on the analysis results of the multiple source files, all highlight segments can be extracted from the multiple source files and then spliced ​​together to obtain the target video.

[0257] For example, based on the analysis results of multiple material files, the videos in the multiple material files can be edited and the pictures can be filtered. Then, the edited video segments and the filtered pictures can be spliced ​​to obtain the target video.

[0258] It should be noted that if, after changing the analysis strategy for Video 1 to the specified analysis strategy, the analysis strategy for the entire pending video has timed out, Video 1's analysis strategy will also be changed to the fixed-length analysis strategy when the analysis strategies for all remaining unanalyzed videos in the pending video are changed to the fixed-length analysis strategy. In other words, if the analysis of the entire pending video has timed out, Video 1 will be analyzed using the fixed-length analysis strategy instead of the specified analysis strategy.

[0259] Step 715: If the entire analysis of the video to be processed has not timed out, determine whether the analysis progress of the video to be processed is behind schedule.

[0260] The lagging analysis progress of the videos to be processed means that the analysis progress of the videos to be processed significantly lags behind the expected progress, and the expected progress refers to the analysis progress indicated by the allocated duration of each video in the videos to be processed.

[0261] In an embodiment of the present application, it is possible to determine whether the analysis progress of the video to be processed is lagging behind based on the analyzed duration and the expected analysis duration of at least one video.

[0262] As an example, whether the analysis progress of the video to be processed is behind schedule can be determined based on the analyzed time, expected analysis time, maximum timeout tolerance, and the total allocated time of unanalyzed videos in the video to be processed.

[0263] For example, the analysis progress of a video to be processed may be determined to be behind schedule when the analyzed time, expected analysis time, maximum timeout tolerance, and total allocated time of unanalyzed videos in the video to be processed meet the following conditions:

[0264] (Analyzed duration of the video to be processed + total allocated duration of the unanalyzed videos in the video to be processed) - expected analysis duration of the video to be processed ≥ maximum timeout tolerance of the video to be processed / second preset value

[0265] The second preset value may be pre-set, for example, the second preset value may be 3 or 4.

[0266] It should be noted that if the analysis progress of the video to be processed is not behind, the process jumps to step 706 to continue to determine whether there is an unanalyzed video in the video to be processed, so as to continue to analyze the next unanalyzed video according to the above analysis process.

[0267] Step 716: If the analysis progress of the to-be-processed video is behind schedule, the expected analysis duration of the unanalyzed video in the to-be-processed video is determined, and the unanalyzed video in the to-be-processed video is used as the to-be-processed video, and the process jumps to step 704.

[0268] When the analysis progress of the videos to be processed falls behind, in order to catch up with the analysis progress, a re-planning process can be triggered to re-plan the allocated time and analysis strategy of the unanalyzed videos so that the unanalyzed videos can be analyzed according to the re-planned allocated time and analysis strategy.

[0269] In the embodiment of the present application, a new video to be processed can be determined by using an unanalyzed video in the video to be processed as the video to be processed. After jumping to step 704 based on the expected analysis duration of the new video to be processed, the analysis strategy for the new video to be processed can be replanned, and then the new video to be processed can be analyzed according to the replanned analysis strategy.

[0270] In the embodiment of the present application, after each video is analyzed, it can be determined whether the analysis progress of the video to be processed is behind schedule. If the analysis progress is behind schedule, the re-planning process is triggered.

[0271] As an example, the input parameters of the re-planning process may include the following three parameters: target range, expected analysis duration, and maximum tolerance for timeout duration.

[0272] Among them, during the initial planning:

[0273] 1) Target scope: all input videos.

[0274] 2) Expected analysis time: The recommended upper limit of the analysis time. For example, the difference between the target analysis time and the image analysis time.

[0275] 3) Maximum timeout tolerance: the product of the first preset time, the expected analysis time and the second preset ratio, or the larger of the first two, for example, the larger of (expected analysis time × 20%) and 10 seconds.

[0276] When the re-planning process is triggered, the above three parameters need to be re-planned, and then the re-planned three parameters are used as input parameters of step 704, that is, step 704 is executed based on the re-determined three parameters to re-plan the allocation time and analysis strategy of the unanalyzed video based on the re-planned three parameters.

[0277] Among them, during re-planning:

[0278] 1) Target range: all remaining unanalyzed videos.

[0279] 2) Expected analysis duration: the total allocated duration of the unanalyzed videos planned last time + (T(remaining upper limit) - the total allocated duration of the unanalyzed videos planned last time) / the third preset duration).

[0280] The third preset duration can be set in advance according to needs, for example, the third preset duration is 3 or 4.

[0281] Wherein, T(remaining upper limit)=T(upper limit)−the total duration of the analyzed videos within the target range planned last time.

[0282] Wherein, T (upper limit) = the expected analysis duration of the last plan + the maximum tolerance of the timeout duration of the last plan.

[0283] 3) Maximum tolerance for timeout: T (remaining upper limit) – the expected analysis duration for this plan.

[0284] For example, suppose a total of five videos need to be analyzed, with an expected analysis time of 15 seconds. The analysis time allocated to each video is 5 seconds, 4 seconds, 3 seconds, 2 seconds, and 1 second, for a total of 15 seconds. Assume that the analysis of the first video actually takes 9 seconds to complete.

[0285] Assume that the judgment condition for the analysis progress falling behind is: (analyzed time of the video to be processed + total allocated time of the unanalyzed videos in the video to be processed) - expected analysis time of the video to be processed ≥ maximum timeout tolerance of the video to be processed / 3. The third preset time is 3.

[0286] When initially planning:

[0287] Target scope: 5 videos;

[0288] Expected analysis time: 15 seconds

[0289] Maximum timeout tolerance: the larger value of (15s*20%) and (10s), whichever is greater, 10s.

[0290] After the first video analysis is completed, determine whether the analysis progress is behind schedule: compare 9s + (4s + 3s + 2s + 1s) - 15s = 4s with 10s / 3 = 3.3s. Since 4s ≥ 3.3s, the analysis progress is determined to be behind schedule, triggering replanning.

[0291] During re-planning:

[0292] T (upper limit) = expected analysis time of the last plan + maximum tolerance of the last planned timeout = 15s + 10s = 25s;

[0293] T(remaining upper limit) = T(upper limit) – total duration of analyzed videos within the previously planned target range = 25s - 9s = 16s;

[0294] Target scope for re-planning: the remaining 4 videos;

[0295] Replanned expected analysis duration: the total allocated duration of the unanalyzed videos planned last + (T(remaining upper limit) - the total allocated duration of the unanalyzed videos planned last) / 3) = (4s + 3s + 2s + 1s) + (16s - (4s + 3s + 2s + 1s)) / 3 = 12s;

[0296] The maximum tolerance for the re-planning timeout is: T (remaining upper limit) – expected analysis time for this plan = 16 seconds - 12 seconds = 4 seconds.

[0297] The video processing monitoring and scheduling method provided in the embodiments of the present application can monitor and schedule the video analysis process using an adaptive monitoring and scheduling algorithm during the video analysis process to ensure that the video analysis is completed within a reasonable time limit, thereby improving video analysis efficiency. For example, during the video analysis process, the analyzed video duration can be monitored. If the video analysis timeout is determined based on the analyzed video duration, the video analysis is stopped and the location of the video highlight segments is determined based on the analyzed video results. This prevents excessive video analysis time and ensures that the video analysis is completed within a reasonable time limit. Furthermore, the method can monitor whether the overall analysis of all videos has timed out. If so, a fixed-length analysis strategy is used to analyze the remaining unanalyzed videos. This can further accelerate video analysis and improve video analysis efficiency. Furthermore, the method can monitor whether the analysis progress of all videos is lagging behind. If so, the allocated analysis time and analysis strategy can be re-planned. This ensures that the analysis is completed within a reasonable time limit while dynamically planning the analysis time allocation and analysis strategy allocation, making full use of the analysis time and analysis strategy to achieve better analysis results and improve video analysis efficiency.

[0298] The specific implementation process of the monitoring and scheduling method for video processing provided by the embodiment of the present application is described below with reference to the accompanying drawings.

[0299] Figure 9 This is a schematic flow chart of a video processing monitoring scheduling method provided by an embodiment of the present application, which can be executed by an electronic device. The hardware structure diagram of the electronic device can be as follows: Figure 1 As shown, the software structure diagram of the electronic device can be as follows Figure 2 As shown, but the embodiments of the present application are not limited thereto.

[0300] S101, the application editing business layer receives the user's operation on the one-click film option.

[0301] The application editing business layer includes a one-click film creation application. This application is presented as a "One-click Film Creation" option on the interface. When a user clicks the "One-click Film Creation" option, the application editing business layer activates the application.

[0302] S102 , in response to a user operation, the application editing service layer loads and displays pictures and videos.

[0303] For example, a one-touch capture option is displayed in the UI interface of a camera application of an electronic device. When a user clicks the one-touch capture option, the UI interface changes to a gallery interface. The gallery interface displays multiple images and multiple videos, and a confirmation control (e.g., √) for the one-touch capture option is displayed.

[0304] S103: The application editing service layer receives the user's selection of videos and pictures and confirms the execution of the one-click film-making operation.

[0305] Among them, the user first selects one or more videos and one or more pictures in the gallery interface, and then clicks the √ control to confirm the one-click filming.

[0306] Exemplarily, the user selected video materials such as video 1, video 2, and video 3, and also selected picture materials such as picture 1 and picture 2. Based on video 1, video 2, video 3, picture 1, and picture 2, the electronic device will analyze the highlight clips or wonderful moments therein, and then splice the highlight clips or wonderful moments in the videos and pictures, and finally generate a video, completing one-click filming. In the one-click filming process, it is necessary to analyze the highlight clips at different positions of multiple video files within a limited time to avoid taking too much time. In this regard, the improvement of the embodiment of the present application is that a suitable duration is allocated to each video through a duration allocation algorithm, and the video highlight clip analysis is performed, and then the highlight clips of multiple videos are spliced ​​together to complete the one-click filming. The following will step by step explain how the present application solution reasonably allocates duration to each video.

[0307] After the user triggers the one-click slice generation, the modules at different layers interact with each other to complete parameter and algorithm initialization. For the specific initialization process, see the following steps.

[0308] S104, the application editing business layer sends a one-click film-making instruction to the application function layer. In response to the instruction, the application function layer calls the media middle platform framework to initialize each interface.

[0309] The application function layer instructs the media center framework layer to initialize the image highlight segment analysis interface, theme summary interface, and analysis performance query interface. Each interface in the media center framework layer completes initialization separately.

[0310] S105: The performance analysis query interface sends initialization parameters to the one-key slice module of the FWK layer through the channel interface.

[0311] The initialization parameter is used to instruct the initialization of the highlight segment analysis algorithm.

[0312] S106 , the one-key slice module of the FWK layer instructs the algorithm module of the HAL layer to initialize.

[0313] S107 , the algorithm module of the HAL layer initializes the highlight segment analysis algorithm.

[0314] S108, the algorithm module of the HAL layer reports the message of successful initialization to the channel interface in the media middle platform framework layer; wherein, the message of successful initialization is transmitted to the channel interface through the one-key film module of the FWK layer.

[0315] S109, the channel interface in the media middle platform framework layer calls the chip analysis speed interface to obtain the chip analysis speed from the HAL layer.

[0316] S110, the algorithm module of the HAL layer returns the chip analysis speed to the channel interface in the media middle platform framework layer.

[0317] Exemplarily, the chip analysis speed is 5 times the speed. For example, for a 60-second video, if the analysis is performed at a 5-times-speed chip analysis speed, it will take 12 seconds to complete the analysis of the video.

[0318] Among them, the call of the chip analysis speed interface and the return of the chip analysis speed are both transmitted through the one-click chip module of the FWK layer.

[0319] S111, the channel interface in the media middle platform framework layer sends the chip analysis speed to the analysis performance query interface.

[0320] The chip analysis speed obtained from the performance query interface here can be used to calculate the analysis speed of each video in the following steps.

[0321] S112, the analysis performance query interface in the media middle platform framework layer reports the initialization success message to the application layer.

[0322] Therefore, each module of the video editing service layer and the application function layer of the application layer can be informed that the corresponding initialization has been completed.

[0323] After initialization, the application layer will query the analysis performance query interface for the analysis speed of each video in turn.

[0324] S113 , the video editing service layer of the application layer sends query parameters to the analysis performance query interface to query the analysis speed of video 1 .

[0325] The query parameters include the file descriptor of the video, and the file descriptor of video 1 is recorded as fd1.

[0326] The file descriptor of a video can be used to uniquely identify the video. According to the file descriptor of the video, relevant information of the video, such as the resolution and frame rate of the video, can be obtained.

[0327] S114, analyzing the performance query interface to obtain the resolution and frame rate of video 1 according to the file descriptor fd1 of video 1.

[0328] S115 , the analysis performance query interface calculates the analysis speed of video 1 according to the resolution and frame rate of video 1 and the chip analysis speed.

[0329] As mentioned above, the analysis speed of a single video ultimately depends on the video decoding speed and the chip analysis speed. Specifically, the video decoding speed of Video 1 can be calculated based on its resolution and frame rate. It's understandable that since the resolution and frame rate of each video may vary, the video decoding speed may also vary. The specific process for calculating the analysis speed of Video 1 can be found above and will not be further elaborated here.

[0330] S116 , the analysis performance query interface returns the analysis speed of video 1 to the video editing service layer of the application layer.

[0331] After the application layer video editing service obtains the analysis speed of video 1, S113-S116 are executed again to continue to query the analysis speed of the next video. Thus, the application layer video editing service sequentially queries the analysis speeds of video 1, video 2, and video 3.

[0332] It should be noted that the analysis speed of the video is mainly considered here, without considering the analysis speed of the image.

[0333] S117: The video editing service layer of the application layer calculates the target total analysis time according to the analysis speed of all returned videos.

[0334] For example, assuming that the video length L1 of video 1 is 120 seconds, the video length L2 of video 2 is 40 seconds, and the video length L3 of video 3 is 15 seconds; the analysis speed S1 of video 1 is 5 times the speed, the analysis speed S2 of video 2 is 4 times the speed, and the analysis speed S3 of video 3 is 3 times the speed.

[0335] Combined with the above formula (2), we can know that:

[0336] The expected analysis duration of Video 1 is min(L1 / S1, 20s), that is, min(120 / 5, 20s)=20s.

[0337] The expected analysis duration of Video 2 is min(L2 / S2, 20s), that is, min(40 / 4, 20s) = 10s.

[0338] The expected analysis duration of Video 3 is min(L3 / S3, 20s), that is, min(15 / 3, 20s)=3s.

[0339] The total expected analysis time of all videos is 20s+10s+3s=33s.

[0340] As mentioned above, when multiple materials to be analyzed include videos and pictures, the target total analysis time is obtained by adding the sum of the expected analysis times of all videos and the sum of the expected analysis times of all pictures. The expected analysis time of each picture is a fixed value, such as 1 second. Here, the sum of the expected analysis times of Picture 1 and Picture 2 is 2 seconds. Therefore, the target total analysis time of 35 seconds is obtained by adding the sum of the expected analysis times of all videos, 33s, and the sum of the expected analysis times of all pictures, 2s. It should be noted that the target total analysis time is the expected analysis time preliminarily determined for all materials, and the target total analysis time will be allocated to each video and picture according to actual needs.

[0341] Figure 10 Another schematic flow chart of the monitoring scheduling method for video processing provided by an embodiment of the present application is shown. Figure 9 ,like Figure 10 As shown, after S117, the monitoring scheduling method for video processing further includes the following S118-S143.

[0342] S118, the video editing business layer of the application layer calls the image highlight segment analysis interface, and sends the target analysis total time and the file descriptors fd of all material files to be analyzed to the policy monitoring module of the media middle platform framework layer.

[0343] After S118, all the material files to be analyzed are analyzed. Optionally, each image can be analyzed sequentially first, and after all images are analyzed, each video can be analyzed sequentially. Alternatively, each video can be analyzed sequentially first, and after all videos are analyzed, each image can be analyzed sequentially. For ease of explanation, this example uses the example of analyzing images first and then videos.

[0344] For example, Figure 10 Schematic diagram of analysis pictures and videos provided in the embodiment of the present application is shown. Figure 10 As shown, each picture is analyzed in turn, and after all pictures are analyzed, each video is analyzed in turn. For example, the analysis time of each picture is 1 second.

[0345] The image analysis process is described below. It involves analyzing multiple images one by one. During each analysis, the image is first processed and then analyzed within a preset analysis duration to determine whether it is a highlight segment.

[0346] S119, the policy monitoring module of the media middle platform framework layer instructs the channel interface to analyze picture 1.

[0347] S120 , the channel interface performs processing such as decoding, reducing resolution, and converting format on the picture 1 , and stores the processed picture 1 .

[0348] S121: The channel interface sends the frame data address of picture 1 to the HAL layer through the one-key slice module of the FWK layer.

[0349] S122 , the algorithm module of the HAL layer finds the processed picture 1 according to the frame data address, analyzes the picture 1 using the highlight segment analysis algorithm, and obtains an analysis result (such as a score) of the picture 1 .

[0350] The highlight segment analysis algorithm is an image processing algorithm provided by the image chip. This algorithm scores each frame based on its color, texture, quality, interpolation with previous and next frames, and edge change rate. The score can be used to determine whether a frame represents a highlight segment or a wonderful moment. For example, if the score is greater than or equal to 60, the frame is considered a highlight segment or a wonderful moment.

[0351] S123, the algorithm module of the HAL layer transmits the analysis result of picture 1 to the policy monitoring module of the media middle platform framework layer.

[0352] The analysis results of Figure 1 are transmitted through the one-click film-making module of the FWK layer and the channel interface of the media middle platform framework layer.

[0353] S119-S123 illustrate the analysis process for a single image. S119-S123 can be executed in a loop to analyze each image in turn. After analyzing a single image, S119-S123 are executed again to analyze the next image. For example, after analyzing image 1, S119-S123 are executed again to analyze image 2.

[0354] For example, the analysis result of picture 1 is 80 points, that is, picture 1 is analyzed as a wonderful moment.

[0355] For another example, the analysis result of picture 2 is 50 points, that is, picture 2 is analyzed as a non-wonderful moment.

[0356] For example, the allocated duration for picture 1 and picture 2 is 1 second each, so the picture analysis takes 2 seconds.

[0357] The following describes the video analysis process. It involves analyzing multiple videos individually. During each analysis, an analysis duration is allocated to a video (referred to as the first video), an analysis strategy is set for the first video, and the first video is processed. Finally, the first video is analyzed again within the allocated analysis duration according to the analysis strategy to identify the highlights within the first video.

[0358] S124, the policy monitoring module of the media middle platform framework layer allocates analysis time for each video according to the expected analysis time of all videos.

[0359] The expected analysis time for all videos is the sum of the expected analysis times for all videos, and the expected analysis time for all videos is the difference between the total target analysis time and the time required to analyze all images.

[0360] For example, suppose the user selects 3 videos and 2 pictures as the materials to be analyzed for one-click filming. Taking the target analysis time as 35 seconds, the sum of the expected analysis time of these 2 pictures (that is, the time required to analyze all pictures) is 2 seconds, and the sum of the expected analysis time of these 3 videos is 33 seconds.

[0361] In an embodiment of the present application, the analysis time allocated to each video can be determined based on the total number of all videos, the duration of each video, the sum of the durations of all videos, the analysis speed of each video, the sum of the analysis speeds of all videos, and the sum of the expected analysis durations of all videos (these parameters are collectively referred to as first parameters).

[0362] For example, 50% of the total expected analysis duration of all videos (called the video time to be allocated) is evenly distributed to each video, and then the remaining 50% of the video time to be allocated is allocated to each video according to the weight ratio.

[0363] It should be noted that first, an analysis time is allocated to video 1, then an analysis strategy for video 1 is set, and then video 1 is analyzed according to the analysis strategy for video 1 to determine the highlight segments in video 1.

[0364] S125, the policy monitoring module dynamically sets the analysis policy.

[0365] Among them, the analysis strategies include full analysis strategy, intensive key segment analysis strategy, sparse key segment analysis strategy, or simple analysis strategy based on I frame.

[0366] For example, you can follow the above Figure 7 The analysis strategy is dynamically set in the manner described in step 705 of the embodiment.

[0367] For example, the analysis strategy set for video 1 is used as a full analysis strategy as an example for explanation.

[0368] S126, the policy monitoring module sends the file descriptor of video 1 and the set analysis policy to the channel interface.

[0369] S127 , the channel interface performs processing such as decoding, format conversion, and resolution reduction on the video 1 , and stores the processed video 1 .

[0370] In the embodiment of the present application, the policy monitoring module sends the file descriptor fd1 of video 1 and the analysis policy (full analysis policy) set for video 1 to the channel interface. Then, the channel interface executes the following steps A1 to A5.

[0371] In step A1, the channel interface decodes all video frames of video 1 based on the file descriptor fd1 of video 1 and the analysis policy (full analysis policy) set for video 1. It will be appreciated that the policy monitoring module, by issuing the analysis policy (full analysis policy) set for video 1, enables the channel interface to determine that the video frames to be decoded are all video frames of video 1 based on the policy.

[0372] It should be noted that to increase the processing speed of the highlight segment algorithm interface, the channel interface can reduce the resolution of the video frames before performing highlight analysis. Furthermore, because the resolution reduction algorithm only supports the second format, it is necessary to first convert all video frames from the first format to the second format, then reduce the resolution of the video frames, and then convert the reduced-resolution video frames back to the original first format for storage in the memory.

[0373] Step A2: The channel interface converts all decoded video frames from the first format to the second format.

[0374] Exemplarily, the first format is nv12 format, and the second format is i420 format.

[0375] In step A3, the channel interface reduces all the video frames after the format change from the first resolution to the second resolution.

[0376] Exemplarily, the first resolution is 1080p and the second resolution is 480p.

[0377] In some embodiments, different analysis strategies may correspond to different second resolutions. Therefore, the strategy monitoring module sends the analysis strategy (full analysis strategy) set for video 1, so that the channel interface can determine the second resolution corresponding to the full analysis strategy.

[0378] In step A4, the channel interface converts all video frames with reduced resolution from the second format to the first format.

[0379] In step A5 , the channel interface stores all video frames converted into the first format again in a memory (buffer).

[0380] S128: The channel interface sends the frame data address of video 1 to the HAL layer through the one-key slice module of the FWK layer.

[0381] Among them, the one-click segmentation module of the FWK layer can complete data packaging and provide data and program operation services.

[0382] S129, the algorithm module of the HAL layer finds the processed video 1 according to the frame data address, and analyzes the video 1 using a highlight segment analysis algorithm.

[0383] The purpose of analyzing the video 1 by the highlight segment analysis algorithm is to obtain the position of the highlight segment of the video 1. The position of the highlight segment is specifically represented by the start time point and the end time point of the highlight segment. Figure 13 As shown, the duration of video 1 is 60 seconds. Video 1 is analyzed using a highlight segment analysis algorithm, and the segments between the 20th second (i.e., the start time) and the 30th second (i.e., the end time) are determined to be highlight segments (e.g., multiple consecutive frames of images containing a person's smiling face).

[0384] S130 , when the algorithm module of the HAL layer analyzes the video 1 using the highlight segment analysis algorithm, the policy monitoring module monitors the analyzed duration of the video 1 .

[0385] The analyzed duration of video 1 refers to the time taken to analyze video 1.

[0386] The policy monitoring module can monitor the analyzed duration of video 1 through a timer. For example, when the policy monitoring module sends the file descriptor of video 1 and the set analysis policy to the channel interface, it can start the timer and use the timer duration as the analyzed duration of video 1. In addition, the policy monitoring module can also start the timer and use the timer duration as the analyzed duration of video 1 when it detects that the algorithm module of the HAL layer begins to analyze video 1. This embodiment of the present application does not limit this.

[0387] S131, the policy monitoring module determines whether the analysis of video 1 has timed out based on the analyzed duration of video 1.

[0388] The policy monitoring module can be Figure 7 The method described in step 708 of the embodiment determines whether the analysis of video 1 has timed out, and the embodiment of the present application will not be repeated here.

[0389] If the policy monitoring module determines that video 1 has not been analyzed and has timed out, no processing is performed.

[0390] S132, the algorithm module of the HAL layer transmits the analysis results of video 1 to the policy monitoring module of the media middle platform framework layer.

[0391] Among them, the analysis results of Video 1 pass through the one-click film-making module of the FWK layer and the channel interface of the media middle platform framework layer during transmission.

[0392] The analysis result of video 1 includes the position of the highlight segment, for example, video 1: [20s, 30s].

[0393] S133: The policy monitoring module determines whether the overall analysis of all videos has timed out.

[0394] The policy monitoring module can monitor the analyzed duration of all videos and determine whether the overall analysis of all videos has timed out based on the analyzed duration of all videos.

[0395] The analyzed duration of all videos refers to the time taken to analyze all videos.

[0396] The policy monitoring module can be Figure 7 The method described in step 711 of the embodiment determines whether the overall analysis of all videos has timed out, and the embodiment of the present application will not be repeated here.

[0397] S134: If all videos have not been analyzed as a whole within the time limit, the policy monitoring module determines whether the analysis progress of all videos is behind schedule.

[0398] The policy monitoring module can be Figure 7 The method described in step 715 of the embodiment determines whether the analysis progress of all videos is behind schedule, and the embodiment of the present application will not be repeated here.

[0399] S126-S134 illustrate the analysis process of a video. S126-S124 can be executed in a loop to analyze each video in turn. For example, after the analysis of video 1 is completed, the analysis of video 2 can be continued in the same manner as S126-S124.

[0400] S135, if the analysis progress of all videos is not behind, the policy monitoring module sends the file descriptor of video 2 and the set analysis policy to the channel interface.

[0401] S136 , the channel interface decodes, converts the format, reduces the resolution, and performs other processing on the video 2 , and stores the processed video 2 .

[0402] S137: The channel interface sends the frame data address of video 2 to the HAL layer through the one-key slice module of the FWK layer.

[0403] S138 , the algorithm module of the HAL layer finds the processed video 2 according to the frame data address, and analyzes the video 2 using a highlight segment analysis algorithm.

[0404] S139 , when the algorithm module of the HAL layer analyzes the video 2 using the highlight segment analysis algorithm, the policy monitoring module monitors the analyzed duration of the video 2 .

[0405] S140: The policy monitoring module determines whether the analysis of video 2 has timed out based on the analyzed duration of video 2.

[0406] If the policy monitoring module determines that video 1 has not been analyzed and has timed out, no processing is performed.

[0407] S141 , if the analysis of video 2 times out, the policy monitoring module sends a stop instruction to the algorithm module of the HAL layer, where the stop instruction is used to instruct the algorithm module of the HAL layer to stop analyzing video 2 .

[0408] For example, the stop instruction may carry the file descriptor of video 2 to instruct the algorithm module of the HAL layer to stop analyzing video 2.

[0409] S142 : The algorithm module of the HAL layer stops analyzing the highlight segment position of video 2 according to the stop instruction.

[0410] When the HAL layer's algorithm module stops analyzing the highlight location of Video 2, the location of Video 2's highlight segment may already be included in the analyzed results. This means that the algorithm module has already analyzed and obtained the location of Video 2's highlight segment and sent it to the policy monitoring module. Alternatively, the location of Video 2's highlight segment may not yet be included in the analyzed results. This means that the algorithm module has not yet analyzed and obtained the location of Video 2's highlight segment. The policy monitoring module can perform different operations for these two situations.

[0411] For example, after sending a stop instruction to the algorithm module of the HAL layer, the policy monitoring module can determine whether the highlight segment position of video 2 has been obtained from the algorithm module of the HAL layer. If not, jump to S143; if so, jump to S143.

[0412] S143: If the strategy monitoring module has not yet obtained the highlight segment position of video 2, the strategy monitoring module changes the analysis strategy of video 2 to a simple analysis strategy based on I frames.

[0413] That is, the pre-set analysis strategy for video 2 is changed to a simple analysis strategy based on I frames.

[0414] Figure 11 Another schematic flow chart of the monitoring scheduling method for video processing provided by an embodiment of the present application is shown. Figure 10 ,like Figure 11 As shown, after S143, the monitoring scheduling method for video processing further includes the following S144-S157.

[0415] S144, the policy monitoring module determines whether the overall analysis of all videos has timed out.

[0416] S145: If all videos have not been analyzed as a whole within the time limit, the policy monitoring module determines whether the analysis progress of all videos is behind schedule.

[0417] S146: If the analysis progress of all videos is not behind, the policy monitoring module sends the file descriptor of video 2 and the simple analysis policy based on I frame to the channel interface.

[0418] Since the analysis of Video 2 has not yet been completed, it can continue to be analyzed in the next round of analysis. Therefore, the file descriptor of Video 2 can be sent to the channel interface. In addition, since the analysis strategy set for Video 2 has been changed, the changed I-frame-based simple analysis strategy can be sent to the channel interface so that the algorithm module can analyze Video 2 using this I-frame-based simple analysis strategy.

[0419] S147 , the channel interface decodes, converts the format of, and reduces the resolution of the video 2 , and stores the processed video 2 .

[0420] S148: The channel interface sends the frame data address of video 2 to the HAL layer through the one-key slice module of the FWK layer.

[0421] S149, the algorithm module of the HAL layer finds the processed video 2 according to the frame data address, and analyzes the video 2 using the highlight segment analysis algorithm.

[0422] S150 , when the algorithm module of the HAL layer analyzes the video 2 using the highlight segment analysis algorithm, the policy monitoring module monitors the analyzed duration of the video 2 .

[0423] S151, the policy monitoring module determines whether the analysis of video 2 has timed out based on the analyzed duration of video 2.

[0424] If video 2 is not analyzed and times out, the policy monitoring module does not process it.

[0425] S152, the algorithm module of the HAL layer transmits the analysis results of video 2 to the policy monitoring module of the media middle platform framework layer.

[0426] S153: The policy monitoring module determines whether the overall analysis of all videos has timed out.

[0427] If the overall analysis of all videos times out, the process jumps to S154. If the overall analysis of all videos does not time out, the process jumps to S156.

[0428] S154: If the overall analysis of all videos times out, the policy monitoring module changes the analysis policies of all remaining unanalyzed videos to fixed-length analysis policies.

[0429] S155 , the strategy monitoring module adopts a fixed-length analysis strategy to sequentially analyze the positions of highlight segments of the remaining unanalyzed videos to obtain analysis results for each of the remaining unanalyzed videos.

[0430] After S155 is completed, the policy monitoring module can obtain the analysis results of all images and all videos, and then jump to S168 to report the analysis results of all materials to the application.

[0431] S156: If all videos have not been analyzed as a whole within the time limit, the policy monitoring module determines whether the analysis progress of all videos is behind schedule.

[0432] S157, if the analysis progress of all videos is behind schedule, the policy monitoring module re-plans the target range, expected analysis time and maximum tolerance of timeout, and jumps to S124 based on the re-planned target range, expected analysis time and maximum tolerance of timeout.

[0433] The target range of re-planning is the remaining unanalyzed videos. Figure 7 In the embodiment, the method described in step 716 is used to re-plan the target range, the expected analysis duration, and the maximum tolerance of the timeout duration.

[0434] After jumping to S124, the policy monitoring module can re-plan the target range's allocated duration and analysis strategy based on the re-planned target range, the expected analysis duration, and the maximum timeout tolerance. For example, the policy monitoring module can allocate analysis duration for each video in the re-planned target range (the remaining unanalyzed videos) based on the re-planned expected analysis duration, and then dynamically set the analysis strategy for each video in the re-planned target range. Then, based on the re-planned allocated duration and analysis strategy, each video in the re-planned target range is analyzed sequentially.

[0435] Figure 12 Another schematic flow chart of the monitoring scheduling method for video processing provided by an embodiment of the present application is shown. Figure 11 ,like Figure 12 As shown, after S157, the monitoring scheduling method for video processing further includes the following S158-S177.

[0436] As an example, the re-planning process may be implemented through the following steps S158-S168.

[0437] S158: The policy monitoring module allocates analysis time to each of the remaining unanalyzed videos according to the re-planned expected analysis time.

[0438] S159, the strategy monitoring module dynamically sets the analysis strategy.

[0439] The policy monitoring module can dynamically set analysis policies for each of the remaining unanalyzed videos.

[0440] S160, the policy monitoring module sends the file descriptor of video 3 and the set analysis policy to the channel interface.

[0441] S161: The channel interface decodes, converts the format of, and reduces the resolution of the video 3, and stores the processed video 3.

[0442] S162: The channel interface sends the frame data address of video 3 to the HAL layer through the one-key slice module of the FWK layer.

[0443] S163 , the algorithm module of the HAL layer finds the processed video 3 according to the frame data address, and analyzes the video 3 using a highlight segment analysis algorithm.

[0444] S164 , when the algorithm module of the HAL layer analyzes the video 3 using the highlight segment analysis algorithm, the policy monitoring module monitors the analyzed duration of the video 3 .

[0445] S165 , the policy monitoring module determines whether the analysis of video 3 has timed out based on the analyzed duration of video 3 .

[0446] If video 3 is not analyzed and times out, the policy monitoring module does not process it.

[0447] S166, the algorithm module of the HAL layer transmits the analysis results of video 3 to the policy monitoring module of the media middle platform framework layer.

[0448] S167, the policy monitoring module determines whether the overall analysis of all videos has timed out.

[0449] S168: If all videos have not been analyzed as a whole within the time limit, the policy monitoring module determines whether the analysis progress of all videos is behind schedule.

[0450] If the analysis progress of all videos is not behind, the policy monitoring module will continue to process the next video.

[0451] S160-S168 illustrate the analysis process for a video. S160-S168 can be executed in a loop to analyze each remaining unanalyzed video in turn. After the analysis of a video is completed, S160-S168 are executed again to continue analyzing the next video. For example, after the analysis of video 3 is completed, S160-S168 are executed again to continue analyzing video 4.

[0452] S169, the policy monitoring module of the media middle platform framework layer calls the image highlight segment analysis interface to notify the application function layer of the application layer that the analysis of all material files has been completed, and reports the analysis results of all material files to the application function layer of the application layer.

[0453] S170 , the application function layer of the application layer edits and filters the material files selected by the user according to the analysis results of all the material files.

[0454] For example, see Figure 14 The analysis results of all material files include: Picture 1 contains a highlight clip (the highlight clip is marked as 1), Picture 2 does not contain a highlight clip, the highlight clip of Video 1 is [20s, 30s] (the highlight clip is marked as 2), the highlight clips of Video 2 are [5s, 10s] (the highlight clip is marked as 3) and [20s, 25s] (the highlight clip is marked as 4), and the highlight clips of Video 3 are [1s, 5s] (the highlight clip is marked as 5) and the 12th second (the highlight clip is marked as 6).

[0455] Then, based on the analysis results of all material files, the material files selected by the user are edited and filtered to retain the highlight clips.

[0456] S171, the application function layer of the application layer calls the theme summary interface of the media middle platform framework to obtain the theme template.

[0457] S172, the theme summary interface of the media platform framework layer determines the theme template that matches the scene based on the scene in the highlight clip.

[0458] S173, the theme summary interface of the media middle platform framework layer returns the theme template to the application function layer of the application layer.

[0459] For example, assuming that most of the scenes in the highlight clips are parent-child scenes, it can be determined that the theme template that matches the scene is a parent-child theme.

[0460] S174 , the application function layer of the application layer sends the theme template, the edited video clips, and the screened pictures to the basic capability layer.

[0461] S175 , the basic capability layer splices the edited video clips and the screened pictures, and applies the theme template to obtain the target video.

[0462] For example, please refer to Figure 14 , you can splice Image 1, [20s, 30s] of Video 1, [5s, 10s] and [20s, 25s] of Video 2, and [1s, 5s] and the 12th second of Video 3, i.e., splice highlight clips 1, 2, 3, 4, 5, and 6. Alternatively, apply a parent-child themed template to the spliced ​​videos to obtain the target video.

[0463] S176: The basic capability layer instructs the video editing service layer to display the target video.

[0464] S177, the video editing service layer displays the target video in the gallery interface.

[0465] In an embodiment of the present application, the media middle platform framework layer is used to decode video and image files, convert data formats into a unified format, monitor remaining time and adjust operation strategies, send data, control algorithm operation and termination, obtain results and return to the application layer, etc. The FKW layer is used to complete data packaging and provide data and program operation services. After receiving the command sent by the media middle platform framework layer, the HAL layer performs highlight analysis according to the command, and returns the parameter calculation results of the highlight analysis to the media middle platform framework layer. The final result of the algorithm is collected and organized by the media middle platform framework layer, and then sent to the application layer for processing. The application layer can present the editing application interface, video and image file options, and present the final result of the algorithm.

[0466] After the user starts the one-click film-making function, select the video and picture files that need to be edited (for example, a maximum of 30 files are supported). After waiting for a while, the "one-click film-making" application automatically edits to the highlight clips of the video, and combines the highlight clips and pictures according to the algorithm results to generate the edited short video, which can be previewed and played.

[0467] The video processing monitoring and scheduling method provided in the embodiments of the present application can monitor and schedule the video analysis process using an adaptive monitoring and scheduling algorithm during the video analysis process to ensure that the video analysis is completed within a reasonable time limit, thereby improving video analysis efficiency. For example, during the video analysis process, the analyzed video duration can be monitored. If the video analysis timeout is determined based on the analyzed video duration, the video analysis is stopped and the location of the video highlight segments is determined based on the analyzed video results. This prevents excessive video analysis time and ensures that the video analysis is completed within a reasonable time limit. Furthermore, the method can monitor whether the overall analysis of all videos has timed out. If so, a fixed-length analysis strategy is used to analyze the remaining unanalyzed videos. This can further accelerate video analysis and improve video analysis efficiency. Furthermore, the method can monitor whether the analysis progress of all videos is lagging behind. If so, the allocated analysis time and analysis strategy can be re-planned. This ensures that the analysis is completed within a reasonable time limit while dynamically planning the analysis time allocation and analysis strategy allocation, making full use of the analysis time and analysis strategy to achieve better analysis results and improve video analysis efficiency.

[0468] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (such as a coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0469] The above are optional embodiments provided for this application and are not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the technical scope disclosed in this application should be included in the scope of protection of this application.

Claims

1. A video processing monitoring and scheduling method, characterized in that: Applied to electronic equipment, the method includes: receiving a first operation, wherein the first operation is used to trigger splicing of multiple source files into a target video, the multiple source files including at least one video; In response to the first operation, performing a video analysis operation on the at least one video, the video analysis operation being configured to analyze a highlight segment of each video in the at least one video; During the process of performing the video analysis operation on the at least one video, monitoring the analyzed duration of a first video, the first video being the video being analyzed in the at least one video; If it is determined that the analysis of the first video has timed out based on the analyzed duration of the first video, stopping the analysis of the first video and determining the position of the highlight segment of the first video based on the analyzed result of the first video; The determining of the highlight segment position of the first video according to the analyzed result of the first video includes: If the analysis result of the first video contains a highlight segment position of the first video, obtaining the highlight segment position of the first video in the analysis result of the first video; If the highlight segment position of the first video does not exist in the analyzed results of the first video, a second analysis strategy different from the first analysis strategy is used to analyze the first video, where the first analysis strategy is the analysis strategy used to analyze the first video before stopping the analysis of the first video, and the video analysis time consumed by the second analysis strategy is less than the video analysis time consumed by the first analysis strategy.

2. The method according to claim 1, wherein The determining, based on the analyzed duration of the first video, that the first video analysis timeout has occurred, includes: If the analyzed duration of the first video meets a first preset condition, determining that the analysis of the first video has timed out; The first preset condition includes one or more of the following conditions: The analyzed duration of the first video is greater than or equal to the product of the allocated duration of the first video and a first preset ratio, where the allocated duration of the first video refers to the analysis duration pre-allocated for the first video; The analyzed duration of the first video is greater than or equal to the product of the maximum timeout duration tolerance of the at least one video and a first value, and the maximum timeout duration tolerance of the at least one video is used to indicate the allowed timeout duration of each video in the at least one video.

3. The method according to claim 2, wherein The first value is the product of the number of videos of the at least one video and a first preset value.

4. The method according to claim 2, wherein Before determining that the first video analysis times out, the method further includes: Determining a maximum timeout tolerance for the at least one video based on an expected analysis time of the at least one video; The maximum tolerance of the timeout duration is the larger value of the specified duration and the first preset duration, and the specified duration refers to the product of the expected analysis duration and the second preset ratio.

5. The method according to claim 1, wherein The method further comprises: During the process of performing the video analysis operation on the at least one video, monitoring the analyzed duration of the at least one video; After the first video analysis is completed, if it is determined that the overall analysis of the at least one video has timed out based on the analyzed duration of the at least one video, a third analysis strategy is used to analyze the unanalyzed videos in the at least one video in sequence to obtain the highlight segment position of each video in the unanalyzed video.

6. The method according to claim 5, wherein The third analysis strategy is a fixed-length analysis strategy, which refers to an analysis strategy that analyzes a preset position in the video as a highlight segment position.

7. The method according to claim 5, wherein The first video analysis ends in either of the following two situations: If it is determined that the first video analysis has exceeded the time limit according to the video analysis time of the first video, the first video analysis is stopped; If it is determined that the analysis of the first video has not exceeded the time limit according to the video analysis time of the first video, the analysis of the first video is completed.

8. The method according to claim 5, wherein The determining, based on the analyzed duration of the at least one video, that the at least one video has been analyzed for a timeout, includes: If the analyzed duration of the at least one video meets a second preset condition, determining that the overall analysis of the at least one video has timed out; The second preset condition includes one or more of the following conditions: The analyzed duration of the at least one video is greater than or equal to the product of the expected analysis duration of the at least one video and a third preset ratio; The analyzed duration of the at least one video is greater than or equal to the sum of the expected analysis duration of the at least one video and a second preset duration.

9. The method according to claim 1, wherein Before performing the video analysis operation on the at least one video, the method further includes: Allocate an analysis duration for each of the at least one video according to the expected analysis duration of the at least one video, the analysis duration allocated to each video being the allocated duration of each video; determining an analysis strategy for each video in the at least one video based on the allocated duration of each video in the at least one video; The performing a video analysis operation on the at least one video includes: A video analysis operation is performed on the at least one video according to the analysis strategy for each video in the at least one video.

10. The method according to claim 5, wherein The method further comprises: If it is determined that the at least one video has not been analyzed as a whole for a timeout based on the analyzed duration of the at least one video, determining whether the analysis progress of the at least one video is behind schedule based on the analyzed duration of the at least one video and the expected analysis duration; If the analysis progress of the at least one video falls behind, taking the unanalyzed video included in the at least one video as the at least one video, and replanning the expected analysis time of the at least one video; According to the re-planned expected analysis duration, return to the step of performing a video analysis operation on the at least one video.

11. The method according to claim 10, wherein The step of returning to the step of performing the video analysis operation on the at least one video according to the re-planned expected analysis duration includes: Re-allocating an analysis time for each video in the at least one video according to the re-planned expected analysis time, wherein the re-allocated analysis time for each video is the re-planned allocated time for each video; Re-determining an analysis strategy for each video in the at least one video according to the re-planned allocated duration of each video in the at least one video, wherein the re-determined analysis strategy for each video is the re-planned analysis strategy; A video analysis operation is performed on the at least one video according to the re-planned analysis strategy for each of the at least one video.

12. The method according to claim 10, wherein The determining, based on the analyzed duration and the expected analysis duration of the at least one video, whether the analysis progress of the at least one video is behind schedule includes: Whether the analysis progress of the at least one video is behind schedule is determined based on the analyzed time, expected analysis time, and maximum timeout tolerance of the at least one video, as well as the total allocated time of unanalyzed videos in the at least one video.

13. The method according to claim 10, wherein The determining whether the analysis progress of the at least one video is behind schedule according to the analyzed time, the total allocated time, the expected analysis time, and the maximum tolerance of the timeout time of the at least one video includes: If the difference between the second value and the expected analysis time of the at least one video is greater than or equal to the ratio of the maximum tolerance of the timeout time of the at least one video to a second preset value, it is determined that the analysis progress of the at least one video is behind schedule, and the second value is the sum of the analyzed time of the at least one video and the total allocated time of the unanalyzed videos in the at least one video.

14. The method according to any one of claims 1 to 13, wherein: The first operation is an operation in which the user selects the plurality of material files in the gallery and confirms to perform image stitching. Before receiving the first operation, the method further includes: In response to the second operation of the user, the videos and pictures in the gallery are loaded and displayed.

15. The method according to any one of claims 1 to 13, wherein: The performing a video analysis operation on the at least one video includes: Performing a preset process on the first video to obtain a processed first video; wherein the preset process includes at least one of the following: decoding, format conversion, and resolution reduction; The processed first video is analyzed.

16. The method according to any one of claims 1 to 13, characterized in that: The plurality of material files further include at least one picture, and before performing the video analysis operation on the at least one video, the method further includes: The at least one picture is analyzed in sequence to obtain an analysis result of each picture in the at least one picture, where the analysis result of each picture is used to indicate whether the corresponding picture is a highlight segment.

17. The method according to any one of claims 1 to 13, wherein: The method further comprises: In a case where the multiple material files also include pictures, all highlight segments are extracted from all videos and all pictures in the multiple material files, and all the highlight segments are spliced ​​together to obtain the target video.

18. The method according to claim 17, wherein After splicing all the highlight clips to obtain the target video, the method further includes: The target video is displayed in a gallery.

19. The method according to claim 18, characterized in that After displaying the target video in the gallery, the method further includes: In response to the user's third operation, the target video is post-processed, and the post-processing includes adding a theme, adding background music, re-editing, and / or video sharing.

20. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 19 when executed by the processor.

21. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 19.

Images