Video decoding method and electronic equipment

Abstract

The invention provides a video decoding method and electronic equipment, relates to the technical field of video data processing, and is used for improving the decoding efficiency of a to-be-analyzed material video when the electronic equipment analyzes the to-be-analyzed material video by using a one-key large-area function. The method comprises the following steps: after acquiring a first coding format and a first media parameter (namely a video parameter) of a to-be-analyzed material video, the electronic equipment determines that a first decoder which is in a working state and has a decoding parameter the same as the video parameter of the to-be-analyzed material video exists; and if so, the first decoder can be directly used for decoding the to-be-analyzed material video. And if the first decoder does not exist in the electronic equipment, when the electronic equipment determines that a second decoder which is in an idle state and has the same encoding format as the first encoding format exists, the electronic equipment starts the second decoder to decode the to-be-analyzed material video after setting the media parameter of the second decoder as the first media parameter.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of video data processing, and in particular to a video decoding method and an electronic device. Background Art

[0002] At present, some electronic devices can provide a one-click blockbuster function. Specifically, in response to the user's operation of using the one-click blockbuster function after selecting a material video, the electronic device can automatically analyze and extract the highlight segments in the material video through an algorithm, and automatically combine the highlight segments into a complete video.

[0003] Before the electronic device automatically analyzes the source video by algorithm, it needs to decode the source video. The existing video decoding method is inefficient and time-consuming, resulting in a long time spent on generating a finished video. Summary of the invention

[0004] The present application provides a video decoding method and an electronic device, which are used to improve the decoding efficiency of a material video to be analyzed when the electronic device uses a one-click blockbuster function to analyze the material video to be analyzed.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a video decoding method is provided, the method comprising: first, an electronic device obtains a first encoding format and a first media parameter (i.e., a video parameter) of a material video to be analyzed. Then, the electronic device queries whether there is a first decoder in the electronic device that is in a working state and whose decoding parameters are the same as the video parameters of the material video to be analyzed. Here, the decoding parameters of the first decoder are the same as the video parameters of the material video to be analyzed, which means that the second encoding format in the decoding parameters is the same as the first encoding format, and the second media parameters in the decoding parameters are the same as the first media parameters. Then, if the first decoder exists in the electronic device, the electronic device directly uses the first decoder to decode the material video to be analyzed. Then, if the first decoder does not exist in the electronic device, the electronic device queries whether there is a second decoder in an idle state and whose encoding format is the same as the first encoding format. Then, if the second decoder exists in the electronic device, the electronic device sets the media parameters of the second decoder to the first media parameters, and then enables the second decoder to decode the material video to be analyzed.

[0007] In summary, after the electronic device in the present application obtains the first encoding parameter and the first media parameter of the material video to be analyzed, it will first query whether there is a decoder that can be reused. For example, the decoder that can be reused refers to: a first decoder that is in a working state, the encoding parameter is the same as the first encoding parameter, and the media parameter is the same as the first media parameter, or a second decoder that is idle and the encoding parameter is the same as the first encoding parameter. For the first decoder, the electronic device can directly reuse the first decoder to decode the material video to be analyzed, without switching the first decoder between the working state and the idle state. Usually, in the process of switching the decoder from the idle state to the working state, a configuration operation needs to be performed. In the present application, it is not necessary to switch the first decoder between the working state and the idle state, so there is no need to frequently perform configuration operations, thereby improving decoding efficiency. For the second decoder, the electronic device only needs to set the media parameter of the second decoder to the first media parameter (that is, complete the configuration operation), and then enable the second decoder to decode the material video to be analyzed, thereby saving the time of creating the decoder, thereby improving decoding efficiency.

[0008] In a possible implementation of the first aspect, if the second decoder does not exist in the electronic device, the electronic device creates a third decoder whose encoding format is the first encoding format, and after setting the media parameters of the third decoder to the first media parameters, enables the third decoder to decode the video of the material to be analyzed.

[0009] In another possible implementation of the first aspect, the above-mentioned creation of the third decoder whose encoding format is the first encoding format may include: first, the electronic device queries whether the current number of decoders in the decoder cache pool of the electronic device is the maximum number of decoders in the decoder cache pool (i.e., N, N is a positive integer greater than or equal to 2). Then, if the current number of decoders in the decoder cache pool is equal to the maximum number of decoders in the decoder cache pool, the electronic device needs to release the decoder with the longest idle time in the decoder cache pool, and then create the third decoder in the decoder cache pool. If the current number of decoders in the decoder cache pool is less than the maximum number of decoders in the decoder cache pool, the electronic device can directly create the third decoder.

[0010] In another possible implementation of the first aspect, after the electronic device obtains the video parameters of the material video to be analyzed, the video decoding method provided by the present application may further include: first, the electronic device queries whether there is a fifth decoder in the electronic device that is in a working state and has the same encoding format as the first encoding format but different media parameters from the first media parameters. Then, if the fifth decoder exists in the electronic device, after the fifth decoder is in an idle state, the electronic device sets the media parameters of the fifth decoder to the first media parameters, and then enables the fifth decoder to decode the material video to be analyzed.

[0011] In this implementation, there may be multiple decoders in working state in the electronic device at the same time.

[0012] In another possible implementation of the first aspect, the electronic device may include multiple material videos to be analyzed. Before the electronic device queries whether the electronic device has a first decoder, the video decoding method provided by the present application may further include: the electronic device divides multiple material videos to be analyzed with the same first encoding format into one group to obtain at least one video group.

[0013] Wherein, each video group may include at least one video subgroup, and the media parameters of the material video to be analyzed in each video subgroup are the same. The electronic device may decode the material video to be analyzed in each video subgroup in turn. For each video subgroup: if there is a first decoder in the electronic device, the electronic device may directly use the first decoder to decode the material video to be analyzed in the video subgroup. Alternatively, if there is a second decoder, the electronic device sets the media parameters of the second decoder to the first media parameters, and then enables the second decoder to decode the material video to be analyzed in the video subgroup. Alternatively, after creating a third decoder, the electronic device sets the media parameters of the third decoder to the first media parameters, and then enables the third decoder to decode the material video to be analyzed. Alternatively, if there is a fifth decoder, the electronic device sets the media parameters of the fifth decoder to the first media parameters after the fifth decoder is in an idle state, and then enables the fifth decoder to decode the material video to be analyzed.

[0014] Based on this, for the material videos to be analyzed in the same video group, the electronic device can decode multiple material videos to be analyzed in one video group through the same decoder in the above decoders. In this way, during this process, the decoder can always work (i.e., there is no need to frequently switch between the working state and the idle state), and there is no need to repeat the configuration operation, which can further save time and thus further improve the video decoding efficiency.

[0015] In another possible implementation manner of the first aspect, the electronic device may arrange the to-be-analyzed material videos having the same first media parameter in each video group adjacent to each other, thereby dividing each video group into at least one video subgroup.

[0016] In another possible implementation of the first aspect, before the above-mentioned acquisition of video parameters of the material video to be analyzed, the video decoding method provided by the embodiment of the present application may also include: the electronic device acquires the material video to be analyzed and the file descriptor of the material video to be analyzed. Among them, the file descriptor of a material video to be analyzed is used to uniquely identify the material video to be analyzed. The above-mentioned acquisition of video parameters of the material video to be analyzed may include: first, the electronic device creates a media extractor corresponding to the file descriptor of the material video to be analyzed. Then, the electronic device uses the media extractor to acquire the video data of the material video to be analyzed from the material video to be analyzed, and extracts the video parameters of the material video to be analyzed from the video data.

[0017] In another possible implementation of the first aspect, the above-mentioned use of a decoder in an electronic device (such as at least one of the first decoder, the second decoder, the third decoder, and the fifth decoder) to decode the material video to be analyzed includes: the electronic device uses the decoder to decode each first image frame of a first number of first image frames in the material video to be analyzed.

[0018] The first number of first image frames may be evenly distributed at various positions of the material video to be analyzed; or, the first number of first image frames are I frames in the material video to be analyzed.

[0019] In another possible implementation of the first aspect, after decoding each of the first number of first image frames in the material video to be analyzed using the decoder in the electronic device, the video decoding method provided by the present application may also include: first, the electronic device performs image analysis on each decoded first image frame to obtain the image score of each corresponding first image frame. Then, the electronic device determines the area where multiple first image frames are located within a second preset time before and after the first target image frame as the target area of ​​the material video to be analyzed. At this time, the first target image frame is the first image frame with the highest image score among all the first image frames in the material video to be analyzed. Then, the electronic device obtains the second number of second image frames from the target area. Finally, the electronic device uses the decoder in the electronic device to decode each of the second number of second image frames in the target area. Wherein, the second number of image frames are evenly distributed at various positions of the target area of ​​the material video to be analyzed; or, the second number of image frames are I frames in the target area of ​​the material video to be analyzed.

[0020] In a second aspect, an electronic device is provided, the electronic device comprising a memory and one or more processors. The memory is coupled to the processor. The memory stores computer program code, the computer program code comprising instructions. When the instructions are executed by the processor, the electronic device performs the method as described in any one of the above-mentioned first aspects.

[0021] According to a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on an electronic device, the electronic device can execute any one of the methods according to the first aspect.

[0022] According to a fourth aspect, a computer program product comprising instructions is provided, which, when executed on an electronic device, enables the electronic device to execute any one of the methods according to the first aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method of the first aspect.

[0024] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of an interface provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0028] Figure 4 One of the flowcharts of a video decoding method provided in an embodiment of the present application;

[0029] Figure 5 A second flowchart of a video decoding method provided in an embodiment of the present application;

[0030] Figure 6 A third flowchart of a video decoding method provided in an embodiment of the present application;

[0031] Figure 7 A fourth flowchart of a video decoding method provided in an embodiment of the present application;

[0032] Figure 8 A fifth flowchart of a video decoding method provided in an embodiment of the present application;

[0033] Fig. 9 A sixth flowchart of a video decoding method provided in an embodiment of the present application;

[0034] Fig.10 A seventh flowchart of a video decoding method provided in an embodiment of the present application;

[0035] Fig.11 FIG8 is a flowchart of a video decoding method provided in an embodiment of the present application;

[0036] Fig.12 A ninth flowchart of a video decoding method provided in an embodiment of the present application;

[0037] Fig.13 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.

[0039] The terms "exemplary" or "for example" and the like in the embodiments of the present application are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the related concepts in a specific way.

[0040] The terms "coupling" and "connection" involved in the embodiments of the present application should be understood in a broad sense. For example, they may refer to a direct physical connection, or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0041] Electronic devices can provide a one-click blockbuster function. For example, an electronic device is installed with an image processing application (such as a gallery), which can support processing operations for editing materials such as pictures and videos. The electronic device can decode the material video to be analyzed through the image processing application, and then automatically analyze the material video to be analyzed through an algorithm to extract the highlight clips in the material video to be analyzed. Finally, the electronic device uses the image processing application to splice all the highlight clips extracted from the material video to be analyzed into a complete video, and display it on the display interface.

[0042] The video decoding method in conventional technology is inefficient and time-consuming, resulting in a long time spent on generating a finished video based on the material video to be analyzed, and a poor user experience.

[0043] To this end, an embodiment of the present application provides a video decoding method. After the electronic device obtains the first encoding parameter and the first media parameter of the material video to be analyzed, it will preferentially query whether there is a decoder that can be reused. For example, a decoder that can be reused refers to: a first decoder that is in a working state, whose encoding parameters are the same as the first encoding parameters, and whose media parameters are the same as the first media parameters, or a second decoder that is idle and whose encoding parameters are the same as the first encoding parameters. For the first decoder, the electronic device can directly reuse the first decoder to decode the material video to be analyzed. For the second decoder, the electronic device needs to set the media parameters of the second decoder to the first media parameters (that is, complete the configuration operation), and then enable the second decoder to decode the material video to be analyzed.

[0044] Based on this, the multiplexed decoder can avoid frequent repeated creation, configuration, and release of decoders, thereby improving video decoding efficiency.

[0045] The electronic device involved in the embodiments of the present application may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, or a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the electronic device.

[0046] Figure 1 A schematic diagram of an interface provided by an embodiment of the present application is shown. Figure 1 The application scenario of the video decoding method provided in the embodiment of the present application is exemplified.

[0047] In one embodiment, if Figure 1 The desktop 110 of the mobile phone shown in A in FIG. 1 shows an application icon 111 of the gallery. In response to a click operation on the application icon 111 of the gallery, as shown in FIG. Figure 1 As shown in B in FIG. 1 , the mobile phone displays the first gallery interface 120. The first gallery interface 120 includes a "one-click blockbuster" control 121. In response to a click operation on the "one-click blockbuster" control 121, as shown in FIG. Figure 1 As shown in C in FIG. 1 , the mobile phone displays the second gallery interface 130. The second gallery interface 130 may include multiple videos and multiple pictures. In response to a click operation on any video in the second gallery interface 130, as shown in FIG. Figure 1 As shown in D in FIG. 1 , the mobile phone displays the third gallery interface 140. The third gallery interface 140 includes all videos and all pictures. In response to the selection operation of at least one video in the third gallery interface 140, as shown in FIG. Figure 1 As shown in D in the figure, a first pop-up window 141 is displayed in the third gallery interface 140. The first pop-up window 141 includes a selected video and a video generation control 142. For example, the video generation control may be a one-click blockbuster control. It should be understood that the one-click blockbuster control in the first pop-up window 141 is different from the "one-click blockbuster" control 121 included in the first gallery interface 120. In response to a click operation on the video generation control 142, the mobile phone analyzes the selected video, selects highlight clips from the selected video, and generates a finished video based on the highlight clips. In this process, Figure 1 As shown in E in FIG. 1 , a second pop-up window 143 may be displayed in the third gallery interface 140. The second pop-up window 143 includes the analysis progress so that the user can intuitively view the analysis progress. Figure 1 As shown in F in FIG. 1 , the mobile phone displays a fourth gallery interface 150. The fourth gallery interface 150 displays the finished video.

[0048] In another embodiment, continue as Figure 1 As shown in E in FIG. 1 , the second pop-up window 143 of the third gallery interface 140 also displays a prompt message. For example, when the number of selected videos is less than N (such as N=3), the prompt message may be “More than 3 materials have better effects.” For another example, when the number of selected videos is greater than M (such as M=30), the prompt message may be “Select up to 30 materials” ( Figure 1 E in is not shown).

[0049] It should be noted that the videos and pictures displayed in the second gallery interface 130 and the third gallery interface 140 may be taken by a mobile phone, recorded by a mobile phone, or sent to the mobile phone by other electronic devices and then stored in the gallery. The embodiment of the present application does not limit the source of the videos and pictures displayed in the second gallery interface 130 and the third gallery interface 140.

[0050] Figure 2 The hardware structure diagram of an electronic device provided by an embodiment of the present application is shown. Figure 2 Introduce the hardware structure of electronic equipment.

[0051] Take the mobile phone as an example. Figure 2 As shown, the electronic device 200 may include: a processor 210, a memory 220, a universal serial bus (USB) interface 230, a power management module 240, an antenna, a communication module 250, a display screen 260, an audio module 270, a camera 280, a sensor module 290, etc.

[0052] The processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. The controller can be the nerve center and command center of the electronic device 200. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0053] The memory 220 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in the memory 220. The memory 220 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an interface display function, etc.). The data storage area may store data (such as notification messages) created during the use of the electronic device, etc. In addition, the memory 220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0054] The power management module 240 is used to connect the battery and the processor 210. The power management module 240 receives the battery and / or power input to power the processor 210, the memory 220, the communication module 250, the display screen 260, the audio module 270 and the camera 280. The power management module 240 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 240 can also be set in the processor 210.

[0055] The communication module 250 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 200. The communication module 250 can be one or more devices integrating at least one communication processing module. The communication module 250 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The communication module 250 can also receive the signal to be sent from the processor 210, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0056] In some embodiments, the antenna of the electronic device 200 is coupled to the communication module 250 so that the electronic device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, global navigation satellite system (GNSS), WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), Beidou navigation satellite system (BDS), global navigation satellite system (GLONASS) and / or Galileo satellite navigation system (GALILEO).

[0057] The electronic device 200 implements the display function through a GPU, a display screen 260, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 260 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0058] The display screen 260 is used to display images, videos (such as video clips), etc. The display screen 260 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-OLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In the embodiment of the present application, the display screen 260 can be used to display video clips.

[0059] The electronic device 200 can realize the shooting function through the ISP, the camera 280, the video codec, the GPU, the display screen 260 and the application processor.

[0060] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0061] The camera 280 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0062] The sensor module 290 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, among others.

[0063] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0064] Generally speaking, the realization of the one-click blockbuster function in electronic devices requires not only hardware support but also software cooperation. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. For example, combined with Figure 3 The software architecture of the electronic device involved in the embodiments of the present application is introduced.

[0065] Figure 3 A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application is shown.

[0066] like Figure 3 As shown in FIG. 1 , the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android operating system It is divided into four layers, from top to bottom: application (APP) layer, media middle platform framework layer, application framework (FWK) layer and hardware abstraction layer (HAL).

[0067] The application layer can include a series of application packages, such as camera, gallery, third-party video editing software, calendar, map and navigation, etc. When these application packages are running, they can access the various service modules provided by the media platform framework layer and the application framework layer through the application programming interface (API) and execute corresponding intelligent services.

[0068] In one embodiment, the camera is used to take photos, videos, slow-motion images, panoramic images, etc. in response to user operations. After these images are taken by the camera, or after the user triggers the mobile phone to take a screenshot, or after the user triggers the mobile phone to record the screen, or after the terminal device downloads images from other devices, the terminal device can save these images in the gallery, so that the user can perform video editing operations on the images in the gallery, such as a one-click blockbuster operation.

[0069] The embodiment of the present application divides the gallery into: a business layer, an application function layer and a basic function layer from top to bottom.

[0070] Among them, the business layer is also called the video editing business layer. The business layer includes multiple modules such as multi-lens video automatic filming, AI music short films, one-click blockbusters and wonderful moments, which are used to provide corresponding services respectively. These services are presented in the user interface (UI) of the gallery or the gallery interface in the form of controls. The user can trigger the camera to perform the corresponding video processing action by operating a certain control. For example, after the user selects one or more materials (the materials include material videos and / or material pictures) and clicks on the one-click blockbuster control of the gallery, the electronic device will decode the material to be analyzed, and automatically analyze and extract the highlight clips in the material video and the highlight pictures in the material pictures through the algorithm, and then combine all the highlight clips and all the highlight pictures into a edited video. Among them, the material to be analyzed can be all or part of the material selected by the user.

[0071] Among them, the application function layer includes an automatic editing framework. Each business in the business layer can call the automatic editing framework to provide automatic editing services for the material. Exemplarily, the automatic editing framework may include functional modules such as fragment optimization, storyline organization, layout splicing and special effects beautification. Fragment optimization is used to call the channel interface to decode, reduce resolution, convert formats, etc. of the material to be analyzed, and store the processed material to be analyzed. Fragment optimization is also used to call the image highlight fragment analysis interface, policy monitoring module, etc. to extract highlight fragments and / or highlight pictures from the material to be analyzed. Storyline organization is used to sequentially splice multiple materials to be analyzed in the form of a storyline based on the content of the material to be analyzed. Layout splicing is used to adjust the interface layout of the material to be analyzed. Special effects beautification is used to adjust the beautification effect of the material to be analyzed, such as adjusting the brightness of the picture and beautifying the face of the character.

[0072] Among them, the basic function layer is used to perform basic function processing on the edited video clips after the automatic editing framework edits multiple materials to be analyzed. Exemplarily, the basic function layer may include basic function modules such as video splicing, synthesis and preservation, video speed change, audio and video effect processing, etc. Among them, video splicing is used to splice all the extracted highlight clips and / or highlight pictures to obtain a finished video. Synthesis and preservation are used to store the finished video obtained after splicing. Video speed change is used to perform speed change processing on the finished video. For example, video speed change is used to add a slow playback effect to the part of the spliced ​​finished video that includes highlight actions. The audio and video effect processing module is used to add video effects and sound effects to the finished video obtained after splicing. For example, adding style filters and themes to the finished video, adding background music, etc.

[0073] The media middle platform framework layer is a software layer set between the application layer and the application framework. The media middle platform framework layer can include analysis performance query interface, policy monitoring module, channel (pipeline) interface, topic summary interface, initialization interface, image highlight segment analysis interface and decoder cache pool, etc.

[0074] The analysis performance query interface is used to obtain the estimated analysis duration of each video to be analyzed.

[0075] Among them, the strategy monitoring module is used to obtain the material to be analyzed from all materials, determine the image frames that need to be analyzed in each material video to be analyzed (such as the first image frame, the second image frame, etc. mentioned below), determine the target area of ​​each material video to be analyzed, determine the highlight segment of each material video to be analyzed, etc.

[0076] Among them, the channel interface is used to decode, convert formats, reduce resolution, etc. of the material to be analyzed, and forward the frame data address of the processed material to be analyzed to the hardware abstraction layer through the application framework layer, and then report the analysis results of the analyzed image frames returned by the hardware abstraction layer to the application function layer.

[0077] The channel interface is also used to perform frame reduction processing on the decoded material video to be analyzed, and retain some image frames in each material video to be analyzed. In this way, the time for electronic equipment to perform operations such as format conversion and resolution reduction can be reduced, which can improve efficiency.

[0078] The theme summary interface is used to call the theme algorithm, obtain the theme type that matches all highlight clips and / or highlight pictures, and send the theme type to the application function layer. The application function layer obtains the theme template corresponding to the theme type.

[0079] The initialization interface is used to initialize the algorithms related to the hardware abstraction layer.

[0080] The decoder buffer pool is used to create a decoder and record decoder status, decoding parameters, etc.

[0081] It should be noted that this application is described by taking the one-click blockbuster function provided by the gallery as an example, which does not limit the embodiments of this application. In actual implementation, third-party video editing software can use the video clip processing method provided in the embodiments of this application to combine multiple pictures and videos selected by the user into a complete video with one click.

[0082] The application framework layer, referred to as the framework layer, can be used to support the operation of various modules in the media platform framework layer. For example, the framework layer can include service interfaces such as one-click blockbuster interface, parameter management interface, image data transmission interface, theme analysis interface, and performance analysis interface.

[0083] The hardware abstraction layer may include a capability query interface and an algorithm module. The capability query interface is used to obtain the algorithm capabilities supported by the hardware abstraction layer. The algorithm module may include a highlight segment algorithm, a face detection algorithm, a video acceleration algorithm, etc.

[0084] The highlight segment algorithm is used to analyze the image frames in the video material to be analyzed to determine whether the image frames belong to the highlight segment. The highlight segment algorithm is also used to analyze the pictures in the pictures of the material to be analyzed to determine whether the pictures belong to the highlight pictures.

[0085] Among them, the face detection algorithm is used to perform face detection on image frames (or pictures).

[0086] The video acceleration algorithm is used to accelerate the video (such as the first completed video).

[0087] It should be noted that Figure 3 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the terminal device. In other embodiments, the terminal device may include more layers than shown in the figure, such as a system library (FWK LIB) layer and a kernel layer. Each layer may include more or fewer components than shown in the figure. In addition, the above-mentioned functional modules may also be combined into one functional module, and the layers may also be combined into one layer, such as the media middle platform framework layer can be set in the application framework layer.

[0088] It is understandable that, in order to implement the method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiment.

[0089] Figure 4 FIG. 1 shows one of the flow charts of a video decoding method provided in an embodiment of the present application. Figure 4 As shown, the method may include:

[0090] S401: The electronic device displays at least one candidate video and candidate picture.

[0091] For example, Figure 1 As shown in C, the mobile phone displays the second gallery interface 130. The second gallery interface 130 includes at least one video and at least one picture. The video can be called a candidate video, and the picture can be called a candidate picture.

[0092] The candidate video may be obtained by shooting or recording an electronic device, or may be received from other electronic devices. The candidate picture may be obtained by shooting or recording an electronic device, or may be received from other electronic devices. This application does not limit the source of the candidate video and candidate picture.

[0093] S402: In response to a selection operation on a candidate video, the electronic device obtains a material video from the candidate video, and in response to a selection operation on a candidate picture, the electronic device obtains a material picture from the candidate picture.

[0094] The material video is the selected candidate video. In other embodiments, the electronic device may obtain the material picture from the candidate pictures in response to the selection operation of the candidate picture. The material picture is the selected candidate picture.

[0095] In some embodiments, the electronic device may acquire the material video from the candidate videos only in response to the selection operation on the candidate videos. Alternatively, the electronic device may acquire the material picture from the candidate pictures only in response to the selection operation on the candidate pictures.

[0096] It should be noted that when an electronic device obtains a material video, it also obtains the file descriptor table (FD) of the material video. The file descriptor of a material video can uniquely identify the material video. In other words, the file descriptor and the material video are one-to-one corresponding. Similarly, when an electronic device obtains a material picture, it also obtains the file descriptor of the material picture.

[0097] S403: The electronic device obtains the video material to be analyzed and the picture material to be analyzed from all the materials according to the total analysis time and the recommended upper limit value of the total analysis time.

[0098] The total analysis duration may represent the sum of the durations required to analyze all materials selected by the user. The recommended upper limit of the total analysis duration represents the recommended value of the maximum duration to analyze all materials. The recommended upper limit of the total analysis duration may be calculated based on the estimated analysis duration of each material. Generally, the recommended upper limit of the total analysis duration is greater than the total analysis duration.

[0099] In the embodiment of the present application, the estimated analysis duration can be determined according to the following method:

[0100] In one embodiment, the estimated analysis duration of a material video may be the duration of analyzing one image frame in the material video, that is, the minimum analysis duration corresponding to the material video. For example, if the duration of analyzing one image frame by the algorithm module is 200ms, then the estimated analysis duration of each material video is 200ms.

[0101] In another embodiment, the estimated analysis duration of a material video may be the duration corresponding to analyzing all image frames in the material video.

[0102] In another embodiment, the estimated analysis duration of a material video may be the duration corresponding to analyzing M image frames in the material video, where M is a positive integer greater than 1, and M is less than the number of all image frames of the material video.

[0103] In another embodiment, the estimated analysis duration of a material picture is the duration of analyzing one image frame.

[0104] In the embodiment of the present application, the total analysis time can be determined according to the following method:

[0105] In one embodiment, the material only includes material pictures. In this case, the processing time for processing one material picture can be directly determined according to the algorithm capability. The sum of the estimated analysis times of all material pictures is determined by multiplying the number of material pictures by the processing time for processing one material picture (i.e., the estimated analysis time). In this case, the total analysis time is the sum of the estimated analysis times of all material pictures.

[0106] In another embodiment, the material only includes material videos. In this case, the estimated analysis durations of all material videos can be accumulated to obtain the sum of the estimated analysis durations of all material videos. In this case, the total analysis duration is the sum of the estimated analysis durations of all material videos.

[0107] In another embodiment, the material includes material videos and material pictures. The sum of the estimated analysis durations of all material videos and the sum of the estimated analysis durations of all material pictures are added together to obtain the total analysis duration of all materials.

[0108] The material to be analyzed may be all the material or part of all the material. For example, the material includes material pictures and material videos: the material to be analyzed may include all the material videos and all the material pictures; or, the material to be analyzed may include part of the material pictures and part of the material videos. For another example, the material includes material pictures: the material to be analyzed may include all the material pictures, or part of the material pictures. For another example, the material includes material videos: the material to be analyzed may include all the material videos, or part of the material videos.

[0109] The following article takes the materials including material pictures and material videos as examples to introduce the materials to be analyzed:

[0110] In one embodiment, if the total analysis duration is less than or equal to the recommended upper limit value of the total analysis duration, the materials to be analyzed may be all material pictures and all material videos.

[0111] In another embodiment, if the total analysis duration is greater than the upper limit of the total analysis duration, the material to be analyzed may be a portion of material pictures randomly selected from all material pictures based on a random selection strategy, and a portion of material videos randomly selected from all material videos. The random selection strategy is related to the importance of the material pictures and the material videos.

[0112] Exemplarily, the importance of the material pictures takes precedence over the importance of the material videos. At this time, the random selection strategy can be: every time P material pictures are selected, Q material videos are selected until the total analysis time of the selected materials reaches the recommended value of the upper limit of the analysis time, or the material pictures have been taken out. Among them, Q<P, and P and Q are both positive integers.

[0113] Exemplarily, the importance of the material video takes precedence over the importance of the material picture. At this time, the random selection strategy can be: each time A material videos are selected, B material pictures are selected until the total analysis time of the selected materials reaches the recommended value of the upper limit of the analysis time, or the material videos have been exhausted. Wherein, B<A, and A and B are positive integers.

[0114] It should be noted that the material to be analyzed is further processed below, while the material not to be analyzed is not further processed. In other words, the highlight clips come from the material to be analyzed.

[0115] S404: The electronic device performs image analysis on each of the material pictures to be analyzed, and selects highlight pictures from all the material pictures to be analyzed.

[0116] Specifically, the electronic device sequentially performs the following processing operations on each material picture to be analyzed: decoding, format conversion, resolution reduction, image analysis, etc. The specific process can be referred to the detailed description below, and the embodiment of the present application will not be described in detail here.

[0117] S405: The electronic device obtains video parameters of the material video to be analyzed.

[0118] The video parameters of the material video to be analyzed may include: a first encoding format and a first media parameter of the material video to be analyzed.

[0119] Specifically, the encoding format refers to a format used to store or transmit video. For example, the first encoding format may include the H26X series, but is not limited thereto. Wherein, X=4, 5, and other positive integers.

[0120] Specifically, media parameters refer to basic parameters of a video. For example, the first media parameter may include resolution, frame rate, rotation angle, etc. Resolution is used to indicate the number of pixels in the horizontal and vertical directions of a video. The higher the resolution, the more details and the higher the clarity of the video. Frame rate is used to indicate the number of image frames transmitted per second. The higher the frame rate, the smoother the video. Rotation angle is used to indicate whether the angle of the video is horizontal or vertical. Usually, the horizontal angle differs from the vertical angle by 90°.

[0121] In one embodiment, the electronic device obtains video parameters of a material video to be analyzed, which may include: first, the electronic device creates a media extractor corresponding to the file descriptor of the material video to be analyzed. Then, the electronic device uses the media extractor to separate the video data and audio data of the material video to be analyzed. Finally, the electronic device uses the media extractor to extract the video parameters of the material video to be analyzed from the video data of the material video to be analyzed.

[0122] Since the media extractor corresponds to the file descriptor one by one, and the file descriptor corresponds to the material video to be analyzed one by one, a media extractor can only extract the video parameters of a corresponding material video to be analyzed. Based on this, when the electronic device obtains the video parameters of a corresponding material video to be analyzed through a media extractor, the electronic device will release the media extractor.

[0123] S406: The electronic device queries whether there is a first decoder in working state and whose decoding parameters are the same as those of the material video to be analyzed.

[0124] The decoding parameters are the same as the video parameters of the material video to be analyzed, and may include: the second encoding format in the decoding parameters is the same as the first encoding format, and the second media parameters in the decoding parameters are the same as the first media parameters. Referring to the above introduction to the encoding format, it can be seen that the second encoding format can include the H26X series, but is not limited thereto. Referring to the above introduction to the media parameters, it can be seen that the second media parameters can include: resolution, frame rate, rotation angle, etc.

[0125] S407: If the first decoder exists, the electronic device uses the first decoder to decode the material video to be analyzed based on the target analysis strategy.

[0126] In one embodiment, when the electronic device uses the first decoder to decode the material video clip to be analyzed, the electronic device has completed the current decoding task through the first decoder. Then, the electronic device can directly decode the material video to be analyzed through the first decoder. In this process, the electronic device does not need to put the first decoder in an idle state (even if the first decoder is in a stopped state) and then start the first decoder to decode the material video to be analyzed. In this way, it can not only ensure that the electronic device can complete the decoding operation of each corresponding video to be decoded, but also does not need to frequently switch the first decoder between the working state and the idle state. Usually, in the process of switching the decoder from the idle state to the working state, it is necessary to perform a configuration operation. In this embodiment, it is not necessary to switch the first decoder between the working state and the idle state, so it is not necessary to frequently perform a configuration operation. Based on this, the efficiency of video decoding can be improved.

[0127] The configuration operation refers to that the electronic device uses the media parameters of the video to be decoded as the media parameters of the decoder. For example, the resolution of the video to be decoded is used as the resolution of the decoder, the frame rate of the video to be decoded is used as the frame rate of the decoder, and the rotation angle of the video to be decoded is used as the rotation angle of the decoder. The video to be decoded is the material video to be analyzed that needs to be decoded.

[0128] In one embodiment, the target analysis strategy may be an overview analysis-key area fine analysis strategy. Specifically, overview analysis refers to performing image analysis on a first number of image frames in a material video to be analyzed to obtain a target area of ​​the material video to be analyzed. Key area fine analysis refers to performing image analysis on a second number of image frames in a target area of ​​a material video to be analyzed to obtain highlight segments from the target area.

[0129] The first number may be greater than the second number; or, the first number may be equal to the second number; or, the first number may be equal to the second number, which is not limited in the embodiments of the present application.

[0130] The first number of image frames may be evenly distributed at various positions of the material video to be analyzed; or, the first number of image frames may be arbitrarily distributed at various positions of the material video to be analyzed; or, the first number of image frames are I frames in the material video to be analyzed. Similarly, the second number of image frames may be evenly distributed at various positions of the target area of ​​the material video to be analyzed; or, the second number of image frames may be arbitrarily distributed at various positions of the material video to be analyzed; or, the second number of image frames are I frames in the material video to be analyzed.

[0131] As can be seen from the above, the electronic device needs to decode the same material video to be analyzed twice. The first decoding is in the overview analysis stage. At this time, the electronic device only needs to decode the first number of image frames in the material video to be analyzed. The second decoding is in the key area detailed analysis stage. At this time, the electronic device only needs to decode the second number of image frames in the target area of ​​the material video to be analyzed.

[0132] For the convenience of description, each image frame in the first number of image frames is referred to as a first image frame, and each image frame in the second number of image frames is referred to as a second image frame.

[0133] It should be noted that the overview analysis process and the key area detailed analysis process will be described in detail below, and the embodiments of the present application will not be described in detail here. In other words, the process of obtaining the first number of first image frames in each material video to be analyzed and the process of determining the target area in each material video to be analyzed will be described in detail below, and will not be described in detail here.

[0134] In this embodiment, at the same time, the electronic device can decode the corresponding multiple videos to be decoded through multiple decoders at the same time. That is, at the same time, multiple decoders can be in working state in the electronic device. Or, at the same time, the electronic device can decode one video to be decoded through one decoder. That is, at the same time, only one decoder in the electronic device is in working state.

[0135] S408: If the first decoder does not exist, the electronic device queries whether there is a second encoder that is in an idle state and has a coding format that is the same as the first coding format.

[0136] In the embodiment of the present application, a decoder cache pool is provided in the electronic device. The decoder cache pool can be used to create a decoder, record the state of the decoder, and the decoding parameters of the decoder, etc. When the electronic device fails to query the first decoder, the electronic device will query whether there is a second decoder in the decoder cache pool.

[0137] S409: If there is a second decoder, the electronic device sets the media parameters of the second decoder to the first media parameters, and then enables the second decoder to decode the material video to be analyzed based on the target analysis strategy.

[0138] When the electronic device needs to enable a decoder in an idle state, the electronic device needs to configure the decoder in the idle state, and then enable the configured decoder to decode the video to be decoded. Therefore, in this embodiment, if the electronic device queries the second decoder, before using the second decoder to decode the material video to be analyzed, the electronic device needs to set the media parameters of the second decoder to the first media parameters of the material video to be analyzed, and then enable the second decoder.

[0139] S410: If the second decoder does not exist, the electronic device creates a third decoder whose encoding format is the first encoding format, sets the media parameters of the third decoder to the first media parameters, and then enables the third decoder to decode the material video to be analyzed based on the target analysis strategy.

[0140] When the electronic device does not find the second decoder in the decoder cache pool, it indicates that the second decoder does not exist in the electronic device. At this time, the electronic device needs to create a decoder with the same encoding format as the video to be decoded. Then, the electronic device configures the newly created decoder using the media parameters of the video to be decoded, and then enables the configured decoder to decode the video to be decoded.

[0141] Optionally, a decoder cache pool can store at most N decoders, where N is a positive integer greater than or equal to 2. That is, N is the maximum number of decoders in the decoder cache pool. When an electronic device creates a new decoder in the decoder cache pool, the electronic device needs to determine whether the current number of decoders in the decoder cache pool is equal to the maximum number.

[0142] In one embodiment, if the electronic device determines that the current number of decoders in the decoder cache pool is equal to the maximum number, it indicates that the decoder cache pool cannot continue to store new decoders. At this time, the electronic device needs to release the fourth decoder in the decoder cache pool and create a third decoder in the decoder cache pool. If the electronic device determines that the current number of decoders in the decoder cache pool is less than the maximum number, it indicates that new decoders can continue to be stored in the decoder cache pool. At this time, the electronic device directly creates the third decoder in the decoder cache pool.

[0143] Among them, the duration between the moment when the fourth decoder performs the decoding operation and the current moment is the longest. In other words, the fourth decoder is idle for the longest time in the decoder buffer pool. At this time, in order to release the buffer space of the decoder buffer pool, the fourth decoder can be released to store the newly created third decoder.

[0144] In one embodiment, after decoding the material video to be analyzed, the electronic device will also perform frame reduction processing to retain some image frames in each material video to be analyzed. For example, the material video to be analyzed includes 60 frames per second. After the frame reduction is completed, the material video to be analyzed includes 30 frames per second. In this way, the time for the electronic device to perform operations such as format conversion and resolution reduction can be reduced, and efficiency can be improved.

[0145] S411 , the electronic device performs image analysis on the decoded material video to be analyzed to obtain all highlight segments.

[0146] Specifically, referring to the introduction in S407 above, it can be known that the electronic device uses a decoder to decode the material video to be analyzed twice. The following describes the following situations:

[0147] In one embodiment, after the electronic device decodes a material video to be analyzed for the first time, it obtains the frame data addresses of each first image frame in the material video to be analyzed. Then, based on the frame data addresses of each first image frame, the electronic device obtains the corresponding first image frames from the material video to be analyzed, and performs image analysis on each first image frame to obtain the corresponding analysis results of each first image frame. Among them, the analysis results of the first image frames include the image scores of the first image frames. Then, the electronic device finds the first target image frame with the highest image score from each first image frame. Finally, the electronic device determines the area where multiple consecutive first image frames within a first preset time length before and after the first target image frame are located as the target area of ​​the material video to be analyzed.

[0148] Among them, the first preset duration is related to the actual duration of the material video to be analyzed, the recommended duration of the highlight segment, the first quantity, etc.

[0149] In one embodiment, after decoding a material video to be analyzed for the second time, the electronic device obtains the frame data addresses of each second image frame in the target area of ​​the material video to be analyzed. Then, based on the frame data addresses of each second image frame, the electronic device obtains the corresponding second image frames from the material video, and performs image analysis on each second image frame to obtain the corresponding analysis results of each second image frame. Among them, the analysis results of the second image frames include the image scores of the second image frames. Then, the electronic device finds the second target image frame with the highest image score from each second image frame. Finally, the electronic device determines the area where multiple consecutive second image frames within a second preset time length before and after the second target image frame are located as the highlight segment of the material video to be analyzed.

[0150] It can be seen that there is no strict order between the steps in the embodiment of the present application. For example, part of the process in S411 can be interspersed with part of the steps in S407-S410 above.

[0151] S412: The electronic device obtains a first complete video based on all highlight segments and all highlight images.

[0152] In one embodiment, after the electronic device obtains multiple highlight segments, it can directly splice the multiple highlight segments to obtain a first complete video.

[0153] In another embodiment, after the electronic device obtains all the highlight clips and all the highlight pictures, it first obtains a theme template corresponding to the content of all the highlight clips and all the highlight pictures, and then directly applies the theme template to all the highlight clips and all the highlight pictures after splicing them to obtain a first complete video. The theme template can be obtained by the electronic device applying a theme algorithm to analyze all the highlight clips and all the highlight pictures.

[0154] In another embodiment, after the electronic device obtains all the highlight clips and all the highlight pictures, as well as the theme templates corresponding to all the highlight clips and all the highlight pictures, it splices all the highlight clips and all the highlight pictures to obtain a second video. Then, the electronic device performs video re-editing processing such as speed change processing and adding special effects on the second video to obtain a third video. Finally, the electronic device applies the theme template to the third video to obtain the first video.

[0155] S413: The electronic device displays the first completed video.

[0156] For example, Figure 1 As shown in F in FIG. 1 , the mobile phone displays a fourth gallery interface 150. The fourth gallery interface 150 includes finished videos.

[0157] Optionally, after S405 and before S406, the electronic device may group the multiple materials to be analyzed so that the multiple material videos to be analyzed with the same video parameters are in one video group.

[0158] In one embodiment, the electronic device may group multiple material videos to be analyzed with the same first encoding format into one group to obtain at least one video group; wherein each video group includes at least one video subgroup, and the media parameters of the material videos to be analyzed in each video subgroup are the same. For example, the electronic device may arrange the material videos to be analyzed with the same first media parameters in each video group adjacently, thereby obtaining at least one video subgroup in each video group.

[0159] Figure 5 FIG2 shows a second flow chart of a video decoding method provided by an embodiment of the present application. Figure 4 ,like Figure 5 As shown, the electronic device may include multiple material videos. After the above S405, the method may include:

[0160] S501: The electronic device divides the to-be-analyzed material videos having the same first encoding format into one group to obtain at least one video group.

[0161] In this embodiment, only one decoder in the electronic device is in working state at the same time. Based on this, after obtaining the video parameters of each material video to be analyzed, the electronic device first divides all the material videos to be analyzed with the same first encoding format into one group to obtain at least one video group. In this way, the electronic device can reuse the decoder of the same encoding format to decode all the material videos to be analyzed in one video group, saving the time of the electronic device to query the decoder, thereby further improving the video decoding efficiency.

[0162] S502: The electronic device arranges the to-be-analyzed material videos with the same first media parameter in each video group adjacently.

[0163] In order to further improve the efficiency of video decoding, the electronic device can sort multiple material videos to be analyzed in the same video group based on the first media parameters of each material video to be analyzed in the same video group, so that the material videos to be analyzed with the same first media parameters in the same video group are arranged adjacent to each other, and each video group is divided into at least one video group. Then, the electronic device can decode the material videos to be analyzed in each video group in turn. Based on this, for the material videos to be analyzed in the same video group, the electronic device can decode multiple material videos to be analyzed in one video group through the same decoder in the above decoders.

[0164] Exemplarily, for each video group: if there is a first decoder in the electronic device, the electronic device directly uses the first decoder to decode the material video to be analyzed in the video group. Alternatively, if there is a second decoder, the electronic device sets the media parameters of the second decoder to the first media parameters, and then enables the second decoder to decode the material video to be analyzed in the video group. Alternatively, after creating a third decoder, the electronic device sets the media parameters of the third decoder to the first media parameters, and then enables the third decoder to decode the material video to be analyzed. Alternatively, if there is a fifth decoder, the electronic device sets the media parameters of the fifth decoder to the first media parameters after the fifth decoder is in an idle state, and then enables the fifth decoder to decode the material video to be analyzed.

[0165] During this process, the decoder can work all the time (ie, it does not need to frequently switch between the working state and the idle state), and does not need to repeat the configuration operation. In this way, the electronic device can further save time, thereby further improving the video decoding efficiency.

[0166] It should be noted that, for the first material video to be analyzed in each video group, the electronic device needs to configure the decoder according to the specific situation of the decoder found, so that the decoder is in a working state, or the decoder is in an idle state, etc. However, for the non-first material video to be analyzed in each video group, the electronic device can directly use the decoder used for the first material video to be analyzed in the corresponding video group for decoding.

[0167] Exemplarily, the first media parameter may include resolution, frame rate, and rotation angle. For multiple material videos to be analyzed in the same video group, first, the electronic device may arrange the material videos to be analyzed in descending order according to the resolution; then, the electronic device may arrange the material videos to be analyzed in descending order according to the frame rate; finally, the electronic device may arrange the material videos to be analyzed in descending order according to the rotation angle.

[0168] Figure 6 FIG3 shows a flow chart of a video decoding method provided by an embodiment of the present application. Figure 4 ,like Figure 6 As shown, before the above S408, the method may include:

[0169] S601: The electronic device queries whether there is a fifth decoder which is in working state and has the same encoding format as the first encoding format of the material video to be analyzed, but has different media parameters from the first media parameters of the material video to be analyzed.

[0170] In this embodiment, at the same time, there may be multiple decoders in working state in the electronic device. Therefore, after the electronic device fails to find a first decoder in working state and whose decoding parameters are the same as the video parameters of the material video to be analyzed, the electronic device may check whether there is a decoder in working state and whose decoding parameters are different from the video parameters of the material video to be analyzed.

[0171] The decoding parameters of the decoder are different from the video parameters, which may include: the encoding format of the decoder is different from the first encoding format, but the media parameters of the decoder are the same as the first media parameters; or the encoding format of the decoder is the same as the first encoding format, but the media parameters of the decoder are different from the first media parameters; or the encoding format of the decoder is different from the first encoding format, and the media parameters of the decoder are different from the first media parameters. In this embodiment, a decoder in a working state, whose encoding format is the same as the first encoding format of the material video to be analyzed, but whose media parameters are different from the first media parameters of the material video to be analyzed is called a fifth decoder.

[0172] That is, after failing to find the first decoder, the electronic device firstly queries whether there is a fifth decoder in a working state, rather than directly querying whether there is a second decoder in an idle state.

[0173] S602: If there is a fifth decoder, the electronic device sets the media parameters of the fifth encoder to the first media parameters after the fifth decoder is in an idle state, and then enables the fifth decoder to decode the material video to be analyzed based on the target analysis strategy.

[0174] This step may refer to the introduction in the above S407, and will not be described in detail in this embodiment of the application.

[0175] Optionally, if the fifth decoder does not exist, the electronic device may execute the above-mentioned S408 and subsequent steps, which will not be described in detail in this embodiment of the application.

[0176] It should be noted that, in this embodiment, after obtaining the video parameters of all the material videos to be analyzed, the electronic device may group all the material videos to be analyzed with reference to the above S501 and S502, or may not group them, which is not limited in this embodiment of the present application.

[0177] The above text introduces the video decoding method provided by the embodiment of the present application with the electronic device as the execution subject. In the specific implementation process of the method, the support of various software and hardware modules in the electronic device is required. Figure 7-Figure 12 , through the interaction of various software and hardware modules in an electronic device, the video decoding method provided by an embodiment of the present application is introduced.

[0178] Figure 7 FIG4 shows a flow chart of a video decoding method provided by an embodiment of the present application. Figure 7 As shown, the method may include:

[0179] S01. The one-key blockbuster module of the business layer receives an operation by a user to enable a one-key blockbuster function.

[0180] In one embodiment, if Figure 1 As shown in B in FIG. 1 , the operation of enabling the one-key blockbuster function may be: a click operation on the one-key blockbuster control 121 in the first gallery interface 120 .

[0181] S02. The one-click blockbuster module of the business layer loads and displays candidate images and candidate videos.

[0182] In one embodiment, if Figure 1As shown in C, in response to a click operation on the one-key blockbuster control 121 in the first gallery interface 120, the one-key blockbuster module loads and displays multiple pictures and multiple videos in the second gallery interface 130. The videos in the second gallery interface 130 are candidate videos. The pictures in the second gallery interface 130 are candidate pictures.

[0183] In another embodiment, when there are only videos in the gallery, S02 is a one-key blockbuster module of the business layer that loads and displays candidate videos. Alternatively, when there are only pictures in the gallery, S02 is a one-key blockbuster module of the business layer that loads and displays candidate pictures.

[0184] S03. The one-key movie module of the business layer receives the user's operation of selecting a material video, and determines the operation of executing the one-key movie function.

[0185] The material video is a candidate video selected by the user. Of course, the one-key blockbuster module of the business layer in this step can receive the user's operation of selecting a material picture. The material picture is a candidate picture selected by the user.

[0186] For example, Figure 1 As shown in D in FIG. 1 , the operation of selecting the material video and the material picture may be: clicking the candidate video and the candidate picture in the third gallery interface 140, etc. Figure 1 As shown in D in FIG. 1 , the operation of determining to execute the one-key film-making function may be: a click operation on the video generation control 142 in the third gallery interface 140 .

[0187] S04. The one-click blockbuster module of the business layer calls the initialization interface of the media middle platform framework layer through the application function layer to initialize the relevant algorithms of the hardware abstraction layer.

[0188] The relevant algorithms refer to the algorithms of the business functions to be implemented by the business layer. The business function here is the one-click blockbuster function. For example, the relevant algorithms may include highlight segment algorithms, face detection algorithms, video acceleration algorithms, image super-resolution algorithms, etc.

[0189] S05. The initialization interface of the media middle platform framework layer sends initialization parameters to the algorithm module of the hardware abstraction layer through the channel interface of the media middle platform framework layer and the service interface corresponding to the application framework layer.

[0190] Among them, the service interface is used to perform data transparent transmission between the channel interface and the algorithm module. Figure 7 etc.) The application framework layer is not shown.

[0191] S06. The algorithm module of the hardware abstraction layer uses an initialization algorithm to initialize each algorithm.

[0192] S07. The algorithm module of the hardware abstraction layer sends an initialization success message to the channel interface of the media middle platform framework layer through the service interface of the application framework layer.

[0193] S08. The channel interface of the media middle platform framework layer calls the capability query interface of the hardware abstraction layer through the service interface of the application framework layer to query the algorithm capabilities supported by the algorithm module.

[0194] Among them, the algorithm capability supported by the algorithm module can be the chip analysis speed of the chip (such as an image signal processor) corresponding to the algorithm module. The chip analysis speed of different chips is different. For an electronic device that has been on the market, the chip in the electronic device is fixed, so the chip analysis speed corresponding to the electronic device is fixed. Based on this, the algorithm capability of the algorithm module in the electronic device is fixed.

[0195] In one embodiment, the algorithm capability may be a multiple. The larger the multiple, the stronger the algorithm capability and the faster the image analysis speed.

[0196] For example, take a 60-second material video as an example. If the multiple is 1, the analysis time of the material video is 60 seconds. If the multiple is 2, the analysis time of the material video is 30 seconds.

[0197] S09. The capability query interface of the hardware abstraction layer sends the algorithm capability to the initialization interface of the media middle platform framework layer through the service interface of the application framework layer and the channel interface of the media middle platform framework layer.

[0198] In one embodiment, the capability query interface may also send other relevant performance parameters of various algorithms to the initialization interface through the service interface and the channel interface.

[0199] S10. The initialization interface of the media middle platform framework layer sends an initialization success message to the one-click blockbuster module of the business layer through the application function layer.

[0200] The initialization success message may include the algorithm capability of the algorithm module in the hardware abstraction layer.

[0201] It should be noted that after S10, the following S11-S13 may be performed for each material video. The following description will be made by taking a material video 1 as an example.

[0202] S11. The one-click blockbuster module of the business layer sends a first query message to the analysis performance query interface of the media middle platform framework layer through the application function layer.

[0203] The first query message includes a file descriptor of a material video and the algorithm capability of the algorithm module. The following description will be made by taking a material video 1 as an example.

[0204] S12. The analysis performance query interface of the media middle platform framework layer obtains the estimated analysis time of video 1 according to the file descriptor of video 1.

[0205] For the introduction of estimating the analysis time, reference may be made to the relevant introduction in S403 above.

[0206] S13. The analysis performance query interface of the media middle platform framework layer sends the estimated analysis duration of video 1 to the one-click blockbuster module of the business layer through the application function layer.

[0207] S14. The one-click blockbuster module of the business layer obtains analysis parameters according to the estimated analysis duration of all materials.

[0208] Among them, all materials may include all material videos; or, all materials may include all material pictures; or, all materials may include all material videos and all material pictures.

[0209] Among them, the analysis parameters may include the total analysis duration, the recommended upper limit of the total analysis duration, the maximum duration of the highlight clip, the minimum duration of the highlight clip, the total recommended duration of the highlight clip, the recommended duration of the highlight clip, whether to force each material video to output a highlight clip, the selected highlight clip, etc.

[0210] The default value of whether to force each material video to output a highlight clip is yes. That is to say, in this embodiment, a highlight clip needs to be output for each material video. Of course, in other embodiments, whether to force each material video to output a highlight clip can be set according to actual conditions.

[0211] For the introduction of the total analysis time and the recommended upper limit of the total analysis time, please refer to the relevant introduction in S403 above.

[0212] The maximum duration of the highlight segment indicates the expected maximum duration of a highlight segment. The minimum duration of the highlight segment indicates the expected minimum duration of a highlight segment. The maximum duration of the highlight segment and the minimum duration of the highlight segment may be preset values.

[0213] The total recommended duration of highlight clips indicates the recommended value of the sum of the durations of highlight clips of all source videos. The total recommended duration of highlight clips can be determined based on the number of source videos, the maximum duration of highlight clips, and the minimum duration of highlight clips.

[0214] The recommended duration of the highlight segment indicates a recommended value of the duration of the highlight segment of a material video. The recommended duration of the highlight segment can be determined based on the maximum duration of the highlight segment and the minimum duration of the highlight segment.

[0215] S15. The one-click blockbuster module of the business layer sends the file descriptors and analysis parameters of all materials to the policy monitoring module through the image highlight segment analysis interface of the application function layer and the media middle platform framework layer.

[0216] After the policy monitoring module of the media platform framework layer obtains the file descriptors and analysis parameters of all materials (such as material pictures and / or material videos), in one embodiment, Figure 8 As shown, the electronic device may continue to execute the following S16-S21.

[0217] Figure 8 FIG5 shows a flowchart of a video decoding method provided by an embodiment of the present application. Figure 7 ,like Figure 8 As shown, after the above S15, the method may include:

[0218] S16. The policy monitoring module of the media middle platform framework layer obtains the material to be analyzed from all the materials based on the total analysis time and the recommended upper limit of the total analysis time.

[0219] Here, this step may refer to the relevant introduction in the above S403.

[0220] Generally, since the analysis process of the picture is relatively simple and time-consuming, when the material to be analyzed includes the material picture to be analyzed and the material video to be analyzed, the material picture to be analyzed will be processed first. For each material picture to be analyzed, the following S17-S21 can be performed. The following is an example of a material picture to be analyzed being picture 1.

[0221] S17. The policy monitoring module of the media middle platform framework layer sends a first indication message to the channel interface of the media middle platform framework layer.

[0222] The first indication message may include the file descriptor of picture 1.

[0223] S18. The channel interface of the media middle platform framework layer obtains picture 1 according to the file descriptor of picture 1, and processes picture 1 such as decoding, reducing resolution and converting format, and stores the processed picture 1.

[0224] Generally, in order to save the time of analyzing the image by the electronic device, the electronic device may reduce the resolution of the image before analyzing the image.

[0225] Exemplarily, first, the electronic device decodes the picture 1 to obtain the frame data address of the picture 1. Then, the electronic device converts the decoded picture 1 from the first format to the second format. Then, the electronic device reduces the resolution of the changed picture 1 from the first resolution to the second resolution. Then, the electronic device converts the picture 1 with the reduced resolution from the second format to the first format. Finally, the electronic device stores the picture 1 whose format is converted again.

[0226] It should be noted that since the resolution reduction algorithm only supports the second format, it is necessary to first convert Image 1 from the first format to the second format, then reduce the resolution of Image 1, and then convert the reduced resolution Image 1 to the original first format and store it in the memory.

[0227] Of course, in some embodiments, if the resolution reduction algorithm can support the first format, the electronic device does not need to perform format conversion processing on the picture 1, and can directly reduce the resolution after decoding. This embodiment of the present application does not limit this.

[0228] S19. The channel interface of the media middle platform framework layer sends the frame data address of picture 1 to the algorithm module of the hardware abstraction layer through the service interface of the application framework layer.

[0229] S20 . The algorithm module of the hardware abstraction layer obtains the picture 1 according to the frame data address of the picture 1 , and performs image analysis on the picture 1 to obtain an analysis result of the picture 1 .

[0230] After the algorithm module obtains the picture 1, the algorithm module can use the highlight segment algorithm to score the picture 1 according to the image color, image texture characteristics, image quality and edge change rate value of the picture 1 to obtain the analysis result. The analysis result may include the image score, or the analysis result may include the image score and the scoring result.

[0231] For example, when the image score of picture 1 is 0-19 points, the scoring result of picture 1 is "low". When the image score of picture 1 is 20-49 points, the scoring result of picture 1 is "medium". When the image score of picture 1 is 50-79 points, the scoring result of picture 1 is "high". When the image score of picture 1 is 80-100 points, the scoring result of picture 1 is "high".

[0232] In one embodiment, if the image score of a material picture to be analyzed is greater than or equal to 50 points, or the scoring result is "relatively high" or "high", the electronic device may determine the material picture to be analyzed as a highlight picture.

[0233] S21. The algorithm module of the hardware abstraction layer sends the analysis result of picture 1 to the policy monitoring module of the media middle platform framework layer through the channel interface of the application framework layer and the media middle platform framework layer.

[0234] After the policy monitoring module of the media platform framework layer obtains the analysis result of Picture 1, if there are other material pictures to be analyzed (such as Picture 2), the electronic device can continue to execute the above S17-S21 to obtain the analysis results of all material pictures to be analyzed.

[0235] After obtaining the analysis results of all the material images to be analyzed, if the material to be analyzed includes the material video to be analyzed, then for each material video to be analyzed, the electronic device can execute the following S22-S38 to obtain the analysis results of all the material videos to be analyzed. If the material to be analyzed does not include the material video to be analyzed, the electronic device directly outputs the first piece of video spliced ​​by all the highlight pictures. The following takes one of the material videos to be analyzed (such as video 1) as an example to introduce the image analysis process of video 1.

[0236] Fig. 9 FIG6 shows a sixth flow chart of a video decoding method provided by an embodiment of the present application. Figure 8 ,like Fig. 9 As shown, after the above S21, the method may include:

[0237] S22. The policy monitoring module of the media platform framework layer determines the first number of first image frames of each material video to be analyzed.

[0238] The first image frame refers to an image frame that needs to be analyzed in a material video to be analyzed in the overview analysis stage.

[0239] The first number may be greater than a preset minimum number (such as 5 frames), and the first number of a material video to be analyzed is less than the maximum number of image frames of the material video to be analyzed. The first number may be obtained by allocating the first total analysis number of the first image frames of all the material videos to be analyzed to each material video to be analyzed.

[0240] In one embodiment, in this step: first, the policy monitoring module determines the maximum number and basic number of the first image frames in each material video to be analyzed according to the video length and algorithm capability of each material video to be analyzed. Among them, the maximum number is greater than the basic number. Then, the policy monitoring module can determine the first total analysis number of the first image frames of all material videos to be analyzed according to the recommended upper limit of the total analysis time. Then, the policy monitoring module evenly distributes the first total analysis number to each material video to be analyzed. If there is still a remainder of the first total analysis number after the average distribution, the remaining number can be preferentially allocated to the material videos to be analyzed with longer durations until the first total analysis number is distributed.

[0241] The following describes the process of determining the number of first image frames in different situations:

[0242] If the duration of the material video to be analyzed is less than or equal to the first threshold, the number of the first image frame is 1.

[0243] If the duration of the material video to be analyzed is greater than the first threshold and less than or equal to the second threshold, the number of the first image frames is: the duration of the material video to be analyzed divided by S, rounded up, and then added to m. Where S is a positive integer. For example, S is 5 seconds, and the first threshold is 3 seconds.

[0244] If the duration of the material video to be analyzed is greater than the second threshold and less than or equal to the third threshold, the number of the first image frames is the sum of: the second threshold divided by S and rounded, the difference between the duration of the material video to be analyzed and the second threshold divided by K and rounded, and m. K is a positive integer, the third threshold is greater than the second threshold, and the second threshold is greater than the first threshold. For example, K is 10 seconds, the second threshold is 30 seconds, and the third threshold is 90 seconds.

[0245] It should be noted that if the duration of the material video to be analyzed is longer, more thresholds may be set, for example, a fourth threshold, a fifth threshold, etc. In the process of determining the maximum number and the basic number of first image frames according to the above method, it is determined that each threshold corresponding to the maximum number is less than or equal to each threshold corresponding to the basic number.

[0246] The following describes the process of determining the first total analysis quantity of the first image frame in different situations:

[0247] If the first multiple of the recommended upper limit of the total analysis time (such as 0.3) is less than the sum of the processing times of the basic number of first image frames in all the material videos to be analyzed, then the first total analysis number of the first image frames is the sum of the basic number of first image frames in all the material videos to be analyzed.

[0248] If the first multiple of the recommended upper limit of the total analysis time is greater than the sum of the processing times of the maximum number of first image frames in all the material videos to be analyzed, then the first total analysis number of first image frames is the sum of the maximum number of first image frames in all the material videos to be analyzed.

[0249] If the first multiple of the upper limit of the total analysis duration is greater than the sum of the processing durations of the basic number of first image frames in all the material videos to be analyzed, and less than the sum of the processing durations of the maximum number of first image frames in all the material videos to be analyzed, then the first total analysis number of the first image frames is: the first multiple of the upper limit of the total analysis duration divided by T, rounded to the integer. T is the time to process one image frame.

[0250] If the recommended upper limit of the total analysis time is less than the sum of the processing times of the basic number of first image frames in all the material videos to be analyzed, then the first total analysis number of the first image frames is: the recommended upper limit of the total analysis time divided by T and rounded to the integer.

[0251] It should be noted that after the first number of first image frames in each material video to be analyzed is determined, the following S23-S38 may be performed for each material video to be analyzed.

[0252] S23. The policy monitoring module of the media middle platform framework layer sends the file descriptor of video 1, video 1, and the position of the first image frame in video 1 to the channel interface of the media middle platform framework layer.

[0253] The first first image frame may be a first image frame at any position in the video 1 .

[0254] S24. The channel interface of the media platform framework layer creates a target channel with preset functions.

[0255] The preset function refers to: sequentially performing decoding, first format conversion, resolution reduction, second format conversion, etc. Alternatively, the preset function refers to: sequentially performing decoding, resolution reduction, etc.

[0256] S25. The channel interface of the media platform framework layer creates a media extractor corresponding to the file descriptor of video 1 in the target channel, and extracts the video parameters of video 1 through the media extractor.

[0257] The video parameters of video 1 may include: a first encoding format and a first media parameter.

[0258] In one embodiment, before querying the decoder, the channel interface can obtain the video parameters of all the material videos to be analyzed, and then arrange the material videos to be analyzed with the same video parameters based on the video parameters of all the material videos to be analyzed. Then, the following S26 and subsequent steps are performed. Specifically, the arrangement method can refer to the relevant introduction in the above S501 and S502.

[0259] S26. The channel interface of the media middle platform framework layer sends a second query message to the decoder cache pool of the media middle platform framework layer.

[0260] The second query message includes the video parameters of video 1. The second query message is used to instruct the decoder buffer pool to query whether there is a first decoder in the decoder buffer pool that is in working state and has the same decoding parameters as the video parameters of video 1.

[0261] S27. The decoder cache pool of the media platform framework layer queries whether there is a first decoder that is in working state and whose decoding parameters are the same as those of video 1.

[0262] Here, this step may refer to the relevant introduction in the above S406.

[0263] It should be noted that after the first decoder completes the current decoding task in the channel interface, it will return to the decoder buffer pool.

[0264] S28. If the first decoder exists, the decoder cache pool of the media platform framework layer returns the first decoder to the channel interface of the media platform framework layer.

[0265] S29. The channel interface of the media platform framework layer directly decodes the first first image frame through the first decoder in the target channel according to the position of the first first image frame.

[0266] After the decoder decodes the first first image frame, it can obtain the frame data address of the first first image frame. After the channel interface decodes the first first image frame in the target channel, it can also perform format conversion, resolution reduction and other processing on the first first image frame, and store the processed first first image frame.

[0267] In this step, the first decoder is used to decode the first first image frame. For example, reference may be made to the above-mentioned S407 for the description of the first decoder decoding the first first image frame.

[0268] S30: If the first decoder does not exist, the decoder cache pool of the media platform framework layer queries whether the second decoder exists.

[0269] The second decoder is in an idle state, and the encoding format is the same as the first encoding format. This step can refer to the relevant introduction in the above S408.

[0270] S31. If a second decoder exists, the decoder cache pool of the media platform framework layer returns the second decoder to the channel interface of the media platform framework layer.

[0271] S32. The channel interface of the media platform framework layer sets the media parameters of the second decoder to the first media parameters of video 1 in the target channel, and then enables the second decoder to decode the first first image frame.

[0272] For this step, reference may be made to the relevant introduction in S407 above.

[0273] S33. If the second decoder does not exist, the decoder cache pool of the media platform framework layer creates a third decoder whose encoding format is the first encoding format of video 1.

[0274] The process of creating the third decoder in this step may refer to the relevant introduction in the above S410.

[0275] S34. The decoder cache pool of the media middle platform framework layer sends the third decoder to the channel interface of the media middle platform framework layer.

[0276] S35. The channel interface of the media platform framework layer sets the media parameter of the third decoder to the first media parameter of video 1 in the target channel, and then enables the third decoder to decode the first first image frame.

[0277] S36. The channel interface of the media middle platform framework layer sends the frame data address of the first image frame to the algorithm module of the hardware abstraction layer through the service interface of the application framework layer.

[0278] S37. The algorithm module of the hardware abstraction layer obtains the first first image frame from video 1 according to the frame data address of the first first image frame, and performs image analysis on the first first image frame based on the highlight segment algorithm to obtain an analysis result of the first first image frame.

[0279] In this step, the image analysis process of the first image frame can refer to the relevant content of the image analysis of picture 1 in S20 above, and the present embodiment of the application will not be repeated here. Similarly, the analysis result of the first image frame can refer to the analysis result of picture 1 in S20 above.

[0280] S38. The algorithm module of the hardware abstraction layer sends the analysis result of the first image frame to the policy monitoring module of the media middle platform framework layer through the service interface of the application framework layer and the channel interface of the media middle platform framework layer.

[0281] It should be noted that, for a material video to be analyzed: after the policy monitoring module obtains the analysis result of the first first image frame, if there is a second first image frame, the above S23-S38 will continue to be executed until the number of analyzed first image frames reaches the first number.

[0282] In some embodiments, Fig.10 As shown, if there are multiple decoders in working state in the target channel at the same time, after S27, the method may include:

[0283] S271. If the first decoder does not exist, the decoder cache pool of the media platform framework layer queries whether there is a fifth decoder that is in working state and has the same encoding format as the first encoding format of video 1, but has media parameters that are different from the first media parameters of video 1.

[0284] S272. If the fifth decoder exists, the decoder cache pool of the media middle station framework layer returns the fifth decoder to the channel interface of the media middle station framework layer.

[0285] S273. After the fifth decoder is in an idle state, the channel interface of the media middle station framework layer sets the media parameters of the fifth decoder to the first media parameters of video 1 in the target channel, and then enables the fifth decoder to decode the first first image frame.

[0286] For this step, reference may be made to the relevant description in S602. If the fifth decoder does not exist, then the above S30 is continued to be executed. Otherwise, after S273, S36 is directly executed.

[0287] In one embodiment, the channel interface releases the target channel after processing all the material videos to be analyzed.

[0288] After obtaining the analysis results of all the first image frames in video 1, in some embodiments, refer to Fig.11 , the electronic device can continue to execute the following S39-S42.

[0289] Fig.11 FIG8 shows a flowchart of a video decoding method provided by an embodiment of the present application. Fig. 9 ,like Fig.11 As shown, after the above S38, the method may include:

[0290] S39. The policy monitoring module of the media platform framework layer determines the target area of ​​video 1 based on the analysis result of the first image frame in video 1.

[0291] The strategy monitoring module may determine the region where a plurality of first image frames are located within a first preset time before and after the first target image frame as the target region of the material video to be analyzed. The first target image frame is the first image frame with the highest image score among all the first image frames in the material video to be analyzed. A material video to be analyzed may include at least one target region.

[0292] It should be noted that the first preset duration is related to the actual duration of the material video to be analyzed, the recommended duration of the highlight segment, the first quantity, etc.

[0293] S40. The policy monitoring module of the media platform framework layer determines the second number of second image frames that need to be analyzed in the target area of ​​video 1.

[0294] The second number refers to the number of second image frames that need to be analyzed in the target area of ​​each material video to be analyzed within the remaining analysis time.

[0295] In one embodiment, first, the policy monitoring module determines a second total analysis number of second image frames to be analyzed in the target area of ​​all the material videos to be analyzed. Then, the policy monitoring module determines a second number of second image frames in the target area of ​​each material video to be analyzed based on the duration of the target area of ​​each material video to be analyzed and the score of the target area.

[0296] Exemplarily, the policy monitoring module may determine the second total analysis quantity according to the ratio of the remaining analysis time to the processing time of one image frame, but this is certainly not limited thereto.

[0297] Exemplarily, the policy monitoring module may determine the second number of second image frames for each target area according to a ratio of the number of all image frames included in each target area to the total number of image frames of all target areas.

[0298] S41, the policy monitoring module of the media middle platform framework layer, the channel interface, the decoder cache pool, the service interface of the application framework layer, and the algorithm module of the hardware abstraction layer cooperate with each other to obtain the analysis results of each second image frame in the target area of ​​video 1.

[0299] It should be noted that this step can refer to the relevant introduction in the above S23-S38, and will not be repeated in the embodiment of the present application.

[0300] S42. The policy monitoring module of the media platform framework layer determines the highlight segment of video 1 based on the analysis results of each second image frame in the target area of ​​video 1.

[0301] The policy monitoring module determines the region where multiple consecutive second image frames are located within a second preset time before and after the second target image frame as a highlight segment. The second target image frame is the second image frame with the highest image score among all second image frames in video 1.

[0302] It should be noted that the second preset duration is related to the actual duration of the target area, the recommended duration of the highlight segment, the second quantity, etc.

[0303] In the case where the material to be analyzed includes the material video to be analyzed and the material picture to be analyzed, after obtaining the highlight clips of all the material videos to be analyzed, in some embodiments, reference is made to Fig.12 , the electronic device can continue to execute the following S43-S52.

[0304] Fig.12 FIG9 shows a flow chart of a video decoding method provided by an embodiment of the present application. Fig.11 ,like Fig.12 As shown, after the above S42, the method may include:

[0305] S43. The policy monitoring module of the media middle platform framework layer sends the analysis results of all materials to be analyzed to the application function layer through the image highlight segment analysis interface of the media middle platform framework layer.

[0306] The analysis result of the material to be analyzed may include the position of the highlight picture and the position of the highlight segment.

[0307] S44. The application function layer clips and screens all the materials to be analyzed according to the analysis results of all the materials to be analyzed, and obtains all the highlight images and highlight clips.

[0308] S45. The application function layer calls the theme summary interface of the media middle platform framework to request the theme type.

[0309] Among them, the topic summary interface includes the topic algorithm.

[0310] S46. The topic summary interface of the media platform framework obtains the topic type.

[0311] S47. The topic summary interface of the media middle platform framework sends the topic type to the application function layer.

[0312] S48. The application function layer obtains a theme template corresponding to the theme type.

[0313] The application function layer includes a template set, and the template set includes a plurality of templates. After the application function layer obtains the theme type, it obtains a template corresponding to the theme type from the template set as the theme template.

[0314] S49. The application function layer sends the theme template, all highlight clips, and all highlight pictures to the basic capability layer.

[0315] S50: The basic capability layer splices all highlight clips and all highlight images, and applies a theme template to obtain a first complete video.

[0316] Among them, this step can refer to the relevant introduction in the above S413, and the embodiment of the present application will not be repeated here.

[0317] S51. The basic capability layer sends a second instruction message for playing a completed video to the business layer.

[0318] S52: The business layer plays the first completed video.

[0319] like Fig.13 As shown, an embodiment of the present application also provides a chip system. The chip system 1300 includes at least one processor 1301 and at least one interface circuit 1302. At least one processor 1301 and at least one interface circuit 1302 can be interconnected via lines. The processor 1301 is used to support the electronic device to implement each step in the above method embodiment, and at least one interface circuit 1302 can be used to receive signals from other devices (such as memory) or send signals to other devices (such as communication interfaces). The chip system may include chips and may also include other discrete devices.

[0320] An embodiment of the present application also provides a computer storage medium, which includes instructions. When the instructions are executed on the above-mentioned electronic device, the electronic device executes each step in the above-mentioned method embodiment.

[0321] The embodiment of the present application also provides a computer program product including instructions, and when the instructions are executed on the above electronic device, the electronic device executes each step in the above method embodiment.

[0322] Regarding the technical effects of the chip system, computer storage medium, and computer program product, refer to the technical effects of the previous method embodiments.

[0323] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0324] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0325] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0326] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0327] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one device or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0328] In addition, each functional module in each embodiment of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.

[0329] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer storage medium, or transmitted from one computer storage medium to another computer storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with a medium. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0330] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video decoding method, characterized in that: Applied to electronic equipment, the method comprises: Obtaining video parameters of the material video to be analyzed; wherein the video parameters include a first encoding format and a first media parameter of the video; If there is a first decoder in working state and the decoding parameters are the same as the video parameters, the first decoder is used to decode the material video to be analyzed; wherein the decoding parameters are the same as the video parameters, including: the second encoding format in the decoding parameters is the same as the first encoding format, and the second media parameters in the decoding parameters are the same as the first media parameters; If the first decoder does not exist, query whether there is a second decoder which is in an idle state and has a coding format that is the same as the first coding format; If the second decoder exists, after setting the media parameters of the second decoder to the first media parameters, the second decoder is enabled to decode the material video to be analyzed.

2. The method according to claim 1, characterized in that The method further comprises: If the second decoder does not exist, a third decoder whose encoding format is the first encoding format is created, and after the media parameters of the third decoder are set to the first media parameters, the third decoder is enabled to decode the material video to be analyzed.

3. The method according to claim 1 or 2, characterized in that: The step of creating a third decoder whose encoding format is the first encoding format comprises: Query whether the current number of decoders in the decoder cache pool of the electronic device is equal to N; wherein N is the maximum number of decoders in the decoder cache pool, and N is a positive integer greater than or equal to 2; If the current number is equal to N, the third decoder is created in the decoder buffer pool after the fourth decoder is released; wherein the fourth decoder is the decoder with the longest idle time in the decoder buffer pool; If the current number is less than the N, the third decoder is created.

4. The method according to any one of claims 1 to 3, characterized in that After obtaining the video parameters of the material video to be analyzed, the method further includes: If there is a fifth decoder in working state, and the encoding format is the same as the first encoding format, but the media parameters are different from the first media parameters, then after the fifth decoder is in idle state, the media parameters of the fifth decoder are set to the first media parameters, and then the fifth decoder is enabled to decode the material video to be analyzed.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device includes a plurality of material videos to be analyzed; before the first decoder is in working state and has the same decoding parameters as the video parameters, the method further includes: Divide a plurality of to-be-analyzed material videos having the same first encoding format into one group to obtain at least one video group; wherein each video group includes at least one video subgroup, and the media parameters of the to-be-analyzed material videos in each video subgroup are the same; The electronic device decodes the material video to be analyzed in each video group in turn; for each video group, if the first decoder exists, the first decoder is used to decode the material video to be analyzed in the video group; for each video group, if the second decoder exists, the media parameters of the second decoder are set to the first media parameters, and then the second decoder is enabled to decode the material video to be analyzed in the video group.

6. The method according to any one of claims 1 to 5, characterized in that Before obtaining the video parameters of the material video to be analyzed, the method further includes: obtaining the material video to be analyzed and a file descriptor of the material video to be analyzed; wherein a file descriptor of the material video to be analyzed is used to uniquely identify the material video to be analyzed; The step of obtaining the video parameters of the material video to be analyzed includes: Creating a media extractor corresponding to the file descriptor of the material video to be analyzed; wherein the media extractor is used to separate the video data and the audio data in the video, and extract the video parameters of the video from the video data; Using the media extractor to obtain video data of the material video to be analyzed from the material video to be analyzed; The media extractor is used to extract video parameters of the material video to be analyzed from the video data.

7. The method according to any one of claims 1 to 6, characterized in that Decoding the material video to be analyzed using a decoder in the electronic device includes: Decoding each first image frame of a first number of first image frames in the material video to be analyzed using a decoder in the electronic device; The decoder in the electronic device includes the first decoder and / or the second decoder; the first number of first image frames are evenly distributed at various positions of the material video to be analyzed; or, the first number of first image frames are I frames in the material video to be analyzed.

8. The method according to claim 7, characterized in that After using the decoder in the electronic device to decode each first image frame of the first number of first image frames in the material video to be analyzed, the method further includes: Obtaining an image score for each first image frame; Determine an area where a plurality of first image frames are located within a second preset time length before and after a first target image frame as a target area of ​​the material video to be analyzed; wherein the first target image frame is the first image frame with the highest image score among all first image frames in the material video to be analyzed; acquiring a second number of second image frames from the target area; Decoding each second image frame in a second number of second image frames in the target area using a decoder in the electronic device; The second number of image frames are evenly distributed at various positions in the target area of ​​the material video to be analyzed; or the second number of image frames are I frames in the target area of ​​the material video to be analyzed.

9. An electronic device, characterized in that: The electronic device comprises one or more processors and a memory coupled to the processor; the memory stores computer program code, the computer program code comprising instructions; when the instructions are executed by the processor, the electronic device executes the method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 8.

11. A computer program product, characterized in that The method comprises instructions which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 8.

Images