A video processing method, device, storage medium and program product

Abstract

The application relates to the technical field of multimedia data processing, and discloses a video processing method, a device, a storage medium and a program product. In the video processing method, in a video frame extraction stage, an electronic device can determine a decoder for extracting a video frame based on the matching degree between the processing parameters of a video frame extractor in a software decoder and a hardware decoder and the video parameters of a video to be processed. Then, after the video frame is extracted, the electronic device can determine a decoder for decoding the extracted video frame based on the matching degree between the processing parameters of a video frame decoder in the software decoder and the hardware decoder and the video parameters of the video to be processed. In this way, by combining the software decoder and the hardware decoder, the time for video processing can be shortened, and the efficiency of video processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of multimedia data processing technology, and in particular to a video processing method, device, storage medium and program product. Background Art

[0002] Currently, when electronic devices such as mobile phones perform rapid detection and recognition of video content, they usually divide a long video into multiple short videos, extract the first frame image from each short video and decode it, and then detect whether each first frame image contains the target content. Figure 1 As shown, taking the example of mobile phone 10 detecting whether there is vehicle 100 in video A of 60S, a brief introduction to the scene of rapid detection and recognition of video content is given. Figure 1 As shown in (a) in FIG, video A is a long video with a duration of 60 seconds. At this time, the electronic device can divide video A into four short videos according to the duration to detect whether there is a vehicle 100 in video A. Figure 1 As shown in (b), the electronic device can divide video A into four short videos: video B, video C, video D, and video E. Each short video is 15 seconds long, and frames 101, 102, 103, and 104 are the first frames of each short video. The electronic device can extract and decode the four first frames and then detect the content within each frame. If vehicle 100 is detected in frame 102, it indicates that vehicle 100 appeared in video A for 60 seconds, thus completing rapid detection and recognition of video content.

[0003] Currently, electronic devices typically use hardware decoders to extract and decode the first frame of a video. However, due to hardware limitations, such as the time required to configure hardware resources or initialize the hardware decoder, it takes a long time for electronic devices to detect the content of the first frame of a video.

[0004] Therefore, how to shorten the extraction and decoding time of the first frame image of the electronic device video, thereby improving the speed of video content detection, is an urgent problem that needs to be solved. Summary of the Invention

[0005] To solve the above problems, embodiments of the present application provide a video processing method, device, storage medium, and program product.

[0006] In a first aspect, the present application provides a video processing method, which is applied to an electronic device, wherein the electronic device includes a hardware decoder; and the method includes: obtaining a first video to be processed; determining a decoding method for the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device; wherein the decoding method includes an extraction method for extracting video frames from the first video and a decoding method for decoding the video frames extracted from the first video; and the extraction method includes extracting video frames using a hardware decoder or extracting video frames using a software decoder; the decoding method includes decoding video frames using a hardware decoder or decoding video frames using a software decoder.

[0007] In this application, the first video may be the current video to be processed mentioned later.

[0008] In an embodiment of the present application, the electronic device can determine a decoder for extracting video frames based on the degree of match between processing parameters (e.g., video frame rate, resolution, etc.) of the video frame extractor in each decoder and video parameters (e.g., video frame rate, resolution) of the video to be processed. If the degree of match between the processing parameters and the video parameters is high, it indicates that the video frame extractor has high support for extracting the first video, and the video frame extractor is used to extract the video frame.

[0009] In other embodiments, the electronic device may further determine a decoder for decoding the extracted video frame based on a degree of match between processing parameters (e.g., encoding parameters, encoding format, etc.) of the video frame decoder in each decoder and video parameters (e.g., encoding format information, such as encoding parameters, encoding format, etc.) of the video to be processed. If the degree of match between the processing parameters and the video parameters is high, it indicates that the video frame decoder has high decoding support for the first video, and the video frame decoder is used to decode the extracted video frame.

[0010] Based on the above method, the video frame extractor in the matching decoder is used to extract the video frames in the first video, and the video frame decoder in the matching decoder is used to decode the extracted video frames, which can save the extraction and decoding time of the video frames.

[0011] In a possible implementation of the first aspect above, it includes: determining the first extraction support of the hardware decoder for the first video based on the matching degree between the various processing parameters of the video frame extractor in the hardware decoder and the various parameters of the video file format of the first video; and determining the second extraction support of the software decoder for the first video based on the matching degree between the various processing parameters of the video frame extractor in the software decoder and the various parameters of the video file format of the first video.

[0012] In the present application, the first extraction support degree can be the extraction support degree of the hardware decoder mentioned hereinafter to the current video file, for example, can be the extraction support degree of the hardware decoder 200 to the video; the second extraction support degree can be the extraction support degree of the software decoder mentioned hereinafter to the current video file, for example, can be the extraction support degree of the software decoder 300 to the video.

[0013] In some embodiments, the electronic device can determine the extraction support degree of each video frame extractor to the current video based on the matching degree between the processing parameters of each video frame extractor and the video parameters of the video to be processed, so as to determine the extraction support degree of the decoder corresponding to the video frame extractor to the current video.

[0014] In some embodiments, the parameters of the video file format of the first video are stored in the header information of the first video file, and the processor can obtain the parameters by reading the header information in the first video file; the processor can also interact with the hardware decoder or the software decoder to obtain the current state information (such as the processing parameters of the video frame rate, resolution, etc.) of the hardware decoder or the software decoder. In this way, based on the extraction support degree, the decoder that is most matched with the current video can be selected to extract the video frame.

[0015] In a possible implementation of the first aspect, the processing parameters include but are not limited to video resolution and video frame rate.

[0016] In some embodiments, the electronic device can determine the extraction support degree based on the matching degree between the processing parameters (such as video frame rate, resolution, etc.) of each video frame extractor and the video parameters (such as video frame rate, resolution) of the video to be processed. For example, if the video frame rate supported by the video frame extractor in the software decoder is 30 frames per second to 60 frames per second, and the supported video resolution is 720P and 1080P; the video frame rate supported by the video frame extractor in the hardware decoder is 45 frames per second to 60 frames per second, and the supported video resolution is 1080P and 2K; and the video frame rate of video A is 40 frames per second, and the video resolution is 1080P. Then the extraction support degree of the software decoder to video A is 100%, and the extraction support degree of the hardware decoder to video A is 50%. In this way, the decoder with the maximum extraction support degree can be selected to extract the video frame of the first video, so as to improve the video frame extraction efficiency.

[0017] In a possible implementation of the first aspect above, it includes: determining the first decoding support of the hardware decoder for the first video based on the matching degree between the various processing parameters of the video frame decoder in the hardware decoder and the various parameters of the video encoding format information of the first video; and determining the second decoding support of the software decoder for the first video based on the matching degree between the various processing parameters of the video frame decoder in the software decoder and the various parameters of the video encoding format information of the first video.

[0018] In the present application, the first decoding support can be the decoding support of the hardware decoder mentioned later for the current video file, for example, it can be the decoding support of the hardware decoder 200 for the video; the second decoding support can be the decoding support of the software decoder mentioned later for the current video file, for example, it can be the decoding support of the software decoder 300 for the video.

[0019] In some embodiments, the electronic device can determine the decoding support of each video frame decoder for the first video based on the matching degree between the processing parameters of each video frame decoder and the video parameters of the first video, thereby determining the decoding support of the decoder corresponding to the video frame decoder for the first video.

[0020] In some embodiments, the parameters of the video encoding format information of the first video are stored in the header information of the first video file. The processor can obtain the parameters by reading the header information in the first video file. The processor can also interact with a hardware decoder or a software decoder to obtain the current status information of the hardware decoder or the software decoder (e.g., processing parameters such as encoding parameters and encoding formats that can be processed). In this way, based on the support of each decoding, the decoder that best matches the current video can be selected to decode the extracted video frame.

[0021] In a possible implementation of the first aspect, the processing parameters include but are not limited to video resolution, video frame rate, bit rate, and encoding format.

[0022] In some embodiments, the electronic device can determine the decoding support degree based on the matching degree between the processing parameters (e.g., video resolution, video frame rate, code rate, encoding format, etc.) of each video frame decoder and the video parameters (e.g., video resolution, video frame rate, code rate, encoding format, etc.) of the video to be processed. For example, if the video encoding format supported by the video frame decoders in the software decoder is H.264, H.265, and the supported code rate is 100kbps-10000kbps; and the video encoding format supported by the video frame decoders in the hardware decoder is H.264, H.265, and the supported code rate is 500kbps-15000kbps; and if the video A has a video encoding format of H.264 and a video code rate of 300kbps. Then the decoding support degree of the software decoder for video A is 100%, and the decoding support degree of the hardware decoder for video A is 50%. In this way, the decoder with the maximum decoding support degree can be selected to decode the video frames extracted from the first video, so as to improve the decoding efficiency.

[0023] In a possible implementation of the first aspect, the determination of the decoding mode of the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device comprises: selecting the decoder with the maximum extraction support degree for the first video from the hardware decoder and the software decoder to extract video frames from the first video; and selecting the decoder with the maximum decoding support degree for the first video from the hardware decoder and the software decoder to decode the video frames extracted from the first video.

[0024] In this application, the electronic device can determine the decoder for extracting video frames based on the extraction support degree of the video frame extractor for the video file. For example, if it is determined that the extraction support degree of the video frame extractor in the software decoder for the video file is high, the video frames are extracted through the software decoder; if it is determined that the extraction support degree of the video frame extractor in the hardware decoder for the video file is high, the video frames are extracted through the hardware decoder.

[0025] In some embodiments, after extracting the video frames, the electronic device can also determine the decoder for decoding the video frames based on the decoding support degree of the video frame decoder for the video coding format information. For example, if the decoding support degree of the video frame decoder in the software decoder for the video coding format information is high, the extracted video frames are decoded through the software decoder; if the decoding support degree of the video frame decoder in the hardware decoder for the video coding format information is high, the extracted video frames are decoded through the hardware decoder.

[0026] Thus, by combining the use of the software decoder and the hardware decoder, in the video frame extraction and decoding stage, a more suitable video frame extractor can be used to extract the video frame, and a suitable video frame decoder can be used to decode the extracted video frame. Thus, the time for video processing can be shortened, and the efficiency of video processing can be improved.

[0027] In a possible implementation of the first aspect, the matching degrees between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device are used to determine the decoding mode of the first video, including: selecting a decoder with the highest extraction support for the first video from the hardware decoder and the software decoder, and extracting the video frame from the first video; and using a preset decoder to decode the video frame extracted from the first video.

[0028] In the present application, the electronic device can determine the decoder for extracting the video frame according to the extraction support of the video frame extractor for the video file. For example, if it is determined that the extraction support of the video frame extractor in the software decoder for the video file is high, the video frame is extracted by the software decoder; if it is determined that the extraction support of the video frame extractor in the hardware decoder for the video file is high, the video frame is extracted by the hardware decoder.

[0029] In some embodiments, after the video frame is extracted, the electronic device can also use a preset or system default video frame decoder (such as a video frame decoder in the hardware decoder) to decode the extracted video frame. Thus, the decoder with the highest extraction support can be selected to extract the video frame from the first video to improve the efficiency of video frame extraction.

[0030] In a possible implementation of the first aspect, the matching degrees between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device are used to determine the decoding mode of the first video, including: selecting a decoder with the highest extraction support for the first video from the hardware decoder and the software decoder, and extracting the video frame from the first video; and using a preset decoder to decode the video frame extracted from the first video.

[0031] In the present application, the electronic device can determine the decoder for extracting the video frame according to the extraction support of the video frame extractor for the video file. For example, if it is determined that the extraction support of the video frame extractor in the software decoder for the video file is high, the video frame is extracted by the software decoder; if it is determined that the extraction support of the video frame extractor in the hardware decoder for the video file is high, the video frame is extracted by the hardware decoder.

[0032] In some embodiments, after extracting the video frames, the electronic device may also use the decoder that extracted the video frames from the first video to decode the video frames extracted from the first video. For example, if the electronic device determines that the video frame extractor in the software decoder has high support for video file extraction, the electronic device may use the software decoder to extract the video frames from the current video file and simultaneously use the software decoder to decode the extracted video frames. In this way, the decoder with the highest extraction support can be selected to extract the video frames from the first video, thereby improving video frame extraction efficiency.

[0033] In a possible implementation of the first aspect above, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: using a preset decoder to extract video frames from the first video; selecting a decoder with the greatest decoding support for the first video from the hardware decoder and the software decoder, and decoding the video frames extracted from the first video.

[0034] In this application, the electronic device can use a preset or system default video frame extractor (e.g., a video frame extractor in a hardware decoder) to extract video frames, then select a video frame decoder with high decoding support for video codec format information, and use the video frame decoder to decode the extracted video frames. In this way, the decoder with the highest decoding support can be selected to decode the video frames extracted from the first video, thereby improving decoding efficiency.

[0035] In a possible implementation of the first aspect, the first video includes but is not limited to digital rights management video, high dynamic range video, and standard dynamic range video.

[0036] In this application, the first video can be any type of video, for example, any type of video such as a digital rights management (DRM) video, a high dynamic range (HDR) video, or a standard dynamic range (SDR) video that has been processed using encryption technology. It should be understood that the video processing method mentioned in this application can be applied to any type of video, and this application does not limit this.

[0037] In a possible implementation of the first aspect above, corresponding to the first video being a high dynamic range video, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: when the video parameters of the first video match the processing parameters of the software decoder, using the software decoder to extract video frames in the first video; when the video parameters of the first video do not match the processing parameters of the software decoder, and the video parameters of the first video match the processing parameters of the hardware decoder, using the hardware decoder to extract video frames in the first video.

[0038] In the present application, when the video type of the first video is HDR video, if the video frame extractor in the software decoder and the video frame extractor in the hardware decoder both support the video format of the current HDR video (i.e., parameter matching), the video frame extractor in the software decoder will have higher support for the extraction of the current HDR video file than the video frame extractor in the hardware decoder.

[0039] In some embodiments, if the HDR video format is supported by the software decoder, the software decoder is used to extract the video frames. If the software decoder does not support the current HDR video format, the hardware decoder is used to extract the video frames. In this way, a matching decoder can be selected to extract the video frames from the HDR video, thereby improving the efficiency of video frame extraction.

[0040] In a possible implementation of the first aspect above, corresponding to the first video being a digital rights management video and a standard dynamic range video, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: when the electronic device supports processing the first video, determining the extraction method for video frames of the first video based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device.

[0041] In this application, if the video type of the first video is DRM video, the electronic device can determine whether it supports processing DRM video. For example, the electronic device can determine whether AOSP supports processing DRM video by viewing the documentation of the Android Open Source Project (AOSP), detecting the code library of AOSP, and viewing the hardware specifications supported by AOSP. TM The basic code framework in the operating system, for Android TM The device provides basic functions and features.

[0042] In some embodiments, if the video file type of the first video is a common video (e.g., SDR video, etc.), or is a DRM video and the current electronic device supports processing DRM video, the electronic device can determine the extraction support based on the matching degree between the processing parameters (e.g., video frame rate, resolution, etc.) of each video frame extractor and the video parameters (e.g., video frame rate, resolution) of the video to be processed, thereby determining the extraction method for extracting video frames from the first video. In this way, a matching decoder can be selected to extract video frames from the first video to improve the efficiency of video frame extraction.

[0043] In a possible implementation of the first aspect above, the above-mentioned extraction method for extracting video frames from the first video is determined based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: determining the first extraction support of the hardware decoder for the first video based on the matching degree between the various processing parameters of the video frame extractor in the hardware decoder and the various parameters of the video file format of the first video; determining the second extraction support of the software decoder for the first video based on the matching degree between the various processing parameters of the video frame extractor in the software decoder and the various parameters of the video file format of the first video; establishing a video frame extractor capability table based on the first extraction support and the second extraction support; and selecting the decoder corresponding to the largest extraction support in the video frame extractor capability table to extract the video frames in the first video.

[0044] In this application, the first extraction support can be the extraction support of the hardware decoder mentioned later for the first video file, for example, it can be the extraction support of the hardware decoder 200 for the video; the second extraction support can be the extraction support of the software decoder mentioned later for the first video file, for example, it can be the extraction support of the software decoder 300 for the video.

[0045] In some embodiments, the electronic device can determine the extraction support of each video frame extractor for the current video based on the matching degree between the processing parameters of each video frame extractor and the video parameters of the video to be processed. In addition, the electronic device can establish a video frame extractor capability table based on the extraction support of each video frame extractor for the video file format. The electronic device can query the video frame extractor capability table and select the video frame extractor with the highest extraction support for the first video file, that is, select the best matching decoder to perform video frame extraction on the first video. In this way, a matching decoder can be selected to perform video frame extraction on the first video to improve the efficiency of video frame extraction.

[0046] In a possible implementation of the first aspect above, the decoding method of the first video is determined based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: determining the first decoding support of the hardware decoder for the first video based on the matching degree between the various processing parameters of the video frame decoder in the hardware decoder and the various parameters of the video encoding format information of the first video; determining the second decoding support of the software decoder for the first video based on the matching degree between the various processing parameters of the video frame decoder in the software decoder and the various parameters of the video encoding format information of the first video; establishing a video frame decoder capability table based on the first decoding support and the second decoding support; in the video frame decoder capability table, selecting the decoder corresponding to the largest decoding support to decode the video frame extracted from the first video.

[0047] In the present application, the first decoding support can be the decoding support of the first video file by the hardware decoder mentioned later, for example, it can be the decoding support of the video by the hardware decoder 200; the second decoding support can be the decoding support of the first video file by the software decoder mentioned later, for example, it can be the decoding support of the video by the software decoder 300.

[0048] In some embodiments, the electronic device can determine the decoding support of each video frame decoder for the current video based on the matching degree between the processing parameters of each video frame decoder and the video parameters of the video to be processed. Furthermore, the electronic device can establish a video frame decoder capability table based on the decoding support of each video frame decoder for the video codec format information. The electronic device can query the video frame decoder capability table and select the video frame decoder with the highest decoding support for the first video file, that is, select the best matching decoder to decode the video frames extracted from the first video. In this way, a matching decoder can be selected to decode the video frames of the first video to improve decoding efficiency.

[0049] In a second aspect, the present application provides an electronic device comprising: a memory and a processor, the memory being used to store instructions executed by one or more processors of the electronic device, and the processor being one of the one or more processors of the electronic device, and being used to execute the video processing method mentioned in the present application.

[0050] In a third aspect, the present application provides a readable storage medium, wherein the readable storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the video processing method mentioned in the present application.

[0051] In a fourth aspect, the present application provides a computer program product, comprising: computer instructions, which, when executed on an electronic device, enable the electronic device to execute the video processing method mentioned in the present application.

[0052] The beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions of the first aspect and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 According to some embodiments of the present application, a schematic diagram of a scenario for rapid detection and identification of video content is shown;

[0054] Figure 2A According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown;

[0055] Figure 2B According to some embodiments of the present application, a schematic structural diagram of a hardware decoder is shown;

[0056] Figure 2C According to some embodiments of the present application, a schematic structural diagram of a software decoder is shown;

[0057] Figure 3 According to some embodiments of the present application, a schematic diagram of a scenario in which a video preview thumbnail is generated when a file list is loaded is shown;

[0058] Figure 4 According to some embodiments of the present application, a schematic diagram of a flow chart of a video processing method executed by an electronic device is shown;

[0059] Figure 5A According to some embodiments of the present application, a framework diagram of a video processing system is shown;

[0060] Figure 5B According to some embodiments of the present application, a schematic diagram of a specific flow chart of each functional module executing a video processing method is shown;

[0061] Figure 6 According to some embodiments of the present application, a schematic diagram of a flow chart of interaction between a software decoder decoding module and a software decoder is shown;

[0062] Figure 7 According to some embodiments of the present application, a schematic diagram of a flow chart of the interaction between a hardware decoder decoding module and a hardware decoder is shown. DETAILED DESCRIPTION

[0063] The illustrative embodiments of the present application include, but are not limited to, a video processing method, apparatus, storage medium, and program product.

[0064] First, a brief description is given of the application background of the video processing method provided in the embodiment of the present application.

[0065] It is understood that the electronic devices applicable to the embodiments of the present application may be any electronic device capable of processing video data, such as a mobile phone, a computer, a tablet computer, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, etc. This application does not limit this.

[0066] The following uses a mobile phone as an example to exemplify a hardware structure diagram of an electronic device according to an embodiment of the present application.

[0067] For example, Figure 2A Schematic diagram of the hardware structure of the mobile phone 10 is shown. Figure 2A As shown, the mobile phone 10 may include a processor 110, an external memory interface 120, an internal memory 121, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a screen 194, and a hardware decoder 200, etc.

[0068] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the mobile phone 10. In other embodiments of the present application, the mobile phone 10 may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 110 may include one or more processing units, for example: the processor 110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a controller, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU), and a graphics processing unit (GPU). Among them, different processing units can be independent devices or integrated into one or more processors. In this application, the processor 110 can be used to execute the video processing method mentioned in this application.

[0070] In some embodiments, the processor 110 may execute instructions related to a software decoder to implement data processing of audio files or video files. For example, the mobile phone 10 may use the CPU to execute relevant code of the FFMPEG software decoder in the Fast Forward Moving Picture Experts Group (FFMPEG) open source platform to implement video decoding.

[0071] exist Figure 2A In the example, the hardware decoder 200 is a hardware device or chip for decoding audio or video data. TM system, the hardware decoder 200 may be a media codec (MediaCodec), wherein the MediaCodec hardware decoder may be an advanced video coding (AVC) hardware decoder supporting the H.264 encoding format, etc. In addition, when the mobile phone 10 uses the hardware decoder 200 to decode the video, corresponding resource configuration is required, for example, allocating memory and processor resources to the hardware decoder 200, or selecting an appropriate hardware decoder (such as an H.264 decoder, an H.265 decoder, etc.); or loading a hardware driver, etc.

[0072] It should be understood that both the software decoder and the hardware decoder 200 may include a functional module for extracting video frames and a functional module for decoding the extracted video frames. Figure 2B As shown, the hardware decoder 200 includes a video frame extractor 201 and a video frame decoder 202. The video frame extractor 201 is responsible for extracting video frames from a video file and passing the extracted video frames to the video frame decoder 202 for decoding and processing.

[0073] For example, Figure 2C As shown, the software decoder 300 includes a video frame extractor 301 and a video frame decoder 302. Furthermore, the video frame extractor 301 is responsible for extracting video frames from a video file and passing the extracted video frames to the video frame decoder 302 for decoding and processing. It should be understood that the extraction and decoding of video frames can be implemented by the software decoder 300 or by the hardware decoder 200.

[0074] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, at least one application required for a function, etc. The data storage area can store data created during the use of the mobile phone 10 (such as audio data, phone book, etc.). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 10 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor 110.

[0075] exist Figure 2A In the example, the screen 194 can be used to display video images, for example, playing the above Figure 1 The audio module 170 can be used to play the audio in the video. The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the mobile phone 10. The wireless communication module 160 can provide solutions for wireless communications such as wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks) and Bluetooth (BT) applied to the mobile phone 10.

[0076] As mentioned above, when detecting and identifying video content, in order to avoid the problem of slow efficiency in detecting video content frame by frame from the first frame to the last frame, electronic devices usually divide the video into multiple short videos and extract the first frame image of each short video for content detection, thereby saving detection time.

[0077] Currently, electronic devices generally use a video frame extractor in a hardware decoder (e.g., a MediaCodec hardware decoder) to extract the first frame image of a video, and pass the extracted first frame image of the video to a video frame decoder in the hardware decoder, which decodes the extracted first frame image.

[0078] However, when using a hardware decoder to extract and decode the first frame image of a video, the electronic device needs to configure hardware resources or perform operations such as initializing the hardware decoder, such as loading the hardware driver, and then extract and decode the first frame image of the video through the hardware decoder. As a result, it takes a long time for the electronic device to detect the content of the first frame image of the video.

[0079] To solve the above problems, the present application provides a video processing method. When extracting a video frame, a decoder for extracting the video frame can be determined according to the extraction support degree of the video frame extractor to the video file. For example, if the extraction support degree of the video frame extractor 301 in the software decoder 300 to the video file is high, the video frame is extracted by the software decoder 300; if the extraction support degree of the video frame extractor 201 in the hardware decoder 200 to the video file is high, the video frame is extracted by the hardware decoder 200. After the video frame is extracted, a decoder for decoding the video frame can be determined according to the decoding support degree of the video frame decoder to the video coding format information. For example, if the decoding support degree of the video frame decoder 302 in the software decoder 300 to the coding format information of the video is high, the extracted video frame is decoded by the software decoder 300; if the decoding support degree of the video frame decoder 202 in the hardware decoder 200 to the coding format information of the video is high, the extracted video frame is decoded by the hardware decoder 200.

[0080] In this way, by using the software decoder and the hardware decoder together, a more suitable video frame extractor can be used to extract the video frame and a more suitable video frame decoder can be used to decode the extracted video frame in the video frame extraction and decoding stages. In this way, the time for video processing can be shortened and the efficiency of video processing can be improved.

[0081] The extraction support degree indicates the ability of the video frame extractor to extract the video frame in the video to be processed, such as accuracy and efficiency. At the same time, the extraction support degree can be determined based on the matching degree between the processing parameters (such as video frame rate, resolution, etc.) of the video frame extractor and the video parameters (such as video frame rate, resolution) of the video to be processed. For example, the extraction support degree of the video frame extractor 201 to the video A is high if the supported video frame rate of the video frame extractor 201 is 30 frames per second to 60 frames per second and the video frame rate of the video A is 40 frames per second; the extraction support degree of the video frame extractor 201 to the video A is low if the video frame rate of the video A is 25 frames per second. Figure 2B For example, if the extraction support degree of the video frame extractor 301 in the software decoder 300 to the video file is high, the video frame is extracted by the software decoder 300; if the extraction support degree of the video frame extractor 201 in the hardware decoder 200 to the video file is high, the video frame is extracted by the hardware decoder 200. After the video frame is extracted, a decoder for decoding the video frame can be determined according to the decoding support degree of the video frame decoder to the video coding format information. For example, if the decoding support degree of the video frame decoder 302 in the software decoder 300 to the coding format information of the video is high, the extracted video frame is decoded by the software decoder 300; if the decoding support degree of the video frame decoder 202 in the hardware decoder 200 to the coding format information of the video is high, the extracted video frame is decoded by the hardware decoder 200.

[0082] In addition, the above decoding support indicates the ability of the video frame decoder to decode the extracted video frames, such as accuracy, efficiency, etc. At the same time, the decoding support can be determined based on the matching degree between the processing parameters of the video frame decoder (such as video resolution, video frame rate, bit rate, encoding parameters, encoding format, etc.) and the video parameters of the video to be processed (such as encoding parameters, encoding format, and other encoding format information). For example, still taking the above Figure 2B For example, if the encoding formats supported by the video frame decoder 202 are H.264 and H.265, and if the video encoding format of video A is H.264, then the video frame decoder 202 has high decoding support for video A; if the video encoding format of video A is H.263, then the video frame decoder 202 has low decoding support for video A. The video parameters are stored in the header information of the video file, and the processor can obtain the video parameters by reading the header information in the video file; the processor can also interact with the hardware decoder 200 or the software decoder 300 to obtain the current status information of the hardware decoder 200 or the software decoder 300 (e.g., processing parameters such as encoding parameters and encoding formats that can be processed).

[0083] In other embodiments of the present application, the electronic device may further select only a video frame extractor with high support for video file extraction, extract video frames through the video frame extractor, and then use a preset or system default video frame decoder (e.g., a video frame decoder in a hardware decoder) to decode the extracted video frames. Alternatively, the electronic device may further use a preset or system default video frame extractor (e.g., a video frame extractor in a hardware decoder) to extract video frames, and then select a video frame decoder with high support for decoding video codec format information, and use the video frame decoder to decode the extracted video frames.

[0084] In addition, in other embodiments of the present application, the electronic device may further select a video frame extractor with high support for video file extraction and extract video frames using the video frame extractor. Simultaneously, the electronic device selects the decoder containing the video frame extractor to decode the extracted video frames. For example, if the electronic device determines that the video frame extractor 301 in the software decoder 300 has high support for video file extraction, the electronic device may use the software decoder 300 to extract the video frames from the current video file and simultaneously use the software decoder 300 to decode the extracted video frames.

[0085] In the embodiments of the present application, the video can be any type of video, for example, a digital rights management (DRM) video processed by encryption technology, a high dynamic range (HDR) video, a standard dynamic range (SDR) video, or any other type of video. It should be understood that the video processing method mentioned in the present application can be applied to any type of video, which is not limited in the present application.

[0086] In some embodiments, the resolution of the HDR video is generally significantly higher than that of other videos. However, when the video type is an HDR video, if both the video frame extractor 301 in the software decoder 300 and the video frame extractor 201 in the hardware decoder 200 support the video format of the current HDR video, the extraction support of the video frame extractor 301 in the software decoder 300 for the current HDR video file will be higher than that of the video frame extractor 201 in the hardware decoder 200. For example, the resolution of the HDR video is 8K, and both the video frame extractor 301 in the software decoder 300 and the video frame extractor 201 in the hardware decoder 200 support processing of the video with 8K resolution, and the extraction support of the video frame extractor 301 in the software decoder 300 for the HDR video will be higher. That is, in the embodiments of the present application, if the video format of the HDR video is supported by the software decoder 300, the video frame is extracted through the software decoder 300; if the software decoder 300 does not support the format of the current HDR video, the video frame can be extracted through the hardware decoder 200.

[0087] In some embodiments, when extracting the video frame of the current video, the video frame extractor capability table can also be established based on the extraction support of each video frame extractor for the current video file. When extracting the video frame, the electronic device can first query the video frame extractor capability table to determine the best decoder for extracting the video frame of the current video file. For example, for the current video file, if the extraction support of the video frame extractor 301 in the software decoder 300 for the video file is the highest, the video frame can be extracted through the software decoder 300; if the extraction support of the video frame extractor 201 in the hardware decoder 200 for the video file is the highest, the video frame can be extracted through the hardware decoder 200. In this way, the video frame is extracted by the video frame extractor in the matched decoder, which can save the extraction time of the video frame.

[0088] Among them, the video frame extractor capability table is determined based on the extraction support of each video frame extractor for the video file format. In addition, the extraction support can be determined based on the matching degree between the processing parameters of each video frame extractor (such as video frame rate, resolution, etc.) and the video parameters of the video to be processed (such as video frame rate, resolution). For example, if the video frame extractor 301 in the software decoder 300 supports a video frame rate of 30 frames per second to 60 frames per second and supports video resolutions of 720P and 1080P; and the video frame extractor 201 in the hardware decoder 200 supports a video frame rate of 45 frames per second to 60 frames per second and supports video resolutions of 1080P and 2K. If the video frame rate of video A is 40 frames per second and the video resolution is 1080P, a video frame extractor capability table can be established: the extraction support of the software decoder 300 for video A is 100%, and the extraction support of the hardware decoder 200 for video A is 50%. Therefore, the electronic device can query the video frame extractor capability table to determine that the software decoder 300 has the highest support for extracting video A, that is, determine that the best decoder for extracting the video frame of video A is the software decoder 300.

[0089] In some embodiments, when decoding the extracted video frames, a video frame decoder capability table can also be established based on the decoding support of each video frame decoder for the current video codec format information. After extracting the video frame, the electronic device can obtain the codec format information of the current video, and then, based on the video codec format information, search the video frame decoder capability table for a decoder that decodes the video frame. For example, for the current video file, if the video frame decoder 302 in the software decoder 300 has the highest decoding support for the video coding format information, the video frame can be decoded by the software decoder 300; if the video frame decoder 202 in the hardware decoder 200 has the highest decoding support for the video coding format information, the video frame can be decoded by the hardware decoder 200. In this way, using the video frame decoder in the matching decoder to decode the extracted video frame can save the decoding time of the video frame.

[0090] The video frame decoder capability table is determined based on the decoding support of each video frame decoder for the video's codec format information. Furthermore, the decoding support can be determined based on the degree of match between the processing parameters (e.g., encoding parameters, encoding format, etc.) of each video frame decoder and the video parameters (e.g., encoding format information, such as encoding parameters, encoding format, etc.) of the video to be processed. For example, if the video frame decoder 302 in the software decoder 300 supports H.264 and H.265 video encoding formats with supported bit rates of 100kbps to 10,000kbps, and the video frame decoder 202 in the hardware decoder 200 supports H.264 and H.265 video encoding formats with supported bit rates of 500kbps to 15,000kbps, then a video frame decoder capability table can be established: the software decoder 300's decoding support for video A is 100%, while the hardware decoder 200's decoding support for video A is 50%. Therefore, the electronic device can query the video frame decoder capability table to determine that the software decoder 300 has the highest decoding support for video A, that is, determine that the best decoder for decoding the video frame of video A is the software decoder 300 .

[0091] In some embodiments, the video processing method mentioned in this application can be applied to the above Figure 1 In the scenario of rapid detection of video content mentioned in . Specifically, when the electronic device extracts the first frame image of each video segment, it first determines the decoder for extracting the first frame of the video based on the extraction support of the video frame extractor 201 and the video frame extractor 301 for the video file. If it is determined that the adapted decoder is the software decoder 300, the electronic device extracts the first frame image of each video segment through the software decoder 300. Next, the electronic device determines the decoder for decoding the first frame of the video based on the decoding support of the video codec format information by the video frame decoder 202 and the video frame decoder 302. If it is determined that the adapted decoder is the software decoder 300, the electronic device decodes the extracted first frames of each video through the software decoder 300. Finally, the electronic device detects whether the content in the first frame of each video includes the target content. In this way, the rapid detection and identification of the video content can be completed.

[0092] In some embodiments, the time can be shortened and the efficiency can be improved when the first frame image of a video is extracted using the software decoder 300 and the extracted first frame image is decoded using the software decoder 300. For example, a simple comparison is made between the processing efficiency of the hardware decoder 200 and the software decoder 300, taking the extraction and decoding of the first frame image of 5 videos as an example. The 5 videos have a time length of 10 seconds, a video resolution of 1080P, a video frame rate of 30 frames per second, and a video encoding format of AVC (i.e., H.264 encoding format).

[0093] In some embodiments, the electronic device can extract and decode the first frame image of the 5 videos using the AVC hardware decoder and the FFMPEG software decoder, respectively. The average time length of extracting and decoding the first frame image of each video using the AVC hardware decoder is greater than the average time length of extracting and decoding the first frame image of each video using the FFMPEG software decoder. For details, refer to the decoding time length shown in Table 1 below:

[0094] Table 1:

[0095] AVC hardware decoder FFMPEG (8 threads) FFMPEG (single thread) Video 1 698 381 210 Video 2 571 382 178 Video 3 598 401 213 Video 4 607 395 177 Video 5 563 388 179 average value 607.4 389.4 191.4

[0096] It should be understood that the time length of extracting and decoding the first frame image of a video includes the start-up time of the decoder.

[0097] In Table 1, the average time length of extracting and decoding the first frame image of the 5 videos using the AVC hardware decoder is 607.4 seconds, which is greater than the average time length of extracting and decoding the first frame image of each video using the FFMPEG software decoder.

[0098] In Table 1, the average time length of extracting and decoding the first frame image of the 5 videos using the AVC hardware decoder is 607.4 seconds, which is greater than the average time length of extracting and decoding the first frame image of each video using the FFMPEG software decoder.

[0099] Therefore, by the video processing method mentioned in the present application, when extracting and decoding the first frame image of each video, the electronic device can determine that the decoder adapted to extract and decode the first frame of the video is a software decoder based on the video frame extractor capability table and the video frame decoder capability table, and then the electronic device extracts and decodes the first frame image of each video through the software decoder. In this way, the time for extracting and decoding the first frame of each video can be shortened, and the efficiency of extracting and decoding the first frame of each video can be improved.

[0100] In addition, the video processing method mentioned in the present application can also be applied to the scene of generating video preview thumbnails for a video cover and a loaded video file list. For example, when promoting a movie, a landmark image frame in the movie can be selected as a video cover by the method in the present application to attract the attention of users. Specifically, when extracting the landmark image frame in the movie, the electronic device first determines the decoder adapted to extract the landmark image frame based on the extraction support of the video frame extractor 201 and the video frame extractor 301 for the movie video file. For example, if it is determined that the adapted decoder is a software decoder 300, the electronic device can extract the landmark image frame in the movie through the software decoder 300. Next, the electronic device determines the decoder adapted to decode the landmark image frame based on the decoding support of the video frame decoder 202 and the video frame decoder 302 for the encoding format information of the movie video. For example, if it is determined that the adapted decoder is a software decoder 300, the electronic device can decode the extracted landmark image frame through the software decoder 300. Finally, the electronic device displays the decoded landmark image frame as the video cover of the movie on the screen.

[0101] For another example, when loading a video file list, the preview thumbnails of each video can be generated by the method mentioned in the present application to shorten the loading time of the file list. For example, as shown in (a) of FIG. 13, taking the electronic device PC 30 as an example, the interface 310 of the PC 30 shows that when the PC 30 detects the operation of the user opening the video folder 320, the PC 30 can generate preview thumbnails for the user to view by the method in the present application. Figure 3 Figure 3 As shown in (b) of FIG. 13, in the interface 330, for each video file in the video folder 320, the PC 30 generates a preview thumbnail of each video file for the user to view.

[0102] ​The process of generating preview thumbnails for each video file can be as follows: When processing each video file, PC30 first determines the decoder for extracting video frames based on the extraction support of the video frame extractor 201 and the video frame extractor 301 for the video file. For example, if it is determined that the adapted decoder is the software decoder 300, PC30 can extract the video frame that can be used as the preview thumbnail through the software decoder 300, such as the first frame, or extract the frame containing the target with the most appearances by identifying each video frame. Next, PC30 determines the decoder for decoding the video frame based on the decoding support of the video codec format information by the video frame decoder. For example, if it is determined that the adapted decoder is the software decoder 300, PC30 can decode the extracted video frame through the software decoder 300. Finally, PC30 displays the decoded video frame on the screen as the preview thumbnail of the video file.

[0103] It should be understood that, as shown in Table 1 above, when the software decoder 300 simultaneously creates multiple task processes (e.g., 8 threads), that is, the average speed at which the software decoder 300 simultaneously generates preview thumbnails for multiple video files is faster than the speed at which the hardware decoder 200 generates preview thumbnails for a single video file. Therefore, the video processing method described in this application can shorten the time it takes to extract and decode the first frame of a video on an electronic device and increase the speed at which a video file list is loaded.

[0104] The following is based on Figure 4 The flowchart shown in FIG. briefly describes the process of the video processing method mentioned in this application. The method can be applied to electronic devices, such as the above-mentioned Figure 3 PC30 or Figure 2A The mobile phone 10 shown. Figure 4 Specifically, the method includes the following steps:

[0105] S401: Obtain a first video to be processed.

[0106] In an embodiment of the present application, the electronic device receives an instruction to extract video frames from a first video, for example, receives an instruction to quickly detect the content of the first video, generate a cover for the first video, generate a preview thumbnail of the first video, or detects that a user opens a video folder, and the electronic device first needs to obtain the video file of the first video to be processed.

[0107] S402: Determine a decoding method for the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder.

[0108] In an embodiment of the present application, the electronic device can determine a decoder for extracting video frames based on the degree of match between the processing parameters (e.g., video frame rate, resolution, etc.) of each video frame extractor and the video parameters (e.g., video frame rate, resolution) of the video to be processed. If the degree of match between the processing parameters and the video parameters is high, it indicates that the video frame extractor has high support for extracting the first video, and the video frame extractor is used to extract the video frame.

[0109] In an embodiment of the present application, the electronic device may further determine a decoder for decoding the extracted video frame based on a degree of match between processing parameters (e.g., encoding parameters, encoding format, etc.) of each video frame decoder and video parameters (e.g., encoding format information, such as encoding parameters, encoding format, etc.) of the video to be processed. If the degree of match between the processing parameters and the video parameters is high, it indicates that the video frame decoder has high decoding support for the first video, and the video frame decoder is used to decode the extracted video frame.

[0110] In an embodiment of the present application, when extracting video frames, if the video type of the first video is HDR video, if the software decoder 300 supports processing HDR video, the video frames of the HDR video are extracted through the software decoder 300. When the video type of the first video is DRM video and the electronic device supports processing DRM video, or when the video type of the first video is other video (e.g., SDR video), the electronic device can obtain the extraction support of the video frame extractor 201 and the video frame extractor 301 for the current video, and select the video frame extractor with high extraction support to extract the video frame.

[0111] In addition, in an embodiment of the present application, for any type of video (such as DRM video, HDR video, SDR video, etc.), when the electronic device decodes the extracted video, the electronic device can obtain the decoding support of the video frame decoder 202 and the video frame decoder 302 for the current video, and select a video frame decoder with high decoding support to decode the extracted video frame.

[0112] In this way, the video frame extractor in the matching decoder is used to extract the video frames in the first video, and the video frame decoder in the matching decoder is used to decode the extracted video frames, which can save the time of extracting and decoding the video frames.

[0113] Through the above method, a matching video frame extractor and video frame decoder can be used to extract and decode the video frames of the first video, thereby shortening the video processing time and improving the efficiency of video processing by combining the use of software decoders and hardware decoders.

[0114] In some embodiments, a video processing system based on the video processing method mentioned in this application can be divided into several functional modules, and the video processing method mentioned in this application is implemented through interaction between the modules.

[0115] like Figure 5A FIG. 4 shows a schematic diagram of the framework structure of a video processing system 400. The video processing system 400 can be divided into several functional modules, and each functional module can interact with each other. Specifically, the video processing system 400 includes a video source acquisition module 410, a decision module 420, a software decoder extraction module 430, a hardware decoder extraction module 440, a software decoder decoding module 450, and a hardware decoder decoding module 460.

[0116] Among them, the video source acquisition module 410 can be used to obtain video files and send the video files to the decision module 420, which determines the decoder to process the video files. The decision module 420 is used to determine the decoder to extract video frames and the decoder to decode the video frames. The software decoder extraction module 430 is used to interact with the software decoder 300 and extract video frames through the software decoder 300. The hardware decoder extraction module 440 is used to interact with the hardware decoder 200 and extract video frames through the hardware decoder 200. The software decoder decoding module 450 is used to interact with the software decoder 300 and decode the extracted video frames through the software decoder 300. The hardware decoder decoding module 460 is used to interact with the hardware decoder 200 and decode the extracted video frames through the hardware decoder 200.

[0117] It should be understood that the video processing system 400 may also be split or combined into more or fewer modules, and this embodiment of the present application does not limit this.

[0118] The following takes the hardware decoder 200 as the MediaCodec hardware decoder, the software decoder 300 as the FFMPEG software decoder as an example, and the video file types including DRM video, HDR video, etc. as an example, combined with the above Figure 5A The algorithm framework diagram mentioned in Figure 5B The flowchart specifically introduces the internal specific execution process of each functional module mentioned in this application when executing the video processing method. Specifically, Figure 5B The method shown includes the following steps:

[0119] 501: The video source acquisition module 410 acquires a video file.

[0120] In an embodiment of the present application, the video source acquisition module 410 can acquire a video file based on a video file path input by a user. Alternatively, a user can search for a video on an electronic device in an application such as a website or video player. The video source acquisition module 410 can retrieve the video file from a database within the application such as the website or video player based on user-triggered operations such as detecting video content. This application does not limit the source of the video file.

[0121] 502: The decision module 420 determines whether the video is a DRM video. If so, the process proceeds to step 503; if not, the process proceeds to step 504.

[0122] In the embodiment of the present application, the video may be a non-encrypted ordinary video, or a DRM video (ie, a video processed with encryption technology).

[0123] In some embodiments, the decision module 420 can receive the video file obtained by the video source acquisition module 410 and determine whether the video file is an encrypted DRM video. If the video is a DRM video, the process goes to step 503 to determine whether the Android open source project (AOSP) supports the DRM video. TM Whether the device can access the content in the DRM video; if the video is not a DRM video, go to step 504 to determine whether the video type is HDR video.

[0124] 503: The decision module 420 determines whether the DRM video is supported by AOSP. If so, the process proceeds to step 507; if not, the process proceeds to step 518.

[0125] It should be understood that AOSP is Android TM The basic code framework in the operating system, for Android TM The device provides basic functions and features. However, if the electronic device running AOSP wants to be able to handle DRM videos, it needs to add a DRM code framework to AOSP, that is, integrate and be compatible with the corresponding DRM functions in AOSP. DRM is a technology used to protect digital content, which can encrypt video files and restrict electronic devices from accessing video content.

[0126] In an embodiment of the present application, when the decision module 420 determines that the current video is a DRM video through the above step 502, the decision module 420 also needs to determine whether AOSP supports DRM video. If AOSP does not support DRM video, it means that the DRM video cannot be accessed in the electronic device where the AOSP is located, that is, the electronic device cannot extract and decode the video frames in the DRM video, then go to step 518, and the electronic device (such as a processor) can end the execution of the method flow. If AOSP supports DRM video, for example, the decision module 420 detects that there is a DRM code framework in AOSP, then go to step 507, query the video frame extractor capability table, and obtain the best matching video frame extractor.

[0127] It should be understood that when an electronic device decodes a DRM video, it decrypts the encrypted DRM video.

[0128] 504: The decision module 420 determines whether the video is an HDR video. If so, the process proceeds to step 505; if not, the process proceeds to step 507.

[0129] In this embodiment of the present application, when the decision module 420 determines that the current video is an unencrypted, ordinary video through step 502, it is also necessary to determine whether the current video type is an HDR video. If it is an HDR video, the process proceeds to step 505 to determine whether the FFMPEG software decoder supports HDR video. If it is not an HDR video, the process proceeds to step 507 to query the video frame extractor capability table to obtain the best matching video frame extractor. HDR is a processing technology that improves the brightness and contrast of a video. Compared to ordinary video, HDR video can provide more image detail, richer colors, and more vivid images that are closer to what the human eye sees.

[0130] In some embodiments, the electronic device may first execute step 502 to determine that the current video type is not a DRM video, and then execute step 504 to determine whether the current video is an HDR video. Alternatively, the electronic device may first execute step 504 to determine whether the current video type is an HDR video. If it is determined that the current video type is not an HDR video, the electronic device may execute step 502 to determine whether the current video type is a DRM video. If it is determined that the current video type is not a DRM video through step 502, the electronic device may directly execute step 507 to query the video frame extractor capability table and select the best matching video frame extractor. In other words, this application does not limit the execution order of steps 502 and 504.

[0131] 505: The decision module 420 determines whether HDR video is supported by FFMPEG. If so, the process proceeds to step 510; if not, the process proceeds to step 506.

[0132] In the embodiments of the present application, when the decision module 420 determines that the current video is an HDR video through the above step 504, it is further needed to determine whether the FFMPEG software decoder supports processing the HDR video. If yes, go to step 510 to create the FFMPEG extractor; if not, go to step 506 to determine whether the HDR video is supported by AOSP.

[0133] In some embodiments, when the video type is an HDR video, if the video frame extractor in the FFMPEG software decoder and the video frame extractor in the MediaCodec hardware decoder both support the video format of the current HDR video, the support degree of the video frame extractor in the FFMPEG software decoder to the current HDR video file is higher than that of the video frame extractor in the MediaCodec hardware decoder. For example, the resolution of the HDR video is 8K, and both the video frame extractor in the FFMPEG software decoder and the video frame extractor in the MediaCodec hardware decoder support processing the video with 8K resolution, so the support degree of the video frame extractor in the FFMPEG software decoder to the HDR video is higher.

[0134] In some embodiments, if the video format of the HDR video is supported by the FFMPEG software decoder, the video frame can be extracted through the FFMPEG software decoder. If the FFMPEG software decoder does not support the format of the current HDR video, the video frame can be extracted through the MediaCodec hardware decoder.

[0135] 506: The decision module 420 determines whether the HDR video is supported by AOSP. If yes, go to step 511; if not, go to step 518.

[0136] In the embodiments of the present application, when the decision module 420 determines that the current video is an HDR video through the above step 504, and determines that the FFMPEG software decoder does not support the HDR video file at the moment through the above step 505, the decision module 420 further needs to determine whether the AOSP supports the HDR video. If yes, go to step 511 to create the MediaCodec extractor; if not, go to step 518, and the electronic device (such as a processor) can end the execution of the method flow.

[0137] In some embodiments, the decision that the FFMPEG does not support the HDR video file at the moment through the above step 505 can extract the video frame through the MediaCodec hardware decoder. Before the video frame is extracted through the MediaCodec hardware decoder, it is further needed to determine whether the MediaCodec hardware decoder supports processing the current HDR video.

[0138] It should be understood that MediaCodec is a hardware decoder provided by Android TM System, and AOSP is a basic code framework in Android TM System, when AOSP supports HDR video, it means that the MediaCodec hardware decoder also supports the current HDR video. Among them, the decision module 420 can determine whether AOSP supports the current HDR video file by checking the document of AOSP, detecting the code library of AOSP, and checking the hardware specifications supported by AOSP.

[0139] In some embodiments, if AOSP supports the current HDR video file, that is, the MediaCodec hardware decoder supports the current HDR video file, the video frame can be extracted by the video frame extractor in the MediaCodec hardware decoder. If AOSP does not support the current HDR video file, that is, the MediaCodec hardware decoder does not support the current HDR video file, and the decision module 420 also determines through the above step 505 that the FFMPEG software decoder also does not support the current HDR video file, go to step 518, the electronic device (such as a processor) can end the execution of the method flow.

[0140] 507: The decision module 420 queries the video frame extractor capability table to obtain the best matching video frame extractor.

[0141] In the embodiments of the present application, when the decision module 420 determines through the above steps 502 and 503 that the current video is an AOSP supported DRM video, or the decision module 420 determines through the above steps 502 and 504 that the current video is a non-DRM video and a non-HDR video, the video frame extractor capability table can be queried to select the video frame extractor with the highest extraction support for the current video file, that is, the best matching video frame extractor.

[0142] In some embodiments, the video frame extractor capability table is determined based on the extraction support of each video frame extractor for the video file format. In addition, the extraction support can be determined based on the matching degree between the processing parameters of each video frame extractor (such as video frame rate, resolution, etc.) and the video parameters of the video to be processed (such as video frame rate, resolution). For example, if the video frame rate supported by the video frame extractor in the FFMPEG software decoder is 30 frames per second to 60 frames per second, and the supported video resolution is 720P and 1080P; and the video frame rate supported by the video frame extractor in the MediaCodec hardware decoder is 45 frames per second to 60 frames per second, and the supported video resolution is 1080P and 2K. If the video frame rate of video A is 40 frames per second and the video resolution is 1080P, a video frame extractor capability table can be established: the extraction support of the FFMPEG software decoder for video A is 100%, and the extraction support of the MediaCodec hardware decoder for video A is 50%.

[0143] 508: The decision module 420 determines whether the video frame extractor in the software decoder is optimal. If so, the process proceeds to step 510; if not, the process proceeds to step 509.

[0144] In this embodiment of the present application, after the decision module 420 obtains the best matching video frame extractor through the above step 507, if it is determined that the best matching video frame extractor is the video frame extractor in the FFMPEG software decoder, the process proceeds to step 510 to create the FFMPEG extractor. If not, the process proceeds to step 509 to determine whether the video frame extractor in the MediaCodec hardware decoder is the best.

[0145] 509 : The decision module 420 determines whether the video frame extractor in the hardware decoder is optimal. If so, the process proceeds to step 511 ; if not, the process proceeds to step 518 .

[0146] In an embodiment of the present application, when the decision module 420 determines through the above step 508 that the video frame extractor in the FFMPEG software decoder is not the optimal video frame extractor, it is also necessary to determine whether the video frame extractor in the MediaCodec hardware decoder is the optimal video frame extractor. If so, go to step 511, create a MediaCodec extractor, and extract video frames through the video frame extractor in the MediaCodec hardware decoder; if not, it means that neither the FFMPEG software decoder nor the MediaCodec hardware decoder supports extracting video frames of the current video file, then go to step 518, and the electronic device (such as a processor) can end the execution of the method flow.

[0147] In some embodiments, the electronic device can perform step 508 first and then perform step 509. Alternatively, the electronic device can perform step 509 first, and if it is determined that the video frame extractor in the MediaCodec hardware decoder is not the best choice, the electronic device can perform step 508 to determine whether the video frame extractor in the FFMPEG software decoder is the best video frame extractor. That is, the present application does not limit the order of performing step 508 and step 509.

[0148] In some embodiments of the present application, steps 508 and 509 can be combined into one step. For example, when the electronic device performs step 508, if it is determined that the video frame extractor in the FFMPEG software decoder is the best video frame extractor, the electronic device proceeds to step 510 to create the FFMPEG extractor; if it is determined that the video frame extractor in the FFMPEG software decoder is not the best video frame extractor, the electronic device directly proceeds to step 511 to create the MediaCodec extractor. Alternatively, the electronic device can directly perform step 509 without performing step 508, that is, if it is determined that the video frame extractor in the MediaCodec hardware decoder is the best video frame extractor, the electronic device proceeds to step 511 to create the MediaCodec extractor; if it is determined that the video frame extractor in the MediaCodec hardware decoder is not the best video frame extractor, the electronic device directly proceeds to step 510 to create the FFMPEG extractor.

[0149] 510: The software decoder extraction module 430 creates the FFMPEG extractor.

[0150] In the embodiments of the present application, if the decision module 420 determines that the FFMPEG software decoder supports the current HDR video through step 505 described above, or if the decision module 420 determines that the video frame extractor in the FFMPEG software decoder is the best video frame extractor through step 508 described above, the decision module 420 can pass the video file to the software decoder extraction module 430, and the software decoder extraction module 430 creates the FFMPEG extractor, that is, the software decoder extraction module 430 interacts with the FFMPEG software decoder to extract the video frame through the FFMPEG software decoder.

[0151] 511: The hardware decoder extraction module 440 creates the MediaCodec extractor.

[0152] In an embodiment of the present application, when the decision module 420 determines through the above step 506 that AOSP supports the current HDR video, or the decision module 420 determines through the above step 509 that the video frame extractor in the MediaCodec hardware decoder is the optimal video frame extractor, the decision module 420 can pass the video file to the hardware decoder extraction module 440, and the hardware decoder extraction module 440 creates a MediaCodec extractor, that is, the hardware decoder extraction module 440 interacts with the MediaCodec hardware decoder and extracts video frames through the MediaCodec hardware decoder.

[0153] 512: The decision module 420 obtains video encoding format information.

[0154] In an embodiment of the present application, after extracting the video frame through the above step 510 or step 511, it is necessary to select the best video frame decoder based on the video encoding format information of the current video to decode the extracted video frame. The video frame encoding format information may include: video compression encoding format (H.264, H.265, etc.), video resolution (1080P, 480P, etc.), video bit rate (500kbps, 600kbps, etc.), and other video encoding format information.

[0155] 513: The decision module 420 queries the video frame decoder capability table to obtain the best matching video frame decoder.

[0156] In an embodiment of the present application, after the decision module 420 obtains the video frame encoding format information through the above step 512, it can query the corresponding video frame decoder capability table and select the video frame decoder with the highest support for decoding the current video file, that is, select the best matching video frame decoder.

[0157] In some embodiments, the video frame decoder capability table is determined based on the decoding support of each video frame decoder for the encoding and decoding format information of the video. In addition, the decoding support can be determined based on the matching degree between the processing parameters of each video frame decoder (such as encoding parameters, encoding format, etc.) and the video parameters of the video to be processed (such as encoding format information such as encoding parameters, encoding format, etc.). For example, if the video encoding format supported by the video frame decoder in the FFMPEG software decoder is H.264 and H.265, the supported bit rate is 100kbps to 10000kbps; and the video encoding format supported by the video frame decoder in the MediaCodec hardware decoder is H.264 and H.265, and the supported bit rate is 500kbps to 15000kbps. If the video encoding format of video A is H.264 and the video bit rate is 300kbps, a video frame decoder capability table can be established: the decoding support of the FFMPEG software decoder for video A is 100%, and the decoding support of the MediaCodec hardware decoder for video A is 50%.

[0158] 514: The decision module 420 determines whether the video frame decoder in the software decoder is optimal. If so, the process proceeds to step 516; if not, the process proceeds to step 515.

[0159] In this embodiment of the present application, after the decision module 420 obtains the best matching video frame decoder through the above step 513, if it is determined that the best matching video frame decoder is the video frame decoder in the FFMPEG software decoder, the process proceeds to step 516 to create the FFMPEG decoder. If not, the process proceeds to step 515 to determine whether the video frame decoder in the MediaCodec hardware decoder is the best.

[0160] 515: The decision module 420 determines whether the video frame decoder in the hardware decoder is optimal. If so, the process proceeds to step 517; if not, the process proceeds to step 518.

[0161] In an embodiment of the present application, when the decision module 420 determines through the above step 514 that the video frame decoder in the FFMPEG software decoder is not the optimal video frame decoder, it is also necessary to determine whether the video frame decoder in the MediaCodec hardware decoder is the optimal video frame decoder. If so, go to step 517, create a MediaCodec decoder, and decode the extracted video frame through the video frame decoder in the MediaCodec hardware decoder; if not, it means that neither the FFMPEG software decoder nor the MediaCodec hardware decoder supports decoding the current video file, then go to step 518, and the electronic device (such as a processor) can end the execution of the method flow.

[0162] In some embodiments, the electronic device may first execute step 514 and then execute step 515. Alternatively, the electronic device may first execute step 515. If it is determined that the video frame decoder in the MediaCodec hardware decoder is not the optimal choice, the electronic device may execute step 514 to determine whether the video frame decoder in the FFMPEG software decoder is the optimal video frame decoder. In other words, this application does not limit the order in which steps 514 and 515 are executed.

[0163] In other embodiments of the present application, the above-mentioned steps 514 and 515 can also be combined into one step. For example, when the electronic device executes step 514, if it is determined that the video frame decoder in the FFMPEG software decoder is the best video frame decoder, it goes to step 516 to create an FFMPEG decoder; if it is determined that the video frame decoder in the FFMPEG software decoder is not the best video frame decoder, it goes directly to step 517 to create a MediaCodec decoder. Alternatively, the electronic device may not execute step 514 and directly execute step 515, that is, if it is determined that the video frame decoder in the MediaCodec hardware decoder is the best video frame decoder, it goes to step 517 to create a MediaCodec decoder; if it is determined that the video frame decoder in the MediaCodec hardware decoder is not the best video frame decoder, it goes directly to step 516 to create an FFMPEG decoder.

[0164] In an embodiment of the present application, the electronic device may further perform any part of steps 502 to 509 and steps 512 to 515 in the above steps. That is, the electronic device may only perform steps 502 to 509, select the best video frame extractor, extract the video frame through the video frame extractor, and then use the default video frame decoder (e.g., the video frame decoder in the hardware decoder) or the same decoder to decode the extracted video frame. Alternatively, the electronic device may also use the default video frame extractor (e.g., the video frame extractor 201 in the hardware decoder 200) to extract the video frame, and then perform steps 512 to 515, select the best video frame decoder, and use the video frame decoder to decode the extracted video frame.

[0165] 516: Software Decoder The decoding module 450 creates an FFMPEG decoder.

[0166] In an embodiment of the present application, after the decision module 420 determines through the above step 514 that the video frame decoder in the FFMPEG software decoder is the optimal video frame decoder, the decision module 420 or the video frame extractor can pass the extracted video frame to the software decoder decoding module 450, and the software decoder decoding module 450 creates the FFMPEG decoder, that is, the software decoder decoding module 450 interacts with the FFMPEG software decoder and decodes the extracted video frame through the FFMPEG software decoder.

[0167] The process of interaction between the software decoder decoding module 450 and the FFMPEG software decoder can be referred to as follows Figure 6 It is described in , so I will not repeat it here.

[0168] 517: The hardware decoder decoding module 460 creates a MediaCodec decoder.

[0169] In an embodiment of the present application, after the decision module 420 determines through the above step 515 that the video frame decoder in the MediaCodec hardware decoder is the optimal video frame decoder, the decision module 420 or the video frame extractor can pass the extracted video frame to the hardware decoder decoding module 460, and the hardware decoder decoding module 460 creates a MediaCodec decoder, that is, the hardware decoder decoding module 460 interacts with the MediaCodec hardware decoder and decodes the extracted video frame through the MediaCodec hardware decoder.

[0170] The process of interaction between the hardware decoder decoding module 460 and the MediaCodec hardware decoder can be referred to as follows Figure 7 It is described in , so I will not repeat it here.

[0171] 518: The processor ends the task.

[0172] In an embodiment of the present application, when the decision module 420 determines through the above steps 502 and 503 that the current video is a DRM video and AOSP does not support DRM video; or the decision module 420 determines through the above steps 504 and 506 that the current video is an HDR video and AOSP does not support HDR video; or the decision module 420 determines through the above steps 508 and 509 that neither the FFMPEG software decoder nor the MediaCodec hardware decoder supports extracting video frames of the current video file; or when the decision module 420 determines through the above steps 514 and 515 that neither the FFMPEG software decoder nor the MediaCodec hardware decoder supports decoding the current video file, the processor in the electronic device can end execution of the method flow.

[0173] Thus, through the above flowchart, by combining the use of software decoders and hardware decoders to extract and decode video frames, the time of video processing can be shortened and the efficiency of video processing can be improved.

[0174] The following still takes the software decoder 300 as the FFMPEG software decoder as an example, based on Figure 6 The flowchart shown in FIG. 4 illustrates the interaction between the software decoder decoding module 450 and the FFMPEG software decoder. FFMPEG is an open source audio and video processing tool that includes a variety of FFMPEG software decoders. In this application, the FFMPEG code module can be integrated into the code framework of the electronic device.

[0175] Specifically, Figure 6 The method shown includes the following steps:

[0176] 601: Software decoder The decoding module 450 creates an FFMPEG software decoder.

[0177] In some embodiments, when the electronic device uses the FFMPEG software decoder, the software decoder decoding module 450 first needs to create an FFMPEG software decoder object. For example, the FFMPEG decoder object can be a structure in the FFMPEG open source library, which includes parameters of each FFMPEG software decoder (e.g., the output video format of each FFMPEG software decoder) and data structure.

[0178] 602: The software decoder decoding module 450 loads the operating environment.

[0179] In some embodiments, the software decoder decoding module 450 also needs to load the operating environment required by the FFMPEG software decoder, for example, loading the library file in the FFMPEG software decoder into the system directory of the electronic device, and loading the library file in the application where the current video needs to be extracted and decoded.

[0180] 603: The software decoder decoding module 450 controls the initialization of the FFMPEG software decoder.

[0181] In some embodiments, after the software decoder decoding module 450 creates the FFMPEG decoder object through step 601 and loads the operating environment through step 602, it is also necessary to initialize the created decoder object.

[0182] The specific initialization process may include the following steps 6031 to 6034, wherein the FFMPEG module 470 is a functional module that integrates FFMPEG open source code in the electronic device and can execute the corresponding functions mentioned in this application.

[0183] 6031: The FFMPEG module 470 selects a matching software decoder.

[0184] In some embodiments, FFMPEG is an open source audio and video processing tool that includes multiple FFMPEG software decoders. In this application, the FFMPEG code module (ie, FFMPEG module 470) can be integrated into the code framework of the electronic device.

[0185] In some embodiments, the FFMPEG module 470 needs to select a matching software decoder from a plurality of FFMPEG software decoders. For example, a matching software decoder can be selected based on the names of the various FFMPEG software decoders. For example, for a video file encoded in H.264 format, a software decoder named libx264 can be selected as the matching software decoder. Alternatively, the FFMPEG documentation can be consulted to select a matching software decoder based on the current video format (e.g., frame rate, encoding format, etc.). This application is not limited to this.

[0186] 6032: FFMPEG module 470 allocates the operating environment.

[0187] In some embodiments, after selecting a matching software decoder, the FFMPEG module 470 also needs to allocate an operating environment to the matching software decoder, for example, allocating processor resources to the matching software decoder, providing operating system support, etc.

[0188] 6033: FFMPEG module 470 allocates memory space.

[0189] In the embodiment of the present application, the FFMPEG module 470 also needs to allocate memory space to the matching software decoder, providing the matching software decoder with sufficient memory for storing data and temporary variables required by the software decoder.

[0190] 6034: FFMPEG module 470 completes the integration of the matching software decoder.

[0191] In an embodiment of the present application, after completing steps 6031 to 6034 above, the FFMPEG module 470 can complete the initialization process of the FFMPEG software decoder, that is, integrate the matching software decoder into the system of the electronic device and / or into the application where the current video is located.

[0192] 604: The software decoder decoding module 450 obtains an input buffer.

[0193] In an embodiment of the present application, the software decoder decoding module 450 may apply for temporary storage space as an input buffer for storing video frame data to wait for the above-mentioned matching software decoder to read and process it.

[0194] 605: The software decoder extraction module 430 reads the video data.

[0195] In the embodiment of the present application, the software decoder extraction module 430 can extract the video frame and pass the video frame data to the software decoder decoding module 450, which is decoded by the software decoder decoding module 450. Figure 5B As mentioned in, I will not elaborate on this.

[0196] 606: The software decoder decoding module 450 puts the data into the input buffer.

[0197] In the embodiment of the present application, the software decoder decoding module 450 can obtain the video frame data sent by the software decoder extraction module 430 and put the video frame data into the input buffer obtained in step 604.

[0198] 607: The software decoder decoding module 450 decodes.

[0199] In this embodiment of the present application, after the electronic device completes the initialization of the matching software decoder through steps 6031 to 6034 and places the video frame data into the input buffer through steps 604 to 606, the electronic device can read the data in the input buffer through the matching software decoder and perform decoding. The specific decoding process may include steps 6071 to 6073.

[0200] 6071: The FFMPEG module 470 passes the data in the input buffer to the matching software decoder.

[0201] In an embodiment of the present application, the FFMPEG module 470 can read the video frame data in the input buffer and send the video frame data to a matching software decoder for decoding.

[0202] 6072: FFMPEG module 470 puts the decoded data into the output buffer.

[0203] In an embodiment of the present application, after the matching software decoder decodes the video frame data, the decoded video frame data can be placed in an output buffer through the FFMPEG module 470 to wait for reading the decoded video frame data.

[0204] 6073: FFMPEG module 470 reads data in the output buffer.

[0205] In the embodiment of the present application, the FFMPEG module 470 can read the data in the output buffer and send the data to the software decoder decoding module 450 for processing. In this way, the process of decoding the extracted video frame data can be completed through steps 6071 to 6073.

[0206] 608: The software decoder decoding module 450 puts the data into the output buffer.

[0207] In the embodiment of the present application, after the software decoder decoding module 450 obtains the decoded data frames sent by the FFMPEG module 470 in step 6073, it can sequentially place the decoded data frames into a buffer queue, waiting for the electronic device to read the data in the buffer queue and perform content detection, etc. It should be understood that the buffer queue is a different buffer from the output buffer in step 6072, wherein the data in the buffer queue needs to wait for the electronic device to read and perform operations such as content detection; the data in the output buffer in step 6072 needs to wait for the FFMPEG module 470 to read and send to the software decoder decoding module 450 for processing.

[0208] 609: The software decoder decoding module 450 reads the data in the output buffer.

[0209] In the embodiment of the present application, after the software decoder decoding module 450 places the data into the output buffer through the above step 608, the software decoder decoding module 450 can read the data in the output buffer.

[0210] 610: The software decoder decoding module 450 determines that the read data is the required video frame.

[0211] In an embodiment of the present application, after the software decoder decoding module 450 reads the data in the output buffer through the above-mentioned step 609, it can determine whether the read data (i.e., the decoded video frame data) is the required video frame. If the software decoder decoding module 450 determines that the read data is the required video frame, for example, in video content detection, the video frame data contains the image data required for detection, etc., then steps 611 to 613 can be executed to release memory space and end the execution method flow. If the software decoder decoding module 450 determines that the read data is not the required video frame, it is necessary to re-execute step 609 and read the other data in the output buffer in sequence.

[0212] 611: The software decoder decoding module 450 releases the input buffer and the output buffer.

[0213] In the embodiment of the present application, after the software decoder decoding module 450 determines that the read data is the required video frame in step 610, the method flow can be terminated. In the process of terminating the method flow, the input buffer and output buffer space need to be released, for example, the data in the input buffer in step 604 and the output buffer in step 608 may be cleared and reset.

[0214] 612: The software decoder decoding module 450 releases the FFMPEG software decoder.

[0215] In an embodiment of the present application, when the execution method flow ends, the software decoder decoding module 450 also needs to release the FFMPEG software decoder object created in the above step 601, for example, clear the parameters and release the space occupied by the FFMPEG software decoder object.

[0216] 613: FFMPEG module 470 releases memory space.

[0217] In an embodiment of the present application, when the execution method flow ends, the FFMPEG module 470 also needs to release the memory space allocated for the matching software decoder in step 6033.

[0218] In this way, through the above process, the interaction between the software decoder decoding module 450 and the FFMPEG software decoder can be completed, and the process of decoding the extracted video frames through the FFMPEG software decoder can be realized.

[0219] The following takes the hardware decoder 200 as the MediaCodec hardware decoder 480 as an example, based on Figure 7 The flow chart shown above Figure 4 The process of interaction between the hardware decoder decoding module 460 and the MediaCodec hardware decoder 480 mentioned in is described.

[0220] 701: The hardware decoder decoding module 460 creates a MediaCodec hardware decoder.

[0221] In some embodiments, when an electronic device uses a MediaCodec hardware decoder, the hardware decoder decoding module 460 first needs to create a MediaCodec hardware decoder, for example, by configuring and instantiating a hardware decoder through the MediaCodec class on the electronic device.

[0222] 702: The hardware decoder decoding module 460 initializes the MediaCodec hardware decoder.

[0223] In some embodiments, the hardware decoder decoding module 460 needs to initialize the created MediaCodec hardware decoder, where the initialization may include allocating a running environment, allocating memory space, and the like.

[0224] 703: The MediaCodec hardware decoder 480 starts working.

[0225] In some embodiments, after the hardware decoder decoding module 460 creates and initializes the MediaCodec hardware decoder, the configuration of the MediaCodec hardware decoder can be completed, that is, the MediaCodec hardware decoder 480 can start working.

[0226] 704: The MediaCodec hardware decoder 480 places the decoded data into an output buffer.

[0227] In some embodiments, the MediaCodec hardware decoder 480 can send the decoded video frame data to an output buffer for subsequent processing, for example, detecting whether the content in the video frame includes a target image.

[0228] 705: The hardware decoder decoding module 460 reads the data in the output buffer.

[0229] In some embodiments, the hardware decoder decoding module 460 can read the decoded video data from the output buffer.

[0230] 706: The hardware decoder decoding module 460 determines whether the output buffer is empty. If so, the process proceeds to step 709; if not, the process proceeds to step 707.

[0231] In some embodiments, if the hardware decoder decoding module 460 determines that the data read out in step 705 is empty, indicating that the output buffer is empty, that is, the MediaCodec hardware decoder 480 has not decoded the video frame data, then the process proceeds to step 709 to obtain the input buffer to input data into the input buffer for decoding by the MediaCodec hardware decoder 480. If the hardware decoder decoding module 460 determines that the data read out in step 705 is not empty, then the process proceeds to step 707 to determine whether the data is the required video frame.

[0232] 707: The hardware decoder decoding module 460 determines whether the data is the required video frame. If so, the process proceeds to step 708; if not, the process proceeds to step 705.

[0233] In some embodiments, if the data read by the hardware decoder decoding module 460 is not empty, the read data (i.e., the decoded video frame data) can be determined to be the required video frame. If the hardware decoder decoding module 460 determines that the read data is the required video frame, for example, in video content detection, the video frame data contains image data required for detection, then step 708 can be executed to release the output buffer. If the hardware decoder decoding module 460 determines that the read data is not the required video frame, step 705 needs to be re-executed to sequentially read other data in the output buffer.

[0234] 708: The MediaCodec hardware decoder 480 releases the output buffer.

[0235] In some embodiments, if the hardware decoder decoding module 460 determines through step 707 that the read data is the required video frame, the output buffer may be released.

[0236] 709: The MediaCodec hardware decoder 480 obtains the input buffer.

[0237] In some embodiments, if the hardware decoder decoding module 460 determines through the above step 706 that the output buffer is empty, indicating that the MediaCodec hardware decoder 480 has not decoded the video frame data, then the MediaCodec hardware decoder 480 needs to obtain the input buffer, read the video frame data to be processed input in the input buffer and decode it.

[0238] 710: The hardware decoder decoding module 460 determines whether the input buffer is available. If so, the process proceeds to step 711; if not, the process repeats step 709.

[0239] In some embodiments, the hardware decoder decoding module 460 also needs to determine whether the input buffer obtained by the MediaCodec hardware decoder 480 is an available buffer. For example, if the memory space of the input buffer is larger than the storage space required for the video frame, the input buffer is available, and the process goes to step 711, where the hardware decoder decoding module 460 obtains the video frame data to be decoded. If the input buffer is not available, step 709 is repeated to obtain the input buffer again.

[0240] 711: The hardware decoder decoding module 460 obtains the video frame data to be decoded.

[0241] In some embodiments, the hardware decoder decoding module 460 can obtain the video frame data to be decoded and send the data to the input buffer, wherein the video frame data to be decoded is extracted by the video frame extractor. For detailed description, please refer to the above Figure 5B As mentioned in, I will not elaborate on this.

[0242] 712: The hardware decoder decoding module 460 determines whether all video frames have been extracted. If so, the process proceeds to steps 713 and 714; if not, the process proceeds to step 713.

[0243] In some embodiments, after the hardware decoder decoding module 460 obtains the video frame data to be decoded, it is also necessary to determine whether all the video frame data has been obtained. Figure 1 As shown in , when performing rapid content detection on video A, video A is divided into four short videos and the first frame image is extracted from each of them. The hardware decoder decoding module 460 can determine whether the acquired video frame data is four video frame data. If so, go to step 713 and step 714 to obtain the data in the input buffer and determine whether the method flow is completed; if not, go to step 713, the MediaCodec hardware decoder 480 obtains the data in the input buffer and repeatedly obtains the video frame data to be decoded.

[0244] 713: The MediaCodec hardware decoder 480 obtains data in the input buffer.

[0245] In some embodiments, the MediaCodec hardware decoder 480 can obtain the data to be decoded in the buffer and decode it, wherein the decoded video frame data can be sent to the output buffer, that is, the above steps 705 to 708 are repeatedly executed.

[0246] 714: The hardware decoder decoding module 460 determines whether the decoding of the video frame is completed. If so, the process proceeds to step 715; if not, the process proceeds to step 705.

[0247] In some embodiments, the hardware decoder decoding module 460 can determine whether the decoding of the video frame is complete. If so, the method proceeds to step 715 to release the MediaCodec hardware decoder and end the method flow. If not, the method proceeds to step 705 to continue reading data in the output buffer.

[0248] The judgment condition for whether the decoding of the video frame is completed can be whether the above step 707 or the above step 712 is satisfied. If so, it means that the decoded data is the required video frame data or all video frame data has been decoded, then the MediaCodec hardware decoder is released and the method flow ends. If not, it means that the decoded data is not the required video frame data or there is still undecoded video frame data, then the above steps are repeated.

[0249] 715: The hardware decoder decoding module 460 releases the MediaCodec hardware decoder.

[0250] In some embodiments, after the hardware decoder decoding module 460 determines that the decoding of the video frame is completed through the above step 714, the MediaCodec hardware decoder ends the execution of the method flow, for example, clears parameters and releases the space occupied by the MediaCodec hardware decoder.

[0251] In this way, through the above flow, the interaction between the hardware decoder decoding module 460 and the MediaCodec hardware decoder 480 is completed, and the process of decoding the extracted video frame by the MediaCodec hardware decoder 480 is realized.

[0252] In this way, through the method mentioned in the present application, by combining the use of a software decoder and a hardware decoder, the time of video processing can be shortened, and the efficiency of video processing can be improved.

[0253] The embodiment of the present application provides an electronic device, including a memory and a processor, the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device and is used to execute the video processing method mentioned in the present application.

[0254] The embodiment of the present application provides a readable storage medium, and the readable storage medium stores instructions, and the instructions are executed on the electronic device to make the electronic device execute the image processing method mentioned in the present application.

[0255] The embodiment of the present application further provides a computer program product, including computer instructions, when the computer instructions are executed on the electronic device, the electronic device executes the image processing method mentioned in the present application.

[0256] The video processing method provided by the embodiment of the present application can be applied to an electronic device, and the electronic device can be any electronic device such as a mobile phone, a computer, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, and the like, and the embodiment of the present application does not make any limitation.

[0257] The embodiments disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as computer programs or program codes executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memories and / or storage elements), at least one input device and at least one output device.

[0258] The program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices, in known fashion. For purposes of this application, a processing system includes any system that has a processor, such as a digital signal processor, microcontroller, application specific processor, or microprocessor.

[0259] The program code can be implemented in a high level procedural or object oriented programming language to communicate with a processing system. The program code can be implemented in assembly or machine language, if desired. In fact, the mechanisms described herein are not limited in scope to any particular programming language. In any case, the language can be a compiled or interpreted language.

[0260] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable medium, such as a computer-readable medium, which can be read and executed by one or more processors. For example, a machine-readable medium can include a floppy diskette, an optical disk, a compact disk, a magnetic tape, a flash memory, a paper tape or punch card, an optical mark sheet, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a hard disk drive, a memory stick, or a solid-state drive. The disclosed embodiments can also be embodied in a computer program product which can be executed by one or more processors. Therefore, a machine-readable medium can include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including without limitation a floppy diskette, an optical disk, a compact disk, a magnetic tape, a flash memory, a paper tape or punch card, an optical mark sheet, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a hard disk drive, a memory stick, or a solid-state drive. Therefore, a machine-readable medium includes any type of medium for storing or transmitting information in a form readable by a machine (e.g., a computer).

[0261] In the drawings, some of the structural or methodological acts can be shown in a particular arrangement and / or order. However, it should be understood that such is merely an example, and that other arrangements and / or orders can be made without departing from the scope of the present disclosure. Moreover, the inclusion of a structural or methodological act in a particular figure does not imply that the act is required in all examples, and in some examples the act can not be included or can be combined with other acts in a different manner.

[0262] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0263] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further qualifications, an element defined by the statement "comprises a" does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0264] While the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A video processing method, applied to electronic equipment, characterized in that: The electronic device includes a hardware decoder; and The method comprises: Get the first video to be processed; determining a decoding method for the first video based on a degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device; The video parameters of the first video include parameters in the video file format of the first video and parameters in the encoding format information of the first video; Furthermore, determining a decoding method for the first video based on a matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device includes: Based on a degree of matching between parameters in the video file format of the first video and processing parameters of a video frame extractor in the hardware decoder and processing parameters of a video frame extractor in the software decoder, one of the hardware decoder and the software decoder is used as a first target decoder, and video frames are extracted from the first video by the first target decoder; Based on the matching degree between the parameters in the encoding format information of the first video and the processing parameters of the video frame decoder in the hardware decoder and the processing parameters of the video frame decoder in the software decoder, one of the hardware decoder and the software decoder is used as a second target decoder, and the video frames extracted from the first video are decoded by the second target decoder.

2. The method according to claim 1, characterized in that include: determining a first extraction support level of the hardware decoder for the first video based on a matching degree between processing parameters of a video frame extractor in the hardware decoder and parameters of a video file format of the first video; Based on the matching degree between each processing parameter of the video frame extractor in the software decoder and each parameter of the video file format of the first video, the second extraction support degree of the software decoder for the first video is determined.

3. The method according to claim 2, characterized in that The processing parameters of the video frame extractor include one or more of video resolution and video frame rate.

4. The method according to claim 1, wherein include: determining a first decoding support level of the hardware decoder for the first video based on a matching degree between processing parameters of a video frame decoder in the hardware decoder and parameters of the video encoding format information of the first video; Based on the matching degree between each processing parameter of the video frame decoder in the software decoder and each parameter of the video encoding format information of the first video, a second decoding support degree of the software decoder for the first video is determined.

5. The method according to claim 4, characterized in that The processing parameters of the video frame decoder include one or more of video resolution, video frame rate, bit rate, and encoding format.

6. The method according to any one of claims 2 to 5, characterized in that The determining a decoding method for the first video based on a matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, includes: Selecting the first target decoder with the greatest support for extraction of the first video from the hardware decoder and the software decoder, and performing video frame extraction on the first video; The second target decoder with the greatest decoding support for the first video is selected from the hardware decoder and the software decoder to decode the video frame extracted from the first video.

7. The method according to claim 1, characterized in that The first video includes one or more of digital rights management video, high dynamic range video, and standard dynamic range video.

8. The method according to claim 7, characterized in that Corresponding to the first video being the high dynamic range video, determining a decoding method for the first video based on a degree of matching between video parameters of the first video and processing parameters of the hardware decoder and processing parameters of a software decoder on the electronic device, includes: extracting video frames from the first video using the software decoder when video parameters of the first video match processing parameters of the software decoder; In a case where the video parameters of the first video do not match the processing parameters of the software decoder and the video parameters of the first video match the processing parameters of the hardware decoder, the hardware decoder is used to extract video frames in the first video.

9. The method according to claim 7, characterized in that Corresponding to the first video being the digital rights management video and the standard dynamic range video, determining a decoding method for the first video based on a degree of matching between video parameters of the first video and processing parameters of the hardware decoder and processing parameters of a software decoder on the electronic device, respectively, includes: When the electronic device supports processing the first video, a method for extracting video frames from the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device.

10. The method according to claim 9, characterized in that The determining of an extraction method for extracting video frames from the first video based on a matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, includes: determining a first extraction support level of the hardware decoder for the first video based on a matching degree between processing parameters of a video frame extractor in the hardware decoder and parameters of a video file format of the first video; determining, based on a degree of matching between processing parameters of a video frame extractor in the software decoder and parameters of a video file format of the first video, a second extraction support level of the software decoder for the first video; establishing a video frame extractor capability table based on the first extraction support and the second extraction support; In the video frame extractor capability table, the first target decoder corresponding to the largest extraction support is selected to extract the video frame in the first video.

11. The method according to claim 7, characterized in that The determining a decoding method for the first video based on a matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, includes: determining a first decoding support level of the hardware decoder for the first video based on a matching degree between processing parameters of a video frame decoder in the hardware decoder and parameters of the video encoding format information of the first video; determining, based on a degree of matching between processing parameters of a video frame decoder in the software decoder and parameters of the video encoding format information of the first video, a second decoding support level of the software decoder for the first video; Establishing a video frame decoder capability table based on the first decoding support and the second decoding support; In the video frame decoder capability table, the second target decoder corresponding to the maximum decoding support is selected to decode the video frame extracted from the first video.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the video processing method according to any one of claims 1 to 11.

13. A readable storage medium, characterized in that: The readable storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the video processing method according to any one of claims 1 to 11.

14. A computer program product, characterized in that include: Computer instructions, when the computer instructions are executed on an electronic device, enable the electronic device to execute the video processing method according to any one of claims 1 to 11.

Images