Video processing method and device, storage medium and program product

Abstract

The invention relates to the technical field of multimedia data processing, and discloses a video processing method and device, a storage medium and a program product. The video processing method comprises the following steps: in a stage of extracting a video frame, the electronic equipment can determine a decoder for extracting the video frame based on a matching degree between a processing parameter of a video frame extractor in a software decoder and a hardware decoder and a video parameter of a video to be processed. Then, after the video frame is extracted, the electronic device can also determine a decoder for decoding the extracted video frame based on the matching degree between the processing parameter of the video frame decoder in the software decoder and the hardware decoder and the video parameter of the video to be processed. Therefore, through combined use of the software decoder and the hardware decoder, the video processing time can be shortened, and the video processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of multimedia data processing, and in particular to a video processing method, device, storage medium and program product. Background Art

[0002] Currently, when mobile phones and other electronic devices quickly detect and identify video content, they usually divide a long video into multiple short videos, then extract the first frame image in each short video and decode it, and then detect whether each first frame image contains the target content. Figure 1 As shown, taking the example of a mobile phone 10 detecting whether a 60S video A contains a vehicle 100, a brief introduction is given to the scenario of rapid detection and recognition of video content. Figure 1 As shown in (a) in FIG. 1 , video A is a long video with a duration of 60 seconds. At this time, the electronic device can divide video A into four short videos according to the duration to detect whether there is a vehicle 100 in video A. Figure 1 As shown in (b), the electronic device can divide video A into four short videos, video B, video C, video D and video E, wherein the duration of each short video is 15 seconds, and screen 101, screen 102, screen 103 and screen 104 are the first frame images in each short video. The electronic device can extract the four first frame images and decode them, and then the electronic device detects the contents therein respectively. If the vehicle 100 is detected in the screen 102, it means that the vehicle 100 appears in the 60-second video A, and the video content can be quickly detected and identified.

[0003] Currently, electronic devices usually use hardware decoders to extract and decode the first frame of a video. However, due to the limitations of hardware devices, such as the need to spend time configuring hardware resources or initializing hardware decoders, it takes a long time for electronic devices to detect the content of the first frame of a video.

[0004] Therefore, how to shorten the extraction and decoding time of the first frame image of the electronic device video, thereby improving the speed of video content detection, is a problem that needs to be solved urgently. Summary of the invention

[0005] To solve the above problems, the embodiments of the present application provide a video processing method, device, storage medium and program product.

[0006] In a first aspect, the present application provides a video processing method, which is applied to an electronic device, wherein the electronic device includes a hardware decoder; and the method includes: obtaining a first video to be processed; determining a decoding method for the first video based on the degree of matching between video parameters of the first video and processing parameters of a hardware decoder and processing parameters of a software decoder on the electronic device; wherein the decoding method includes an extraction method for extracting video frames from the first video and a decoding method for decoding video frames extracted from the first video; and the extraction method includes extracting video frames using a hardware decoder or extracting video frames using a software decoder; the decoding method includes decoding video frames using a hardware decoder or decoding video frames using a software decoder.

[0007] In the present application, the first video may be the current video to be processed mentioned later.

[0008] In the embodiment of the present application, the electronic device can determine the decoder for extracting the video frame based on the matching degree between the processing parameters (such as video frame rate, resolution, etc.) of the video frame extractor in each decoder and the video parameters (such as video frame rate, resolution) of the video to be processed. If the matching degree between the processing parameters and the video parameters is high, it means that the video frame extractor has a high support for extracting the first video, and the video frame extractor is used to extract the video frame.

[0009] In other embodiments, the electronic device may also determine a decoder for decoding the extracted video frame based on the matching degree between the processing parameters (e.g., encoding parameters, encoding formats, etc.) of the video frame decoder in each decoder and the video parameters (e.g., encoding format information such as encoding parameters, encoding formats, etc.) of the video to be processed. If the matching degree between the processing parameters and the video parameters is high, it means that the video frame decoder has a high decoding support for the first video, and the video frame decoder is used to decode the extracted video frame.

[0010] Based on the above method, a video frame extractor in a matching decoder is used to extract video frames in the first video, and a video frame decoder in a matching decoder is used to decode the extracted video frames, which can save the time of extracting and decoding the video frames.

[0011] In a possible implementation of the first aspect above, it includes: determining the first extraction support of the hardware decoder for the first video based on the matching degree between each processing parameter of the video frame extractor in the hardware decoder and each parameter of the video file format of the first video; and determining the second extraction support of the software decoder for the first video based on the matching degree between each processing parameter of the video frame extractor in the software decoder and each parameter of the video file format of the first video.

[0012] In the present application, the first extraction support may be the extraction support of the hardware decoder mentioned later for the current video file, for example, it may be the extraction support of the hardware decoder 200 for the video; the second extraction support may be the extraction support of the software decoder mentioned later for the current video file, for example, it may be the extraction support of the software decoder 300 for the video.

[0013] In some embodiments, the electronic device can determine the extraction support of each video frame extractor for the current video based on the matching degree between the processing parameters of each video frame extractor and the video parameters of the video to be processed, thereby determining the extraction support of the decoder corresponding to the video frame extractor for the current video.

[0014] In some embodiments, the parameters of the video file format of the first video are stored in the header information of the first video file, and the processor can obtain the parameters by reading the header information in the first video file; the processor can also interact with the hardware decoder or software decoder to obtain the current status information of the hardware decoder or software decoder (e.g., the processing parameters such as the video frame rate and resolution that can be processed). In this way, based on the extraction support, the decoder that best matches the current video can be selected to extract the video frame.

[0015] In a possible implementation of the first aspect, the processing parameters include but are not limited to video resolution and video frame rate.

[0016] In some embodiments, the electronic device can determine the extraction support based on the matching degree between the processing parameters of each video frame extractor (such as video frame rate, resolution, etc.) and the video parameters of the video to be processed (such as video frame rate, resolution). For example, if the video frame rate supported by the video frame extractor in the software decoder is 30 frames per second to 60 frames per second, and the supported video resolution is 720P and 1080P; and the video frame rate supported by the video frame extractor in the hardware decoder is 45 frames per second to 60 frames per second, and the supported video resolution is 1080P and 2K; at the same time, the video frame rate of video A is 40 frames per second, and the video resolution is 1080P. Then the extraction support of the software decoder for video A is 100%, and the extraction support of the hardware decoder for video A is 50%. In this way, the decoder with the largest extraction support can be selected to extract the video frame of the first video to improve the efficiency of video frame extraction.

[0017] In a possible implementation of the first aspect above, it includes: determining the first decoding support of the hardware decoder for the first video based on the matching degree between each processing parameter of the video frame decoder in the hardware decoder and each parameter of the video encoding format information of the first video; and determining the second decoding support of the software decoder for the first video based on the matching degree between each processing parameter of the video frame decoder in the software decoder and each parameter of the video encoding format information of the first video.

[0018] In the present application, the first decoding support may be the decoding support of the hardware decoder mentioned later for the current video file, for example, it may be the decoding support of the hardware decoder 200 for the video; the second decoding support may be the decoding support of the software decoder mentioned later for the current video file, for example, it may be the decoding support of the software decoder 300 for the video.

[0019] In some embodiments, the electronic device can determine the decoding support of each video frame decoder for the first video based on the matching degree between the processing parameters of each video frame decoder and the video parameters of the first video, thereby determining the decoding support of the first video by the decoder corresponding to the video frame decoder.

[0020] In some embodiments, the parameters of the video encoding format information of the first video are stored in the header information of the first video file, and the processor can obtain the parameters by reading the header information in the first video file; the processor can also interact with the hardware decoder or software decoder to obtain the current status information of the hardware decoder or software decoder (e.g., the encoding parameters that can be processed, the encoding format and other processing parameters). In this way, based on the support of each decoding, the decoder that best matches the current video can be selected to decode the extracted video frame.

[0021] In a possible implementation of the first aspect, the processing parameters include but are not limited to video resolution, video frame rate, bit rate, and encoding format.

[0022] In some embodiments, the electronic device can determine the decoding support based on the matching degree between the processing parameters of each video frame decoder (such as video resolution, video frame rate, bit rate, encoding format, etc.) and the video parameters of the video to be processed (such as video resolution, video frame rate, bit rate, encoding format rate). For example, if the video encoding format supported by the video frame decoder in the software decoder is H.264, H.265, and the supported bit rate is 100kbps to 10000kbps; and the video encoding format supported by the video frame decoder in the hardware decoder is H.264, H.265, and the supported bit rate is 500kbps to 15000kbps; at the same time, if the video encoding format of video A is H.264, and the video bit rate is 300kbps. Then the decoding support of the software decoder for video A is 100%, and the decoding support of the hardware decoder for video A is 50%. In this way, the decoder with the largest decoding support can be selected to decode the video frame extracted from the first video to improve the decoding efficiency.

[0023] In a possible implementation of the first aspect, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: selecting a decoder with the greatest support for extraction of the first video from the hardware decoder and the software decoder, and extracting video frames from the first video; selecting a decoder with the greatest support for decoding the first video from the hardware decoder and the software decoder, and decoding the video frames extracted from the first video.

[0024] In the present application, the electronic device can determine the decoder for extracting video frames by the extraction support of the video frame extractor for the video file. For example, if it is determined that the video frame extractor in the software decoder has a high extraction support for the video file, the video frame is extracted by the software decoder; if it is determined that the video frame extractor in the hardware decoder has a high extraction support for the video file, the video frame is extracted by the hardware decoder.

[0025] In some embodiments, after extracting the video frames, the electronic device may also determine the decoder for decoding the video frames through the decoding support of the video codec format information by the video frame decoder. For example, if the video frame decoder in the software decoder has high decoding support for the codec format information of the video, the extracted video frames are decoded by the software decoder; if the video frame decoder in the hardware decoder has high decoding support for the codec format information of the video, the extracted video frames are decoded by the hardware decoder.

[0026] Thus, by combining the software decoder and the hardware decoder, in the video frame extraction and decoding stages, a more suitable video frame extractor can be used to extract the video frame, and a suitable video frame decoder can be used to decode the extracted video frame. Thus, the video processing time can be shortened and the video processing efficiency can be improved.

[0027] In a possible implementation of the first aspect above, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: selecting a decoder with the greatest support for the extraction of the first video from the hardware decoder and the software decoder, and extracting video frames from the first video; using a preset decoder to decode the video frames extracted from the first video.

[0028] In the present application, the electronic device can determine the decoder for extracting video frames by the extraction support of the video frame extractor for the video file. For example, if it is determined that the video frame extractor in the software decoder has a high extraction support for the video file, the video frame is extracted by the software decoder; if it is determined that the video frame extractor in the hardware decoder has a high extraction support for the video file, the video frame is extracted by the hardware decoder.

[0029] In some embodiments, after extracting the video frame, the electronic device may also use a preset or system default video frame decoder (e.g., a video frame decoder in a hardware decoder) to decode the extracted video frame. In this way, the decoder with the largest extraction support may be selected to extract the video frame of the first video to improve the efficiency of video frame extraction.

[0030] In a possible implementation of the first aspect above, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, and also includes: selecting a decoder with the greatest support for extraction of the first video from the hardware decoder and the software decoder to extract video frames for the first video; and using a decoder that extracts video frames for the first video to decode the video frames extracted from the first video.

[0031] In the present application, the electronic device can determine the decoder for extracting video frames by the extraction support of the video frame extractor for the video file. For example, if it is determined that the video frame extractor in the software decoder has a high extraction support for the video file, the video frame is extracted by the software decoder; if it is determined that the video frame extractor in the hardware decoder has a high extraction support for the video file, the video frame is extracted by the hardware decoder.

[0032] In some embodiments, after extracting the video frames, the electronic device may also use a decoder that extracts video frames from the first video to decode the video frames extracted from the first video. For example, if the electronic device determines that the video frame extractor in the software decoder has a high support for extracting video files, the electronic device may use the software decoder to extract the video frames in the current video file, and use the software decoder to decode the extracted video frames. In this way, the decoder with the highest extraction support may be selected to extract video frames from the first video to improve the efficiency of video frame extraction.

[0033] In a possible implementation of the first aspect above, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: using a preset decoder to extract video frames from the first video; selecting a decoder with the greatest decoding support for the first video from the hardware decoder and the software decoder, and decoding the video frames extracted from the first video.

[0034] In the present application, the electronic device may use a preset or system default video frame extractor (e.g., a video frame extractor in a hardware decoder) to extract the video frame, and then select a video frame decoder with high decoding support for the video codec format information, and use the video frame decoder to decode the extracted video frame. In this way, a decoder with the highest decoding support may be selected to decode the video frame extracted from the first video, so as to improve decoding efficiency.

[0035] In a possible implementation of the first aspect, the first video includes but is not limited to a digital rights management video, a high dynamic range video, and a standard dynamic range video.

[0036] In the present application, the first video may be any type of video, for example, any type of video such as a digital rights management (DRM) video, a high dynamic range (HDR) video, a standard dynamic range (SDR) video, etc. processed by encryption technology. It should be understood that the video processing method mentioned in the present application may be applied to any type of video, and the present application does not limit this.

[0037] In a possible implementation of the first aspect above, corresponding to the first video being a high dynamic range video, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: when the video parameters of the first video match the processing parameters of the software decoder, using the software decoder to extract video frames in the first video; when the video parameters of the first video do not match the processing parameters of the software decoder and the video parameters of the first video match the processing parameters of the hardware decoder, using the hardware decoder to extract video frames in the first video.

[0038] In the present application, when the video type of the first video is HDR video, if the video frame extractor in the software decoder and the video frame extractor in the hardware decoder both support the video format of the current HDR video (i.e., parameter matching), the video frame extractor in the software decoder will have higher support for the extraction of the current HDR video file than the video frame extractor in the hardware decoder will have.

[0039] In some embodiments, if the video format of the HDR video is a video format supported by the software decoder, the video frame is extracted by the software decoder; if the software decoder does not support the format of the current HDR video, the video frame can be extracted by the hardware decoder. In this way, a matching decoder can be selected to extract the video frame of the HDR video to improve the efficiency of video frame extraction.

[0040] In a possible implementation of the first aspect above, corresponding to the first video being a digital rights management video and a standard dynamic range video, the decoding method of the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: when the electronic device supports processing of the first video, determining an extraction method for extracting video frames from the first video based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device.

[0041] In the present application, if the video type of the first video is a DRM video, the electronic device can determine whether it supports processing DRM videos. For example, the electronic device can determine whether AOSP supports processing DRM videos by viewing the Android Open Source Project (AOSP) document, detecting the AOSP code base, and viewing the hardware specifications supported by AOSP. TM The basic code framework in the operating system, for Android TM The device provides basic functions and features.

[0042] In some embodiments, if the video file type of the first video is a common video (e.g., SDR video, etc.), or a DRM video and the current electronic device supports processing DRM video, the electronic device can determine the extraction support based on the matching degree between the processing parameters of each video frame extractor (e.g., video frame rate, resolution, etc.) and the video parameters of the video to be processed (e.g., video frame rate, resolution), thereby determining the extraction method for extracting video frames from the first video. In this way, a matching decoder can be selected to extract video frames from the first video to improve the efficiency of video frame extraction.

[0043] In a possible implementation of the first aspect, the extraction method for extracting video frames from the first video is determined based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: determining a first extraction support of the hardware decoder for the first video based on the matching degree between the processing parameters of the video frame extractor in the hardware decoder and the parameters of the video file format of the first video; determining a second extraction support of the software decoder for the first video based on the matching degree between the processing parameters of the video frame extractor in the software decoder and the parameters of the video file format of the first video; establishing a video frame extractor capability table based on the first extraction support and the second extraction support; and selecting a decoder corresponding to the largest extraction support in the video frame extractor capability table to extract video frames from the first video.

[0044] In the present application, the first extraction support may be the extraction support of the hardware decoder mentioned later for the first video file, for example, it may be the extraction support of the hardware decoder 200 for the video; the second extraction support may be the extraction support of the software decoder mentioned later for the first video file, for example, it may be the extraction support of the software decoder 300 for the video.

[0045] In some embodiments, the electronic device can determine the extraction support of each video frame extractor for the current video based on the matching degree between the processing parameters of each video frame extractor and the video parameters of the video to be processed. In addition, the electronic device can establish a video frame extractor capability table based on the extraction support of each video frame extractor for the video file format. The electronic device can query the video frame extractor capability table and select the video frame extractor with the highest extraction support for the first video file, that is, select the best matching decoder to extract video frames for the first video. In this way, a matching decoder can be selected to extract video frames for the first video to improve the efficiency of video frame extraction.

[0046] In a possible implementation of the first aspect, the decoding method of the first video is determined based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, including: determining the first decoding support of the hardware decoder for the first video based on the matching degree between the processing parameters of the video frame decoder in the hardware decoder and the parameters of the video encoding format information of the first video; determining the second decoding support of the software decoder for the first video based on the matching degree between the processing parameters of the video frame decoder in the software decoder and the parameters of the video encoding format information of the first video; establishing a video frame decoder capability table based on the first decoding support and the second decoding support; in the video frame decoder capability table, selecting a decoder corresponding to the largest decoding support to decode the video frame extracted from the first video.

[0047] In the present application, the first decoding support may be the decoding support of the first video file by the hardware decoder mentioned later, for example, it may be the decoding support of the video by the hardware decoder 200; the second decoding support may be the decoding support of the first video file by the software decoder mentioned later, for example, it may be the decoding support of the video by the software decoder 300.

[0048] In some embodiments, the electronic device can determine the decoding support of each video frame decoder for the current video based on the matching degree between the processing parameters of each video frame decoder and the video parameters of the video to be processed. In addition, the electronic device can establish a video frame decoder capability table based on the decoding support of each video frame decoder for the encoding and decoding format information of the video. The electronic device can query the video frame decoder capability table and select the video frame decoder with the highest decoding support for the first video file, that is, select the best matching decoder to decode the video frame extracted from the first video. In this way, a matching decoder can be selected to decode the video frame of the first video to improve decoding efficiency.

[0049] In a second aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory being used to store instructions executed by one or more processors of the electronic device, and the processor being one of the one or more processors of the electronic device, and being used to execute the video processing method mentioned in the present application.

[0050] In a third aspect, the present application provides a readable storage medium, wherein the readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the video processing method mentioned in the present application.

[0051] In a fourth aspect, the present application provides a computer program product, comprising: computer instructions, which, when executed on an electronic device, enable the electronic device to execute the video processing method mentioned in the present application.

[0052] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect mentioned above and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 According to some embodiments of the present application, a schematic diagram of a scenario for quickly detecting and identifying video content is shown;

[0054] Figure 2A According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown;

[0055] Figure 2B According to some embodiments of the present application, a schematic diagram of the structure of a hardware decoder is shown;

[0056] Figure 2C According to some embodiments of the present application, a schematic diagram of the structure of a software decoder is shown;

[0057] Figure 3 According to some embodiments of the present application, a schematic diagram of a scenario of generating a video preview thumbnail when loading a file list is shown;

[0058] Figure 4 According to some embodiments of the present application, a schematic diagram of a flow chart of an electronic device executing a video processing method is shown;

[0059] Figure 5A According to some embodiments of the present application, a framework diagram of a video processing system is shown;

[0060] Figure 5B According to some embodiments of the present application, a schematic diagram of a specific flow chart of each functional module executing a video processing method is shown;

[0061] Figure 6 According to some embodiments of the present application, a schematic diagram of a flow chart of the interaction between a software decoder decoding module and a software decoder is shown;

[0062] Figure 7 According to some embodiments of the present application, a schematic diagram of a flow chart of the interaction between a hardware decoder decoding module and a hardware decoder is shown. DETAILED DESCRIPTION

[0063] The illustrative embodiments of the present application include, but are not limited to, a video processing method, an apparatus, a storage medium, and a program product.

[0064] First, the application background of the video processing method provided in the embodiment of the present application is briefly described.

[0065] It is understood that the electronic device to which the embodiments of the present application are applicable may be any electronic device that can process video data, such as a mobile phone, a computer, a tablet computer, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, etc. The present application does not limit this.

[0066] The following uses a mobile phone as an example to exemplarily illustrate a hardware structure diagram of an electronic device of an embodiment of the present application.

[0067] For example, Figure 2A FIG. 1 shows a schematic diagram of the hardware structure of the mobile phone 10. Figure 2A As shown, the mobile phone 10 may include a processor 110, an external memory interface 120, an internal memory 121, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a screen 194, and a hardware decoder 200, etc.

[0068] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 10. In other embodiments of the present application, the mobile phone 10 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 110 may include one or more processing units, for example, the processor 110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a controller, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU) and a graphics processing unit (GPU). Among them, different processing units can be independent devices or integrated into one or more processors. In the present application, the processor 110 can be used to execute the video processing method mentioned in the present application.

[0070] In some embodiments, the processor 110 can execute instructions related to the software decoder to implement data processing of audio files or video files. For example, the mobile phone 10 can execute the relevant code of the FFMPEG software decoder in the fast forward moving picture experts group (FFMPEG) open source platform through the CPU to implement video decoding.

[0071] exist Figure 2A In the embodiment, the hardware decoder 200 is a hardware device or chip for decoding audio or video data. For example, if the operating system of the mobile phone 10 is Android TM system, the hardware decoder 200 may be a media codec (MediaCodec), wherein the MediaCodec hardware decoder may be an advanced video coding (AVC) hardware decoder supporting the H.264 encoding format, etc. In addition, when the mobile phone 10 uses the hardware decoder 200 to decode the video, corresponding resource configuration is required, for example, allocating memory and processor resources to the hardware decoder 200, or selecting a suitable hardware decoder (such as an H.264 decoder, an H.265 decoder, etc.); or loading a hardware driver, etc.

[0072] It should be understood that both the software decoder and the hardware decoder 200 may include a functional module for extracting video frames and a functional module for decoding the extracted video frames. Figure 2B As shown, the hardware decoder 200 includes a video frame extractor 201 and a video frame decoder 202. In addition, the video frame extractor 201 is responsible for extracting video frames from a video file and passing the extracted video frames to the video frame decoder 202 for decoding and processing.

[0073] For example, Figure 2C As shown, the software decoder 300 includes a video frame extractor 301 and a video frame decoder 302. In addition, the video frame extractor 301 is responsible for extracting video frames from the video file and passing the extracted video frames to the video frame decoder 302 for decoding and processing. It should be understood that the extraction and decoding of video frames can be implemented by the software decoder 300, or can also be implemented by the hardware decoder 200.

[0074] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data created during the use of the mobile phone 10 (such as audio data, phone book, etc.). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 10 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor 110.

[0075] exist Figure 2A In the embodiment, the screen 194 can be used to display video images, for example, playing the above Figure 1 The audio module 170 can be used to play the audio in the video. The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the mobile phone 10. The wireless communication module 160 can provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (blue tooth, BT) and the like applied on the mobile phone 10.

[0076] As mentioned above, when detecting and identifying video content, in order to avoid the problem of slow efficiency in detecting the video content frame by frame from the first frame to the last frame, electronic devices usually divide the video into multiple short videos and extract the first frame image of each short video for content detection, thereby saving detection time.

[0077] Currently, electronic devices usually use a video frame extractor in a hardware decoder (such as a MediaCodec hardware decoder) to extract the first frame image of a video, and pass the extracted first frame image of the video to a video frame decoder in the hardware decoder, which decodes the extracted first frame image.

[0078] However, when using a hardware decoder to extract and decode the first frame of a video, the electronic device needs to configure hardware resources or perform operations such as initializing the hardware decoder, such as loading a hardware driver, and then extracting and decoding the first frame of the video through the hardware decoder. As a result, it takes a long time for the electronic device to detect the content of the first frame of the video.

[0079] In order to solve the above problems, the present application provides a video processing method. When extracting video frames, the decoder for extracting video frames can be determined by the extraction support of the video frame extractor for the video file. For example, if it is determined that the video frame extractor 301 in the software decoder 300 has a high extraction support for the video file, the video frame is extracted by the software decoder 300; if it is determined that the video frame extractor 201 in the hardware decoder 200 has a high extraction support for the video file, the video frame is extracted by the hardware decoder 200. After the video frame is extracted, the decoder for decoding the video frame can be determined by the decoding support of the video codec format information of the video frame decoder. For example, if the video frame decoder 302 in the software decoder 300 has a high decoding support for the codec format information of the video, the extracted video frame is decoded by the software decoder 300; if the video frame decoder 202 in the hardware decoder 200 has a high decoding support for the codec format information of the video, the extracted video frame is decoded by the hardware decoder 200.

[0080] Thus, by combining the software decoder and the hardware decoder, in the video frame extraction and decoding stages, a more suitable video frame extractor can be used to extract the video frame, and a suitable video frame decoder can be used to decode the extracted video frame. Thus, the video processing time can be shortened and the video processing efficiency can be improved.

[0081] The extraction support represents the ability of the video frame extractor to extract video frames from the video to be processed, such as accuracy, efficiency, etc. At the same time, the extraction support can be determined based on the matching degree between the processing parameters of the video frame extractor (such as video frame rate, resolution, etc.) and the video parameters of the video to be processed (such as video frame rate, resolution). Figure 2B For example, if the video frame rate supported by the video frame extractor 201 is 30 to 60 frames per second, if the video frame rate of video A is 40 frames per second, then the video frame extractor 201 has high support for the extraction of video A; if the video frame rate of video A is 25 frames per second, then the video frame extractor 201 has low support for the extraction of video A. The video parameters are stored in the header information of the video file, and the processor can obtain the video parameters by reading the header information in the video file; the processor can also interact with the hardware decoder 200 or the software decoder 300 to obtain the current status information of the hardware decoder 200 or the software decoder 300 (such as the processing parameters such as the video frame rate and resolution that can be processed).

[0082] In addition, the above decoding support indicates the ability of the video frame decoder to decode the extracted video frames, such as accuracy, efficiency, etc. At the same time, the decoding support can be determined based on the matching degree between the processing parameters of the video frame decoder (such as video resolution, video frame rate, bit rate, encoding parameters, encoding format, etc.) and the video parameters of the video to be processed (such as encoding parameters, encoding format, etc. encoding format information). Figure 2B For example, if the encoding formats supported by the video frame decoder 202 are H.264 and H.265, if the video encoding format of video A is H.264, then the video frame decoder 202 has high decoding support for video A; if the video encoding format of video A is H.263, then the video frame decoder 202 has low decoding support for video A. The video parameters are stored in the header information of the video file, and the processor can obtain the video parameters by reading the header information in the video file; the processor can also interact with the hardware decoder 200 or the software decoder 300 to obtain the current status information of the hardware decoder 200 or the software decoder 300 (such as the encoding parameters that can be processed, the encoding format and other processing parameters).

[0083] In other embodiments of the present application, the electronic device may also select only a video frame extractor with high support for video file extraction, extract video frames through the video frame extractor, and then use a preset or system default video frame decoder (such as a video frame decoder in a hardware decoder) to decode the extracted video frames. Alternatively, the electronic device may also use a preset or system default video frame extractor (such as a video frame extractor in a hardware decoder) to extract video frames, and then select a video frame decoder with high support for decoding video codec format information, and use the video frame decoder to decode the extracted video frames.

[0084] In addition, in other embodiments of the present application, the electronic device may also select a video frame extractor with a high support for video file extraction, and extract video frames through the video frame extractor. At the same time, the electronic device selects the decoder where the video frame extractor is located to decode the extracted video frames. For example, if the electronic device determines that the video frame extractor 301 in the software decoder 300 has a high support for video file extraction, the electronic device may use the software decoder 300 to extract the video frames in the current video file, and use the software decoder 300 to decode the extracted video frames.

[0085] In the embodiments of the present application, the video may be any type of video, for example, any type of video such as a digital rights management (DRM) video, a high dynamic range (HDR) video, a standard dynamic range (SDR) video, etc. processed by encryption technology. It should be understood that the video processing method mentioned in the present application may be applied to any type of video, and the present application does not limit this.

[0086] In some embodiments, the resolution of HDR video is usually significantly higher than the resolution of other videos. However, when the video type is HDR video, if the video frame extractor 301 in the software decoder 300 and the video frame extractor 201 in the hardware decoder 200 both support the video format of the current HDR video, the video frame extractor 301 in the software decoder 300 supports the extraction of the current HDR video file, which is higher than the video frame extractor 201 in the hardware decoder 200 supports the extraction of the current HDR video file. For example, if the resolution of the HDR video is 8K, and the video frame extractor 301 in the software decoder 300 and the video frame extractor 201 in the hardware decoder 200 both support processing videos with 8K resolution, the video frame extractor 301 in the software decoder 300 supports the extraction of the HDR video more. That is, in the embodiment of the present application, if the video format of the HDR video is a video format supported by the software decoder 300, the video frame is extracted by the software decoder 300; if the software decoder 300 does not support the format of the current HDR video, the video frame can be extracted by the hardware decoder 200.

[0087] In some embodiments, when extracting video frames of the current video, a video frame extractor capability table may also be established based on the extraction support of each video frame extractor for the current video file. When extracting video frames, the electronic device may first query the video frame extractor capability table to determine the best decoder for extracting video frames of the current video file. For example, for the current video file, if the video frame extractor 301 in the software decoder 300 has the highest extraction support for the video file, the video frame may be extracted by the software decoder 300; if the video frame extractor 201 in the hardware decoder 200 has the highest extraction support for the video file, the video frame may be extracted by the hardware decoder 200. In this way, using the video frame extractor in the matching decoder to extract video frames can save the time of extracting video frames.

[0088] Among them, the video frame extractor capability table is determined based on the extraction support of each video frame extractor for the video file format. In addition, the extraction support can be determined based on the matching degree between the processing parameters (such as video frame rate, resolution, etc.) of each video frame extractor and the video parameters (such as video frame rate, resolution) of the video to be processed. For example, if the video frame rate supported by the video frame extractor 301 in the software decoder 300 is 30 frames per second to 60 frames per second, and the supported video resolution is 720P and 1080P; and the video frame rate supported by the video frame extractor 201 in the hardware decoder 200 is 45 frames per second to 60 frames per second, and the supported video resolution is 1080P and 2K. If the video frame rate of video A is 40 frames per second and the video resolution is 1080P, a video frame extractor capability table can be established: the extraction support of the software decoder 300 for video A is 100%, and the extraction support of the hardware decoder 200 for video A is 50%. Therefore, the electronic device can query the video frame extractor capability table to determine that the software decoder 300 has the highest support for extracting video A, that is, determine that the best decoder for extracting the video frame of video A is the software decoder 300 .

[0089] In some embodiments, when decoding the extracted video frames, a video frame decoder capability table may also be established based on the decoding support of each video frame decoder for the current video codec format information. After extracting the video frames, the electronic device may obtain the codec format information of the current video, and then search the video frame decoding capability table for a decoder based on the codec format information of the video. For example, for the current video file, if the video frame decoder 302 in the software decoder 300 has the highest decoding support for the video coding format information, the video frame may be decoded by the software decoder 300; if the video frame decoder 202 in the hardware decoder 200 has the highest decoding support for the video coding format information, the video frame may be decoded by the hardware decoder 200. In this way, decoding the extracted video frames using the video frame decoder in the matching decoder may save the decoding time of the video frames.

[0090] Among them, the video frame decoder capability table is determined based on the decoding support of each video frame decoder for the encoding and decoding format information of the video. In addition, the decoding support can be determined based on the matching degree between the processing parameters (such as encoding parameters, encoding formats, etc.) of each video frame decoder and the video parameters (such as encoding format information such as encoding parameters, encoding formats, etc.) of the video to be processed. For example, if the video encoding format supported by the video frame decoder 302 in the software decoder 300 is H.264 and H.265, the supported bit rate is 100kbps to 10000kbps; and the video encoding format supported by the video frame decoder 202 in the hardware decoder 200 is H.264 and H.265, and the supported bit rate is 500kbps to 15000kbps. If the video encoding format of video A is H.264 and the video bit rate is 300kbps, a video frame decoder capability table can be established: the decoding support of the software decoder 300 for video A is 100%, and the decoding support of the hardware decoder 200 for video A is 50%. Therefore, the electronic device can query the video frame decoder capability table to determine that the software decoder 300 has the highest decoding support for video A, that is, determine that the best decoder for decoding the video frame of video A is the software decoder 300 .

[0091] In some embodiments, the video processing method mentioned in this application can be applied to the above Figure 1 In the scenario of rapid detection of video content mentioned in . Specifically, when the electronic device extracts the first frame image of each video segment, it first determines the decoder for extracting the first frame of the video based on the extraction support of the video frame extractor 201 and the video frame extractor 301 for the video file. If it is determined that the adapted decoder is the software decoder 300, the electronic device extracts the first frame image of each video segment through the software decoder 300. Next, the electronic device determines the decoder for decoding the first frame of the video based on the decoding support of the video codec format information by the video frame decoder 202 and the video frame decoder 302. If it is determined that the adapted decoder is the software decoder 300, the electronic device decodes the extracted first frames of each video through the software decoder 300. Finally, the electronic device detects whether the content in the first frame of each video includes the target content. In this way, rapid detection and identification of video content can be completed.

[0092] In some embodiments, when the software decoder 300 is used to extract the first frame of a video and the software decoder 300 is used to decode the extracted first frame, the time can be shortened and the efficiency can be improved. For example, taking the extraction of the first frames of images in 5 videos and decoding of each first frame as an example, a simple comparison is made between the processing efficiency of the hardware decoder 200 and the software decoder 300. Among them, the duration of the 5 videos is 10S, the video resolution is 1080P, the video frame rate is 30 frames per second, and the video encoding format is AVC format (i.e., H.264 encoding format).

[0093] In some embodiments, the electronic device can extract and decode the first frame images of the five videos respectively through the AVC hardware decoder and the FFMPEG software decoder. In addition, the average duration of extracting and decoding the first frame images of each video through the AVC hardware decoder is greater than the average duration of extracting and decoding the first frame images of each video through the FFMPEG software decoder. For details, please refer to the decoding duration shown in the following Table 1:

[0094] Table 1:

[0095] AVC hardware decoder FFMPEG (8 threads) FFMPEG (single thread) Video 1 698 381 210 Video 2 571 382 178 Video 3 598 401 213 Video 4 607 395 177 Video 5 563 388 179 average value 607.4 389.4 191.4

[0096] It should be understood that the duration of extracting and decoding the first frame of the video includes the startup time of the decoder.

[0097] In Table 1, the average duration for the electronic device to extract and decode the first frame images of the five videos respectively through the AVC hardware decoder is 607.4S, which is longer than the average duration for extracting and decoding the first frame images of each video through the FFMPEG software decoder.

[0098] Among them, 8 threads means that the FFMPEG software decoder creates 8 task processes at the same time. For example, when the FFMPEG software decoder extracts the first frame of video 1, it can also execute the processing instructions of other 7 video files at the same time; single thread means that the FFMPEG software decoder only executes 1 task process at the same time. In Table 1, when the FFMPEG software decoder executes the 8-thread task process or the single-thread task process, the average processing speed of extracting and decoding the first frame of 5 videos is faster than the average processing speed of the AVC hardware decoder extracting and decoding the first frame of 5 videos.

[0099] Therefore, through the video processing method mentioned in this application, when extracting and decoding the first frame image of each of the above videos, the electronic device can determine that the decoders adapted for extracting and decoding the first frame of the video are all software decoders based on the video frame extractor capability table and the video frame decoder capability table, and then the electronic device extracts and decodes the first frame image of each video through the software decoder. In this way, the time for extracting and decoding the first frame of each video can be shortened, and the efficiency of extracting and decoding the first frame of each video can be improved.

[0100] In addition, the video processing method mentioned in the present application can also be applied to the video cover and the scene of loading the video file list to generate the video preview thumbnail. For example, when promoting a movie, the iconic image frame in the movie can be selected as the video cover by the method in the present application to attract the user's attention. Specifically, when the electronic device extracts the iconic image frame in the movie, it first determines the decoder for extracting the iconic image frame based on the extraction support of the video frame extractor 201 and the video frame extractor 301 for the movie video file. For example, if it is determined that the adapted decoder is the software decoder 300, the electronic device can extract the iconic image frame therein through the software decoder 300. Next, the electronic device determines the decoder for decoding the iconic image frame based on the decoding support of the encoding and decoding format information of the movie video by the video frame decoder 202 and the video frame decoder 302. For example, if it is determined that the adapted decoder is the software decoder 300, the electronic device can decode the extracted iconic image frame through the software decoder 300. Finally, the electronic device displays the decoded iconic image frame on the screen as the video cover of the movie.

[0101] For another example, when loading a video file list, a preview thumbnail of each video can be generated by the method mentioned in this application to shorten the loading time of the file list. Figure 3 As shown in (a) of FIG. 1 , taking the electronic device as PC 30 as an example, the interface 310 of PC 30 shows that when PC 30 detects that the user opens the video folder 320, PC 30 can generate a preview thumbnail for the user to view using the method of the present application. Figure 3 As shown in (b) in the interface 330, for each video file in the video folder 320, the PC 30 generates a preview thumbnail of each video file for the user to view.

[0102] The process of generating preview thumbnails of each video file can be as follows: when processing each video file, PC30 first determines the decoder for extracting video frames based on the extraction support of the video frame extractor 201 and the video frame extractor 301 for the video file. For example, if it is determined that the adapted decoder is the software decoder 300, PC30 can extract the video frame that can be used as the preview thumbnail through the software decoder 300, such as the first frame, or extract the frame where the target with the most appearances is located by identifying each video frame. Next, PC30 determines the decoder for decoding the video frame based on the decoding support of the video codec format information by the video frame decoder. For example, if it is determined that the adapted decoder is the software decoder 300, PC30 can decode the extracted video frame through the software decoder 300. Finally, PC30 displays the decoded video frame on the screen as the preview thumbnail of the video file.

[0103] It should be understood that, referring to Table 1 above, when the software decoder 300 creates multiple task processes (e.g., 8 threads) at the same time, that is, the average speed at which the software decoder 300 generates preview thumbnails for multiple video files at the same time is faster than the speed at which the hardware decoder 200 generates preview thumbnails for a single video file. Therefore, through the video processing method mentioned in this application, the time for extracting and decoding the first frame of the video of the electronic device can be shortened, and the loading speed of the video file list can be increased.

[0104] The following is based on Figure 4 The flowchart shown in FIG. 1 briefly describes the process of the video processing method mentioned in the present application. The method can be applied to electronic devices, such as the above Figure 3 The PC30 or Figure 2A The mobile phone 10 shown. Figure 4 Specifically, the method comprises the following steps:

[0105] S401: Obtain a first video to be processed.

[0106] In an embodiment of the present application, the electronic device receives an instruction to extract video frames from a first video, for example, receives an instruction to quickly detect the content of the first video, generate a cover for the first video, generate a preview thumbnail of the first video, or detects that a user has opened a video folder. The electronic device first needs to obtain the video file of the first video to be processed.

[0107] S402: Determine a decoding method for the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder.

[0108] In the embodiment of the present application, the electronic device can determine the decoder for extracting the video frame based on the matching degree between the processing parameters (such as video frame rate, resolution, etc.) of each video frame extractor and the video parameters (such as video frame rate, resolution) of the video to be processed. If the matching degree between the processing parameters and the video parameters is high, it means that the video frame extractor has a high support for extracting the first video, and the video frame extractor is used to extract the video frame.

[0109] In the embodiment of the present application, the electronic device can also determine the decoder for decoding the extracted video frame based on the matching degree between the processing parameters (such as encoding parameters, encoding format, etc.) of each video frame decoder and the video parameters (such as encoding format information such as encoding parameters, encoding format, etc.) of the video to be processed. If the matching degree between the processing parameters and the video parameters is high, it means that the video frame decoder has a high decoding support for the first video, and the video frame decoder is used to decode the extracted video frame.

[0110] In the embodiment of the present application, when extracting video frames, when the video type of the first video is HDR video, if the software decoder 300 supports processing HDR video, the video frames of the HDR video are extracted through the software decoder 300. When the video type of the first video is DRM video and the electronic device supports processing DRM video, or when the video type of the first video is other video (e.g., SDR video), the electronic device can obtain the extraction support of the video frame extractor 201 and the video frame extractor 301 for the current video, and select a video frame extractor with high extraction support to extract the video frame.

[0111] In addition, in an embodiment of the present application, for any type of video (such as DRM video, HDR video, SDR video, etc.), when the electronic device decodes the extracted video, the electronic device can obtain the decoding support of the video frame decoder 202 and the video frame decoder 302 for the current video, and select a video frame decoder with high decoding support to decode the extracted video frame.

[0112] In this way, the video frame extractor in the matching decoder is used to extract the video frame in the first video, and the video frame decoder in the matching decoder is used to decode the extracted video frame, which can save the time of extracting and decoding the video frame.

[0113] Through the above method, a matching video frame extractor and video frame decoder can be used to extract and decode the video frames of the first video, thereby shortening the video processing time and improving the efficiency of video processing by combining the use of software decoders and hardware decoders.

[0114] In some embodiments, a video processing system based on the video processing method mentioned in the present application can be divided into several functional modules, and the video processing method mentioned in the present application is implemented through the interaction between the modules.

[0115] like Figure 5A As shown, a schematic diagram of the framework structure of a video processing system 400 is shown. The video processing system 400 can be divided into several functional modules, and the functional modules can interact with each other. Specifically, the video processing system 400 includes a video source acquisition module 410, a decision module 420, a software decoder extraction module 430, a hardware decoder extraction module 440, a software decoder decoding module 450, and a hardware decoder decoding module 460.

[0116] Among them, the video source acquisition module 410 can be used to acquire a video file and send the video file to the decision module 420, and the decision module 420 determines the decoder for processing the video file. The decision module 420 is used to determine the decoder for extracting the video frame and the decoder for decoding the video frame. The software decoder extraction module 430 is used to interact with the software decoder 300 and extract the video frame through the software decoder 300. The hardware decoder extraction module 440 is used to interact with the hardware decoder 200 and extract the video frame through the hardware decoder 200. The software decoder decoding module 450 is used to interact with the software decoder 300 and decode the extracted video frame through the software decoder 300. The hardware decoder decoding module 460 is used to interact with the hardware decoder 200 and decode the extracted video frame through the hardware decoder 200.

[0117] It should be understood that the video processing system 400 may also be split or combined into more or fewer modules, and this embodiment of the present application does not limit this.

[0118] The following takes the hardware decoder 200 as a MediaCodec hardware decoder, the software decoder 300 as an FFMPEG software decoder as an example, and takes the video file types including DRM video, HDR video, etc. as an example, combined with the above Figure 5A The algorithm framework diagram mentioned in Figure 5B The flowchart specifically introduces the internal specific execution process of each functional module mentioned in this application when executing the video processing method. Specifically, Figure 5B The method shown includes the following steps:

[0119] 501: The video source acquisition module 410 acquires a video file.

[0120] In the embodiment of the present application, the video source acquisition module 410 can acquire the video file based on the video file path input by the user. Alternatively, the user can search for the video in an application such as a website or a video player through an electronic device, and the video source acquisition module 410 acquires the video file from a database in an application such as a website or a video player based on operations such as user triggering detection of video content. The present application does not limit the source of the video file.

[0121] 502: The decision module 420 determines whether the video is a DRM video. If yes, the process proceeds to step 503; if no, the process proceeds to step 504.

[0122] In the embodiment of the present application, the video may be a non-encrypted ordinary video, or a DRM video (ie, a video processed by encryption technology).

[0123] In some embodiments, the decision module 420 can receive the video file obtained by the video source acquisition module 410, and determine whether the video file is an encrypted DRM video. If the video is a DRM video, go to step 503 to determine whether the Android open source project (AOSP) supports the DRM video, that is, determine whether the Android TM Whether the device can access the content in the DRM video; if the video is not a DRM video, go to step 504 to determine whether the video type is an HDR video.

[0124] 503: The decision module 420 determines whether the DRM video is supported by AOSP. If yes, the process proceeds to step 507; if no, the process proceeds to step 518.

[0125] It should be understood that AOSP is Android TM The basic code framework in the operating system, for Android TM The device provides basic functions and features. However, if the electronic device where AOSP is located wants to have the function of processing DRM video, it is also necessary to add a DRM code framework to AOSP, that is, integrate and be compatible with the corresponding DRM function in AOSP. Among them, DRM is a technology used to protect digital content. It can encrypt video files and limit electronic devices from accessing video content.

[0126] In an embodiment of the present application, when the decision module 420 determines that the current video is a DRM video through the above step 502, the decision module 420 also needs to determine whether AOSP supports DRM video. If AOSP does not support DRM video, it means that the DRM video cannot be accessed in the electronic device where the AOSP is located, that is, the electronic device cannot extract and decode the video frames in the DRM video, then go to step 518, and the electronic device (such as a processor) can end the execution of the method flow. If AOSP supports DRM video, for example, the decision module 420 detects that there is a DRM code framework in AOSP, then go to step 507, query the video frame extractor capability table, and obtain the best matching video frame extractor.

[0127] It should be understood that when an electronic device decodes a DRM video, it means that the encrypted DRM video is decrypted.

[0128] 504: The decision module 420 determines whether the video is an HDR video. If yes, the process proceeds to step 505; if no, the process proceeds to step 507.

[0129] In the embodiment of the present application, when the decision module 420 determines that the current video is a non-encrypted ordinary video through the above step 502, it is also necessary to determine whether the current video type is an HDR video. If it is an HDR video, go to step 505 to determine whether the FFMPEG software decoder supports HDR video; if it is not an HDR video, go to step 507 to query the video frame extractor capability table to obtain the best matching video frame extractor. Among them, HDR is a processing technology that improves the brightness and contrast of the video. Compared with ordinary videos, HDR videos can provide more image details, richer colors, and more vivid pictures that are close to what the human eye sees.

[0130] In some embodiments, the electronic device may first execute step 502, determine that the current video type is not a DRM video, and then execute step 504 to determine whether the current video is an HDR video. Alternatively, the electronic device may first execute step 504 to determine whether the current video type is an HDR video. If it is determined that the current video type is not an HDR video, then execute step 502 to determine whether the current video type is a DRM video. If it is determined that the current video type is not a DRM video through step 502, then directly execute step 507 to query the video frame extractor capability table and select the best matching video frame extractor. In other words, this application does not limit the execution order of step 502 and step 504.

[0131] 505: The decision module 420 determines whether the HDR video is supported by FFMPEG. If yes, the process proceeds to step 510; if no, the process proceeds to step 506.

[0132] In the embodiment of the present application, when the decision module 420 determines that the current video is an HDR video through the above step 504, it is also necessary to determine whether the FFMPEG software decoder supports processing HDR video. If it is supported, go to step 510 to create an FFMPEG extractor; if not, go to step 506 to determine whether HDR video is supported by AOSP.

[0133] In some embodiments, when the video type is HDR video, if the video frame extractor in the FFMPEG software decoder and the video frame extractor in the MediaCodec hardware decoder both support the video format of the current HDR video, the video frame extractor in the FFMPEG software decoder will have a higher support for the extraction of the current HDR video file than the video frame extractor in the MediaCodec hardware decoder. For example, if the resolution of the HDR video is 8K, and the video frame extractor in the FFMPEG software decoder and the video frame extractor in the MediaCodec hardware decoder both support processing videos with 8K resolution, the video frame extractor in the FFMPEG software decoder will have a higher support for the extraction of the HDR video.

[0134] In some embodiments, if the video format of the HDR video is a video format supported by the FFMPEG software decoder, the video frame can be extracted by the FFMPEG software decoder. If the FFMPEG software decoder does not support the format of the current HDR video, the video frame can be extracted by the MediaCodec hardware decoder.

[0135] 506: The decision module 420 determines whether HDR video is supported by AOSP. If yes, the process proceeds to step 511; if no, the process proceeds to step 518.

[0136] In the embodiment of the present application, when the decision module 420 determines that the current video is an HDR video through the above step 504, and determines through the above step 505 that the FFMPEG software decoder does not support the current HDR video file, the decision module 420 also needs to determine whether AOSP supports HDR video. If so, go to step 511 to create a MediaCodec extractor; if not, go to step 518, and the electronic device (such as a processor) can end the execution of the method flow.

[0137] In some embodiments, the decision is made through the above step 505 to determine that FFMPEG does not support the current HDR video file, and the video frame can be extracted through the MediaCodec hardware decoder. Before extracting the video frame through the MediaCodec hardware decoder, it is also necessary to determine whether the MediaCodec hardware decoder supports processing the current HDR video.

[0138] It should be understood that MediaCodec is Android TM The hardware decoder provided by the system, and AOSP is Android TM The basic code framework in the system, when AOSP supports HDR video, it means that the MediaCodec hardware decoder also supports the current HDR video. Among them, the decision module 420 can determine whether AOSP supports the current HDR video file by checking the AOSP document, detecting the AOSP code library, and checking the hardware specifications supported by AOSP.

[0139] In some embodiments, if AOSP supports the current HDR video file, that is, the MediaCodec hardware decoder supports the current HDR video file, the video frame can be extracted by the video frame extractor in the MediaCodec hardware decoder. If AOSP does not support the current HDR video file, that is, the MediaCodec hardware decoder does not support the current HDR video file, and the decision module 420 also determines through the above step 505 that the FFMPEG software decoder does not support the HDR video file at that time, then go to step 518, and the electronic device (such as a processor) can end the execution of the method flow.

[0140] 507: The decision module 420 queries the video frame extractor capability table to obtain the best matching video frame extractor.

[0141] In an embodiment of the present application, the decision module 420 determines through the above steps 502 and 503 that the current video is a DRM video supported by AOSP; or the decision module 420 determines through the above steps 502 and 504 that the current video is a non-DRM video and a non-HDR video, then the video frame extractor capability table can be queried to select the video frame extractor with the highest support for the current video file extraction, that is, the best matching video frame extractor is selected.

[0142] In some embodiments, the video frame extractor capability table is determined based on the extraction support of each video frame extractor for the video file format. In addition, the extraction support can be determined based on the matching degree between the processing parameters (such as video frame rate, resolution, etc.) of each video frame extractor and the video parameters (such as video frame rate, resolution) of the video to be processed. For example, if the video frame rate supported by the video frame extractor in the FFMPEG software decoder is 30 frames per second to 60 frames per second, and the supported video resolution is 720P and 1080P; and the video frame rate supported by the video frame extractor in the MediaCodec hardware decoder is 45 frames per second to 60 frames per second, and the supported video resolution is 1080P and 2K. If the video frame rate of video A is 40 frames per second and the video resolution is 1080P, a video frame extractor capability table can be established: the extraction support of the FFMPEG software decoder for video A is 100%, and the extraction support of the MediaCodec hardware decoder for video A is 50%.

[0143] 508: The decision module 420 determines whether the video frame extractor in the software decoder is optimal. If so, the process proceeds to step 510; if not, the process proceeds to step 509.

[0144] In the embodiment of the present application, after the decision module 420 obtains the best matching video frame extractor through the above step 507, if it is determined that the best matching video frame extractor is the video frame extractor in the FFMPEG software decoder, it goes to step 510 to create the FFMPEG extractor. If not, it goes to step 509 to determine whether the video frame extractor in the MediaCodec hardware decoder is the best.

[0145] 509: The decision module 420 determines whether the video frame extractor in the hardware decoder is optimal. If so, the process proceeds to step 511; if not, the process proceeds to step 518.

[0146] In an embodiment of the present application, when the decision module 420 determines through the above step 508 that the video frame extractor in the FFMPEG software decoder is not the best video frame extractor, it is also necessary to determine whether the video frame extractor in the MediaCodec hardware decoder is the best video frame extractor. If so, go to step 511, create a MediaCodec extractor, and extract video frames through the video frame extractor in the MediaCodec hardware decoder; if not, it means that both the FFMPEG software decoder and the MediaCodec hardware decoder do not support the extraction of video frames of the current video file, then go to step 518, and the electronic device (such as a processor) can end the execution of the method flow.

[0147] In some embodiments, the electronic device may first perform step 508 and then perform step 509. Alternatively, the electronic device may also first perform step 509, and if it is determined that the video frame extractor in the MediaCodec hardware decoder is not the best choice, the electronic device may perform step 508 to determine whether the video frame extractor in the FFMPEG software decoder is the best video frame extractor. In other words, the present application does not limit the execution order of step 508 and step 509.

[0148] In some other embodiments of the present application, the above-mentioned steps 508 and 509 can also be combined into one step. For example, when the electronic device executes step 508, if it is determined that the video frame extractor in the FFMPEG software decoder is the best video frame extractor, it goes to step 510 to create the FFMPEG extractor; if it is determined that the video frame extractor in the FFMPEG software decoder is not the best video frame extractor, it goes directly to step 511 to create the MediaCodec extractor. Alternatively, the electronic device may not execute step 508 and directly execute step 509, that is, if it is determined that the video frame extractor in the MediaCodec hardware decoder is the best video frame extractor, it goes to step 511 to create the MediaCodec extractor; if it is determined that the video frame extractor in the MediaCodec hardware decoder is not the best video frame extractor, it goes directly to step 510 to create the FFMPEG extractor.

[0149] 510: The software decoder extraction module 430 creates a FFMPEG extractor.

[0150] In an embodiment of the present application, when the decision module 420 determines through the above step 505 that the FFMPEG software decoder supports the current HDR video, or the decision module 420 determines through the above step 508 that the video frame extractor in the FFMPEG software decoder is the optimal video frame extractor, the decision module 420 can pass the video file to the software decoder extraction module 430, and the software decoder extraction module 430 creates the FFMPEG extractor, that is, the software decoder extraction module 430 interacts with the FFMPEG software decoder and extracts video frames through the FFMPEG software decoder.

[0151] 511: The hardware decoder extraction module 440 creates a MediaCodec extractor.

[0152] In an embodiment of the present application, when the decision module 420 determines through the above step 506 that AOSP supports the current HDR video, or the decision module 420 determines through the above step 509 that the video frame extractor in the MediaCodec hardware decoder is the best video frame extractor, the decision module 420 can pass the video file to the hardware decoder extraction module 440, and the hardware decoder extraction module 440 creates a MediaCodec extractor, that is, the hardware decoder extraction module 440 interacts with the MediaCodec hardware decoder and extracts video frames through the MediaCodec hardware decoder.

[0153] 512: The decision module 420 obtains video encoding format information.

[0154] In the embodiment of the present application, after extracting the video frame through the above step 510 or step 511, it is also necessary to select the best video frame decoder based on the video encoding format information of the current video to decode the extracted video frame. Among them, the video frame encoding format information may include: video compression encoding format (H.264, H.265, etc.), video resolution (1080P, 480P, etc.), video bit rate (500kbps, 600kbps, etc.) and other video encoding format information.

[0155] 513: The decision module 420 queries the video frame decoder capability table to obtain the best matching video frame decoder.

[0156] In an embodiment of the present application, after the decision module 420 obtains the video frame encoding format information through the above step 512, it can query the corresponding video frame decoder capability table to select the video frame decoder with the highest support for decoding the current video file, that is, select the best matching video frame decoder.

[0157] In some embodiments, the video frame decoder capability table is determined based on the decoding support of each video frame decoder for the encoding and decoding format information of the video. In addition, the decoding support can be determined based on the matching degree between the processing parameters of each video frame decoder (such as encoding parameters, encoding format, etc.) and the video parameters of the video to be processed (such as encoding format information such as encoding parameters and encoding format). For example, if the video encoding format supported by the video frame decoder in the FFMPEG software decoder is H.264 and H.265, the supported bit rate is 100kbps to 10000kbps; and the video encoding format supported by the video frame decoder in the MediaCodec hardware decoder is H.264 and H.265, and the supported bit rate is 500kbps to 15000kbps. If the video encoding format of video A is H.264 and the video bit rate is 300kbps, a video frame decoder capability table can be established: the decoding support of the FFMPEG software decoder for video A is 100%, and the decoding support of the MediaCodec hardware decoder for video A is 50%.

[0158] 514: The decision module 420 determines whether the video frame decoder in the software decoder is optimal. If so, the process proceeds to step 516; if not, the process proceeds to step 515.

[0159] In the embodiment of the present application, after the decision module 420 obtains the best matching video frame decoder through the above step 513, if it is determined that the best matching video frame decoder is the video frame decoder in the FFMPEG software decoder, it goes to step 516 to create the FFMPEG decoder. If not, it goes to step 515 to determine whether the video frame decoder in the MediaCodec hardware decoder is the best.

[0160] 515: The decision module 420 determines whether the video frame decoder in the hardware decoder is optimal. If so, the process proceeds to step 517; if not, the process proceeds to step 518.

[0161] In an embodiment of the present application, when the decision module 420 determines through the above step 514 that the video frame decoder in the FFMPEG software decoder is not the best video frame decoder, it is also necessary to determine whether the video frame decoder in the MediaCodec hardware decoder is the best video frame decoder. If so, go to step 517, create a MediaCodec decoder, and decode the extracted video frame through the video frame decoder in the MediaCodec hardware decoder; if not, it means that neither the FFMPEG software decoder nor the MediaCodec hardware decoder supports decoding the current video file, then go to step 518, and the electronic device (such as a processor) can end the execution of the method flow.

[0162] In some embodiments, the electronic device may first execute step 514 and then execute step 515. Alternatively, the electronic device may also first execute step 515, and if it is determined that the video frame decoder in the MediaCodec hardware decoder is not the best choice, the electronic device may execute step 514 to determine whether the video frame decoder in the FFMPEG software decoder is the best video frame decoder. In other words, the present application does not limit the execution order of step 514 and step 515.

[0163] In some other embodiments of the present application, the above-mentioned step 514 and step 515 can also be combined into one step. For example, when the electronic device executes step 514, if it is determined that the video frame decoder in the FFMPEG software decoder is the best video frame decoder, it goes to step 516 to create an FFMPEG decoder; if it is determined that the video frame decoder in the FFMPEG software decoder is not the best video frame decoder, it goes directly to step 517 to create a MediaCodec decoder. Alternatively, the electronic device may not execute step 514, but directly execute step 515, that is, if it is determined that the video frame decoder in the MediaCodec hardware decoder is the best video frame decoder, it goes to step 517 to create a MediaCodec decoder; if it is determined that the video frame decoder in the MediaCodec hardware decoder is not the best video frame decoder, it goes directly to step 516 to create an FFMPEG decoder.

[0164] In the embodiment of the present application, the electronic device may also perform any part of steps 502 to 509 and steps 512 to 515 in the above steps. That is, the electronic device may only perform steps 502 to 509, select the best video frame extractor, extract the video frame through the video frame extractor, and then use the default video frame decoder (e.g., the video frame decoder in the hardware decoder) or the same decoder to decode the extracted video frame. Alternatively, the electronic device may also use the default video frame extractor (e.g., the video frame extractor 201 in the hardware decoder 200) to extract the video frame, and then perform steps 512 to 515, select the best video frame decoder, and use the video frame decoder to decode the extracted video frame.

[0165] 516: Software Decoder The decoding module 450 creates a FFMPEG decoder.

[0166] In an embodiment of the present application, after the decision module 420 determines through the above step 514 that the video frame decoder in the FFMPEG software decoder is the optimal video frame decoder, the decision module 420 or the video frame extractor can pass the extracted video frame to the software decoder decoding module 450, and the software decoder decoding module 450 creates the FFMPEG decoder, that is, the software decoder decoding module 450 interacts with the FFMPEG software decoder, and decodes the extracted video frame through the FFMPEG software decoder.

[0167] The process of interaction between the software decoder decoding module 450 and the FFMPEG software decoder can be referred to as follows Figure 6 It is described in , and will not be repeated here.

[0168] 517: The hardware decoder decoding module 460 creates a MediaCodec decoder.

[0169] In an embodiment of the present application, after the decision module 420 determines through the above step 515 that the video frame decoder in the MediaCodec hardware decoder is the optimal video frame decoder, the decision module 420 or the video frame extractor can pass the extracted video frame to the hardware decoder decoding module 460, and the hardware decoder decoding module 460 creates a MediaCodec decoder, that is, the hardware decoder decoding module 460 interacts with the MediaCodec hardware decoder, and decodes the extracted video frame through the MediaCodec hardware decoder.

[0170] The process of interaction between the hardware decoder decoding module 460 and the MediaCodec hardware decoder can be referred to as follows Figure 7 It is described in , and will not be repeated here.

[0171] 518: The processor ends the task.

[0172] In an embodiment of the present application, when the decision module 420 determines through the above steps 502 and 503 that the current video is a DRM video and AOSP does not support DRM video; or the decision module 420 determines through the above steps 504 and 506 that the current video is an HDR video and AOSP does not support HDR video; or the decision module 420 determines through the above steps 508 and 509 that neither the FFMPEG software decoder nor the MediaCodec hardware decoder supports extracting video frames of the current video file; or when the decision module 420 determines through the above steps 514 and 515 that neither the FFMPEG software decoder nor the MediaCodec hardware decoder supports decoding of the current video file, the processor in the electronic device can end execution of the method flow.

[0173] Thus, through the above flowchart, by combining the software decoder and the hardware decoder to extract and decode the video frame, the video processing time can be shortened and the efficiency of the video processing can be improved.

[0174] The following still takes the software decoder 300 as the FFMPEG software decoder as an example. Figure 6 The flowchart shown describes the process of interaction between the software decoder decoding module 450 and the FFMPEG software decoder mentioned above. Among them, FFMPEG is an open source audio and video processing tool, which includes a variety of FFMPEG software decoders. In this application, the code module of FFMPEG can be integrated into the code framework of the electronic device.

[0175] Specifically, Figure 6 The method shown comprises the following steps:

[0176] 601: Software decoder The decoding module 450 creates a FFMPEG software decoder.

[0177] In some embodiments, when the electronic device uses the FFMPEG software decoder, the software decoder decoding module 450 first needs to create a FFMPEG software decoder object. For example, the FFMPEG decoder object can be a structure in the FFMPEG open source library, which includes parameters of each FFMPEG software decoder (such as the output video format of each FFMPEG software decoder, etc.) and data structure.

[0178] 602: The software decoder decoding module 450 loads the operating environment.

[0179] In some embodiments, the software decoder decoding module 450 also needs to load the operating environment required by the FFMPEG software decoder, for example, loading the library file in the FFMPEG software decoder into the system directory of the electronic device, and loading the library file in the application where the current video that needs to extract and decode the video frame is located.

[0180] 603: The software decoder decoding module 450 controls the initialization of the FFMPEG software decoder.

[0181] In some embodiments, after the software decoder decoding module 450 creates the FFMPEG decoder object through the above step 601 and loads the operating environment through step 602, it is also necessary to initialize the created decoder object.

[0182] The specific initialization process may include the following steps 6031 to 6034, wherein the FFMPEG module 470 is a functional module that integrates the FFMPEG open source code in the electronic device and can execute the corresponding functions mentioned in this application.

[0183] 6031: FFMPEG module 470 selects a matching software decoder.

[0184] In some embodiments, FFMPEG is an open source audio and video processing tool, which includes a variety of FFMPEG software decoders. In the present application, the code module of FFMPEG (ie, FFMPEG module 470) can be integrated into the code framework of the electronic device.

[0185] In some embodiments, the FFMPEG module 470 needs to select a matching software decoder from a plurality of FFMPEG software decoders. For example, a matching software decoder can be selected based on the name of each FFMPEG software decoder. For example, for a video file in H.264 encoding format, a software decoder named libx264 can be selected as a matching software decoder; or, the FFMPEG document can be viewed to select a matching software decoder based on the current video format (e.g., frame rate, encoding format, etc.). This application does not limit this.

[0186] 6032: FFMPEG module 470 allocates the operating environment.

[0187] In some embodiments, after selecting a matching software decoder, the FFMPEG module 470 also needs to allocate an operating environment to the matching software decoder, for example, allocating processor resources to the matching software decoder, providing operating system support, etc.

[0188] 6033: FFMPEG module 470 allocates memory space.

[0189] In the embodiment of the present application, the FFMPEG module 470 also needs to allocate memory space to the matching software decoder, providing the matching software decoder with sufficient memory for storing data and temporary variables required by the software decoder.

[0190] 6034: FFMPEG module 470 completes the integration of the matching software decoder.

[0191] In an embodiment of the present application, after completing the above steps 6031 to 6034, the FFMPEG module 470 can complete the initialization process of the FFMPEG software decoder, that is, integrate the matching software decoder into the system of the electronic device and / or into the application where the current video is located.

[0192] 604: The software decoder decoding module 450 obtains an input buffer.

[0193] In the embodiment of the present application, the software decoder decoding module 450 can apply for temporary storage space as an input buffer for storing video frame data to wait for the above-mentioned matching software decoder to read and process.

[0194] 605: The software decoder extraction module 430 reads the video data.

[0195] In the embodiment of the present application, the software decoder extraction module 430 can extract the video frame and pass the video frame data to the software decoder decoding module 450, which is decoded by the software decoder decoding module 450. For a detailed description, please refer to the above Figure 5B As mentioned in, I will not elaborate on this.

[0196] 606: The software decoder decoding module 450 puts the data into the input buffer.

[0197] In the embodiment of the present application, the software decoder decoding module 450 can obtain the video frame data sent by the software decoder extraction module 430 and put the video frame data into the input buffer obtained in step 604.

[0198] 607: The software decoder decoding module 450 decodes.

[0199] In the embodiment of the present application, the electronic device completes the matching software decoder initialization setting through the above steps 6031 to 6034, and after putting the video frame data into the input buffer through steps 604 to 606, the electronic device can read the data in the input buffer through the matching software decoder and decode it. The specific decoding process may include the following steps 6071 to 6073.

[0200] 6071: The FFMPEG module 470 passes the data in the input buffer to the matching software decoder.

[0201] In an embodiment of the present application, the FFMPEG module 470 can read the video frame data in the input buffer and send the video frame data to a matching software decoder for decoding.

[0202] 6072: FFMPEG module 470 places the decoded data into the output buffer.

[0203] In an embodiment of the present application, after the matching software decoder decodes the video frame data, the decoded video frame data can be placed in an output buffer through the FFMPEG module 470 to wait for reading the decoded video frame data.

[0204] 6073: FFMPEG module 470 reads data in the output buffer.

[0205] In the embodiment of the present application, the FFMPEG module 470 can read the data in the output buffer and send the data to the software decoder decoding module 450 for processing. In this way, the process of decoding the extracted video frame data can be completed through steps 6071 to 6073.

[0206] 608: The software decoder decoding module 450 puts the data into the output buffer.

[0207] In the embodiment of the present application, after the software decoder decoding module 450 obtains the decoded data frame sent by the FFMPEG module 470 in the above step 6073, the decoded data frame can be sequentially placed in the buffer queue, waiting for the electronic device to read the data in the buffer queue and perform content detection, etc. It should be understood that the buffer queue is a different buffer from the output buffer in step 6072, wherein the data in the buffer queue needs to wait for the electronic device to read and perform operations such as content detection; the data in the output buffer in step 6072 needs to wait for the FFMPEG module 470 to read and send to the software decoder decoding module 450 for processing.

[0208] 609: The software decoder decoding module 450 reads the data in the output buffer.

[0209] In the embodiment of the present application, after the software decoder decoding module 450 puts the data into the output buffer through the above step 608, the software decoder decoding module 450 can read the data in the output buffer.

[0210] 610: The software decoder decoding module 450 determines that the read data is the required video frame.

[0211] In the embodiment of the present application, after the software decoder decoding module 450 reads the data in the output buffer through the above step 609, it can determine whether the read data (i.e., the decoded video frame data) is the required video frame. If the software decoder decoding module 450 determines that the read data is the required video frame, for example, in the video content detection, the video frame data contains the image data required for detection, etc., then steps 611 to 613 can be executed to release the memory space and end the execution method flow. If the software decoder decoding module 450 determines that the read data is not the required video frame, it is necessary to re-execute step 609 and read the other data in the output buffer in sequence.

[0212] 611: The software decoder decoding module 450 releases the input buffer and the output buffer.

[0213] In the embodiment of the present application, the software decoder decoding module 450 can terminate the execution method flow after determining that the read data is the required video frame through the above step 610. In the process of terminating the execution method flow, it is necessary to release the input buffer and the output buffer space, for example, clear and reset the data in the input buffer in the above step 604 and the output buffer in the step 608.

[0214] 612: The software decoder decoding module 450 releases the FFMPEG software decoder.

[0215] In an embodiment of the present application, when the execution method flow ends, the software decoder decoding module 450 also needs to release the FFMPEG software decoder object created in the above step 601, for example, clear parameters and release the space occupied by the FFMPEG software decoder object.

[0216] 613: FFMPEG module 470 releases memory space.

[0217] In the embodiment of the present application, when the execution method flow ends, the FFMPEG module 470 also needs to release the memory space allocated for the matching software decoder in step 6033.

[0218] In this way, through the above process, the interaction between the software decoder decoding module 450 and the FFMPEG software decoder can be completed, and the process of decoding the extracted video frames through the FFMPEG software decoder can be realized.

[0219] The following takes the hardware decoder 200 as the MediaCodec hardware decoder 480 as an example. Figure 7 The flow chart shown in the figure is Figure 4 The process of interaction between the hardware decoder decoding module 460 and the MediaCodec hardware decoder 480 mentioned in is described.

[0220] 701: The hardware decoder decoding module 460 creates a MediaCodec hardware decoder.

[0221] In some embodiments, when an electronic device uses a MediaCodec hardware decoder, the hardware decoder decoding module 460 first needs to create a MediaCodec hardware decoder, for example, by configuring and instantiating a hardware decoder through the MediaCodec class on the electronic device.

[0222] 702: The hardware decoder decoding module 460 initializes the MediaCodec hardware decoder.

[0223] In some embodiments, the hardware decoder decoding module 460 needs to initialize the created MediaCodec hardware decoder, where the initialization may include allocating an operating environment, allocating memory space, and the like.

[0224] 703: The MediaCodec hardware decoder 480 starts working.

[0225] In some embodiments, after creating and initializing the MediaCodec hardware decoder, the hardware decoder decoding module 460 can complete the configuration of the MediaCodec hardware decoder, that is, the MediaCodec hardware decoder 480 can start working.

[0226] 704: The MediaCodec hardware decoder 480 puts the decoded data into an output buffer.

[0227] In some embodiments, the MediaCodec hardware decoder 480 may send the decoded video frame data to an output buffer, waiting for subsequent processing, for example, detecting whether the content in the video frame includes a target image.

[0228] 705: The hardware decoder decoding module 460 reads the data in the output buffer.

[0229] In some embodiments, the hardware decoder decoding module 460 can read the decoded video data from the output buffer.

[0230] 706: The hardware decoder decoding module 460 determines whether the output buffer is empty. If so, the process proceeds to step 709; if not, the process proceeds to step 707.

[0231] In some embodiments, if the hardware decoder decoding module 460 determines that the data read out in the above step 705 is empty, indicating that the output buffer is empty, that is, the MediaCodec hardware decoder 480 has not decoded the video frame data, then go to step 709 to obtain the input buffer to input data into the input buffer for decoding by the MediaCodec hardware decoder 480. If the hardware decoder decoding module 460 determines that the data read out in the above step 705 is not empty, then go to step 707 to determine whether the data is the required video frame.

[0232] 707: The hardware decoder decoding module 460 determines whether the data is a required video frame. If yes, it proceeds to step 708; if no, it proceeds to step 705.

[0233] In some embodiments, if the data read out by the hardware decoder decoding module 460 is not empty, the read out data (i.e., the decoded video frame data) can be judged whether it is a required video frame. If the hardware decoder decoding module 460 determines that the read out data is a required video frame, for example, in video content detection, the video frame data contains image data required for detection, etc., then step 708 can be executed to release the output buffer. If the hardware decoder decoding module 460 determines that the read out data is not a required video frame, step 705 needs to be re-executed to read other data in the output buffer in sequence.

[0234] 708: The MediaCodec hardware decoder 480 releases the output buffer.

[0235] In some embodiments, if the hardware decoder decoding module 460 determines through the above step 707 that the read data is the required video frame, the output buffer can be released.

[0236] 709: The MediaCodec hardware decoder 480 obtains the input buffer.

[0237] In some embodiments, if the hardware decoder decoding module 460 determines through the above step 706 that the output buffer is empty, indicating that the MediaCodec hardware decoder 480 has not decoded the video frame data, then the MediaCodec hardware decoder 480 needs to obtain the input buffer, read the video frame data to be processed input in the input buffer and decode it.

[0238] 710: The hardware decoder decoding module 460 determines whether the input buffer is available. If yes, proceed to step 711; if no, repeat step 709.

[0239] In some embodiments, the hardware decoder decoding module 460 also needs to determine whether the input buffer obtained by the MediaCodec hardware decoder 480 is an available buffer. For example, if the memory space of the input buffer is larger than the storage space required for the video frame, the input buffer is available, and then the process goes to step 711, and the hardware decoder decoding module 460 obtains the video frame data to be decoded. If the input buffer is not available, step 709 is repeated to re-acquire the input buffer.

[0240] 711: The hardware decoder decoding module 460 obtains the video frame data to be decoded.

[0241] In some embodiments, the hardware decoder decoding module 460 can obtain the video frame data to be decoded and send the data to the input buffer, wherein the video frame data to be decoded is extracted by the video frame extractor. For a specific description, refer to the above Figure 5B As mentioned in, I will not elaborate on this.

[0242] 712: The hardware decoder decoding module 460 determines whether all video frames have been extracted. If yes, it proceeds to steps 713 and 714; if no, it proceeds to step 713.

[0243] In some embodiments, after the hardware decoder decoding module 460 obtains the video frame data to be decoded, it is also necessary to determine whether all the video frame data has been obtained. Figure 1 As shown in , when performing fast content detection on video A, video A is divided into four short videos and the first frame image is extracted from each of them. The hardware decoder decoding module 460 can determine whether the acquired video frame data is four video frame data. If so, go to step 713 and step 714 to obtain the data in the input buffer and determine whether the method flow is completed; if not, go to step 713, the MediaCodec hardware decoder 480 obtains the data in the input buffer and repeatedly obtains the video frame data to be decoded.

[0244] 713: The MediaCodec hardware decoder 480 obtains the data in the input buffer.

[0245] In some embodiments, the MediaCodec hardware decoder 480 can obtain the data to be decoded in the buffer and decode it, wherein the decoded video frame data can be sent to the output buffer, that is, the above steps 705 to 708 are repeatedly executed.

[0246] 714: The hardware decoder decoding module 460 determines whether the decoding of the video frame is completed. If yes, it proceeds to step 715; if no, it proceeds to step 705.

[0247] In some embodiments, the hardware decoder decoding module 460 can determine whether the decoding of the video frame is completed. If so, step 715 is executed to release the MediaCodec hardware decoder and end the method flow. If not, the method flow goes to step 705 to continue reading data in the output buffer.

[0248] The judgment condition of whether the decoding of the video frame is completed can be whether the above step 707 or the above step 712 is satisfied. If yes, it means that the decoded data is the required video frame data or all the video frame data has been decoded, then the MediaCodec hardware decoder is released and the method flow ends. If no, it means that the decoded data is not the required video frame data, or there is still undecoded video frame data, then the above steps are repeated.

[0249] 715: The hardware decoder decoding module 460 releases the MediaCodec hardware decoder.

[0250] In some embodiments, after the hardware decoder decoding module 460 determines that the decoding of the video frame is completed through the above step 714, the MediaCodec hardware decoder can be released to end the execution method process, for example, clearing parameters and releasing the space occupied by the MediaCodec hardware decoder.

[0251] In this way, through the above process, the interaction between the hardware decoder decoding module 460 and the MediaCodec hardware decoder 480 can be completed, and the process of decoding the extracted video frames through the MediaCodec hardware decoder 480 can be realized.

[0252] In this way, through the method mentioned in this application, by combining the use of software decoders and hardware decoders, the video processing time can be shortened and the efficiency of video processing can be improved.

[0253] An embodiment of the present application provides an electronic device, including: a memory and a processor, the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the video processing method mentioned in the present application.

[0254] An embodiment of the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the image processing method mentioned in the present application.

[0255] An embodiment of the present application also provides a computer program product, including: computer instructions, when the computer instructions are executed on an electronic device, the electronic device executes the image processing method mentioned in the present application.

[0256] The video processing method provided in the embodiment of the present application can be applied to electronic devices, and the electronic devices can be any electronic devices such as mobile phones, computers, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) devices, virtual reality (VR) devices, laptop computers, etc., and the embodiment of the present application is not limited to this.

[0257] The various embodiments disclosed in the present application may be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0258] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application processor, or a microprocessor.

[0259] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0260] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable storage media (e.g., computer-readable storage media), which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), a magnetic card or an optical card, or a tangible machine-readable memory for transmitting information (e.g., carrier wave, infrared signal digital signal, etc.) using the Internet in an electrical, optical, acoustic or other form of propagation signal. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0261] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0262] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0263] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one" do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0264] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A video processing method, applied to electronic equipment, characterized in that: The electronic device includes a hardware decoder; and The method comprises: Get the first video to be processed; Determining a decoding method for the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device; The decoding method includes an extraction method for extracting video frames from the first video and a decoding method for decoding video frames extracted from the first video; and The extraction method includes extracting video frames using the hardware decoder or extracting video frames using the software decoder; The decoding method includes using the hardware decoder to decode the video frame or using the software decoder to decode the video frame.

2. The method according to claim 1, characterized in that include: Determining a first extraction support degree of the hardware decoder for the first video based on a matching degree between various processing parameters of a video frame extractor in the hardware decoder and various parameters of a video file format of the first video; Based on the matching degree between each processing parameter of the video frame extractor in the software decoder and each parameter of the video file format of the first video, the second extraction support degree of the software decoder for the first video is determined.

3. The method according to claim 2, characterized in that The processing parameters include one or more of video resolution and video frame rate.

4. The method according to claim 1, characterized in that include: Determining a first decoding support degree of the hardware decoder for the first video based on a matching degree between each processing parameter of a video frame decoder in the hardware decoder and each parameter of the video encoding format information of the first video; Based on the matching degree between each processing parameter of the video frame decoder in the software decoder and each parameter of the video encoding format information of the first video, the second decoding support degree of the software decoder for the first video is determined.

5. The method according to claim 4, characterized in that The processing parameters include one or more of video resolution, video frame rate, bit rate, and encoding format.

6. The method according to any one of claims 2 to 5, characterized in that: The determining the decoding method of the first video based on the matching degree between the video parameter of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, comprises: Selecting a decoder with the greatest support for extraction of the first video from the hardware decoder and the software decoder, and performing video frame extraction on the first video; A decoder with the greatest decoding support for the first video is selected from the hardware decoder and the software decoder to decode the video frame extracted from the first video.

7. The method according to claim 2, characterized in that The determining the decoding method of the first video based on the matching degree between the video parameter of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, comprises: Selecting a decoder with the greatest support for extraction of the first video from the hardware decoder and the software decoder, and performing video frame extraction on the first video; A preset decoder is used to decode the video frames extracted from the first video.

8. The method according to claim 2, characterized in that: The determining of a decoding method for the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, further includes: Selecting a decoder with the greatest support for extraction of the first video from the hardware decoder and the software decoder, and performing video frame extraction on the first video; The video frames extracted from the first video are decoded using a decoder for extracting video frames from the first video.

9. The method according to claim 4, characterized in that The determining the decoding method of the first video based on the matching degree between the video parameter of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, comprises: Using a preset decoder to extract video frames from the first video; A decoder with the greatest decoding support for the first video is selected from the hardware decoder and the software decoder to decode the video frame extracted from the first video.

10. The method according to claim 1, characterized in that The first video includes one or more of a digital rights management video, a high dynamic range video, and a standard dynamic range video.

11. The method according to claim 10, characterized in that Corresponding to the first video being the high dynamic range video, the determining the decoding method of the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, includes: extracting video frames in the first video using the software decoder when video parameters of the first video match processing parameters of the software decoder; In a case where video parameters of the first video do not match processing parameters of the software decoder, and video parameters of the first video match processing parameters of the hardware decoder, the hardware decoder is used to extract video frames in the first video.

12. The method according to claim 10, characterized in that Corresponding to the first video being the digital rights management video and the standard dynamic range video, the determining the decoding method of the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, respectively, includes: In the case that the electronic device supports processing the first video, a method for extracting video frames from the first video is determined based on the degree of matching between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device.

13. The method according to claim 12, characterized in that The step of determining a method for extracting video frames from the first video based on the matching degree between the video parameters of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, comprises: Determining a first extraction support degree of the hardware decoder for the first video based on a matching degree between various processing parameters of a video frame extractor in the hardware decoder and various parameters of a video file format of the first video; Determining a second extraction support degree of the software decoder for the first video based on a matching degree between various processing parameters of a video frame extractor in the software decoder and various parameters of a video file format of the first video; Establishing a video frame extractor capability table based on the first extraction support and the second extraction support; In the video frame extractor capability table, a decoder corresponding to the maximum extraction support is selected to extract the video frame in the first video.

14. The method according to claim 10, characterized in that The determining the decoding method of the first video based on the matching degree between the video parameter of the first video and the processing parameters of the hardware decoder and the processing parameters of the software decoder on the electronic device, comprises: Determining a first decoding support degree of the hardware decoder for the first video based on a matching degree between each processing parameter of a video frame decoder in the hardware decoder and each parameter of the video encoding format information of the first video; Determining a second decoding support degree of the software decoder for the first video based on a matching degree between each processing parameter of a video frame decoder in the software decoder and each parameter of the video encoding format information of the first video; Establishing a video frame decoder capability table based on the first decoding support and the second decoding support; In the video frame decoder capability table, a decoder corresponding to the maximum decoding support is selected to decode the video frame extracted from the first video.

15. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the video processing method described in any one of claims 1-14.

16. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the video processing method according to any one of claims 1 to 14.

17. A computer program product, characterized in that include: Computer instructions, when the computer instructions are executed on an electronic device, enable the electronic device to execute the video processing method according to any one of claims 1 to 14.

Images