Video playback method, device, electronic device and storage medium

By performing preset image processing in parallel when decoding the current video frame in the video frame queue in the electronic device, the increase in power consumption and lag caused by decoding and image processing during video playback is solved, and more efficient video playback is achieved .

CN113038276BActive Publication Date: 2024-11-19GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110251615.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-08
Publication Date
2024-11-19
Estimated Expiration
2041-03-08

AI Technical Summary

Technical Problem

When electronic devices play video, power consumption increases and video playback is stuttered during decoding and image processing, which is difficult to effectively solve in the prior art.

Method used

By obtaining the video frame queue of the video to be processed, preset image processing is performed in parallel during decoding, avoiding the mutual squeeze of decoding and image processing, reducing the amount of calculation, and using pipeline parallel processing methods to process decoding, preset image processing and display playback respectively.

Benefits of technology

This reduces the power consumption caused by preset image processing of video frames, avoids video playback lag in case of large processing volume, and improves the smoothness and efficiency of video playback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113038276B_ABST
    Figure CN113038276B_ABST
Patent Text Reader

Abstract

The present application discloses a video playback method, device, electronic device and storage medium, the method comprising: obtaining a video frame queue of a video to be processed, the video frame queue comprising a plurality of continuous video frames; when decoding a current video frame in the video frame queue, performing a preset image processing on the decoded video frame in parallel; and displaying and playing the video frame after the preset image processing. The method can reduce the power consumption of the electronic device when the video to be processed needs to be processed by the preset image processing when playing the video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and more specifically, to a video playback method, device, electronic device and storage medium. Background Art

[0002] With the rapid advancement of technology and living standards, electronic devices (such as smart phones, tablet computers, etc.) are widely used by people, and as the configuration of electronic devices becomes better and better, people often use electronic devices to play videos. When playing videos, electronic devices usually perform image processing on the video images to improve the video display effect, but this also increases power consumption. Summary of the invention

[0003] In view of the above problems, the present application proposes a video playback method, device, electronic device and storage medium.

[0004] In a first aspect, an embodiment of the present application provides a video playback method, the method comprising: obtaining a video frame queue of a video to be processed, the video frame queue comprising multiple continuous video frames; when decoding a current video frame in the video frame queue, performing preset image processing on the decoded video frames in parallel; and displaying and playing the video frames after the preset image processing.

[0005] In a second aspect, an embodiment of the present application provides a video playback device, comprising: a frame queue acquisition module, an image processing module and a display module, wherein the frame queue acquisition module is used to acquire a video frame queue of a video to be processed, the video frame queue comprising multiple continuous frames of images; the image processing module is used to perform preset image processing on the decoded video frames in parallel when decoding the current video frame in the video frame queue; the display module is used to display and play the video frames after the preset image processing.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the video playback method provided in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the video playback method provided in the first aspect above.

[0008] The solution provided by the present application obtains a video frame queue of a video to be processed, and the video frame queue includes multiple continuous video frames. When decoding the current video frame in the video frame queue, preset image processing is performed on the decoded video frame in parallel, and the video frame after the preset image processing is displayed and played, thereby achieving preset image processing on other frames in parallel when decoding the video frame, avoiding time squeezed by decoding and preset image processing, thereby reducing power consumption caused by more calculations required for preset image processing of the video frame, and at the same time, avoiding video playback freezes when the processing volume of the preset image processing is large. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flow chart of a video playback method according to an embodiment of the present application is shown.

[0011] Figure 2 A flow chart of a video playback method according to another embodiment of the present application is shown.

[0012] Figure 3 A flow chart of a video playback method according to another embodiment of the present application is shown.

[0013] Figure 4 A schematic diagram showing the principles of a video playback method provided by yet another embodiment of the present application is shown.

[0014] Figure 5 A flow chart of a video playback method according to another embodiment of the present application is shown.

[0015] Figure 6 A schematic diagram showing a principle of a video playback method provided in yet another embodiment of the present application is shown.

[0016] Figure 7 Another principle schematic diagram of a video playback method provided in yet another embodiment of the present application is shown.

[0017] Figure 8 A flow chart of a video playback method according to yet another embodiment of the present application is shown.

[0018] Fig. 9 A flow chart of a video playback method according to yet another embodiment of the present application is shown.

[0019] Fig.10A block diagram of a video playback device according to an embodiment of the present application is shown.

[0020] Fig.11 It is a block diagram of an electronic device according to an embodiment of the present application for executing the video playback method according to an embodiment of the present application.

[0021] Fig.12 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the video playback method according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0023] At present, most electronic devices, such as mobile phones, tablet computers, and computers, can play videos. As the configuration of electronic devices becomes higher and higher, and people's demand for video quality is also increasing, electronic devices usually perform corresponding image processing on video images when playing videos to improve the display effect of video images. For example, when playing videos, electronic devices perform super-resolution processing (super-resolution reconstruction), display enhancement processing, denoising processing, etc. on video images to improve the display effect of video images.

[0024] In the related art, when playing a video, an electronic device usually decodes (decoder), processes the image, and displays (display) each frame of the video image in turn, that is, after decoding, processing, and displaying one frame of the video image, the next frame of the video image is decoded, processed, and displayed. However, the playback frame rate is usually required to be 30FPS (frames per second, Frames Per Second). In this case, the total processing time of the decoder, image processing, and display must be less than 33ms (milliseconds) / frame. Artificial Intelligence (AI) algorithms are usually used to process video images, and the AI ​​algorithm is a task that focuses on a large amount of calculations. Smaller delays mean that more calculations and higher memory access frequencies are required during AI reasoning, resulting in greater power consumption.

[0025] In response to the above problems, the inventors have proposed a video playback method, device, electronic device, and storage medium provided in the embodiments of the present application, which can realize the preset image processing of other frames in parallel when decoding a video frame, avoiding the time occupied by decoding and preset image processing, thereby reducing the power consumption caused by more calculations required for the preset image processing of the video frame, and at the same time, avoiding the video playback freeze when the processing amount of the preset image processing is large. Among them, the specific video playback method is described in detail in the subsequent embodiments.

[0026] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a video playback method provided by an embodiment of the present application. In a specific embodiment, the video playback method is applied to Fig.10 The video playback device 400 and the electronic device 100 ( Fig.11 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment can be a smart phone with a video display function, a tablet computer, a smart watch, a laptop computer, etc., which is not limited here. Figure 1 The process shown is described in detail, and the video playing method may specifically include the following steps:

[0027] Step S110: obtaining a video frame queue of a video to be processed, wherein the video frame queue includes a plurality of continuous video frames.

[0028] In the embodiment of the present application, the video to be processed is a video used by the electronic device for display. Optionally, the electronic device can obtain the video to be processed from a server, can obtain the video to be processed locally, or can obtain the video to be displayed from other electronic devices. The specific method of obtaining the video to be displayed may not be limited.

[0029] In some embodiments, when the video to be processed is obtained by the electronic device from the server, the video to be processed may be downloaded by the electronic device from the server, or obtained online from the server by the electronic device. For example, the video to be processed may be video data downloaded by the electronic device through the installed video playback software, or video data obtained online by the video playback software. Among them, the server may be a cloud server; when the video to be processed is obtained from the local of the electronic device, the video to be processed may be video data pre-downloaded and stored in the local memory by the electronic device; when the video to be processed is obtained by the electronic device from other electronic devices, the video to be processed may be transmitted to the electronic device by other electronic devices through wireless communication protocols, for example, through Wlan protocol, Bluetooth protocol, ZigBee protocol or WiFi protocol, etc., or may be transmitted to the electronic device by other electronic devices through data networks, for example, 2G network, 3G network, 4G network or 5G network, etc., which are not limited here.

[0030] In an embodiment of the present application, the electronic device obtains each frame of the video to be processed and writes it into a video frame queue in sequence, so that each frame of the video is decoded (decoder), preset image processing, and displayed and played in sequence according to the sequence of the video frames.

[0031] Step S120: When decoding the current video frame in the video frame queue, performing preset image processing on the decoded video frame in parallel.

[0032] In the embodiment of the present application, since each frame of video is decoded, preset image processing and displayed and played in sequence, the decoding and preset image processing will take up time, and the thread will use more resources during decoding or preset image processing, resulting in greater power consumption. Therefore, when the electronic device plays the video frames in the video frame queue, it can perform preset image processing on the decoded video frames in parallel while decoding the current video frame. It should be noted that during the playback of multiple consecutive video frames in the video frame queue, the video frames are displayed in the order of the video frames.

[0033] In some embodiments, when the electronic device starts playing the video to be processed, after decoding the first video frame, when decoding the second video frame, the first decoded video frame can be processed in parallel with the preset image processing; when decoding the third video frame, the second decoded video frame can be processed in parallel with the preset image processing, and so on can be repeated. When decoding the current video frame, the previous decoded video frame can be processed in parallel with the preset image processing; the video frame that has been processed with the preset image processing can be displayed and played. Thus, the display and playback of each video frame is placed in at least two time periods, avoiding the decoding, preset image processing and display playback of one video frame before decoding, preset image processing and display playback of the next video frame, which causes the decoding and preset image processing of the video frame to take up time, resulting in an increase in power consumption, or increases the total duration of decoding, preset image processing and display, resulting in freezes during video playback.

[0034] In other embodiments, the electronic device may, when starting to play the video to be processed, perform preset image processing on the multiple video frames in parallel after decoding the multiple video frames; the video frames after the preset image processing can be displayed and played in sequence, and each time a video frame is displayed, the next undecoded video frame is decoded, and it is ensured that multiple decoded video frames exist at the same time, and the preset image processing is performed on the decoded multiple video frames in parallel, thereby further saving power consumption and improving performance ratio.

[0035] In an embodiment of the present application, the preset image processing includes any one or more of the following: super-resolution reconstruction, contrast enhancement, saturation enhancement, image denoising, image defogging, and image deblurring. Among them, super-resolution reconstruction refers to an image processing technology that processes a low-resolution image or image sequence to obtain a high-resolution image; contrast enhancement refers to processing an image to enhance its contrast; saturation enhancement refers to processing an image to enhance its saturation; image denoising refers to removing noise from an image; image defogging refers to defogging an image to eliminate the influence of particle scattering on the image; image deblurring refers to eliminating the blurred content of an image. Of course, the specific preset image processing may not be limited, for example, it may also be display enhancement processing during video playback.

[0036] In some embodiments, the preset image processing may be to process images using an artificial intelligence (AI) algorithm. Since the AI ​​algorithm requires a large amount of processing, the preset image processing with a large amount of processing is performed in parallel on the decoded video frames, and the video frames after the preset image processing are displayed in parallel, thereby avoiding the time occupied by decoding and preset image processing in related technologies, and further reducing the power consumption caused by more calculations and data reading and writing when performing preset image processing on video frames.

[0037] Step S130: displaying the video frame after the preset image processing.

[0038] In the embodiment of the present application, the video frames in the video frame queue that have been subjected to the preset image processing can be displayed and played. The above process is repeated to complete the playback of all video frames of the video to be processed.

[0039] In some embodiments, the video driver transmits the video data after the preset image processing to SurfaceFlinger (layer transfer module) for rendering and display on the screen to complete the display and playback of the video frame. SurfaceFlinger is an independent Service, which receives the Surface of all Windows as input, calculates the position of each Surface in the final composite image according to parameters such as ZOrder, transparency, size, and position, and then passes it to HWComposer or OpenGL (Open Graphics Library) to generate the final display Buffer, which is then displayed on a specific display device.

[0040] In some implementations, the decoding, preset image processing, and display and playback of video frames may be performed in a pipeline parallel processing manner, wherein the decoding, preset image processing, and display and playback may be performed on different video frames respectively to achieve pipeline parallel processing.

[0041] In other embodiments, the preset image processing and display playback can also be used as a processing stage, and processed in parallel with the decoding stage, wherein the decoding and the processing stage process different video frames respectively. For example, in the related art, the time period for decoding, preset image processing, and display playback of each video frame does not exceed 33 milliseconds (ms), then a certain video frame can be decoded within the time period for decoding, preset image processing, and display playback of a video frame, and the preset image processing and display playback of the decoded video frame can be performed in parallel within the time period, thereby avoiding the decoding and preset image processing from occupying each other's time, thereby reducing the power consumption caused by more calculations and data reading and writing when performing preset image processing on the video frame.

[0042] The video playback method provided in the embodiment of the present application performs preset image processing on the decoded video frames in parallel when decoding the current video frame in the video frame queue when playing the video to be processed, and displays and plays the video frames after the preset image processing. That is to say, when the preset image processing is performed on the video frame, the subsequent video frames are decoded at the same time. In this way, when it is really necessary to display the subsequent video frames after performing the preset image processing, the decoding will not occupy the time of the preset image processing and cause high power consumption, thereby saving the power consumption of the electronic device and avoiding the video playback jam.

[0043] See also Figure 2 , Figure 2 A schematic diagram of a video playback method provided by another embodiment of the present application is shown. The video playback method is applied to the above electronic device. Figure 2 The process shown is described in detail, and the video playing method may specifically include the following steps:

[0044] Step S210: obtaining a video frame queue of a video to be processed, wherein the video frame queue includes a plurality of continuous video frames.

[0045] In the embodiment of the present application, step S210 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0046] Step S220: When decoding the Mth video frame in the video frame queue, perform preset image processing on the first N frames of the Mth video frame in parallel, and display and play the video frames after the preset image processing in parallel, where M is a positive integer greater than 2, and N is a positive integer less than M.

[0047] In an embodiment of the present application, the decoding of video frames, preset image processing, and display and playback can be divided into a three-stage pipeline, and pipeline parallel processing can be performed for the decoding of video frames, preset image processing, and display and playback.

[0048] In some embodiments, when decoding the Mth video frame in the video frame queue, the first N frames of the Mth video frame are subjected to preset image processing in parallel, and the video frames after the preset image processing are displayed in parallel. That is to say, decoding, preset image processing, and display playback are all performed simultaneously, but the video frames processed by decoding, preset image processing, and display playback are different, and according to the sequence of video frames, the video frames that come earlier in time sequence are first subjected to the process of decoding, preset image processing, and display playback, thereby avoiding the time squeezed by decoding and display and preset image processing, which requires more calculations and higher memory access frequency during the squeezed time for preset image processing, resulting in increased power consumption.

[0049] In some embodiments, when the video to be processed begins to be played, the starting video frame is decoded; after a decoded video frame exists, subsequent video frames are decoded, and preset image processing is performed on the decoded video frames in parallel; after a video frame after preset image processing exists, subsequent video frames are decoded, preset image processing is performed on the decoded video frames in parallel, and the video frames after preset image processing are displayed and played.

[0050] In some embodiments, when implementing pipeline parallel processing for decoding, preset image processing, and display playback of video frames, different threads can be created for decoding, preset image processing, and display playback, respectively. The threads corresponding to decoding, preset image processing, and display, respectively, can process different video frames in the order in which the video frames are played.

[0051] Optionally, different cache spaces can be set for the three threads respectively, so that the three threads can obtain video frames from different cache spaces to process different video frames. For example, the decoding thread corresponds to the first cache space of the video frame queue, and the decoding thread obtains video frames from the first cache space of the video frame queue for decoding; the decoded video frames can be written to the second cache space, and the thread corresponding to the preset image processing can obtain video frames from the second cache space for preset image processing; the video frames after the preset image processing can be written to the third cache space, and the thread corresponding to the display and playback can obtain video frames from the third cache space for display and playback.

[0052] In the embodiment of the present application, the above N can be 1 or a positive integer greater than 1. When N is 1, the three-stage pipeline of decoding, preset image processing and display playback only processes different 1-frame video frames respectively; when N is a positive integer greater than 1, decoding and display playback process different 1-frame video frames respectively, and the preset image processing processes multiple video frames different from the video frames processed by decoding and display in parallel.

[0053] The video playback method provided in the embodiment of the present application divides the decoding, preset image processing and display playback of video frames into three-level pipelines, and performs pipeline parallel processing for the decoding, preset image processing and display playback of video frames. In other words, the video frames processed by decoding, preset image processing and display playback are different, and according to the order of video frames, the video frames that come earlier in sequence are first decoded, preset image processed and displayed, thereby avoiding the time squeezed by decoding and display and preset image processing, and making the preset image processing require more calculations and higher memory access frequency during the squeezed time, which increases power consumption.

[0054] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of a video playback method provided by another embodiment of the present application. The video playback method is applied to the above electronic device. Figure 3 The process shown is described in detail, and the video playing method may specifically include the following steps:

[0055] Step S310: obtaining a video frame queue of a video to be processed, wherein the video frame queue includes a plurality of continuous video frames.

[0056] In the embodiment of the present application, step S310 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0057] Step S320: when decoding the Mth video frame in the video frame queue, performing preset image processing on the (M-1)th video frame in parallel, and displaying the (M-2)th video frame in parallel, where M is a positive integer greater than 2.

[0058] In an embodiment of the present application, the decoding, preset image processing and display of video frames are divided into a three-level pipeline. When the pipeline is processed in parallel for the decoding, preset image processing and display playback of video frames, when a certain video frame is decoded, the preset image processing can be performed in parallel on a previous decoded video frame, and the subsequent video frame can be displayed in parallel.

[0059] In some embodiments, see Figure 4, in the process of playing the video frames in the video frame queue, first decode the video frame 1 in time period 1; while decoding the video frame 2 in time period 2, perform preset image processing on the video frame 1 in parallel; decode the video frame 3 in time period 3, perform preset image processing on the video frame 2 in parallel, and display and play the video frame 1 in parallel; decode the video frame 4 in time period 4, perform preset image processing on the video frame 3 in parallel, and display and play the video frame 2 in parallel; repeat this until all the video frames are displayed, that is, the playback of the processed video is completed. Since the playback frame rate is usually at least 30FPS to achieve a smoother visual experience, each of the above time periods can be less than 33ms; in the time period of actually displaying and playing each video frame, it is only displayed, and the decoding and preset image processing have been performed in the time period before it, so they will not occupy each other's time, resulting in an increase in power consumption, and the duration of each video frame in the time period of actual display is less than 33ms, which also avoids jamming and ensures the smoothness of video playback.

[0060] In this embodiment, the thread corresponding to decoding starts running in time period 1; the thread corresponding to the preset image processing starts running in time period 2, and the thread corresponding to decoding runs in parallel; in time period 3 and subsequent time periods, the thread corresponding to decoding, the thread corresponding to the preset image processing, and the thread corresponding to display and playback all run in parallel, so that when decoding a certain frame of video, the preset image processing can be performed in parallel on a previously decoded frame of video, and the preset image processing can be performed in parallel on a frame of video that has previously completed the preset image processing.

[0061] The video playback method provided in the embodiment of the present application divides the decoding, preset image processing and display playback of video frames into a three-level pipeline, and performs pipeline parallel processing on the decoding, preset image processing and display playback of video frames. When decoding a certain video frame, preset image processing can be performed in parallel on a previously decoded video frame, and display playback can be performed in parallel on a video frame, thereby avoiding the time occupied by decoding and display and preset image processing, and further reducing the need for more calculations and higher memory access frequency for image processing during the occupied time, which increases power consumption.

[0062] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of a video playback method provided by another embodiment of the present application. The video playback method is applied to the above electronic device. Figure 5 The process shown is described in detail, and the video playing method may specifically include the following steps:

[0063] Step S410: obtaining a video frame queue of a video to be processed, wherein the video frame queue includes a plurality of continuous video frames.

[0064] Step S420: When decoding the Mth video frame in the video frame queue, perform preset image processing on the (MK)th video frame to the (M-1)th video frame in parallel, and display the (MK-1)th video frame in parallel, wherein M is a positive integer greater than 2, and K is a positive integer greater than 1.

[0065] In an embodiment of the present application, the decoding, preset image processing and display playback of video frames are divided into a three-level pipeline. When the pipeline parallel processing is performed for the decoding, preset image processing and display playback of video frames, when a certain video frame is decoded, the preset image processing can be performed in parallel on multiple previously decoded video frames, and the subsequent video frames can be displayed and played in parallel.

[0066] In some implementations, since the processing amount of the preset image processing is large, the preset image processing can be performed during the time period when multiple video frames are displayed and played. Figure 6, in the process of playing the video frames in the video frame queue, firstly, in time period 1, video frame 1 and video frame 2 are decoded; in time period 2, video frame 3 and video frame 4 are decoded, and preset image processing is performed on video frame 1 and video frame 2 in parallel; in time period 3 when displaying and playing video frame 1, part of the preset image processing is performed on video frame 3 and video frame 4 in parallel; in time period 4 when displaying and playing video frame 2, video frame 5 is decoded, and another part of the preset image processing is performed on video frame 3 and video frame 4 in parallel, and because the decoding time is usually short, the preset image processing of video frame 3 and video frame 4 can also be completed within time period 4, and after decoding video frame 5, part of the preset image processing is performed on video frame 5; in time period 5 when displaying and playing video frame 3, video frame 6 is decoded. The video frame 5 is decoded, and the preset image processing is continued on the video frame 5. In the time period 5, after the video frame 6 is decoded, part of the preset image processing is performed on the video frame 6. In the time period 6 of displaying and playing the video frame 4, the video frame 7 is decoded, and the preset image processing is continued on the video frame 5 in parallel, and the preset image processing is continued on the video frame 6 in parallel. In addition, after the preset image processing is completed on the video frame 5 and the video frame 7 is decoded, part of the preset image processing can also be performed on the video frame 7 in parallel. Repeat this, and start decoding one video frame every time one video frame is displayed and played. The preset image processing of the video frames after the video frame 4 spans the display time period of three video frames. In each time period, it is guaranteed that one video frame is decoded, one video frame is displayed and played, and multiple video frames are at least partially processed by the preset image. Therefore, the degree of parallelization is further improved. Since the playback frame rate is usually at least 30FPS to achieve a smoother visual experience, the time period for displaying and playing each video frame can be less than 33ms. In this way, when each video frame is actually displayed, it is only displayed and played, while decoding and preset image processing have been performed in the time period before it, so they will not occupy each other's time, resulting in increased power consumption or playback jams.

[0067] In some other embodiments, when K is 2, see Figure 7, in the process of playing the video frames in the video frame queue, firstly decode the video frame 1 and the video frame 2 in the time period 1; decode the video frame 3 and the video frame 4 in the time period 2, and perform the preset image processing on the video frame 1 and the video frame 2 in parallel; in the time period 3 when the video frame 1 is displayed and played, decode the video frame 5 in parallel, and perform part of the preset image processing on the video frames 3 and 4 in parallel; in the time period 4 when the video frame 2 is displayed and played, decode the video frame 6 in parallel, and perform another part of the preset image processing on the video frames 3 and 4 in parallel; in the time period 5 when the video frame 3 is displayed and played, decode the video frame 7 in parallel, and perform part of the preset image processing on the video frames 5 and 6 in parallel; in the time period when the video frame 4 is displayed 6, decode video frame 8 in parallel, and perform another part of preset image processing on video frames 5 and 6 in parallel; in time period 7 when displaying and playing video frame 5, decode video frame 9 in parallel, and perform a part of preset image processing on video frames 7 and 8 in parallel; in time period 8 when displaying and playing video frame 6, decode video frame 10 in parallel, and perform another part of preset image processing on video frames 7 and 8 in parallel; repeat in this way, for the display time period where each video frame is located, one video frame is decoded, two video frames are processed in parallel with preset image processing, and one video frame is displayed and played in parallel, and finally the decoding, preset image processing and display and playback of each video frame are completed, that is, the playback of the processed video is completed. Since the playback frame rate is usually at least 30FPS to achieve a relatively smooth visual experience, the time period where each video frame is displayed and played can be less than 33ms. In this way, when each video frame is actually displayed, it is only displayed and played, and the decoding and preset image processing have been performed in the time period before it, so they will not occupy each other's time, resulting in increased power consumption or playback jams.

[0068] Of course, the above example is an implementation method when K is 2, and N can also be 3, or 4, etc., which is not limited here.

[0069] In some embodiments, performing preset image processing on the (MK)th video frame to the (M-1)th video frame in parallel may include: using different asynchronous threads to perform preset image processing on the (MK)th video frame to the (M-1)th video frame in parallel. In this embodiment, the thread pool asynchronous architecture can be used to use different asynchronous threads for multiple frames of video frames that are subjected to parallel preset image processing, thereby achieving preset image processing on multiple frames of video frames in parallel.

[0070] In this embodiment, for the threads performing the preset image processing, different devices can be used to complete each step in the preset image processing, so as to make full use of the devices for acceleration. Specifically, each thread can use a different digital signal processor (DSP), an embedded neural network processor (NPU) and a central processing unit (CPU), wherein the DSP, NPU and CPU core are used for different steps in the preset image processing. Exemplarily, when the preset image processing is performed on 2 frames of video, thread 1 corresponds to DSP1, NPU1 and CPU core 1, thread 2 corresponds to DSP2, NPU2 and CPU core 2, and the preset image processing includes the steps of resizing, padding, super-resolution reconstruction and cropping, then for thread 1, DSP1, NPU1 and CPU core 1 can be used to perform different steps, for example, DSP1 is used for scaling, CPU core 1 is used for padding, NPU is used for super-resolution reconstruction, and CPU core 1 is used for cropping; for thread 2, DSP2, NPU2 and CPU core 2 can be used to perform different steps. In addition, since the image includes multiple components, different devices can be used for different components to make full use of the devices for acceleration. For example, for the image Y, U and V components, NPU, CPU and DSP can be used for each processing step of component Y, display processor (DPU, Display Processor Unit), graphics processor (GPU, Graphics Processing Unit) and CPU can be used for each processing step of component U, and accelerated processing unit (Accelerated Processing Unit, APU), tensor processor (TPU) and CPU can be used for each processing step of component V. Of course, the above is only an example and does not represent a limitation on the devices actually allocated to different components. Among them, component Y represents brightness, that is, grayscale value; component U and component V represent chromaticity, which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0071] In some scenarios, such as video playback, when playing a video, it is necessary to reconstruct the video image with super-resolution before displaying it. Since the average user's eyes are not very sensitive to color and are more sensitive to brightness, super-resolution reconstruction can be performed only on component Y, while bilinear interpolation can be used to amplify components U and V. At this time, NPU, CPU, and DSP can be used for each processing step of component Y; DPU and other devices can be used for bilinear interpolation of component U; and GPU and other devices can be used for bilinear interpolation of component V, so that each component and the devices used for the processing steps of each component are different, and the devices are fully utilized for acceleration.

[0072] In some embodiments, since the performance of a multi-core processor is generally better, the remaining resources of the processor can also be obtained when performing preset image processing on multiple frames of images in parallel; if the remaining resources are greater than the first resource threshold, the operating frequency of the processor is reduced.

[0073] Among them, the remaining amount of resources is the amount of resources remaining after deducting the resources occupied by the running program from the total amount of resources of the processor. After detecting the remaining amount of resources of the processor, the remaining amount of resources can be compared with the first resource threshold to determine whether to reduce the operating frequency of the processor to save power consumption according to the comparison result of the remaining amount of resources and the first resource threshold. Among them, the resource threshold can be a critical value indicating the high and low levels of the remaining amount of resources of the AI ​​chip. When the remaining amount of resources of the processor is less than or equal to the above-mentioned first resource threshold, the performance is general at this time, and it may just be enough to smoothly perform preset image processing on multiple frames of images in parallel, so the operating frequency of the processor is not reduced; when the remaining amount of resources of the processor is greater than the above-mentioned first resource threshold, the remaining resources for processing are more, and the preset image processing on multiple frames of images can be performed smoothly in parallel, and the performance is in excess, so the operating frequency of the processor can be reduced to save power consumption of the electronic device. For example, when the remaining resources of the processor are expressed as a percentage of the maximum resources of the processor, the first resource threshold can be 60%-80%. Of course, the above first resource threshold is only an example, and the specific resource threshold may not be used as a limitation in the embodiment of the present application, and can be determined according to actual needs and the performance of the processor. That is to say, when the performance of the processor is strong, since its remaining resources are lower than a certain level, its performance is still sufficient, so its corresponding first resource threshold can be lower than the first resource threshold of the processor with weak performance. For example, the maximum operating frequencies of two processors are 2GHz and 4GHz respectively. If the first resource threshold corresponding to the processor with a maximum operating frequency of 2GHz is 20%, then the resource threshold corresponding to the processor with a maximum operating frequency of 4GHz can be 30%.

[0074] In some video playback scenarios, the video image needs to be super-resolution reconstructed before being displayed and played when playing the video. The power consumption brought about by the video super-resolution reconstruction scenario is relatively large, which places stricter requirements on power consumption. Therefore, on the basis of being able to meet the video frame rate and smoothly perform preset image processing on multiple frames in parallel, when the remaining resources of the processor are greater than the above-mentioned first resource threshold, reducing the operating frequency of the processor can save power consumption of the electronic device.

[0075] In some embodiments, the electronic device can perform preset image processing on one video frame in parallel while decoding the video frame, or can perform preset image processing on multiple video frames in parallel while decoding the video frame. The electronic device can also selectively execute the scheme of the previous embodiment, or execute the scheme of this embodiment. Optionally, the electronic device can obtain the duration of the processor performing preset image processing on images of the same size under the performance condition of playing the video to be processed. If the duration is greater than the preset duration, it means that the processing amount of the preset image processing is large, so the method of performing parallel processing on multiple video frames provided in this embodiment can be executed; if the duration is not greater than the preset duration, it means that the processing amount of the preset image processing is relatively small, so the method of performing parallel processing on one video frame provided in the previous embodiment can be executed.

[0076] The video playback method provided in the embodiment of the present application divides the decoding, preset image processing and display of video frames into a three-level pipeline, and performs pipeline parallel processing on the decoding, preset image processing and display playback of video frames. When decoding a certain video frame, preset image processing can be performed in parallel on multiple previously decoded video frames, and one video frame can be displayed and played in parallel, thereby avoiding the decoding and display and preset image processing from occupying time, which requires more calculations and a higher memory access frequency for the preset image processing during the occupied time, thereby increasing power consumption.

[0077] See also Figure 8 , Figure 8 FIG. 1 is a flow chart of a video playback method provided by another embodiment of the present application. The video playback method is applied to the above electronic device. Figure 8 The process shown is described in detail, and the video playing method may specifically include the following steps:

[0078] Step S510: obtaining a video frame queue of a video to be processed, wherein the video frame queue includes a plurality of continuous video frames.

[0079] Step S520: Obtain the remaining resources of the processor.

[0080] Step S530: If the remaining resource amount is greater than a second resource threshold, start the preset image processing.

[0081] In an embodiment of the present application, considering that preset image processing such as super-resolution reconstruction and display enhancement is performed on video frames, it is necessary to ensure the smoothness of video playback, and the processing volume of these preset image processing is large, so it is also possible to determine whether to turn on the preset image processing when playing the video frame. After detecting the remaining resources of the processor, the remaining resources can be compared with the second resource threshold to determine whether to turn on the preset image processing of the video frame based on the comparison result between the remaining resources and the second resource threshold. Among them, the second resource threshold is used as a condition for whether the playback after the preset image processing can be smoothly performed, and the second resource threshold can be smaller than the aforementioned first resource threshold. If the remaining amount of resources is greater than the second resource threshold, it means that the preset image processing can be performed smoothly, so the preset image processing can be turned on; if the remaining amount of resources is less than or equal to the second resource threshold, it means that the preset image processing cannot be performed smoothly, so the preset image processing can be turned off. When playing the video to be processed, each video frame can be decoded and displayed in turn, that is, after decoding and displaying one video frame, the next video frame can be decoded and displayed. Of course, the decoding and display can also be divided into a two-level pipeline to perform parallel processing on the decoding and display of the video frames.

[0082] Step S540: If the remaining resource amount is greater than a third resource threshold, when decoding the current video frame in the video frame queue, preset image processing is performed in parallel on all areas of the decoded video frame, wherein the third resource threshold is greater than the second resource threshold.

[0083] In an embodiment of the present application, when the preset image processing is turned on, the remaining resource amount can also be compared with a third resource threshold, wherein the third resource threshold is a condition for determining whether to perform the preset image processing on the entire area of ​​the video frame, and the third resource threshold is greater than the second resource threshold. If, according to the comparison result, it is determined that the remaining resource amount is greater than the third resource threshold, it means that the preset image processing can be smoothly performed on the entire area of ​​the video frame. At this time, when the current video frame in the video frame queue is decoded, the preset image processing can be performed in parallel on the entire area of ​​the decoded video frame.

[0084] Step S550: If the remaining resource amount is less than or equal to the third resource threshold, when decoding the current video frame in the video frame queue, performing preset image processing on a partial area of ​​the decoded video frame in parallel.

[0085] If it is determined according to the comparison result that the remaining amount of resources is not greater than the third resource threshold, it means that the preset image processing cannot be smoothly performed on the entire area of ​​the video frame. At this time, when decoding the current video frame in the video frame queue, the preset image processing can be performed in parallel on part of the decoded video frame.

[0086] In some embodiments, the target area of ​​each video frame may be subjected to display enhancement processing. The target area may be an area of ​​a target size at the center position of each video frame, or a certain number of evenly distributed areas of the same size, or a partial area in any half area. Of course, the specific target area is not limited in the embodiments of the present application.

[0087] Step S560: display and play the video frame after the preset image processing.

[0088] The video playback method provided in the embodiment of the present application performs preset image processing on the video frame while decoding the subsequent video frame. In this way, when it is really necessary to perform preset image processing on the subsequent video frame for display and playback, the decoding will not occupy the time of preset image processing and cause high power consumption, thus saving the power consumption of the electronic device and avoiding video playback jams. In addition, whether to start the preset image processing and whether to perform the preset image processing on the entire area of ​​the video frame is controlled according to the operation status of the processor, further avoiding video playback jams.

[0089] See also Fig. 9 , Fig. 9 FIG. 1 is a flow chart showing a video playback method provided by another embodiment of the present application. The video playback method is applied to the above electronic device. Fig. 9 The process shown is described in detail, and the video playing method may specifically include the following steps:

[0090] Step S610: obtaining a video frame queue of a video to be processed, wherein the video frame queue includes a plurality of continuous video frames.

[0091] Step S620: Obtain the remaining resources of the processor.

[0092] Step S630: If the remaining resource amount is less than a fourth resource threshold, a hard decoding method is used to decode the current video frame in the video frame queue, and preset image processing is performed on the decoded video frame in parallel.

[0093] The decoding of video frames can include hardware decoding and software decoding. Hardware decoding is to transfer part of the video data that was originally processed by the central processing unit (CPU) to other devices (such as GPU), which can greatly reduce the load on the CPU. When the CPU occupancy rate is low, some other programs can be run at the same time. Of course, for processors with better performance, software decoding can also be used, that is, decoding is performed by the decoding software program in the CPU.

[0094] In an embodiment of the present application, after obtaining the remaining resources of the processor, the remaining resources can be compared with the fourth resource threshold, wherein the fourth resource threshold is used as a condition for whether soft decoding can be smoothly performed and preset image processing can be performed on the video frame. If the remaining resources are less than the fourth resource threshold, it means that the decoding is performed by soft decoding and the preset image processing is performed on the video frame, which cannot run smoothly. Therefore, the hard decoding method can be used to decode the current video frame in the video frame queue, and the preset image processing can be performed on the decoded video frame in parallel.

[0095] Step S640: If the remaining resource amount is greater than or equal to the fourth resource threshold, the current video frame in the video frame queue is decoded by soft decoding, and preset image processing is performed on the decoded video frame in parallel.

[0096] In an embodiment of the present application, after comparing the remaining resource amount with the fourth resource threshold, if the remaining resource amount is less than the fourth resource threshold, it means that soft decoding is used for decoding, and preset image processing is performed on the video frame, which can run smoothly. Therefore, soft decoding can be used to decode the current video frame in the video frame queue, and preset image processing can be performed on the decoded video frame in parallel.

[0097] Step S650: displaying and playing the video frame after the preset image processing.

[0098] The video playback method provided by the embodiment of the present application performs preset image processing on the video frame and decodes the subsequent video frame at the same time. In this way, when it is really necessary to perform preset image processing on the subsequent video frame for display and playback, the decoding will not occupy the time of preset image processing and cause high power consumption, thus saving the power consumption of the electronic device and avoiding video playback jams. In addition, the decoding method is controlled according to the operation of the processor, further avoiding video playback jams.

[0099] See also Fig.10, which shows a structural block diagram of a video playback device 400 provided in an embodiment of the present application. The video playback device 400 applies the above-mentioned electronic device, and the video playback device 400 includes: a frame queue acquisition module 410, an image processing module 420 and a display module 430. Among them, the frame queue acquisition module 410 is used to obtain a video frame queue of a video to be processed, and the video frame queue includes a continuous multi-frame image; the image processing module 420 is used to perform preset image processing on the decoded video frame in parallel when decoding the current video frame in the video frame queue; the display module 430 is used to display and play the video frame after the preset image processing.

[0100] In some embodiments, when the image processing module 420 decodes the current video frame in the video frame queue, the image processing module 420 performs preset image processing on the decoded video frames in parallel, and the display module 430 displays and plays the video frames after the preset image processing. This may include: when the image processing module 420 decodes the Mth video frame in the video frame queue, the image processing module 420 performs preset image processing on the first N frames of the Mth video frame in parallel, and the display module 430 displays and plays the video frames after the preset image processing in parallel, where M is a positive integer greater than 2, and N is a positive integer less than M.

[0101] As an implementation manner, when the image processing module 420 decodes the Mth video frame in the video frame queue, the image processing module 420 performs preset image processing on the first N frames of the Mth video frame in parallel, and the display module 430 displays and plays the video frames after the preset image processing in parallel. It may include: when the image processing module 420 decodes the Mth video frame in the video frame queue, the image processing module 420 performs preset image processing on the (M-1)th video frame in parallel, and the display module 430 displays and plays the (M-2)th video frame in parallel.

[0102] As another implementation, when the image processing module 420 decodes the Mth video frame in the video frame queue, the image processing module 420 performs preset image processing on the first N frames of the Mth video frame in parallel, and the display module 430 displays and plays the video frames after the preset image processing in parallel. It may include: when the image processing module 420 decodes the Mth video frame in the video frame queue, the image processing module 420 performs preset image processing on the (MK)th video frame to the (M-1)th video frame in parallel, and the display module 430 displays and plays the (MK-1)th video frame in parallel, wherein K is a positive integer greater than 1.

[0103] Optionally, the image processing module 420 performs preset image processing on the (MK)th video frame to the (M-1)th video frame in parallel, which may include: the image processing module 420 uses different asynchronous threads to perform preset image processing on the (MK)th video frame to the (M-1)th video frame in parallel.

[0104] In some embodiments, the video playback device 400 may further include: a first remaining resource acquisition module and a frequency reduction module. The first remaining resource acquisition module is used to obtain the remaining resource amount of the processor; the frequency reduction module is used to reduce the operating frequency of the processor if the remaining resource amount is greater than the first resource threshold.

[0105] In some embodiments, the video playback device 400 may further include: a second remaining resource acquisition module and a function activation module. The second remaining resource acquisition module is used to acquire the remaining resource amount of the processor; the function activation module is used to activate the preset image processing if the remaining resource amount is greater than the second resource threshold.

[0106] Optionally, the image processing module 420 can be specifically used to: if the remaining resource amount is greater than a third resource threshold, perform preset image processing in parallel on the entire area of ​​the decoded video frame, wherein the third resource threshold is greater than the second resource threshold; if the remaining resource amount is less than or equal to the third resource threshold, perform preset image processing in parallel on a partial area of ​​the decoded video frame.

[0107] In some embodiments, the image processing module 420 can be specifically used to: obtain the remaining resources of the processor; if the remaining resources are less than a fourth resource threshold, use hard decoding to decode the current video frame in the video frame queue, and perform preset image processing on the decoded video frame in parallel; if the remaining resources are greater than or equal to the fourth resource threshold, use soft decoding to decode the current video frame in the video frame queue, and perform preset image processing on the decoded video frame in parallel.

[0108] In some embodiments, the preset image processing includes any one or more of the following: super-resolution reconstruction, contrast enhancement, saturation enhancement, image denoising, image defogging, and image deblurring.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0110] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0111] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0112] To summarize, the solution provided by the present application obtains a video frame queue of the video to be processed, which includes multiple continuous video frames. When decoding the current video frame in the video frame queue, preset image processing is performed on the decoded video frames in parallel, and the video frames after the preset image processing are displayed and played, thereby achieving preset image processing on other frames in parallel when decoding the video frames, avoiding time squeezed by decoding and preset image processing, and thus reducing power consumption caused by more calculations required for preset image processing of the video frames. At the same time, it also avoids video playback freezes when the processing volume of the preset image processing is large.

[0113] Please refer to Fig.11 , which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, a smart watch, a laptop computer, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0114] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0115] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0116] Please refer to Fig.12 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0117] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video playback method, characterized in that: The method comprises: Obtain a video frame queue of a video to be processed, wherein the video frame queue includes a plurality of continuous video frames; When decoding the Mth video frame in the video frame queue, performing preset image processing on the (MK)th to (M-1)th video frames in parallel, and displaying and playing the (MK-1)th video frame in parallel, wherein K is a positive integer greater than 1, M is a positive integer greater than 2, and the preset image processing includes super-resolution reconstruction; The step of performing preset image processing on the (MK)th video frame to the (M-1)th video frame in parallel includes: Different asynchronous threads are used to perform preset image processing on the (MK)th frame to the (M-1)th frame of video in parallel, each thread uses multiple devices to complete different steps in the preset image processing, each of the multiple devices is used to execute different steps in the preset image processing, different color components of the video frame use different devices when performing the preset image processing, and different threads use different devices; wherein, when the preset image processing is performed on the video frame, super-resolution reconstruction is performed on the color component Y of the video frame, bilinear interpolation is performed on the color components U and V of the video frame, and each processing step of super-resolution reconstruction of the color component Y is completed by an embedded neural network processor NPU, a central processing unit CPU and a digital signal processor DSP, bilinear interpolation of the color component U is completed by a display processor DPU, and bilinear interpolation of the color component V is completed by a graphics processor GPU.

2. The method according to claim 1, characterized in that: The method further comprises: Get the remaining resources of the processor; If the remaining resource amount is greater than a first resource threshold, the operating frequency of the processor is reduced.

3. The method according to claim 1, characterized in that When decoding the current video frame in the video frame queue, before performing preset image processing on the decoded video frames in parallel, the method further includes: Get the remaining resources of the processor; If the remaining resource amount is greater than a second resource threshold, the preset image processing is started.

4. The method according to claim 3, characterized in that The performing preset image processing on the decoded video frames in parallel includes: If the remaining resource amount is greater than a third resource threshold, performing preset image processing on all areas of the decoded video frame in parallel, wherein the third resource threshold is greater than the second resource threshold; If the remaining resource amount is less than or equal to the third resource threshold, the preset image processing is performed in parallel on a partial area of ​​the decoded video frame.

5. The method according to claim 1, characterized in that When decoding the current video frame in the video frame queue, performing preset image processing on the decoded video frame in parallel, including: Get the remaining resources of the processor; If the remaining resource amount is less than a fourth resource threshold, the current video frame in the video frame queue is decoded by hard decoding, and preset image processing is performed on the decoded video frame in parallel; If the remaining resource amount is greater than or equal to a fourth resource threshold, a soft decoding method is used to decode the current video frame in the video frame queue, and preset image processing is performed on the decoded video frame in parallel.

6. The method according to claim 1, characterized in that The preset image processing also includes any one or more of the following: contrast enhancement, saturation enhancement, image denoising, image defogging and image deblurring.

7. A video playback device, characterized in that: The device comprises: a frame queue acquisition module, an image processing module and a display module, wherein: The frame queue acquisition module is used to acquire a video frame queue of a video to be processed, wherein the video frame queue includes a continuous plurality of frames of images; The image processing module is used to perform preset image processing on the (MK)th to (M-1)th video frames in parallel when decoding the Mth video frame in the video frame queue, and the display module is used to display and play the (MK-1)th video frame in parallel, wherein K is a positive integer greater than 1, M is a positive integer greater than 2, and the preset image processing includes super-resolution reconstruction; The image processing module is also used to use different asynchronous threads to perform preset image processing on the (MK)th frame to the (M-1)th frame of video in parallel, wherein each thread uses multiple devices to complete different steps in the preset image processing, and each of the multiple devices is used to execute different steps in the preset image processing. Different color components of the video frame use different devices when performing the preset image processing, and different threads use different devices; wherein, when the preset image processing is performed on the video frame, super-resolution reconstruction is performed on the color component Y of the video frame, and bilinear interpolation is performed on the color components U and V of the video frame, and each processing step of super-resolution reconstruction of the color component Y is completed by an embedded neural network processor NPU, a central processing unit CPU and a digital signal processor DSP, bilinear interpolation of the color component U is completed by a display processor DPU, and bilinear interpolation of the color component V is completed by a graphics processor GPU.

8. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.