Terminal device and image super-resolution method

By splitting the image super-resolution processing on the terminal device into pipelined processing and configuring appropriate chunking methods and processing algorithms, the problem of high frame rate and low power consumption on the terminal device is solved, and efficient image super-resolution is achieved rate technology application.

WO2024230751A1PCT designated stage expired Publication Date: 2024-11-14HUAWEI TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/091786
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2024-05-08
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

The existing neural network-based image super-resolution technology is difficult to achieve high frame rate and low power consumption simultaneously on terminal devices, mainly due to the computing power limitations of the neural network processing unit and the graphics processing unit.

Method used

By splitting the image super-resolution processing process into pipelined processing, multiple sub-images in each frame are super-resolution processing in turn, and the complexity of the chunking method and super-resolution processing algorithm is reduced by scheduling modules. Running frequency and improving processing efficiency, combined with GPU post-processing to improve frame rate and image quality.

Benefits of technology

It achieves a balance between high frame rate and low power consumption on terminal devices, improves the efficiency and user experience of image super-resolution processing, and adapts to the image quality needs of different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024091786_14112024_PF_FP_ABST
    Figure CN2024091786_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application discloses a terminal device and an image super-resolution method. The terminal device comprises: a scheduling module used to issue multiple sub-images comprised by a current frame to a neural network processing unit (NPU), the current frame being a low-resolution image obtained after low-resolution rendering by a graphics processing unit (GPU); and an NPU used to perform super-resolution processing on the multiple sub-images in sequence, according to a preset sequence. By means of the present application, it is possible to simultaneously achieve the effects of high frame rate and low power consumption on a terminal device.
Need to check novelty before this filing date? Find Prior Art

Description

Terminal device and image super-resolution method

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 11, 2023, with application number 202310533602.2 and invention name “Terminal device and image super-resolution method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of chip technology, and in particular to a terminal device and an image super-resolution method. Background Art

[0003] Image super-resolution technology (also known as image super-resolution technology) is a technology that processes low-resolution images rendered by a graphics processing unit (GPU) into high-resolution images for display. This method usually reads low-resolution game images and some auxiliary information, and uses traditional interpolation algorithms or neural network algorithms to reconstruct high-resolution images. One of the use cases of image super-resolution technology is user gaming scenarios. It has been widely used on personal computers, such as NVIDIA's Deep Learning Super-Sampling (DLSS) technology and AMD's Fidelityfx Super Resolution (FSR) technology.

[0004] However, existing neural network-based image super-resolution technologies require high computing power from GPUs and neural-network processing units (NPUs), requiring users to configure independent graphics cards. Therefore, when adapting this technology to terminal devices, it is difficult to achieve both high frame rates (FPS) and low power consumption due to the computing power limitations of the NPUs and GPUs on these devices.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a terminal device and an image super-resolution method, which can simultaneously achieve high frame rate and low power consumption on the terminal device.

[0007] In the first aspect, the present application provides a terminal device, which includes: a scheduling module for sending multiple sub-images contained in a current frame to a neural network processor NPU, where the current frame is a low-resolution image obtained after low-resolution rendering by a graphics processor GPU; the NPU is used to perform super-resolution processing on the multiple sub-images in sequence according to a preset order.

[0008] From the technical effect point of view, the present application splits the super-resolution processing of each frame into super-resolution processing of each sub-image on the frame in turn (pipeline super-resolution processing). Compared with the super-resolution processing process based on frames in the prior art, the amount of tasks processed by the NPU at a single time in the present application is significantly smaller, so that the NPU can run at a lower frequency, that is, when the frame rate remains unchanged, the power consumption of the terminal device can be effectively reduced. In addition, if the NPU operating frequency is appropriately increased and / or a super-resolution processing algorithm of appropriate complexity is selected (that is, the algorithm complexity should not be too high), the speed of the super-resolution processing process can be increased, thereby achieving a high frame rate effect on the terminal device, and then achieving a higher frame rate and lower power consumption at the same time, that is, achieving a dynamic balance between frame rate and power consumption on the terminal device. That is, the present application can achieve the effects of high frame rate and low power consumption at the same time after the super-resolution technology is applied to the terminal device.

[0009] In a feasible embodiment, the scheduling module is also used to configure the blocking method of the current frame and the complexity of the super-resolution processing algorithm; wherein, the multiple sub-images are obtained after blocking processing according to the blocking method, and the image quality of the image obtained after the super-resolution processing is related to the complexity of the super-resolution processing algorithm.

[0010] The scheduling unit is a hardware module on the terminal device that is independent of the GPU and CPU, or a hardware module integrated into the GPU or CPU.

[0011] The block division method includes setting the size of the sub-image.

[0012] Among them, the configuration of the super-resolution processing algorithm can be displayed on the user interface as different image quality options provided by the graphics card program. For example, the high-quality image quality option in the user interface corresponds to a complex super-resolution processing algorithm, and the low-quality image quality option corresponds to a simple super-resolution processing algorithm.

[0013] From a technical perspective, using a hardware scheduling unit to schedule the super-resolution process significantly improves its efficiency and reduces latency. Furthermore, by configuring the block partitioning method to match the cache size on the terminal device, the efficiency of super-resolution processing is further improved. Furthermore, by providing super-resolution processing algorithms of varying complexity, the system meets user requirements for image quality in different scenarios, demonstrating excellent adaptability.

[0014] In a feasible embodiment, after each sub-image is subjected to the super-resolution processing, a high-resolution image corresponding to each sub-image is obtained; the GPU is used to post-process the high-resolution image corresponding to each sub-image in sequence according to the preset order to obtain a display image corresponding to each sub-image; wherein the post-processing includes halo effect, motion blur and depth of field effect.

[0015] In a feasible embodiment, after each sub-image undergoes the super-resolution processing, a weight matrix corresponding to each sub-image is obtained; the GPU is also used to sequentially fuse each sub-image, the sub-image corresponding to each sub-image in the previous frame, and the weight matrix corresponding to each sub-image, and post-process the fused high-resolution image to obtain a display image corresponding to each sub-image; wherein the post-processing includes halo effect, motion blur, and depth of field effect.

[0016] From a technical perspective, consistent with the pipelined super-resolution processing in the aforementioned embodiment, the GPU also uses a pipelined approach to perform post-processing on each sub-image within each frame, improving GPU processing efficiency and reducing latency. Furthermore, the result of the super-resolution processing can be either a weight matrix or a high-resolution sub-image. This means that the block-based super-resolution processing provided by this application can be flexibly adapted to different processing flows, offering excellent compatibility.

[0017] In a feasible embodiment, the multiple sub-images include a first sub-image, and the first sub-image corresponds to the second sub-image in the previous frame; the position of the first sub-image in the current frame is the same as the position of the second sub-image in the previous frame, and the previous frame is a high-resolution image obtained after the super-resolution processing and the post-processing.

[0018] From a technical effect perspective, in the above-mentioned process of fusing the first sub-image, the accuracy and quality of the fused image are improved by using the second sub-image corresponding to the position of the first sub-image in the previous frame.

[0019] In a feasible implementation manner, the terminal device further includes: a display configured to sequentially display a display image corresponding to each of the sub-images according to the preset order.

[0020] From the technical effect point of view, it is compatible with the pipelined super-resolution processing and pipelined post-processing process in the aforementioned embodiment, and the display unit also performs pipelined display, so that the current frame is displayed correctly and completely on the display unit, and the user is unaware of the entire block-by-block pipeline display process.

[0021] In a feasible implementation manner, the GPU is further configured to pre-process the current frame before the NPU performs the super-resolution processing on the current frame; wherein the pre-processing includes screen space reflection and screen space ambient light occlusion.

[0022] From a technical perspective, for the pre-processing process that cannot be performed in blocks, this application adjusts the pre-processing process to be performed before super-resolution processing, that is, the pre-processing process is performed in frames before super-resolution processing, so that the image finally displayed on the display unit has better picture quality and improves the user experience.

[0023] In a feasible implementation, after completing the low-resolution rendering of the current frame, the GPU starts the low-resolution rendering of the next frame; the priority of the low-resolution rendering is lower than the priority of the post-processing task.

[0024] From a technical perspective, by prioritizing post-processing tasks (also known as post-processing rendering) on ​​the GPU, the continuity of the super-resolution, post-processing, and display processes for each sub-image can be effectively improved. This shows that while this solution performs block-based super-resolution on the current frame, the processing of each sub-image remains fast and continuous, imperceptible to the user. This means that this application achieves high frame rates and low power consumption on the terminal device while effectively ensuring the user's visual experience.

[0025] In a feasible embodiment, the scheduling module is specifically used to send the first storage address of the first sub-image and the second storage address of the second sub-image to the NPU; the NPU is specifically used to obtain the first sub-image and the second sub-image according to the first storage address and the second storage address, respectively, and use the second sub-image to perform the super-resolution processing on the first sub-image to obtain a high-resolution image corresponding to the first sub-image or a corresponding weight matrix.

[0026] From a technical perspective, the scheduling module allocates the storage addresses of the data required for super-resolution processing, thereby improving the efficiency of the super-resolution processing process and reducing latency.

[0027] In a feasible implementation, the scheduling module is also used to send auxiliary parameters corresponding to each sub-image to the NPU, and the auxiliary parameters are used for the super-resolution processing of each sub-image, and the auxiliary parameters include motion vector map, depth map and global brightness information.

[0028] From a technical perspective, this application will also use auxiliary information such as motion vector maps during the super-resolution process to improve the accuracy of the processing results.

[0029] In a feasible implementation manner, any two adjacent sub-images among the multiple sub-images partially overlap or do not overlap.

[0030] From a technical perspective, the super-resolution process for the multiple sub-images obtained by dividing the current frame can be performed sequentially according to the position of each sub-image in the current frame, or in any other order, providing great flexibility. Furthermore, the edges of adjacent sub-images within the resulting multiple sub-images can overlap to a certain degree, allowing for appropriate cropping during display, thereby improving the fault tolerance of the super-resolution process.

[0031] In a feasible implementation manner, the NPU and the GPU are independent processing units or the NPU is integrated into the GPU.

[0032] From a technical perspective, the implementation of NPU and GPU on terminal devices is not restricted and has good flexibility.

[0033] In the second aspect, the present application provides an image super-resolution method, which includes: sending multiple sub-images contained in the current frame to the neural network processor NPU through a scheduling module, wherein the current frame is a low-resolution image obtained after low-resolution rendering by the graphics processor GPU; according to a preset order, the NPU performs super-resolution processing on the multiple sub-images in turn.

[0034] In a feasible embodiment, the method further includes: configuring the blocking method of the current frame and the complexity of the super-resolution processing algorithm through the scheduling module; wherein the multiple sub-images are obtained after blocking processing according to the blocking method, and the image quality of the image obtained after the super-resolution processing is related to the complexity of the super-resolution processing algorithm.

[0035] In a feasible embodiment, after each sub-image is subjected to the super-resolution processing, a high-resolution image corresponding to each sub-image is obtained; the method further includes: according to the preset order, the GPU sequentially post-processes the high-resolution image corresponding to each sub-image to obtain a display image corresponding to each sub-image; wherein the post-processing includes halo effect, motion blur and depth of field effect.

[0036] In a feasible embodiment, after each sub-image is subjected to the super-resolution processing, a weight matrix corresponding to each sub-image is obtained; the method further includes: according to the preset order, the GPU sequentially fuses each sub-image, the sub-image corresponding to each sub-image in the previous frame, and the weight matrix corresponding to each sub-image, and post-processes the fused high-resolution image to obtain a display image corresponding to each sub-image; wherein the post-processing includes halo effect, motion blur and depth of field effect.

[0037] In a feasible embodiment, the multiple sub-images include a first sub-image, and the first sub-image corresponds to the second sub-image in the previous frame; the position of the first sub-image in the current frame is the same as the position of the second sub-image in the previous frame, and the previous frame is a high-resolution image obtained after the super-resolution processing.

[0038] In a feasible implementation, the method further includes: displaying the display image corresponding to each sub-image in sequence on a display according to the preset sequence.

[0039] In a feasible implementation, the method further includes: before the NPU performs the super-resolution processing on the current frame, the GPU pre-processes the current frame; wherein the pre-processing includes screen space reflection and screen space ambient light occlusion.

[0040] In a feasible implementation, after completing the low-resolution rendering of the current frame, the GPU starts the low-resolution rendering of the next frame; the priority of the low-resolution rendering is lower than the priority of the post-processing.

[0041] In a feasible embodiment, the method also includes: the scheduling module sends auxiliary parameters corresponding to each sub-image to the NPU, the auxiliary parameters are used for the super-resolution processing of each sub-image, and the auxiliary parameters include motion vector map, depth map and global brightness information.

[0042] In a feasible implementation manner, any two adjacent sub-images among the multiple sub-images partially overlap or do not overlap.

[0043] In a feasible implementation manner, the NPU and the GPU are independent processing units or the NPU is integrated into the GPU.

[0044] In a third aspect, an embodiment of the present application provides a chip system, which includes at least one processor, a memory and an interface circuit, wherein the memory, the interface circuit and the at least one processor are interconnected through lines, and instructions are stored in the at least one memory; when the instructions are executed by the processor, any method described in the above second aspect is implemented.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the method described in any one of the above-mentioned second aspects is implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed, the method described in any one of the above second aspects is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The following is an introduction to the drawings used in the embodiments of this application.

[0048] FIG1 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

[0049] FIG2 is a specific implementation of a scheduling module provided in an embodiment of the present application;

[0050] Figures 3a and 3b are two specific examples of block division methods provided in the embodiments of the present application;

[0051] FIG4 is a schematic diagram of the structure of another terminal device provided in an embodiment of the present application;

[0052] FIG5 is a schematic structural diagram of another terminal device provided in an embodiment of the present application;

[0053] FIG6 is a schematic diagram of the execution flow of each module in a terminal device provided in an embodiment of the present application;

[0054] FIG7 is a schematic diagram of a process of obtaining an intermediate result when performing image super-resolution processing on a current frame according to an embodiment of the present application;

[0055] FIG8 is a schematic diagram of a data interaction process between modules in a terminal device provided by an embodiment of the present application;

[0056] FIG9 is a schematic diagram of the processing order of sub-images contained in each frame by various modules in a terminal device provided by an embodiment of the present application;

[0057] FIG10 is a flow chart of an image super-resolution method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents or, for example, A / B can represent A or B; "and / or" in the text is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0059] The terms "first", "second", "third" and "fourth" in the specification and claims of this application and the accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices. Reference to "embodiments" herein means that the specific features, structures or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] The following are the professional terms involved in this application

[0061] (1) Super Resolution Processing: It can also be referred to as super-resolution processing. It refers to a method of restoring a high-resolution image from a low-resolution image or image sequence. The current super-resolution methods include: interpolation-based methods, reconstruction-based methods, and learning-based (i.e., deep learning) methods. The super-resolution processing process in this application uses a deep learning method, and this application does not limit the deep learning model of super-resolution processing.

[0062] (2) Cache: A layer in a computer's memory system, located between main memory and the processor, designed to bridge the processing speed gap between the two. Compared to main memory, cache has faster access speeds but smaller capacity. Cache is typically divided into multiple layers, with the layers closer to the CPU having smaller capacity and faster access speeds.

[0063] Please refer to Figure 1, which is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. As shown in Figure 1, the terminal device includes a scheduling module and a neural network processor NPU.

[0064] The following describes the process of the terminal device performing the image super-resolution method in conjunction with FIG1 , which is as follows:

[0065] The scheduling module is used to send multiple sub-images contained in the current frame to the NPU. The current frame is a low-resolution image obtained after the GPU performs low-resolution rendering. The NPU performs super-resolution processing on these multiple sub-images in a preset order.

[0066] The scheduling module can be in the form of hardware or software. When the scheduling module is implemented as hardware, it can be an independent hardware module or a hardware module integrated into other hardware (such as a CPU or GPU, etc.). When the scheduling module is implemented as software, it can be in the form of a program or software thread in the operating system on the terminal device, which is not limited by this application.

[0067] Please refer to Figure 2, which shows a specific implementation of a scheduling module provided in an embodiment of the present application.

[0068] As shown in Figure 2, the scheduling module includes a configuration module and a task queue. The configuration module is used to configure super-resolution processing parameters, including the current frame's block division method and the complexity of the super-resolution algorithm. In other words, the scheduling module can be used to configure the current frame's block division method and the complexity of the super-resolution algorithm.

[0069] Optionally, the super-resolution processing parameters in the configuration module can be configured through game setting options in the user interface or GPU driver, etc.

[0070] After configuring the super-resolution processing parameters, the current frame can be segmented using the configured segmentation method to obtain multiple sub-images contained in the current frame. At the same time, when the NPU subsequently performs super-resolution processing on each sub-image, it calls the super-resolution processing algorithm configured in the configuration module to perform super-resolution processing.

[0071] The complexity of the super-resolution processing algorithm is related to the quality of the image obtained after the super-resolution processing. Optionally, the higher the complexity of the super-resolution processing algorithm, the better the quality of the image obtained after the super-resolution processing.

[0072] Furthermore, the super-resolution processing algorithm can optionally be divided into three levels: simple, medium, and complex, corresponding to three different levels of algorithm complexity. The user interface includes three image quality options: low, medium, and high. Low image quality corresponds to a simple super-resolution processing algorithm, medium image quality corresponds to a medium super-resolution processing algorithm, and high image quality corresponds to a complex super-resolution processing algorithm.

[0073] Further, optionally, the algorithm used for super-resolution processing is a neural network algorithm.

[0074] Optionally, the above-mentioned blocking method specifically includes setting the size of each sub-image obtained after blocking, or setting the shape of each sub-image obtained after blocking, etc., which is not limited in this application.

[0075] For example, the above-mentioned blocking method can be configured by setting the shape of each sub-image after blocking processing to be a square.

[0076] After configuring the above super-resolution processing parameters through the configuration module, the scheduling module will generate super-resolution processing tasks for each sub-image in the current frame in the task queue in a preset order based on parameters such as the size of each sub-image after block processing, and send the super-resolution processing tasks of each sub-image to the GPU in the order in the queue.

[0077] Specifically, the following takes the first sub-image in the current frame as an example to describe the process of the scheduling module sending the super-resolution processing task to the NPU:

[0078] The scheduling module sends the storage address of the first sub-image, the storage address of the second sub-image corresponding to the first sub-image in the previous frame, the storage address of the aforementioned super-resolution processing algorithm, and the storage address of the auxiliary parameters to the NPU.

[0079] The position of the first sub-image in the current frame is the same as or similar to the position of the second sub-image in the previous frame, and the previous frame is a high-resolution image obtained through super-resolution processing or super-resolution processing and post-processing.

[0080] The first sub-image is any sub-image among the multiple sub-images in the current frame.

[0081] After the NPU receives the super-resolution processing task for each sub-image sent by the scheduling module, the NPU performs super-resolution processing on each sub-image in sequence according to the preset order, that is, the NPU performs super-resolution processing on one sub-image at a time.

[0082] The preset order is also the order of the super-resolution processing tasks of each sub-image in the task queue of the scheduling module.

[0083] The following describes the specific process of each super-resolution processing performed by the NPU using the first sub-image as an example:

[0084] The NPU obtains the first sub-image, the second sub-image, auxiliary parameters, and the configured super-resolution processing algorithm through the storage address sent by the scheduling module, inputs the first sub-image, the second sub-image, and the auxiliary parameters into the algorithm, and performs super-resolution processing on the first sub-image through the second sub-image.

[0085] The auxiliary parameters include a motion vector map, a depth map, and global brightness information.

[0086] Optionally, for each sub-image in the current frame, the auxiliary parameters used in super-resolution processing are the motion vector map, depth map and global brightness information of the current frame, or the motion vector map, depth map and global brightness information of the sub-image.

[0087] Optionally, the terminal device may be a mobile terminal device such as a mobile phone, a personal computer, or a tablet, which is not limited in this application.

[0088] Optionally, any two sub-images among the multiple sub-images obtained after the current frame is subjected to block processing partially overlap or do not overlap.

[0089] FIG3a-FIG3b are two specific examples of block division methods provided in the embodiments of the present application.

[0090] Figure 3a is an example of a blocking method provided in an embodiment of the present application. Any two sub-images among the multiple sub-images obtained by this blocking method partially overlap. As shown in Figure 3a, the solid line border represents the size of the current frame, and the dotted line border represents the size of the current frame after the blocking process. After the blocking process, the current frame is divided into 9 sub-images (sub-images 1-9). The shaded portion in Figure 3a is the overlapping portion between adjacent sub-images, and the portion between the border and the dotted line border is the filled portion. Optionally, the pixel value of each pixel point in the filled portion can be 0.

[0091] Among them, sub-image 1 partially overlaps with adjacent sub-image 2, sub-image 4 and sub-image 5, and sub-image 5 partially overlaps with each of the remaining sub-images. The overlapping relationship between the remaining sub-images is not repeated here.

[0092] Figure 3b illustrates another example of a blocking method provided by an embodiment of the present application. This blocking method results in no overlap between any two sub-images in the resulting multiple sub-images. As shown in Figure 3b, using this blocking method, the current frame is divided into nine sub-images, and the borders of any two adjacent sub-images overlap, with no overlapping images.

[0093] Please refer to Figure 4, which is a schematic diagram of the structure of another terminal device provided in an embodiment of the present application, as a supplement to the specific structure of the terminal device shown in Figure 1. Compared with the terminal device shown in Figure 1, the terminal device shown in Figure 4 also includes a GPU, which is used to continue processing the results obtained after the NPU super-resolution processing.

[0094] Specifically, depending on the results obtained by the NPU super-resolution processing, the GPU's subsequent processing process is divided into two methods:

[0095] (1) When the result of super-resolution processing of each sub-image is a high-resolution image corresponding to each sub-image, the GPU performs post-processing on the high-resolution image corresponding to each sub-image in a predetermined order to obtain a display image corresponding to each sub-image. Specifically, taking the first sub-image as an example, after the NPU completes super-resolution processing on the first sub-image, the GPU begins post-processing the first sub-image to obtain a display image corresponding to the first sub-image.

[0096] (2) When the result of super-resolution processing of each sub-image is a weight matrix corresponding to each sub-image, the GPU processes each sub-image in the following order in a preset order: fuses each sub-image, the sub-image corresponding to each sub-image in the previous frame, and the weight matrix corresponding to each sub-image to obtain a high-resolution image corresponding to each sub-image; then the GPU post-processes the high-resolution image corresponding to each sub-image to obtain a display image corresponding to each sub-image. Specifically, taking the first sub-image mentioned above as an example, the GPU fuses the first sub-image, the second sub-image, and the weight matrix corresponding to the first sub-image to obtain a high-resolution image corresponding to the first sub-image, and then post-processes the high-resolution image corresponding to the first sub-image to obtain a display image corresponding to the first sub-image.

[0097] Among them, the above post-processing includes halo effect, motion blur and depth of field effect.

[0098] Please refer to Figure 5, which is a schematic diagram of the structure of another terminal device provided in an embodiment of the present application, as a supplement to the specific structure of the terminal device shown in Figure 4. Compared with the terminal device shown in Figure 4, the terminal device shown in Figure 5 further includes a display.

[0099] After the GPU performs post-processing on each sub-image in sequence to obtain a display image corresponding to each sub-image, the display sequentially displays the display image corresponding to each sub-image.

[0100] For the terminal devices shown in FIG1, FIG4 and FIG5, a storage unit and a cache unit are also included.

[0101] The storage unit is used to store the current frame and the auxiliary parameters, and is a memory on the terminal device. Optionally, the storage unit is a dynamic random access memory (DRAM).

[0102] The cache unit is used to store the sub-images being super-resolution processed, auxiliary parameters, and the high-resolution image or weight matrix obtained after the sub-images are super-resolution processed. It is a cache on the terminal device. Optionally, the cache unit is a static random access memory (SRAM), etc.

[0103] Optionally, before the NPU sequentially performs super-resolution processing on the sub-images included in the current frame, the GPU pre-processes the current frame.

[0104] Specifically, the GPU preprocesses the current frame as a whole. Optionally, the preprocessing includes screen space reflections and screen space ambient occlusion.

[0105] In the above embodiments, after the GPU finishes low-resolution rendering of the current frame, the GPU starts low-resolution rendering of the next frame. The low-resolution rendering is performed as a whole in units of frames.

[0106] Among them, the priority of low-resolution rendering in the GPU is lower than the priority of the aforementioned post-processing. Specifically, when the GPU is performing low-resolution rendering of the next frame, whenever the NPU completes super-resolution processing of a sub-image in the current frame, the GPU will pause the low-resolution rendering of the next frame and start post-processing of this sub-image. After the post-processing is completed, the GPU starts to continue the low-resolution rendering of the next frame.

[0107] Optionally, the NPU and GPU in the above embodiments are independent processing units or the NPU is integrated into the GPU. When the NPU is integrated into the GPU, the processing of the GPU in the above embodiments is implemented by the part of the GPU other than the NPU.

[0108] Please refer to Figure 6, which is a schematic diagram of the execution flow of each module in a terminal device provided in an embodiment of the present application, and is described using the first sub-image and the last sub-image in the current frame as an example. The first sub-image and the last sub-image refer to the first sub-image and the last sub-image to be super-resolution processed in the current frame, respectively.

[0109] As shown in Figure 6, the scheduling module first executes step S610: issuing a low-resolution rendering task. Specifically, the scheduling module may send the storage address of the current frame to the GPU. After receiving the storage address, the GPU performs low-resolution rendering on the current frame and, upon completion of the task, executes step S620: notifying the completion of rendering. After receiving this notification, the scheduling module executes step S630: issuing a first sub-image super-resolution processing task. After receiving this task, the NPU begins super-resolution processing on the first sub-image. After completing super-resolution processing on the first sub-image, the NPU executes step S640: notifying the completion of the super-resolution task. After receiving this notification, the scheduling module executes step S650: issuing a first sub-image post-processing task. After receiving this task, the GPU begins post-processing the super-resolution processing result of the first sub-image. After completing post-processing, the GPU executes step S660: notifying the completion of post-processing. After receiving this notification, the scheduling module executes step S670: outputting the rendering result of the first sub-image for display. This rendering result is the display image in the aforementioned embodiment. The remaining sub-images are then processed sequentially according to the process described above for the first sub-image (steps S630 to S670). For the last sub-image in the current frame, the process is identical to that for the first sub-image, as shown in steps S680 to S6120 in FIG6 , and will not be further described here.

[0110] Specifically, the specific execution process of each step in Figure 6 can be found in the description of the above embodiment, and will not be repeated here.

[0111] Meanwhile, the process of configuring the super-resolution processing parameters in the current frame pre-processing and scheduling module is omitted in the embodiment of FIG6 . For details, please refer to the description in the aforementioned embodiment, which will not be repeated here.

[0112] Please refer to Figure 7, which is a schematic diagram of the process of obtaining an intermediate result when performing image super-resolution processing on the current frame provided by an embodiment of the present application.

[0113] As shown in FIG7 , the image super-resolution processing process of the current frame includes:

[0114] First, the current frame is rendered at low resolution, pre-processed, and divided into blocks to obtain the current frame after division (low-resolution image). As shown in Figure 7, the current frame after division includes four sub-images (sub-images ①-④). Figure 7 takes sub-image ① in the current frame as an example to describe its super-resolution processing and post-processing process. The processing of the remaining sub-images in the current frame is the same and is therefore omitted. Among them, the previous frame also contains four sub-images, and the positions of the sub-images with the same number in the previous frame and the current frame are the same (that is, the position of sub-image ① in the previous frame in the previous frame is the same as the position of sub-image ① in the current frame in the current frame, ... the position of sub-image ④ in the previous frame in the previous frame is the same as the position of sub-image ④ in the current frame in the current frame).

[0115] The super-resolution processing and post-processing of sub-image ① in the current frame are as follows: sub-image ① in the previous frame is super-resolution processed on sub-image ① in the current frame to obtain a processing result (as can be seen from the previous embodiment, this processing result is a high-resolution image or weight matrix corresponding to sub-image ① in the current frame). The GPU then post-processes this processing result to obtain a display image corresponding to sub-image ①, and finally outputs this display image to a display for display.

[0116] Specifically, the specific implementation of the above processing process can be found in the description of the above embodiments, which will not be repeated here.

[0117] Please refer to Figure 8, which is a schematic diagram of the data interaction process between modules in a terminal device provided by an embodiment of the present application, and is described by taking the current frame and any sub-image i in the current frame as an example.

[0118] As shown in Figure 8, the scheduling module executes step ①: notifies the current frame to be rendered at low resolution, pre-processed, and segmented. After receiving the notification, the NPU performs low-resolution rendering, pre-processing, and segmentation on the current frame to obtain the segmented current frame, and then executes step ②: stores the segmented current frame and auxiliary parameters. The scheduling module executes step ③: issues the super-resolution task of sub-image i. After receiving the task, the NPU first reads sub-image i and auxiliary parameters from the storage unit and stores them in the cache unit, and then executes step ④: obtains sub-image i and auxiliary parameters. The NPU uses sub-image i, auxiliary parameters, and the sub-image corresponding to sub-image i in the previous frame (not shown in Figure 8) to perform super-resolution processing to obtain the processing result of sub-image i (the high-resolution image or weight matrix corresponding to sub-image i). The NPU executes step ⑤: stores the high-resolution image or weight matrix corresponding to sub-image i. The GPU executes step ⑥: obtains the high-resolution image or weight matrix corresponding to sub-image i. Then, the GPU performs post-processing on sub-image i based on the acquired data to obtain a display image corresponding to sub-image i, and executes step 7: outputting the display image corresponding to sub-image i. Finally, the display displays the display image corresponding to sub-image i.

[0119] Specifically, the specific implementation process of the above steps can be found in the description of the above embodiments, which will not be repeated here.

[0120] Please refer to Figure 9, which is a schematic diagram of the processing order of the sub-images contained in each frame by each module in a terminal device provided in an embodiment of the present application.

[0121] Figure 9 uses four consecutive frames (i.e., frames A, B, C, and D) as the objects to illustrate the processing order of each module on the terminal device. Frames A, B, C, and D are the nth, n+1th, n+2th, and n+3th frames in the time sequence, respectively. After block processing, each frame contains three sub-images. As shown in Figure 9, frame A contains sub-images A1-A3, frame B contains sub-images B1-B3, frame C contains sub-images C1-C3, and frame D contains sub-images D1-D3.

[0122] As shown in Figure 9 , the NPU first performs super-resolution processing on the three sub-images contained in frame A in sequence. As shown in Figure 9 , the processing order is sub-image A1, sub-image A2, and sub-image A3. Figure 9 omits the GPU's pre-processing of frame A. While the NPU performs super-resolution processing on sub-images A1-A3 in sequence, the GPU pre-processes the next frame after frame A (i.e., frame B). As shown in Figure 9 , after super-resolution processing of sub-image A1 is completed, the GPU pauses pre-processing of frame B and begins post-processing sub-image A1 (i.e., the two processing methods corresponding to the different super-resolution processing results in the aforementioned embodiments are not further described here). The time required for post-processing sub-image A1 is indicated by the dotted line in the pre-processing sequence of frame B. After post-processing of sub-image A1 is completed, the GPU continues pre-processing of frame B. Simultaneously, as shown in Figure 9 , after post-processing of sub-image A1 is completed, the display begins displaying the image corresponding to sub-image A1. Similarly, the NPU, GPU, and display sequentially perform the above processing on sub-images A2 and A3 according to the above processing on sub-image A1.

[0123] As shown in Figure 9, the above-mentioned processing process is expressed in terms of timing as follows: the NPU performs super-resolution processing on the sub-image in each frame in sequence according to a preset order, the GPU performs post-processing on the sub-image in each frame in sequence according to a preset order, and the display displays the display image corresponding to the sub-image in each frame in sequence according to a preset order. For each sub-image, a pipeline processing process is formed from the NPU to the GPU to the display. In this way, the workload of each module on the terminal device is reduced, so that each module can run at a low frequency, thereby reducing the power consumption of the terminal device. Under this low-frequency working mode, the operating frequency of each module can be appropriately increased, which can improve the processing speed and efficiency of each module, thereby achieving the effect of increasing the frame rate. That is, the technical effects of low power consumption and high frame rate can be achieved at the same time through this application.

[0124] After the above processing of frame A is completed, each module on the terminal device starts to perform corresponding processing on frame B, frame C and frame D in sequence. For details, please refer to Figure 9 and will not be repeated here.

[0125] Please refer to Figure 10, which is a flow chart of an image super-resolution method provided by an embodiment of the present application. As shown in Figure 9, the method includes step S1010 and step S1020.

[0126] Step S1010: Send multiple sub-images contained in the current frame to the neural network processor NPU through the scheduling module, and the current frame is a low-resolution image obtained after low-resolution rendering by the graphics processor GPU.

[0127] Step S1020: The NPU performs super-resolution processing on the multiple sub-images in sequence according to a preset order.

[0128] In a feasible embodiment, the method further includes: configuring the blocking method of the current frame and the complexity of the super-resolution processing algorithm through the scheduling module; wherein the multiple sub-images are obtained after blocking processing according to the blocking method, and the image quality of the image obtained after the super-resolution processing is related to the complexity of the super-resolution processing algorithm.

[0129] In a feasible embodiment, after each sub-image is subjected to the super-resolution processing, a high-resolution image corresponding to each sub-image is obtained; the method further includes: according to the preset order, the GPU sequentially post-processes the high-resolution image corresponding to each sub-image to obtain a display image corresponding to each sub-image; wherein the post-processing includes halo effect, motion blur and depth of field effect.

[0130] In a feasible embodiment, after each sub-image is subjected to the super-resolution processing, a weight matrix corresponding to each sub-image is obtained; the method further includes: according to the preset order, the GPU sequentially fuses each sub-image, the sub-image corresponding to each sub-image in the previous frame, and the weight matrix corresponding to each sub-image, and post-processes the fused high-resolution image to obtain a display image corresponding to each sub-image; wherein the post-processing includes halo effect, motion blur and depth of field effect.

[0131] In a feasible embodiment, the multiple sub-images include a first sub-image, and the first sub-image corresponds to the second sub-image in the previous frame; the position of the first sub-image in the current frame is the same as the position of the second sub-image in the previous frame, and the previous frame is a high-resolution image obtained after the super-resolution processing.

[0132] In a feasible implementation, the method further includes: displaying the display image corresponding to each sub-image in sequence on a display according to the preset sequence.

[0133] In a feasible implementation, the method further includes: before the NPU performs the super-resolution processing on the current frame, the GPU pre-processes the current frame; wherein the pre-processing includes screen space reflection and screen space ambient light occlusion.

[0134] In a feasible implementation, after completing the low-resolution rendering of the current frame, the GPU starts the low-resolution rendering of the next frame; the priority of the low-resolution rendering is lower than the priority of the post-processing.

[0135] In a feasible embodiment, the method also includes: the scheduling module sends auxiliary parameters corresponding to each sub-image to the NPU, the auxiliary parameters are used for the super-resolution processing of each sub-image, and the auxiliary parameters include motion vector map, depth map and global brightness information.

[0136] In a feasible implementation manner, any two adjacent sub-images among the multiple sub-images partially overlap or do not overlap.

[0137] In a feasible implementation manner, the NPU and the GPU are independent processing units or the NPU is integrated into the GPU.

[0138] Specifically, the specific execution process of each step in the embodiment of Figure 10 above can be found in the description of the corresponding embodiment above, and will not be repeated here.

[0139] An embodiment of the present application provides a chip system, which includes at least one processor, a memory and an interface circuit. The memory, the interface circuit and the at least one processor are interconnected by lines, and instructions are stored in the at least one memory. When the instructions are executed by the processor, some or all of the steps of any one of the above-mentioned image super-resolution methods are implemented.

[0140] An embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed, part or all of the steps of any one of the above-mentioned image super-resolution methods can be implemented.

[0141] An embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor, some or all of the steps of any one of the above-mentioned image super-resolution methods are implemented.

[0142] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be noted that for the aforementioned method embodiments, in order to simplify the description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited to the order of the actions described, because according to this application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0144] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A terminal device, characterized in that: The terminal device comprises: A scheduling module is used to send multiple sub-images contained in the current frame to the neural network processor NPU, where the current frame is a low-resolution image obtained after low-resolution rendering by the graphics processor GPU; The NPU is used to perform super-resolution processing on the multiple sub-images in sequence according to a preset order.

2. The device according to claim 1, characterized in that The scheduling module is also used for: Configuring a block division method for the current frame and the complexity of a super-resolution processing algorithm; The plurality of sub-images are obtained after block processing according to the block processing method, and the image quality of the image obtained after the super-resolution processing is related to the complexity of the super-resolution processing algorithm.

3. The device according to claim 1 or 2, characterized in that After each of the sub-images is subjected to the super-resolution processing, a high-resolution image corresponding to each of the sub-images is obtained; the GPU is used for: According to the preset sequence, post-processing is performed on the high-resolution image corresponding to each of the sub-images in turn to obtain a display image corresponding to each of the sub-images; The post-processing includes halo effect, motion blur and depth of field effect.

4. The device according to claim 1 or 2, characterized in that After each of the sub-images is subjected to the super-resolution processing, a weight matrix corresponding to each of the sub-images is obtained; and the GPU is further used for: According to the preset order, each of the sub-images, the sub-image corresponding to each of the sub-images in the previous frame, and the weight matrix corresponding to each of the sub-images are sequentially fused, and the fused high-resolution image is post-processed to obtain a display image corresponding to each of the sub-images; The post-processing includes halo effect, motion blur and depth of field effect.

5. The device according to claim 4, characterized in that The multiple sub-images include a first sub-image, and the first sub-image corresponds to the second sub-image in the previous frame; The position of the first sub-image in the current frame is the same as the position of the second sub-image in the previous frame, and the previous frame is a high-resolution image obtained after the super-resolution processing.

6. The device according to any one of claims 1 to 5, characterized in that The terminal device further includes: A display is used to display the display image corresponding to each of the sub-images in sequence according to the preset order.

7. The device according to any one of claims 1 to 6, characterized in that The GPU is also used to: Before the NPU performs the super-resolution processing on the current frame, pre-processing the current frame; The preprocessing includes screen space reflection and screen space ambient occlusion.

8. The device according to any one of claims 3 to 7, characterized in that After completing the low-resolution rendering of the current frame, the GPU starts to perform the low-resolution rendering on the next frame; The priority of the low-resolution rendering is lower than the priority of the post-processing.

9. The device according to any one of claims 1 to 8, characterized in that The scheduling module is also used for: The auxiliary parameters corresponding to each of the sub-images are sent to the NPU, where the auxiliary parameters are used for the super-resolution processing of each of the sub-images, and the auxiliary parameters include a motion vector map, a depth map, and global brightness information.

10. The device according to any one of claims 1 to 9, characterized in that Any two adjacent sub-images among the multiple sub-images partially overlap or do not overlap.

11. The device according to any one of claims 1 to 10, characterized in that The NPU and the GPU are independent processing units or the NPU is integrated in the GPU.

12. An image super-resolution method, characterized in that: The method comprises: Sending multiple sub-images contained in the current frame to the neural network processor NPU through the scheduling module, wherein the current frame is a low-resolution image obtained after low-resolution rendering by the graphics processor GPU; According to a preset order, the NPU performs super-resolution processing on the multiple sub-images in sequence.

13. The method according to claim 12, characterized in that The method further comprises: The scheduling module configures the block division method of the current frame and the complexity of the super-resolution processing algorithm; The plurality of sub-images are obtained after block processing according to the block processing method, and the image quality of the image obtained after the super-resolution processing is related to the complexity of the super-resolution processing algorithm.

14. The method according to claim 12 or 13, characterized in that After each of the sub-images is subjected to the super-resolution processing, a high-resolution image corresponding to each of the sub-images is obtained; the method further comprises: According to the preset sequence, the GPU sequentially post-processes the high-resolution image corresponding to each of the sub-images to obtain a display image corresponding to each of the sub-images; The post-processing includes halo effect, motion blur and depth of field effect.

15. The method according to claim 12 or 13, characterized in that After each of the sub-images is subjected to the super-resolution processing, a weight matrix corresponding to each of the sub-images is obtained; the method further comprises: According to the preset order, the GPU sequentially fuses each of the sub-images, the sub-image corresponding to each of the sub-images in the previous frame, and the weight matrix corresponding to each of the sub-images, and post-processes the fused high-resolution image to obtain a display image corresponding to each of the sub-images; The post-processing includes halo effect, motion blur and depth of field effect.

16. The method according to claim 15, characterized in that The multiple sub-images include a first sub-image, and the first sub-image corresponds to the second sub-image in the previous frame; The position of the first sub-image in the current frame is the same as the position of the second sub-image in the previous frame, and the previous frame is a high-resolution image obtained after the super-resolution processing.

17. The method according to any one of claims 12 to 16, characterized in that: The method further comprises: According to the preset sequence, the display image corresponding to each of the sub-images is displayed in sequence by the display.

18. The method according to any one of claims 12 to 16, characterized in that: The method further comprises: Before the NPU performs the super-resolution processing on the current frame, the GPU performs pre-processing on the current frame; The preprocessing includes screen space reflection and screen space ambient occlusion.

19. The method according to any one of claims 14 to 18, characterized in that After completing the low-resolution rendering of the current frame, the GPU starts to perform the low-resolution rendering on the next frame; The priority of the low-resolution rendering is lower than the priority of the post-processing.

20. The method according to any one of claims 12 to 19, characterized in that The method further comprises: The scheduling module sends auxiliary parameters corresponding to each of the sub-images to the NPU, where the auxiliary parameters are used for the super-resolution processing of each of the sub-images, and the auxiliary parameters include a motion vector map, a depth map, and global brightness information.

21. The method according to any one of claims 12 to 20, characterized in that: Any two adjacent sub-images among the multiple sub-images partially overlap or do not overlap.

22. The method according to any one of claims 12 to 21, characterized in that The NPU and the GPU are independent processing units or the NPU is integrated in the GPU.

23. A chip system, characterized in that: The chip system includes at least one processor, a memory and an interface circuit. The memory, the interface circuit and the at least one processor are interconnected by lines. Instructions are stored in the at least one memory. When the instructions are executed by the processor, any method described in claims 12-22 is implemented.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method described in any one of claims 12 to 22 is implemented.

25. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are executed, the method according to any one of claims 12 to 22 is implemented.

Citation Information

Patent Citations

  • Terminal device and image super-resolution method

    CN118967439A

  • Super-resolution method and device, terminal equipment and storage medium

    CN112862681A

  • Image processing method, electronic device and readable medium thereof

    CN113298712A

  • Method and system for improving fluency of cloud game

    CN113842635A

  • Lightweight video super-resolution reconstruction system and method based on neural network

    CN114331846A