Coloring method, electronic equipment and electronic chip

Abstract

The embodiment of the invention provides a coloring method, electronic equipment and an electronic chip. The method comprises the steps that a first coloring rate texture map for a first image is acquired, the first image is an image generated by variable-resolution coloring rendering according to current image frame data, and the resolution of the first image is lower than the rendering target resolution of an image frame; and performing artificial intelligence super-division on the first image based on the first coloring rate texture map, improving the resolution of the first image, and generating a second image meeting the rendering target resolution. According to the coloring method provided by the embodiment of the invention, image coloring rendering is performed in combination with variable-resolution coloring and artificial intelligence super-resolution, and the fuzzy region in the variable-resolution coloring result is repaired by adopting the artificial intelligence super-resolution, so that the image rendering quality is improved, and the image rendering overhead is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a coloring method, an electronic device, and an electronic chip. Background Art

[0002] With the advancement of real-time rendering, image quality, resolution, and frame rate have continued to improve year by year, placing increasing pressure on graphics processing units (GPUs). Shading is a key step in real-time rendering that consumes significant processing resources. Therefore, a new shading approach is needed to reduce the pressure on GPUs. Summary of the Invention

[0003] In order to solve the problem of how to reduce the pressure of GPU rendering, the present application provides a shading method, an electronic device and an electronic chip. The present application also provides a computer-readable storage medium.

[0004] The embodiments of this application adopt the following technical solutions:

[0005] In a first aspect, the present application provides a coloring method, the method comprising:

[0006] Obtaining a first shading rate texture map (SRM) for a first image, wherein the first image is an image generated by performing variable resolution shading (VRS) rendering based on current image frame data, and a resolution of the first image is lower than a render target resolution of the image frame;

[0007] Perform artificial intelligence (AI) super-resolution on the first image based on the first shading rate texture map to increase the resolution of the first image and generate a second image that meets the rendering target resolution.

[0008] According to the shading method of the first aspect, image shading and rendering are performed in conjunction with VRS (S200) and AI super-resolution (S210+S220). AI super-resolution is used to repair blurred areas in the VRS shading results, significantly improving VRS temporal stability. VRS also enables the use of more aggressive shading rates, effectively resolving temporal stability issues inherent in pure VRS rendering and the overly conservative nature of VRS at low SRM computational overhead. Compared to using VRS alone, the VRS step of the first aspect allows for a more aggressive shading rate, reducing it by approximately 20%, significantly reducing VRS rendering time and power consumption.

[0009] According to the coloring method of the first aspect, the image rendering quality is improved and the image rendering overhead is reduced.

[0010] In an implementation of the first aspect, the method further includes:

[0011] The first shading rate texture map is generated according to the shading rendering result of the image frame previous to the current image frame.

[0012] According to the above implementation method, using the VRS+AI super-resolution rendering and shading result of the previous image frame as the input for calculating SRM in the next frame can greatly improve the quality of SRM and reduce the adverse effects of the feedback loop.

[0013] In an implementation of the first aspect, the method further includes:

[0014] Based on the first shading rate texture map, variable resolution shading rendering is performed according to the current image frame data to generate the first image.

[0015] In an implementation of the first aspect, the method further includes:

[0016] Generate a second shading rate texture map according to a shading rendering result of an image frame previous to the current image frame;

[0017] Based on the second shading rate texture map, variable resolution shading rendering is performed according to the current image frame data to generate the first image.

[0018] In an implementation of the first aspect, generating a second shading rate texture map according to a shading rendering result of an image frame previous to the current image frame includes:

[0019] generating a reference image according to the shading rendering result, wherein the reference image has the same resolution as the first image;

[0020] The second shading rate texture map is generated according to the reference map.

[0021] In an implementation of the first aspect, generating the first shading rate texture map according to a shading rendering result of an image frame previous to the current image frame includes:

[0022] Calculating a first pixel grouping state according to an original resolution color gradient of the shading rendering result;

[0023] For the shading rendering result, pixel extraction is performed using maximum and minimum pooling according to a preset pixel group size to generate a maximum pool coarse-grained map and a minimum pool coarse-grained map;

[0024] Generate a second pixel grouping state according to the maximum pool coarse-grained map and the color gradient of the minimum pool coarse-grained map;

[0025] fusing the first pixel grouping state and the second pixel grouping state to generate a third pixel grouping state;

[0026] The first shading rate texture map is generated according to the third pixel grouping state.

[0027] In an implementation of the first aspect, performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes:

[0028] determining shading rates for different regions on the first image according to the first shading rate texture map;

[0029] According to the coloring rates of different areas on the first image, artificial intelligence super-resolution is performed on different areas on the first image respectively, wherein the image fineness of the artificial intelligence super-resolution of the area with a high coloring rate is higher than the image fineness of the artificial intelligence super-resolution of the area with a low coloring rate.

[0030] According to the above implementation method, in the process of AI super-resolution to improve image resolution, SRM is used to guide the image fineness of different image areas during AI super-resolution, effectively controlling the processing resource consumption of AI super-resolution while satisfying the visual effect.

[0031] In an implementation of the first aspect, the first image includes a first area and a second area, the first area corresponds to a first shading rate, and the second area corresponds to a second shading rate;

[0032] The performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes:

[0033] Based on a first super-resolution ratio, performing artificial intelligence super-resolution on the first region;

[0034] Based on the second super-resolution ratio, artificial intelligence super-resolution is performed on the second area, wherein the ratio of the first super-resolution ratio to the second super-resolution ratio is consistent with the ratio of the first shading rate to the second shading rate.

[0035] In an implementation of the first aspect, performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes:

[0036] Based on the first super-resolution ratio, perform AI super-resolution on the first region to generate a first super-resolution image;

[0037] Based on the second super-resolution ratio, perform AI super-resolution on the second area to generate a second super-resolution image;

[0038] Performing image magnification on the first super-resolved image to generate a first magnified image, wherein a resolution of the first magnified image satisfies the rendering target resolution;

[0039] The second super-resolved image is enlarged to generate a second enlarged image, where the resolution of the second enlarged image meets the rendering target resolution.

[0040] In an implementation of the first aspect, performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes:

[0041] Amplifying the first region by a first expansion ratio to generate a first magnified image;

[0042] A second magnification is used to magnify the second area to generate a second magnified image, wherein a ratio of the second magnification to the first magnification is consistent with a ratio of the first shading rate to the second shading rate;

[0043] Based on the first super-resolution ratio, perform AI super-resolution on the first enlarged image to generate a first super-resolution image, where the resolution of the first super-resolution image meets the rendering target resolution;

[0044] Based on the second super-resolution ratio, AI super-resolution is performed on the second enlarged image to generate a second super-resolution image, where the resolution of the second super-resolution image meets the rendering target resolution.

[0045] In a second aspect, the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps described in the first aspect.

[0046] In a third aspect, the present application provides an electronic chip, comprising a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic chip is triggered to execute the method steps described in the first aspect.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 FIG2 is a simplified structural diagram of a coloring device according to an embodiment of the present application;

[0049] Figure 2 Shown is a flow chart of a coloring method according to an embodiment of the present application;

[0050] Figure 3 FIG2 is a schematic diagram of a rendering process according to an embodiment of the present application;

[0051] Figure 4 FIG2 is a schematic diagram of an AI super-resolution logic architecture according to an embodiment of the present application;

[0052] Figure 5 Schematic diagram of the SRM generation process according to an embodiment of the present application;

[0053] Figure 6 Shown is a flow chart of generating SRM according to an embodiment of the present application;

[0054] Figure 7 FIG2 is a schematic diagram of generating an SRM according to an embodiment of the present application;

[0055] Figure 8 Shown is a flow chart of generating SRM according to an embodiment of the present application;

[0056] Figure 9 FIG2 is a flow chart of a variable resolution rendering method according to an embodiment of the present application;

[0057] Figure 10 FIG2 is a flow chart of a variable resolution rendering method according to an embodiment of the present application;

[0058] Figure 11 FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0061] Variable Rate Shading (VRS) is a key strategy for reducing rendering load. Compared with strategies such as low-resolution rendering, checkerboard rendering, and dynamic resolution rendering, VRS provides a more refined and localized control method. VRS dynamically determines the degree of shading fineness of different areas by analyzing the importance of different areas and the degree of perception of the human eye, thereby effectively reducing the rendering load. VRS can significantly improve performance and energy efficiency in fragment / pixel bound (Fragment / Pixel Bound) application scenarios. An intuitive explanation of VRS is that for large color block areas, VRS does not need to shade each pixel one by one, but only shades the entire area (such as 4x4, a total of 16 pixels) once, thereby improving rendering performance.

[0062] When VRS is enabled, the shading rate of each region in the rendered image must be determined. This rate is recorded in a shading rate map (SRM) with the same resolution as the rendered image. A key step in the VRS algorithm is calculating an appropriate, time-stable SRM. VRS schemes can be categorized as either production-based or runtime-based, depending on the timing of determining the shading rate.

[0063] Taking game rendering as an example, production-based solutions require intervention during the game production phase. These solutions rely on game modifications and determine which render passes, objects, and scenes to enable VRS based on the game's characteristics. This is typically done through a search-based approach. Production-based solutions are highly scene-dependent, making support for existing games relatively difficult. Even for newly developed games, the release of new content (scenes) requires re-searching the VRS configuration, resulting in high maintenance costs.

[0064] The runtime solution does not rely on game changes and directly determines the SRM based on the real-time game screen. The runtime solution is more universal and can be made imperceptible to the game. However, the runtime solution will inevitably introduce the additional overhead of online SRM calculation. Online SRM calculation is an additional operation in the conventional rendering pipeline. Under strict restrictions on its computational overhead, the quality of SRM is difficult to accurately calculate through precise algorithms. Instead, approximate algorithms are used, resulting in a tendency to adopt overly conservative solutions in actual use, making it difficult to fully utilize the true capabilities of VRS. In addition, since the calculation of online SRM depends on the variable resolution rendering results of the previous frame, it will lead to the adverse effect of a multi-frame negative feedback loop, also known as shading rate swing.

[0065] In the runtime solution, smooth local areas can use a lower shading rate. There are many reasons for the smoothness of the rendered image, including low-frequency textures and lighting, shadows, out-of-focus blur, motion blur, and so on. Considering each of these factors individually would inevitably lead to excessive overhead and reduce the algorithm's universality. A common choice in the runtime solution is to take the final rendering result of the previous frame, warp it to the current frame based on the motion vector, and then determine the shading rate of each region in the current frame.

[0066] For example, in a feasible runtime solution (the first one), which does not rely on game modification, the relationship between image quality loss and shading rate is quantitatively analyzed, and a simple and efficient loss estimation function is given. Specifically, (1) the VRS algorithm accepts the final rendering result of the previous frame as input, (2) converts it into a grayscale image and warps it according to the motion vector (MV), (3) divides it into 16*16 tiles, (4) calculates the grayscale value difference of adjacent pixels in the x and y directions, that is, the color gradient of kernel size = 2, and takes the maximum color gradient in the x and y directions within the 16*16 tile, (5) the scene-related threshold of the above gradient card can be used to determine whether each tile can use a 1 / 2 shading rate (2x1, 1x2, 2x1, 2x2), (6) To ensure computational efficiency, the color gradient of kernel size = 4 is calculated based on the color gradient of kernel size = 2 in an approximate manner, and then determines whether each tile can use a more aggressive 1 / 4 shading rate (2x4, 4x2).

[0067] The above-mentioned running state scheme is an approximate parameter of 1 / 4 shading rate set based on the statistical characteristics of natural images. However, the characteristics of game rendering images are very different from natural images, resulting in failure to obtain optimal benefits and the appearance of ghosting in some scenes. Compared with using coarse pixels (coarse pixels) to calculate the actual color gradient of kernel size (kernel size) = 4, although the use of this approximation reduces the cost of calculating SRM, it affects the quality of the rendered image at the same average shading rate. In addition, since the actual kernel size is too small, the above-mentioned running state scheme is sensitive to scene changes and cannot well guarantee the temporal stability of SRM.

[0068] For example, in another feasible running state scheme (the second one), the color gradient of the grayscale image of the previous frame and the card threshold method are used to determine the shading rate. Unlike the first running state scheme, the second running state scheme calculates the real color gradient of kernel size = 4 instead of using an approximate method. The second running state scheme reduces the amount of calculation by using the coarsepixel method. Under the same average shading rate, the second running state scheme can achieve higher rendering image quality than the first running state scheme in various scenarios. At the same time, the actual kernel size of the second running state scheme is larger than that of the first running state scheme, and it performs better in time domain stability.

[0069] However, while the second operational solution can better obtain SRM and ensure temporal stability, its computational overhead is approximately double that of the first operational solution. For fragment / pixel-bound scenarios, its overall benefit is greater than the first operational solution. However, for simple non-fragment / pixel-bound scenarios, the excessive additional SRM computation overhead results in a lower benefit. While its temporal stability is superior to the first operational solution, jitter is still present in dynamic scenes.

[0070] For example, in another feasible runtime solution (the third one), the local shading rate is determined based on Sobel edge detection. The third runtime solution proposes which render passes can apply VRS and proposes that by adjusting the threshold, aggressive / conservative SRM can be generated in a compute pass for different render passes.

[0071] The third runtime solution requires adjusting the VRS calculation threshold on a render pass-by-render basis. In gaming scenarios, this approach requires VRS design during the game development phase. For hardware manufacturers, this solution is not self-contained and requires customized adaptation for specific games. It is also not user-friendly for existing games.

[0072] To address the problems of temporal stability and over-conservatism of VRS under low SRM computational overhead in the above-mentioned solution, an embodiment of the present application proposes a new coloring method.

[0073] Deep Learning Super Sampling (DLSS) / Artificial Intelligence (AI) super-resolution / AI super-sampling is a solution for improving image resolution through AI neural networks. In the shading method proposed in the embodiment of the present application, VRS and AI super-resolution are combined for image shading and rendering. Specifically, VRS is first used for shading rendering to generate a low-resolution image, and then AI super-resolution is used to increase the resolution of the low-resolution image to generate an image that meets the rendering target resolution.

[0074] Furthermore, in the process of AI super-resolution to improve image resolution, the SRM in the VRS solution is introduced. The SRM is used to guide the fineness of the AI ​​super-resolution image when the resolution of each area of ​​the image is improved during the AI ​​super-resolution process, so as to control the processing resource consumption of AI super-resolution.

[0075] According to the shading method of one embodiment of the present application, image shading rendering is performed in combination with VRS and AI super-resolution, so that the temporal stability of VRS is greatly improved. At the same time, VRS can use a more aggressive shading rate, which effectively solves the temporal stability problem existing in pure VRS rendering and the problem of VRS being overly conservative under low SRM computing overhead, thereby improving image rendering quality and reducing image rendering overhead.

[0076] According to the shading method of an embodiment of the present application, there are no constraints on the application (for example, a game application) that generates image data, and the application is not aware of the rendering process.

[0077] The coloring method of the embodiment of the present application is applied to an electronic device. Alternatively, the coloring method of the embodiment of the present application is applied to a GPU or other processor installed in an electronic device.

[0078] Based on the coloring method of the embodiment of the present application, an embodiment of the present application also proposes a coloring device.

[0079] The variable resolution shading device of the embodiment of the present application is installed / built in an electronic device. Alternatively, the variable resolution shading device of the embodiment of the present application is built in a GPU or other processor installed in the electronic device.

[0080] Figure 1 FIG. 1 is a simplified structural diagram of a coloring device according to an embodiment of the present application.

[0081] like Figure 1 As shown, the rendering apparatus 100 includes an SRM acquisition module 120 and an AI super-resolution module 130 .

[0082] The SRM acquisition module 120 is configured to acquire a first SRM for a first image.

[0083] The AI ​​super-resolution module 130 is configured to perform AI super-resolution on the first image based on the first SRM to increase the resolution of the first image to generate a second image that meets the rendering target resolution.

[0084] The first image is an image generated by VRS rendering according to the current image frame data, and the resolution of the first image is lower than the rendering target resolution of the image frame.

[0085] Optionally, in one embodiment, the rendering apparatus 100 further includes a VRS module 140. The VRS module 140 is configured to perform VRS rendering according to the current image frame data to generate a first image.

[0086] Optionally, in one embodiment, the VRS module 140 does not include an SRM calculation unit, and the SRM acquisition module 120 is configured to generate a first SRM for guiding AI super-resolution and a second SRM for performing VRS. The VRS module 140 performs VRS based on the second SRM generated by the SRM acquisition module 120.

[0087] Optionally, in another embodiment, the VRS module 140 includes an SRM calculation unit configured to calculate a second SRM for performing VRS rendering to generate the first image. The SRM acquisition module 120 extracts the second SRM from the VRS module 140 as the first SRM. Alternatively, the SRM acquisition module 120 extracts intermediate parameters for generating the second SRM from the VRS module 140 to generate the first SRM.

[0088] Optionally, considering that in some application scenarios, VRS module 140 is a packaged model and does not provide an output interface for the SRM or intermediate parameters, in another embodiment, VRS module 140 includes an SRM calculation unit for calculating the second SRM used for VRS rendering to generate the first image. The SRM acquisition module 120 does not call the second SRM or intermediate parameters from VRS module 140, but instead independently generates the first SRM used to guide AI super-resolution.

[0089] In the description of the embodiments of the present application, for the convenience of description, the description of the coloring device is divided into various modules according to their functions. The division of each module is only a division of logical functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or one or more software and / or hardware.

[0090] Specifically, the device proposed in the embodiment of the present application can be fully or partially integrated into a physical entity (for example, a GPU or other type of processor) during actual implementation, or it can be physically separated. And these modules can all be implemented in the form of software calling through a processing element; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software calling through a processing element, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. During the implementation process, each step of the above method or each of the above modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.

[0091] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0092] Figure 2 FIG. 4 is a flow chart of a coloring method according to an embodiment of the present application.

[0093] Figure 1 The variable resolution shading device shown performs Figure 2 The following process shown implements shading in image rendering.

[0094] S200 , the VRS module 140 performs VRS rendering according to the current image frame data to generate a first image, where the resolution of the first image is lower than the rendering target resolution of the image frame.

[0095] Specifically, the embodiment of the present application does not limit the specific method of VRS rendering by the VRS module 140. Those skilled in the art can design a specific method of VRS according to actual needs.

[0096] For example, S200 includes: calculating an SRM (second SRM) for performing VRS, enabling a fragment shader (FS) based on the second SRM, and performing VRS rendering.

[0097] For example, Figure 3FIG. 1 is a schematic diagram of a rendering process according to an embodiment of the present application.

[0098] like Figure 3 As shown, an application 300 (e.g., a game application) outputs image data required for image rendering to a geometry processing module 310. The geometry processing module 310 includes a vertex shading module 311, a projection clipping screen mapping module 312, and a compute shading rate module 313. Specifically, the compute shading rate module 313 is used to generate an SRM for VRS.

[0099] The data processed by the geometry processing module (Geometry Processing) 310 is sent to the rasterization module (Rasterization) 320 .

[0100] The processing result of the rasterization module (Rasterization) 320 is sent to the pixel processing module (PixelProcessing) 330.

[0101] The pixel processing module (Pixel Processing) 330 is used to perform VRS. It includes a fragment shading module (Fragment Shading) 331 and a merging module (Merging) 332. The fragment shading module (Fragment Shading) 331 performs region shading based on the SRM calculated by the compute shading rate module (Compute Shading Rate) 313. The merging module (Merging) 332 merges the shading results of each region to generate a complete image.

[0102] S210 : The SRM acquisition module 120 acquires an SRM (first SRM) for guiding AI super-resolution.

[0103] At step S220, the AI ​​super-resolution module 130 performs AI super-resolution on the first image based on the guidance of the first SRM to generate a second image that meets the rendering target resolution of the image frame. During the AI ​​super-resolution, different levels of AI super-resolution are employed for different image regions of the first image based on different shading rates of the different image regions in the first image corresponding to the first SRM.

[0104] Specifically, for the area with a high shading rate in the first SRM, a high-precision AI super-resolution is performed; for the area with a low shading rate in the first SRM, a low-precision AI super-resolution is performed.

[0105] In this way, the image fineness of the area with a high final coloring rate is higher than that of the area with a low final coloring rate.

[0106] According to the method of one embodiment of the present application, in the process of performing AI super-resolution to improve image resolution, SRM is used to guide the image fineness of different image areas during AI super-resolution, effectively controlling the processing resource consumption of AI super-resolution while satisfying the visual effect.

[0107] Specifically, in one embodiment, the colorization rates of different image regions are determined based on the first SRM, and the super-resolution ratio of the image region during AI super-resolution is determined based on the colorization rate of the image region. When performing AI super-resolution, a high super-resolution ratio is used for regions with high colorization rates in the first SRM; a low super-resolution ratio is used for regions with low colorization rates in the first SRM.

[0108] For example, a first region in a first image corresponds to a first shading rate, and a second region corresponds to a second shading rate. The first region in the first image is AI super-resolved using a first super-resolution ratio, and the second region in the first image is AI super-resolved using a second super-resolution ratio, wherein the ratio of the first shading rate to the second shading rate is consistent with the ratio of the first super-resolution ratio to the second super-resolution ratio.

[0109] For example, in one embodiment, the first SRM and the first image are input to the AI ​​super-resolution module 130 together, and the first SRM is used as a modulation signal for AI super-resolution. The first SRM guides the AI ​​super-resolution module 130 to achieve the required super-resolution ratio for the AI ​​super-resolution part during AI super-resolution.

[0110] Specifically, in one embodiment, the first SRM is injected into the AI ​​super-resolution model of the AI ​​super-resolution module 130 via gating (also known as element-wise multiplication), thereby guiding local super-resolution. In other embodiments, other methods may be used to inject the first SRM into the AI ​​super-resolution model of the AI ​​super-resolution module 130.

[0111] For example, Figure 4 FIG2 is a schematic diagram of the AI ​​super-resolution logic architecture according to an embodiment of the present application.

[0112] like Figure 4As shown, the AI ​​super-resolution module 130 includes an SRM guiding stream module (SRM Guiding Stream) 410 and a main stream module (Main Stream) 420.

[0113] The first SRM is input to the SRM guiding stream module (SRMGuiding Stream) 410 by means of SRM embedding (SRM Embedding). The first image is input to the main stream module (Main Stream) 420.

[0114] The SRM Guiding Stream module (SRM Guiding Stream) 410 includes lightweight 1*1 convolution blocks (Lightweight 1*1 Conv) 411, 412, and 413. The main stream module (Main Stream) 420 includes super-resolution convolution blocks (SuperResolution Conv Block) 421, 422, and 423.

[0115] The SRM Guiding Stream module 410 inputs the first SRM into the Main Stream module 420 through gating (also called element-wise multiplication).

[0116] The first image is blended with the input of the main stream module (Main Stream) 420 to generate a second image.

[0117] exist Figure 4 In the illustrated embodiment, since the SRM Guiding Stream module (SRM Guiding Stream) 410 includes Lightweight 1*1Conv, the SRM Guiding Stream module (SRM Guiding Stream) 410 is very lightweight, and the parameter and computational overheads are both less than 3% of the total overhead, thereby effectively controlling the additional overhead of AI super-resolution.

[0118] Furthermore, when performing AI super-resolution, a high super-resolution ratio is used for areas with high shading rates in the first SRM; a low super-resolution ratio is used for areas with low shading rates in the first SRM; and for areas with different resolutions after AI super-resolution, image amplification is used to unify them to the rendering target resolution.

[0119] Alternatively, when performing AI super-resolution, a low magnification is used to amplify the image for areas with a high shading rate in the first SRM; a high magnification is used to amplify the image for areas with a low shading rate in the first SRM; and different super-resolution ratios are used for AI super-resolution for areas with different resolutions after image amplification to unify them to the rendering target resolution.

[0120] Specifically, in one embodiment, for an area with a shading rate of 1, image amplification is not performed, and AI super-resolution is used to improve the area to the rendering target resolution.

[0121] Specifically, those skilled in the art may use any feasible method to magnify the image, and the embodiments of the present application do not limit this.

[0122] For example, in one embodiment, image magnification is performed by copying adjacent pixels, for example, copying and filling one pixel to generate 2*2 pixels to achieve 2*2 image magnification.

[0123] Specifically, in one implementation of S220, the following steps are included:

[0124] Performing AI super-resolution on a first region in the first image using a first super-resolution ratio to generate a first super-resolution image, wherein in a first SRM, the first region corresponds to a first shading ratio;

[0125] Performing AI super-resolution on a second region in the first image using a second super-resolution ratio to generate a second super-resolution image, wherein: in the first SRM, the second region corresponds to the second shading rate; and a ratio of the first shading rate to the second shading rate is consistent with a ratio of the first super-resolution ratio to the second super-resolution ratio;

[0126] Amplifying the first super-resolution image by a first expansion ratio (the first expansion ratio may be 1) to generate a first enlarged image, wherein a resolution of the first enlarged image satisfies a rendering target resolution;

[0127] The second super-resolution image is enlarged using a second enlargement ratio (the second enlargement ratio may be 1) to generate a second enlarged image, wherein a resolution of the second enlarged image meets the rendering target resolution (a ratio of the first shading rate to the second shading rate is consistent with a ratio of the second enlargement ratio to the first enlargement ratio).

[0128] Alternatively, in another implementation of S220, the method includes:

[0129] A first region in the first image is magnified using a first expansion ratio to generate a first magnified image region, wherein in the first SRM, the first region corresponds to a first shading ratio;

[0130] A second region in the first image is magnified using a second expansion ratio to generate a second magnified image region, wherein: in the first SRM, the second region corresponds to the second shading rate; and a ratio of the first shading rate to the second shading rate is consistent with a ratio of the second expansion ratio to the first expansion ratio;

[0131] Performing AI super-resolution on the first enlarged image region using a first super-resolution ratio to generate a first super-resolution image, wherein a resolution of the first super-resolution image meets a rendering target resolution;

[0132] Perform AI super-resolution on the second enlarged image area using the second super-resolution ratio to generate a second super-resolution image, wherein the resolution of the second super-resolution image meets the rendering target resolution (the ratio of the first shading rate to the second shading rate is consistent with the ratio of the first super-resolution ratio to the second super-resolution ratio).

[0133] For example, in one embodiment, assuming AI super-resolution is performed on a first image to generate a second image, the resolution of the first image needs to be expanded to 4*4. In the first SRM, the VRS shading rate for the first image region is 1 (coloring each pixel separately); for the second image region, the VRS shading rate is 1 / 2 (coloring 1*2 pixels).

[0134] Performing AI super-resolution on the first image region and the second image region based on the guidance of the first SRM includes:

[0135] Perform AI super-resolution on the first image area with a super-resolution ratio of 4*4;

[0136] First, the second image area is subjected to AI super-resolution with a super-resolution ratio of 4*2, and then the super-resolution result is enlarged by 1*2; or, first, the second image area is enlarged by 1*2, and then the enlarged result is subjected to AI super-resolution with a super-resolution ratio of 4*2.

[0137] According to the coloring method of one embodiment of the present application, VRS (S200) and AI super-resolution (S210+S220) are combined to perform image coloring rendering, and AI super-resolution is used to repair the blurred areas in the VRS coloring results, so that the VRS temporal stability is greatly improved. At the same time, VRS can use a more aggressive shading rate, which effectively solves the temporal stability problem of pure VRS rendering and the problem of VRS being overly conservative under low SRM computing overhead, thereby improving the image rendering quality and reducing the image rendering overhead.

[0138] Compared with simply using the VRS solution, the VRS step of this solution allows for a more aggressive shading rate, which can be reduced by about 20%, thereby significantly reducing the time and power consumption required for VRS rendering.

[0139] According to the coloring method of one embodiment of the present application, different levels of AI super-resolution are adopted for different image areas. While ensuring the image display effect, the AI ​​super-resolution consumption is effectively controlled, and the existing AI super-resolution module (DLSS module) is used to the greatest extent, extending the traditional super-resolution to local variable magnification super-resolution.

[0140] Furthermore, in S210 , it is necessary to obtain an SRM (first SRM) for guiding AI super-resolution; and in S200 , in order to implement VRS, it is necessary to calculate an SRM (second SRM) for performing VRS.

[0141] Optionally, in one embodiment, the first SRM and the second SRM use the same SRM.

[0142] For example, in S200 , the VRS module 140 calculates the second SRM and performs VRS rendering based on the second SRM to generate a first image. In S210 , the SRM acquisition module 120 calls the second SRM generated by the VRS module 140 as the first SRM.

[0143] For another example, in S210, the SRM acquisition module 120 calculates a first SRM. In S200, the VRS module 140 calls the first SRM calculated by the SRM acquisition module 120 as a second SRM, and performs VRS rendering based on the second SRM to generate a first image.

[0144] For another example, in S200 , the VRS module 140 calculates the second SRM and performs VRS rendering based on the second SRM to generate a first image. In S210 , the SRM acquisition module 120 generates the first SRM based on the same logic as that for calculating the second SRM.

[0145] Optionally, in another embodiment, the first SRM and the second SRM use different SRMs.

[0146] For example, in S200 , the VRS module 140 calculates the second SRM and performs VRS rendering based on the second SRM to generate the first image. In S210 , the SRM acquisition module 120 calls the second SRM generated by the VRS module 140 and / or the intermediate parameters used in generating the second SRM, and generates the first SRM based on the second SRM and / or the intermediate parameters.

[0147] For another example, in S210, the SRM acquisition module 120 calculates a first SRM. In S200, the VRS module 140 calls the first SRM calculated by the SRM acquisition module 120 and / or the intermediate parameters used in generating the first SRM, generates a second SRM based on the first SRM and / or the intermediate parameters, and performs VRS rendering based on the second SRM to generate a first image.

[0148] For another example, in S200 , the VRS module 140 calculates the second SRM and performs VRS rendering based on the second SRM to generate a first image. In S210 , the SRM acquisition module 120 generates the first SRM based on a different logic calculation from that for calculating the second SRM.

[0149] In the embodiments of the present application, there is no limitation on the specific method of calculating the SRM (the first SRM and / or the second SRM), and those skilled in the art can design the specific method of calculating the SRM according to actual needs.

[0150] Specifically, in one embodiment, the SRM (the first SRM and / or the second SRM) is calculated based on the previous image frame (the image frame that has undergone VRS+AI super-resolution). For example, in one embodiment, the previous image frame that has undergone VRS+AI super-resolution is first downsampled, and the SRM of the current image frame is calculated based on the downsampling result.

[0151] Compared with the simple VRS solution, in which the VRS shading result of the previous frame is used as the input for calculating the SRM of the next frame, the variable resolution shading method according to an embodiment of the present application uses the rendering shading result of VRS+AI super-resolution as the input for calculating the SRM of the next frame, which can greatly improve the quality of the SRM, improve the accuracy of the SRM, improve the temporal stability of the VRS, and reduce the adverse effects of the feedback loop.

[0152] For example, in one embodiment, in S200, the first SRM and / or the second SRM are calculated using the first, second, or third operating state scheme described above.

[0153] For another example, in one embodiment, in S210, a first SRM is calculated based on the color gradient of the original resolution of the last rendered frame. In S200, the last rendered frame is down-sampled to generate a reference image with the same resolution as the first image, and a second SRM is calculated based on the color gradient of the reference image.

[0154] For example, Figure 5 Schematic diagram of the SRM generation process according to one embodiment of the present application.

[0155] like Figure 5 As shown, the previous image frame 500 is an image colored according to the coloring method of the present application. The previous image frame 500 is the previous frame of the current image frame. A grayscale map 501 (Frequency map) is generated based on the previous image frame 500. Based on the grayscale map 501, an SRM 503 for the current image frame is generated with reference to the motion vector 502 of the current image frame.

[0156] For example, Figure 6 Shown is a flowchart of generating SRM according to an embodiment of the present application.

[0157] S610, calculating a first pixel grouping state according to the original resolution color gradient of the previous frame image that has been rendered;

[0158] S621: For the last frame of the rendered image, use minmax pooling to extract pixels according to a preset pixel group size (e.g., 2*2) to generate a maximum pool coarse-grained map and a minimum pool coarse-grained map;

[0159] S622, generating a second pixel grouping state according to the color gradient of the maximum pool coarse-grained map and the minimum pool coarse-grained map;

[0160] S630, fusing the first pixel grouping state and the second pixel grouping state to generate a third pixel grouping state;

[0161] S631 , generate an SRM (a first SRM and / or a second SRM) according to the third pixel grouping state.

[0162] For example, Figure 7 FIG. 4 is a schematic diagram of generating an SRM according to an embodiment of the present application.

[0163] like Figure 7 As shown, the last frame of image that has been rendered is a 6*4 pixel image 701 (each square in 701 represents a pixel).

[0164] For image 701, the color gradient between adjacent pixels is calculated. Adjacent pixels whose color gradients fall within a preset gradient threshold are grouped together. The pixel grouping is shown in image 702 (pixels within the same rectangular box are grouped together, while pixels outside the rectangular box are grouped separately). Image 702 shows the pixel grouping for 1*1, 2*1, 1*2, and 2*2 pixel distributions (pixels X11 and X21 are grouped together, pixels X15 and X25 are grouped together, pixels X33, X34, X43 and X44 are grouped together, and pixels X35, X36, X45 and X46 are grouped together).

[0165] The image 701 is segmented into 2*2 pixel groups, and minmax pooling is applied to each pixel group. That is, the pixels with the largest color value in each pixel group are extracted to form a maximum pooling coarse-grained image 703, and the pixels with the smallest color value in each pixel group are extracted to form a minimum pooling coarse-grained image 704.

[0166] The color gradient between corresponding pixels (pixels in the same pixel group) of the maximum pooling coarse-grained image 703 and the minimum pooling coarse-grained image 704 is calculated, as well as the color gradient between adjacent pixels in the maximum pooling coarse-grained image 703 and the color gradient between adjacent pixels in the minimum pooling coarse-grained image 704. Pixels whose color gradients between adjacent pixels in the maximum pooling coarse-grained image 703, the color gradients between adjacent pixels in the minimum pooling coarse-grained image 704, and the color gradients between corresponding pixels in the maximum pooling coarse-grained image 703 and the minimum pooling coarse-grained image 704 are all within a preset gradient threshold are grouped together.

[0167] For example, pixels 711 and 721 are pixels with the largest color values ​​in pixel groups 710 and 720, respectively, and pixels 712 and 722 are pixels with the smallest color values ​​in pixel groups 710 and 720, respectively. If the color gradient between pixels 711 and 721, the color gradient between pixels 712 and 722, the color gradient between pixels 711 and 712, and the color gradient between pixels 721 and 722 are all less than a preset gradient threshold, pixels 711, 721, 712, and 722 are grouped together.

[0168] The pixel grouping state based on the maximum pooling coarse-grained image 703 and the minimum pooling coarse-grained image 704 is shown in image 705 (pixels within the same rectangular box are grouped together, and pixels not within the rectangular box are grouped separately). Image 705 shows the pixel grouping state for 2*4, 4*2, and 4*4 pixel distributions (pixels X33, X34, X43, X44, X35, X36, X45, and X46 are grouped together).

[0169] The grouping states of image 702 and image 705 are fused to generate a grouping state as image 706 (pixels X11 and X21 are grouped together, pixels X15 and X25 are grouped together, and pixels X33, X34, X43, X44, X35, X36, X45 and X46 are grouped together).

[0170] Furthermore, in another embodiment, multiple minmax poolings can be performed based on different pixel group sizes (for example, 2*2, 3*3, 4*4), and multiple sets of different pixel grouping states for different grouping sizes can be generated according to the color gradient of the maximum pool coarse-grained map and the minimum pool coarse-grained map. Finally, all pixel grouping states are integrated to generate the pixel grouping state for generating SRM.

[0171] For example, Figure 8 Shown is a flowchart of generating SRM according to an embodiment of the present application.

[0172] S810, calculating a first pixel grouping state according to the original resolution color gradient of the previous frame image that has been rendered;

[0173] S820: For the previous frame of the rendered image, generate a maximum pooling coarse-grained map and a minimum pooling coarse-grained map according to the pixel group size (2*2) using minmax pooling, and generate a second pixel grouping state based on the color gradients of the maximum pooling coarse-grained map and the minimum pooling coarse-grained map;

[0174] S830: For the previous frame of the rendered image, generate a maximum pooling coarse-grained image and a minimum pooling coarse-grained image according to the pixel group size (3*3) using minmax pooling, and generate a fourth pixel grouping state based on the color gradients of the maximum pooling coarse-grained image and the minimum pooling coarse-grained image;

[0175] S840: Fusion of the first pixel grouping state, the second pixel grouping state, and the fourth pixel grouping state to generate a third pixel grouping state

[0176] S850: Generate an SRM (a first SRM and / or a second SRM) according to the third pixel grouping state.

[0177] Figure 9 FIG. 4 is a flow chart of a variable-resolution rendering method according to an embodiment of the present application.

[0178] like Figure 9 As shown, the SRM acquisition module 920 is based on Figure 5 、 Figure 6 The illustrated scheme generates SRM 901 (the first SRM).

[0179] The SRM 901 is input to the VRS module 940 and the AI ​​super-resolution module 930 .

[0180] The VRS module 940 performs VRS using the SRM 901 (using the first SRM as the second SRM) to generate an image 902 (a first image).

[0181] The image 902 is input to the AI ​​super-resolution module 930 .

[0182] The AI ​​super-resolution module 930 performs AI super-resolution on the image 902 based on the guidance of the SRM 901 to generate an image 903 (a second image).

[0183] The image 903 is fed back to the SRM acquisition module 920 for generating the first SRM for the next frame of image.

[0184] Figure 10 FIG. 4 is a flow chart of a variable-resolution rendering method according to an embodiment of the present application.

[0185] like Figure 10As shown, the SRM acquisition module 1020 reduces the resolution of the previous frame image that has been rendered, generates a reference image with the same resolution as the first image, and calculates SRM1001 (the second SRM) according to the color gradient of the reference image.

[0186] SRM1001 is input to VRS module 1040.

[0187] The VRS module 1040 performs VRS using the SRM 1001 to generate an image 1002 (a first image).

[0188] The image 1002 is input to the AI ​​super-resolution module 1030 .

[0189] The SRM acquisition module 1020 calculates the SRM 1003 (first SRM) according to the original resolution color gradient of the previous frame image that has been rendered.

[0190] SRM1003 is input to the AI ​​super-resolution module 1030.

[0191] The AI ​​super-resolution module 1030 performs AI super-resolution on the image 1002 based on the guidance of the SRM 1003 to generate an image 1004 (a second image).

[0192] The image 1004 is fed back to the SRM acquisition module 1020 for generating an SRM for the next frame of image.

[0193] An embodiment of the present application further provides an electronic device, which is used to execute the method flow or part of the method flow described in the embodiment of the present application.

[0194] Figure 11 FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

[0195] like Figure 11 As shown, the electronic device 2500 includes a memory 2502 for storing computer program instructions and a processor 2501 for executing the program instructions, wherein, when the computer program instructions are executed by the processor 2501, the electronic device 2500 is triggered to execute the method steps as described in the embodiment of the present application.

[0196] Specifically, in one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory 2502, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned electronic device 2500, the above-mentioned electronic device 2500 executes the method steps described in the embodiment of the present application.

[0197] It is understood that the structural description of the electronic device 2500 in the embodiment of the present application does not constitute a specific limitation on the electronic device 2500. In other embodiments of the present application, the electronic device 2500 may include other components besides the processor 2501 and the memory 2502.

[0198] The processor 2501 may be a device on a chip (SOC), and the processor 2501 may include a central processing unit (CPU), and may further include other types of processors.

[0199] The processor involved in processor 2501 may include, for example, a CPU, a DSP, a microcontroller, or a digital signal processor, and may also include a GPU, an embedded neural network processor (NPU), and an image signal processor (ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of this application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0200] The processor 2501 may include one or more processing units. For example, the processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 2500 may also include one or more processors 2501. The controller may generate an operation control signal based on the instruction opcode and the timing signal to complete the control of instruction fetching and execution.

[0201] In some embodiments, the processor 2501 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. Among them, the USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the electronic device, and can also be used to transmit data between the electronic device and peripheral devices.

[0202] Electronic device 2500 may also include an external memory interface for connecting an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device. The external memory card communicates with processor 2501 via the external memory interface to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0203] Memory 2502 may include a code storage area and a data storage area. The code storage area may store an operating system. The data storage area may store data created during the use of electronic device 2500. Furthermore, memory 2502 may include high-speed random access memory and non-volatile memory, such as one or more disk storage components, flash memory components, and universal flash storage (UFS).

[0204] The memory 2502 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any computer-readable medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0205] The processor 2501 and the memory 2502 may be combined into one processing device, or more commonly, they may be independent components.

[0206] One embodiment of the present application further provides an electronic chip. The electronic chip is used to execute the method flow or part of the method flow described in the embodiment of the present application. For example, the electronic chip can be a GPU.

[0207] Specifically, the electronic chip includes a processor for executing program instructions. When the computer program instructions are executed by the processor, the electronic chip is triggered to execute the method steps described in the embodiments of the present application. The processor of the electronic chip can refer to the processor of the above-mentioned electronic device.

[0208] Optionally, the devices, apparatuses, and modules described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0209] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0210] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this application.

[0211] Specifically, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the embodiment of the present application.

[0212] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program product is run on a computer, it enables the computer to execute the method provided in the embodiment of the present application.

[0213] The description of the embodiments in this application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0216] It should also be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0217] In the embodiments of the present application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, commodity, or apparatus comprising the element.

[0218] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0219] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiments.

[0220] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0221] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0222] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A coloring method, characterized in that: The method comprises: Obtaining a first shading rate texture map for a first image, wherein the first image is an image generated by performing variable resolution shading (VRS) rendering based on current image frame data, and a resolution of the first image is lower than a rendering target resolution of the image frame; Perform artificial intelligence super-resolution on the first image based on the first shading rate texture map to improve the resolution of the first image and generate a second image that meets the rendering target resolution.

2. The method according to claim 1, characterized in that The method further comprises: The first shading rate texture map is generated according to the shading rendering result of the image frame previous to the current image frame.

3. The method according to claim 2, characterized in that The method further comprises: Based on the first shading rate texture map, variable resolution shading rendering is performed according to the current image frame data to generate the first image.

4. The method according to claim 3, characterized in that The method further comprises: Generate a second shading rate texture map according to a shading rendering result of an image frame previous to the current image frame; Based on the second shading rate texture map, variable resolution shading rendering is performed according to the current image frame data to generate the first image.

5. The method according to claim 4, characterized in that The generating of a second shading rate texture map according to a shading rendering result of an image frame previous to the current image frame includes: generating a reference image according to the shading rendering result, wherein the reference image has the same resolution as the first image; The second shading rate texture map is generated according to the reference map.

6. The method according to any one of claims 2 to 5, characterized in that The generating the first shading rate texture map according to the shading rendering result of the previous image frame of the current image frame includes: Calculating a first pixel grouping state according to an original resolution color gradient of the shading rendering result; For the shading rendering result, pixel extraction is performed using maximum and minimum pooling according to a preset pixel group size to generate a maximum pool coarse-grained map and a minimum pool coarse-grained map; Generate a second pixel grouping state according to the maximum pool coarse-grained map and the color gradient of the minimum pool coarse-grained map; fusing the first pixel grouping state and the second pixel grouping state to generate a third pixel grouping state; The first shading rate texture map is generated according to the third pixel grouping state.

7. The method according to any one of claims 1 to 6, characterized in that The performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes: determining shading rates for different regions on the first image according to the first shading rate texture map; According to the coloring rates of different areas on the first image, artificial intelligence super-resolution is performed on different areas on the first image respectively, wherein the image fineness of the artificial intelligence super-resolution of the area with a high coloring rate is higher than the image fineness of the artificial intelligence super-resolution of the area with a low coloring rate.

8. The method according to claim 7, characterized in that The first image includes a first area and a second area, the first area corresponds to a first shading rate, and the second area corresponds to a second shading rate; The performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes: Based on a first super-resolution ratio, performing artificial intelligence super-resolution on the first region; Based on the second super-resolution ratio, artificial intelligence super-resolution is performed on the second area, wherein the ratio of the first super-resolution ratio to the second super-resolution ratio is consistent with the ratio of the first shading rate to the second shading rate.

9. The method according to claim 8, characterized in that The performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes: Based on the first super-resolution ratio, perform AI super-resolution on the first region to generate a first super-resolution image; Based on the second super-resolution ratio, perform AI super-resolution on the second area to generate a second super-resolution image; Performing image magnification on the first super-resolved image to generate a first magnified image, wherein a resolution of the first magnified image satisfies the rendering target resolution; The second super-resolved image is enlarged to generate a second enlarged image, where the resolution of the second enlarged image meets the rendering target resolution.

10. The method according to claim 8, characterized in that The performing artificial intelligence super-resolution on the first image based on the first shading rate texture map includes: Amplifying the first region by a first expansion ratio to generate a first magnified image; A second magnification is used to magnify the second area to generate a second magnified image, wherein a ratio of the second magnification to the first magnification is consistent with a ratio of the first shading rate to the second shading rate; Based on the first super-resolution ratio, perform AI super-resolution on the first enlarged image to generate a first super-resolution image, where the resolution of the first super-resolution image meets the rendering target resolution; Based on the second super-resolution ratio, AI super-resolution is performed on the second enlarged image to generate a second super-resolution image, where the resolution of the second super-resolution image meets the rendering target resolution.

11. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method steps according to any one of claims 1 to 10.

12. An electronic chip, characterized in that: The electronic chip comprises a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic chip is triggered to perform the method steps according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 10.

Images