Image frame processing method and device, graphics processor and electronic equipment

Abstract

The invention relates to an image frame processing method and device, a graphics processor and electronic equipment, and relates to the technical field of image processing. The method comprises the following steps: outputting a first image frame, wherein the first image frame comprises a first tile; after the first image frame is output, a second image frame is output, the second image frame comprises a second tile and a third tile, and the first tile in the first image frame and the second tile in the second image frame are at the same pixel position; wherein the color value of the pixel of the second tile, depending on the product of the color value of the pixel of the first tile and the check mask, the coloring ratio of the second tile is different from the coloring ratio of the third tile. Therefore, the visual quality of the image can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and in particular, to an image frame processing method and device, a graphics processing unit, and an electronic device. BACKGROUND

[0002] Variable rate shading (VRS) is an image processing technology that can be used to optimize image rendering performance. VRS technology can apply different shading rates in different areas. In this way, the rendering workload can be greatly reduced without significantly affecting the visual effect. In the current scheme for image rendering using VRS technology, image jaggies appear, resulting in low visual quality of the image. SUMMARY

[0003] Embodiments of the present application provide an image frame processing method and device, a graphics processing unit, and an electronic device, which solve the problem of low visual quality of the image in the prior art.

[0004] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, an image frame processing method is provided, which includes: outputting a first image frame, the first image frame including a first tile. After outputting the first image frame, a second image frame is outputted, the second image frame including a second tile and a third tile, the first tile in the first image frame and the second tile in the second image frame being at the same pixel position. The color value of a pixel of the second tile depends on the product of the color value of a pixel of the first tile and a check mask, and the shading rate of the second tile is different from the shading rate of the third tile.

[0006] In the above technical solution, the color value of the pixel of the second tile is determined by the color value of the pixel of the first tile and the check mask, so the second image frame can refer to the color value of the first image frame. In this way, the color values of the same area of the second image frame and the first image frame will not differ greatly, and the problems of jaggies and flickering can be solved, and the visual quality of the image is improved.

[0007] In a possible implementation manner of the first aspect, the first image frame further includes a fourth tile, the fourth tile of the first image frame and the third tile of the second image frame being at the same pixel position. The check mask of the pixel of the first tile is smaller than the check mask of the pixel of the fourth tile, and the granularity of the shading rate of the second tile is smaller than the granularity of the shading rate of the third tile.

[0008] In the possible implementation manner, the shading ratio map of the second image frame is determined according to the check mask of the first image frame. For a region with a large check mask of the previous frame image, the current frame image has a high degree of reuse of the color value of the previous frame image, and the image quality of the region is relatively stable. Therefore, the granularity of the shading ratio of the region can be large. For a region with a small check mask of the previous frame image, the current frame image has a low degree of reuse of the color value of the previous frame image, that is, the color value of the region changes greatly. Therefore, the region needs to use a small granularity of the shading ratio to ensure the image quality.

[0009] The embodiment dynamically determines the shading ratio map of the second image frame according to the check mask of the first image frame. For example, the temporal anti-aliasing (TAA) technology is used to super-sample and anti-alias the variable resolution rendering result, and the check mask of the previous frame image is output to a variable rate shading (VRS) module of the next frame image as a compensation signal for generating the shading ratio map. In this way, the image quality can be stabilized, the variable resolution shading with higher quality can be implemented, and the rendering load can be further reduced.

[0010] In addition, the embodiment supports using the shading ratio with a large granularity, and can greatly reduce the rendering overhead. In addition, the shading is performed according to the check mask, and the problems of aliasing and flickering can be solved, and the temporal accumulation effect is relatively obvious. In addition, the embodiment does not depend on natural picture features, and can render non-natural pictures such as game pictures. Moreover, the embodiment adds the check mask to the calculation of the shading ratio, does not introduce an additional calculation module, and has a relatively simple calculation process. The embodiment can be compatible with an existing rendering pipeline and multiple TAA technologies such as deep learning super sampling (DLSS). Furthermore, the embodiment has a relatively simple calculation process, and can be applied to a running stage, for example, can be applied to a game running stage, and can be implemented in a game-unaware manner. The embodiment can also be applied to a development stage. Therefore, the embodiment has a wide application scenario.

[0011] In a possible implementation manner of the first aspect, the shading ratio of the second tile is determined according to the check mask of the pixels of the first tile and the rendering parameter. The rendering parameter includes at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame. In the possible implementation manner, the brightness gradient and other parameters of the tile of the previous image frame are adjusted by using the check mask to realize the adjustment of the rendering resolution of different regions. The shading ratio of the second tile is determined according to the check mask of the pixels of the first tile and the rendering parameter, and the rendering parameter includes at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame. The shading ratio can be calculated by using the rendering parameter and the check mask, the calculation manner of the shading ratio is relatively simple, and the existing rendering pipeline can be compatible.

[0012] In a possible implementation manner of the first aspect, the shading ratio of the second tile is determined according to the check mask of the pixels of the first tile and the rendering parameter. The rendering parameter includes at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame. In the possible implementation manner, the brightness gradient and other parameters of the tile of the previous image frame are adjusted by using the check mask to realize the adjustment of the rendering resolution of different regions. The shading ratio of the second tile is determined according to the check mask of the pixels of the first tile and the rendering parameter, and the rendering parameter includes at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame. The shading ratio can be calculated by using the rendering parameter and the check mask, the calculation manner of the shading ratio is relatively simple, and the existing rendering pipeline can be compatible.

[0013] In a possible implementation manner of the first aspect, the shading ratio of the second tile and the shading ratio of the third tile are smoothed. In the possible implementation manner, the shading ratio of the tile of the second image frame is smoothed, and the image quality can be further improved.

[0014] In a second aspect, an image frame processing apparatus is provided, which includes a first output module and a second output module.

[0015] The first output module is configured to output a first image frame, and the first image frame includes a first tile.

[0016] The second output module is further configured to output a second image frame after the first image frame is output, and the second image frame includes a second tile and a third tile, and the first tile in the first image frame and the second tile in the second image frame are located at the same pixel position.

[0017] The color value of the pixel of the second tile is determined according to a product of the color value of the pixel of the first tile and the check mask, and the shading ratio of the second tile is different from the shading ratio of the third tile.

[0018] In a possible implementation of the second aspect, the first image frame further includes a fourth tile, and the fourth tile of the first image frame is at a same pixel position as the third tile of the second image frame.

[0019] The check mask of the pixel of the first tile is smaller than the check mask of the pixel of the fourth tile, and the granularity of the shading ratio of the second tile is smaller than the granularity of the shading ratio of the third tile.

[0020] In a possible implementation of the second aspect, the shading ratio of the second tile is determined according to the check mask of the pixel of the first tile and a rendering parameter.

[0021] The rendering parameter includes at least one of the following: a luminance gradient of the first tile, a motion vector of the second tile, a motion speed of the second tile, and a frame rate of the second image frame.

[0022] In a possible implementation of the second aspect, the shading ratio of the second tile is obtained by correcting an initial shading ratio of the second tile according to the check mask of the pixel of the first tile.

[0023] The initial shading ratio is determined according to a rendering parameter, and the rendering parameter includes at least one of the following: a luminance gradient of the first tile, a motion vector of the second tile, a motion speed of the second tile, and a frame rate of the second image frame.

[0024] In a possible implementation of the second aspect, the shading ratio of the second tile and the shading ratio of the third tile are smoothed.

[0025] In a third aspect, a graphics processing unit (GPU) is provided, and the GPU is configured to perform the method in the first aspect or any possible implementation of the first aspect.

[0026] In a fourth aspect, an electronic device is provided, and the electronic device includes a memory and the GPU in the third aspect, and the memory stores instructions, and when the GPU executes the instructions, the electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect.

[0027] In a fifth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores program code, and the program code is executable by a processor to perform the method in the first aspect or any possible implementation of the first aspect.

[0028] In a sixth aspect, a computer program product is provided, which, when executed on a computer, causes the computer to perform the method provided in the first aspect or any possible implementation manner of the first aspect.

[0029] It can be understood that any of the image frame processing apparatus, the graphics processor, the electronic device, the computer storage medium or the computer program product provided above are used to execute the corresponding method provided above, and thus the beneficial effects achieved thereby can refer to the beneficial effects of the corresponding method provided above, which will not be described here again. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 A schematic diagram of an image frame provided by an embodiment of the present application;

[0031] Figure 2 A flowchart of a time domain anti-aliasing provided by an embodiment of the present application;

[0032] Figure 3 A structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0033] Figure 4 A flowchart of an image frame processing method provided by an embodiment of the present application Figure 1 ;

[0034] Figure 5 A flowchart of an image frame processing method provided by an embodiment of the present application Figure 2 ;

[0035] Figure 6 A flowchart of an image frame processing method provided by an embodiment of the present application Figure 3 ;

[0036] Figure 7 A flowchart of an image frame processing method provided by an embodiment of the present application Figure 4 ;

[0037] Figure 8 A schematic diagram of an image frame processing apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] It should be noted that the terms "first", "second", etc. involved in the embodiments of the present application are only used for the purpose of distinguishing the same type of features, and cannot be understood as indicating relative importance, quantity, order, etc.

[0039] The terms "exemplary" or "for example" used in the embodiments of this application are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0040] The terms "coupling" and "connection" used in the embodiments of this application should be interpreted broadly. For example, they can refer to a physical direct connection or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0041] First, some basic concepts involved in the embodiments of this application will be explained:

[0042] 1. Variable rate shading (VRS) technology

[0043] VRS (Color Rendering System) is an image processing technique used to optimize image rendering performance. Typically, a graphics processing unit (GPU) renders each pixel in a frame of an image once (pixel-by-pixel rendering), resulting in the rendered image for each pixel. Unlike pixel-by-pixel rendering, VRS can apply different rendering ratios to different regions within a frame. The rendering ratio refers to the size of the pixel block corresponding to a single rendering operation. With VRS, a single rendering operation can be applied to a pixel block, resulting in the same rendering for multiple pixels within that block. For example, rendering a 4×4 pixel block requires only one operation instead of sixteen. An image frame can be divided into multiple tiles, each containing multiple pixels. In VRS, each tile corresponds to a rendering ratio, and at least two tiles will have different rendering ratios.

[0044] like Figure 1 As shown, the image frame drawn by the GPU is divided into 15 tiles of 3×5 (3 rows and 5 columns), and each tile includes 64 pixels of 8×8 (8 rows and 8 columns). The image frame includes a subject and a background. The subject can be two partially overlapping triangles, and the background can be the area other than these two triangles.

[0045] The subject is located in the 2nd row, 2nd column tile, the 2nd row, 3rd column tile, the 3rd row, 1st column tile, the 3rd row, 2nd column tile, the 3rd row, 3rd column tile, and the 3rd row, 4th column tile. The subject is a region in the image frame where the pixel value changes greatly and the color ratio change is easily perceived by the human eye. The granularity of the color ratio of the 6 tiles can be set to be small. For example, in the 6 tiles, each 1x1 pixel is taken as a pixel block, and the GPU can perform shading on each 1x1 pixel once, and each pixel has a shading result. The granularity of the color ratio refers to the number of pixels of the pixel block corresponding to a single shading operation.

[0046] The region in the background close to the subject includes the 2nd row, 1st column tile, the 1st row, 2nd column tile, the 1st row, 3rd column tile, the 2nd row, 4th column tile, and the 3rd row, 5th column tile. The granularity of the color ratio of the 5 tiles can be set to be large. For example, in the 5 tiles, each 1x2 pixel is taken as a pixel block, and the GPU can perform shading on each 1x2 pixel once, and the shading results of the 1x2 pixels are the same; or each 2x1 pixel is taken as a pixel block, and the GPU can perform shading on each 2x1 pixel once, and the shading results of the 2x1 pixels are the same. The region in the background far from the subject includes the 1st row, 1st column tile, the 1st row, 4th column tile, and the 1st row, 5th column tile.

[0047] The region in the background far from the subject is a region in the image frame where the pixel value does not change greatly and the color ratio change is not easily perceived by the human eye. The granularity of the color ratio of the 3 tiles can be set to be large. For example, in the 3 tiles, each 2x2 pixel is taken as a pixel block, and the GPU can perform shading on each 2x2 pixel once, and the shading results of the 2x2 pixels are the same; or each 4x4 pixel is taken as a pixel block, and the GPU can perform shading on each 4x4 pixel once, and the shading results of the 4x4 pixels are the same.

[0048] Optionally, the granularity of the color ratio of adjacent tiles differs by a small amount. For example, when the color ratio of the subject is 1x1, the color ratio of the region in the background close to the subject does not jump to 4x4, but is 1x2 or 2x1. In this way, the smoothness of different regions of the image frame can be improved.

[0049] Optionally, the VRS technology can use a smaller granularity color ratio (e.g., 1x1) for a region in an image frame that is relatively important (e.g., the pixel value changes greatly and the color ratio change is easily perceived by the human eye), thereby obtaining higher picture accuracy; and can use a larger granularity color ratio (e.g., 4x4) for a region in an image frame that is relatively unimportant (the pixel value does not change greatly and the color ratio change is not easily perceived by the human eye), thereby reducing the amount of calculation. In this way, the VRS technology can greatly reduce the rendering workload without significantly affecting the visual effect.

[0050] 2. Temporal anti-aliasing (TAA) technique

[0051] The TAA technique is also known as temporal sub-sampling technique, which is an anti-aliasing technique applied in real-time renderers and game engines. The TAA technique can reduce the aliasing effect and flickering phenomenon in images. Especially in moving or dynamic scenes. The TAA technique provides higher visual quality by smoothing the images using information in the temporal domain. The TAA technique can reduce the aliasing effect. The aliasing effect is the pixelated edges caused by resolution limitations. The TAA can integrate multi-frame information to smooth these edges, thereby reducing the aliasing effect. The TAA technique can reduce the flickering phenomenon. In dynamic scenes, especially when objects move quickly, flickering or "wandering" phenomenon may occur. The TAA reduces these unstable visual effects by interpolation between frames in the temporal domain. The TAA technique can improve visual stability. The TAA can provide a smoother visual experience at low frame rates, making fast-moving scenes look more stable and coherent.

[0052] Next, the application scenarios of the embodiments of the present application are introduced. The embodiments of the present application can be applied to electronic devices with image processing functions. As shown in FIG. 1, the electronic device can include a processor, a memory, a multimedia component, a sensor component, and a power supply. Figure 2 The related processes of the TAA technique are illustrated. The history data of the previous frame image is output to the processing module of the current frame. The processing module of the current frame image is rendered by jitter offset to obtain the rendering result of the current frame image. The processing module of the current frame image resamples according to the history data of the previous frame image and the motion vectors of the rendering result of the current frame image. The processing module of the current frame image validates according to the color samples of the current frame image and the resampling result. The processing module of the current frame image blends according to the rendering result of the current frame image and the validation result. The processing module of the current frame image performs post processing on the color accumulation result and sends it to the display device for display. The processing module of the current frame image outputs the color blending result to the processing module of the next frame.

[0053] Next, the application scenarios of the embodiments of the present application are introduced. The embodiments of the present application can be applied to electronic devices with image processing functions. As shown in FIG. 1, the electronic device can include a processor, a memory, a multimedia component, a sensor component, and a power supply. Figure 3

[0054] ​The memory includes an internal memory and an external memory. The internal memory can be a memory and is integrated with the processor. The memory is configured to store data, software programs and software modules, and mainly includes a program storage region and a data storage region. The program storage region can store an operating system and at least one application program required by a function implementation module; and the data storage region can store various data created according to the use state of the electronic device, such as audio data, display data or table data. The memory can further include a high-speed random access memory or a nonvolatile memory, such as at least one disk storage device, a flash memory device or other volatile solid-state memory device.

[0055] The processor is the control center of the electronic device, connects all parts of the device through different interfaces and lines, and realizes various functions of the electronic device by running or executing the application programs stored in the memory and calling various data in the memory. The processor usually includes one or more processing units. In the embodiments of the present application, the processor can include a central processing unit (CPU), a graphics processing unit (GPU). Optionally, the processor can further include an application processor (AP), a modem processor, an image signal processor (ISP), a controller, a video playing codec, a digital signal processor (DSP), a baseband processor and / or a neural-network processing unit (NPU), etc. Taking the processor as the GPU as an example, the memory stores instructions, and when the GPU runs the instructions, the electronic device realizes at least one step in the method embodiments described below.

[0056] The multimedia component includes a display screen and an audio / video. The display screen can be a touch panel including one or more touch sensors configured to receive an input signal of a user, and the input signal can be a sliding, touching or clicking trigger operation of the user on the display screen. The display screen is also configured to receive and display to-be-displayed data. In addition, when the electronic device is in a video playing mode, the audio / video can support the electronic device to play a video.

[0057] The sensor assembly includes one or more sensors, such as a power consumption sensor and an application sensor, for providing state evaluation of various aspects of the electronic device. Through the sensor assembly, the power consumption, orientation, open / close state, relative positioning of components, or temperature change of the electronic device can be detected. The sensor assembly can further include a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor, etc. In addition, the sensor assembly can also include a scene sensor for detecting the application scenario of the electronic device, such as a 2D watch face, time display, sports health, information display, mobile payment, mobile call, video playback, music playback, or 3D watch face, etc. The power supply is used to provide power supply for various components of the electronic device, and the power supply can include a power management system, one or more power supplies, or other components associated with the generation, management, and distribution of power of the electronic device.

[0058] In one possible implementation, the VRS technique can be implemented through a just noticeable difference (JND) loss function. The JND loss function is obtained by quantitative analysis according to the relationship between image quality loss, coloration ratio, and motion. The JND loss function can be used in image quality evaluation, image enhancement, image compression, and other application scenarios. The human visual system (HVS) refers to the fact that humans are less sensitive to some subtle differences in images and more sensitive to other parts. The JND loss function introduces the perception characteristics of the HVS, which can make the model pay more attention to the parts of the image that are more sensitive to human perception, thereby improving the visual effect of the image processing task.

[0059] In this implementation, the JND loss function determines the coloration ratio of different regions of the image frame based on the features of natural pictures. Since there are differences between some game rendering picture features and natural pictures, this implementation cannot achieve the best effect, and unnatural objects may appear in some scenes. In addition, this implementation may have adverse effects of loop feedback, such as sawtooth and flicker problems, resulting in low visual quality of the image.

[0060] In another possible implementation, the VRS technique can be implemented by a Sobel edge detection function. The Sobel edge detection function can detect which pixels are located on the edge of an object, and for the pixels on the edge of the object, a smaller-granularity shading ratio is used to improve the picture accuracy of the object edge. This implementation is based on Sobel edge detection to determine the area where the VRS technique can be used, and by adjusting the threshold, the shading ratio corresponding to different rendering pipelines is determined, such as an aggressive (larger-granularity) or conservative (smaller-granularity) shading ratio. By analyzing different areas in the game scene, different shading ratios are selectively applied. For example, high-detail areas (objects of user interest, important elements in the foreground) can be assigned a smaller-granularity shading ratio to ensure clear visual details. Low-detail areas (such as large flat areas in the background, areas with dark lighting or significant motion blur effects) can be assigned a larger-granularity shading ratio to reduce the workload of the GPU and ensure that the visual experience of the user is not significantly affected.

[0061] In some examples, in the development stage, the developer can manually specify the shading ratio of each rendering target, allowing individual control of different rendering channels (such as color, depth, normal, etc.). Taking an image processing scene in a game as an example, this example applies to the development stage, and the rendering channel that uses the VRS technique is determined according to the characteristics of the game during game production. In addition, this example adjusts the VRS calculation threshold for each rendering channel one by one to determine the shading ratio of each rendering channel.

[0062] In other examples, in the running stage, the shading ratio can be determined according to the complexity, motion, brightness, and other parameters of each pixel or each primitive. This example can automatically analyze and adjust the shading rate during rendering to optimize performance to a greater extent. In addition, this example can be applied to the running stage to achieve game-agnostic and directly determine the shading ratio according to real-time pictures.

[0063] In this implementation, the first example can alleviate the problem of jaggies by adjusting the VRS threshold for each rendering channel. However, this example can only be applied to the development stage and cannot be applied to the running stage, which has a limited application scenario. The second example has the problem of jaggies, resulting in lower visual quality of the image.

[0064] In yet another possible implementation, the VRS technique can be implemented through a content-adaptive metric of judder. The content-adaptive metric of judder can improve the detection and measurement of judder in the game rendering process. Judder is a visual artifact in game rendering, usually occurring in frame rate conversion or fast motion scenes. Judder can cause the picture to appear uneven and jerky, thereby affecting the user's visual experience. The content-adaptive metric of judder dynamically adjusts the detection and evaluation criteria of judder according to game content characteristics related to judder perception, thereby establishing a more accurate measurement system. The game content characteristics related to judder perception used by the model include motion speed, motion direction, frame rate, and content complexity. The motion speed of objects or scenes in the game has a direct impact on judder perception. Fast-moving objects are more likely to produce noticeable judder. Different motion directions have different effects on visual perception, and the perception of judder may differ between horizontal and vertical directions. The frame rate directly affects the smoothness of the video, and inconsistent frame rates or frame loss can cause judder. Content complexity includes the complexity of the game scene, such as the level of detail, brightness changes, etc. Content complexity can affect the perception of judder. For example, scenes with rich details are more likely to make people perceive judder.

[0065] In this implementation, using game content characteristics related to judder perception to determine the shading ratio can solve the problem of jaggies. However, this calculation model calculates the shading ratio based on textures rather than rendering results, and will produce obvious unnatural objects in scenarios such as specular reflection, semi-transparent objects, particle characteristics, etc. Moreover, this calculation model is very complex, with large computational and memory overheads, and is not real-time. Therefore, this calculation model can only be applied in the development stage and cannot be applied in the running stage.

[0066] Embodiments of the present application provide an image frame processing method that can combine TAA technology and VRS technology to solve the problem of jaggies while applying different shading ratios in different areas. The method can be applied to a GPU in an electronic device, such as Figure 4 As shown in the figure, the method can at least include:

[0067] S100: The GPU outputs a first image frame.

[0068] In this embodiment, the GPU divides each image frame it outputs into multiple tiles, and the first image frame includes a first tile.

[0069] S200: After outputting the first image frame, the GPU outputs a second image frame.

[0070] The second image frame includes a second tile and a third tile, and the first tile in the first image frame and the second tile in the second image frame are at the same pixel position.

[0071] In an example, the color value of the pixel of the second tile is determined according to a product of the color value of the pixel of the first tile and a rectification mask, and the shading ratio of the second tile is different from the shading ratio of the third tile.

[0072] In an example, the GPU obtains the rectification mask of the pixel of the first tile of the first image frame according to the TAA technology, and obtains the initial color value of the pixel of the second tile of the second image frame. The color value of the pixel of the second image frame can be equal to: the color value of the pixel of the first tile * the rectification mask of the pixel of the first tile + the initial color value of the pixel of the second tile * (1-the rectification mask of the pixel of the first tile).

[0073] In this embodiment, the color value of the pixel of the second tile is determined according to the color value of the pixel of the first tile and the rectification mask, so that the second image frame can refer to the color value of the first image frame. In this way, the color value of the same region of the second image frame and the first image frame will not be particularly different, and the problems of jaggy and flicker can be solved, and the visual quality of the image can be improved.

[0074] In a possible implementation, as shown in FIG. 1, Figure 4 The first image frame further includes a fourth tile, and the fourth tile of the first image frame is at the same pixel position as the third tile of the second image frame. The rectification mask of the pixel of the first tile is smaller than the rectification mask of the pixel of the fourth tile, and the granularity of the shading ratio of the second tile is smaller than the granularity of the shading ratio of the third tile.

[0075] In an example, the shading rate map (SRM) of the second image frame is determined according to the rectification mask of the pixel of the tile of the first image frame. The shading rate map can include the shading ratio of a plurality of tiles of an image frame. When the rectification mask of the pixel of the first tile is close to 0, a conservative (smaller granularity) shading ratio can be used for the second tile to ensure the rendering effect; when the rectification mask of the pixel of the first tile is close to 1, an aggressive (larger granularity) shading ratio can be used for the second tile to save rendering overhead. Each pixel corresponds to a rectification mask, and the rectification mask of the first tile can be a reference value obtained by averaging, taking the middle value or other calculation methods of a plurality of rectification masks of a plurality of pixels of the first tile. The description of the rectification mask of the pixel of the fourth tile can refer to the description of the rectification mask of the pixel of the first tile, which will not be described herein again.

[0076] In the embodiment, the shading ratio map of the second image frame is determined according to the check mask of the first image frame. For the region with a large check mask of the previous frame image, the current frame image has a high degree of reuse of the color value of the previous frame image, and the image quality of the region is relatively stable. Therefore, the granularity of the shading ratio of the region can be large. For the region with a small check mask of the previous frame image, the current frame image has a low degree of reuse of the color value of the previous frame image, that is, the color value of the region changes greatly. Therefore, the region needs to use a small granularity of the shading ratio to ensure the image quality. The embodiment dynamically determines the shading ratio map of the second image frame according to the check mask of the first image frame. For example, the TAA technology is used to super-sample and anti-alias the variable resolution rendering result, and the check mask of the previous frame image is output to the VRS module of the next frame image as a compensation signal for generating the shading ratio map. In this way, the effect of stabilizing the image quality can be achieved, the variable resolution shading with higher quality can be realized, and the rendering load can be further reduced. Moreover, the embodiment supports using the shading ratio with a large granularity, and the rendering overhead can be greatly reduced. In addition, the shading according to the check mask can solve the problems of aliasing and flickering, and the time-domain accumulation effect is more obvious. Moreover, the embodiment does not depend on natural picture features, and can render non-natural pictures such as game pictures. Furthermore, the embodiment adds the check mask to the calculation of the shading ratio, does not introduce an additional calculation module, and has a simple calculation process. The embodiment can be compatible with the existing rendering pipeline and multiple TAA technologies such as deep learning super sampling (DLSS). In addition, the embodiment has a simple calculation process, and can be applied to the running stage, for example, can be applied to the game running stage, and can be realized without the awareness of the game. The embodiment can also be applied to the development stage. Therefore, the application scenarios of the embodiment are more extensive.

[0077] In a possible implementation, as shown in Figure 5 The shading ratio of the second tile is determined according to the check mask of the pixels of the first tile and the rendering parameter. The rendering parameter includes at least one of the following: the luminance gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame.

[0078] For example, the GPU can run a rendering parameter module and a shading ratio calculation module. The rendering parameter module is configured to determine the rendering parameter and output the rendering parameter to the shading ratio calculation module. The shading ratio calculation module is configured to input the check mask and the rendering parameter and output the shading ratio.

[0079] For example, the rendering parameters include the luminance gradient of the first tile and the motion vector of the second tile. The rendering parameter module inputs the luminance of all pixels of the first tile of the first frame image, and outputs a plurality of luminance gradients corresponding to the first tile of the first frame image. The shading ratio calculation module inputs the minimum value of the absolute values of the motion vectors of all pixels of the second tile of the second frame image, and inputs the plurality of luminance gradients corresponding to the first tile, and inputs the check mask of all pixels of the first tile. The motion vector of each pixel of the second tile is obtained according to the vertex movement position of the corresponding pixel of the first tile. The shading ratio calculation module outputs the shading ratio of the second tile of the second frame image.

[0080] In this embodiment, by using the check mask, the luminance gradient and other parameters of the tile of the previous frame image are multiplexed to adjust the rendering resolution of different regions. The shading ratio of the second tile is determined according to the check mask of the pixels of the first tile and the rendering parameters, and the rendering parameters include at least one of the following: the luminance gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame. The shading ratio can be calculated using the easily obtained rendering parameters and the check mask, the calculation method of the shading ratio is relatively simple, and the existing rendering pipeline can be compatible.

[0081] In another possible implementation, as shown in Figure 6 The shading ratio of the second tile is obtained by correcting the initial shading ratio of the second tile according to the check mask of the pixels of the first tile. The initial shading ratio is determined according to the rendering parameters, and the rendering parameters include at least one of the following: the luminance gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame.

[0082] For example, the GPU can also run a multiplexing check compensation module. The rendering parameter module is used to determine the rendering parameters and output the rendering parameters to the shading ratio calculation module. The shading ratio calculation module is used to input the rendering parameters and output the initial shading ratio. The multiplexing check compensation module is used to input the initial shading ratio and the check mask, correct the shading ratio according to the check mask, and output the corrected shading ratio.

[0083] For example, the rendering parameters include the luminance gradient of the first tile and the motion vector of the second tile. The rendering parameter module inputs the luminance of all pixels of the first tile of the first frame image, and outputs a plurality of luminance gradients corresponding to the first tile of the first frame image. The shading ratio calculation module inputs the minimum value of the absolute values of the motion vectors of all pixels of the second tile of the second frame image, and inputs the plurality of luminance gradients. The motion vector of each pixel of the second tile is obtained according to the vertex movement position of the corresponding pixel of the first tile. The shading ratio calculation module outputs the initial shading ratio of the second tile of the second frame image. The multiplexing verification compensation module inputs the initial shading ratio of the second tile, and inputs the verification mask of all pixels of the first tile, and outputs the corrected shading ratio of the second tile.

[0084] In this embodiment, the initial shading ratio of the second tile is determined according to the rendering parameters, and the change direction and granularity of the shading ratio are corrected by the verification mask. The rendering parameters include at least one of the following: the luminance gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second frame image. The shading ratio can be calculated by using the easily obtained rendering parameters, and then corrected by the verification mask. The calculation method of the shading ratio is relatively simple, and can be compatible with the existing rendering pipeline.

[0085] In one possible implementation, as shown in Figure 5 and Figure 6 , the shading ratio of the second tile and the shading ratio of the third tile are smoothed.

[0086] For example, the GPU can also run a smoothing module. The smoothing module is used to smooth the shading ratio of the tile of the second frame image. For example, the granularity change of the shading ratio of the adjacent regions should not be too large, so as to ensure the smoothness of the image in different regions. In this embodiment, the shading ratio of the tile of the second frame image is smoothed, which can further improve the image quality.

[0087] Next, combined with Figure 7The image frame processing method is exemplified. In a graphics rendering pipeline, a GPU applies input image data to be processed. After geometry processing, the data is rasterized. The geometry processing can include vertex shading, projection clipping, screen mapping, and calculation of a shading rate map, etc. In the calculation of the shading rate map, the GPU calculates a gradient of a tile of a previous frame of image based on color values of the previous frame of image, and calculates the shading rate map based on the gradient of the previous frame of image, a motion vector of a current frame of image, and a validation mask of the previous frame of image. The GPU performs pixel processing (such as fragment shading and blending, etc.) on the rasterized image data based on the shading rate map to obtain initial color values of the current frame of image. The GPU can use the calculated shading rate map to perform fragment shading. The GPU performs history verification based on the color values of the previous frame of image and the initial color values of the current frame of image to obtain the validation mask of the current frame of image. The GPU performs sample accumulation based on the validation mask of the current frame of image and the initial color values of the current frame of image to obtain color values of the current frame of image. The validation mask of the current frame of image and the color values of the current frame of image can be used for calculation of a next frame of image.

[0088] The image frame processing method and the electronic device applying the method are described above. It can be understood that the electronic device includes hardware structures and / or software modules corresponding to each function to realize the above functions. Those skilled in the art should easily realize that the structures and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0089] The embodiments of the present application can divide the functional modules according to the image frame processing apparatus corresponding to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The integrated module can be realized in the form of hardware or software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical functional division. Actual implementation can have another division manner.

[0090] In the case of dividing each functional module according to each function, Figure 8A possible image frame processing apparatus is shown. The image frame processing apparatus includes a first output module and a second output module. The first output module is configured to output a first image frame, the first image frame including a first tile. The second output module is further configured to output a second image frame after the first output module outputs the first image frame, the second image frame including a second tile and a third tile, the first tile in the first image frame and the second tile in the second image frame being at a same pixel position. Wherein a color value of a pixel of the second tile is determined according to a product of a color value of a pixel of the first tile and a check mask, and a shading ratio of the second tile is different from a shading ratio of the third tile.

[0091] In a possible implementation, the first image frame further includes a fourth tile, the fourth tile of the first image frame and the third tile of the second image frame being at the same pixel position. The check mask of the pixel of the first tile is smaller than the check mask of the pixel of the fourth tile, and the granularity of the shading ratio of the second tile is smaller than the granularity of the shading ratio of the third tile.

[0092] In another possible implementation, the shading ratio of the second tile is determined according to the check mask of the pixel of the first tile and a rendering parameter. The rendering parameter includes at least one of the following: a luminance gradient of the first tile, a motion vector of the second tile, a motion speed of the second tile, and a frame rate of the second image frame.

[0093] In another possible implementation, the shading ratio of the second tile is obtained by correcting an initial shading ratio of the second tile according to the check mask of the pixel of the first tile. The initial shading ratio is determined according to a rendering parameter, and the rendering parameter includes at least one of the following: a luminance gradient of the first tile, a motion vector of the second tile, a motion speed of the second tile, and a frame rate of the second image frame.

[0094] In another possible implementation, the shading ratio of the second tile and the shading ratio of the third tile are smoothed.

[0095] The embodiments of the present application further provide an electronic device. The structure of the electronic device can refer to the electronic device shown in Figure 3 The electronic device includes a memory and a GPU. The memory stores instructions, and when the GPU runs the instructions, the electronic device implements one or more steps of the above-mentioned method embodiments.

[0096] The above-mentioned electronic device or GPU provided by the embodiments of the present application are respectively used to realize the functions of the steps of the corresponding image frame processing method. Since the steps have been described in detail in the foregoing image frame processing method embodiments, they will not be described here.

[0097] The embodiment of the present application further provides a computer readable storage medium, which stores program codes, and when the program codes are run on a device (for example, the device can be a single-chip microcomputer, a chip, a computer or a processor, etc.), the program codes can be called by a processor to execute one or more steps in the above method embodiments.

[0098] Based on the understanding, the embodiment of the present application further provides a computer program product containing instructions, the technical solution of the present application or the whole or part of the contribution to the prior art can be embodied in the form of a software product, the computer software product is stored in a storage medium, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor therein execute all or part of the steps of the method described in various embodiments of the present application.

[0099] Finally, it should be noted that: the above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this, any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image frame processing method, characterized in that, The method includes: Output a first image frame, the first image frame including a first tile; After outputting the first image frame, a second image frame is output. The second image frame includes a second tile and a third tile, and the first tile in the first image frame and the second tile in the second image frame are at the same pixel position. The color value of the second tile's pixels depends on the product of the color value of the first tile's pixels and the check mask. The coloring ratio of the second tile is different from that of the third tile.

2. The method according to claim 1, characterized in that, The first image frame also includes a fourth tile, which is at the same pixel position as the third tile of the second image frame; The check mask of the pixels of the first tile is smaller than the check mask of the pixels of the fourth tile, and the granularity of the shading ratio of the second tile is smaller than the granularity of the shading ratio of the third tile.

3. The method according to claim 1 or 2, characterized in that, The shading ratio of the second tile is determined based on the checksum mask of the pixels of the first tile and the rendering parameters; The rendering parameters include at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame.

4. The method according to claim 1 or 2, characterized in that, The shading ratio of the second tile is obtained by correcting the initial shading ratio of the second tile based on the check mask of the pixels of the first tile; The initial shading ratio is determined based on rendering parameters, which include at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame.

5. The method according to any one of claims 1-4, characterized in that, The color ratios of the second tile and the third tile are smoothed.

6. An image frame processing apparatus, characterized in that, The device includes a first output module and a second output module; The first output module is used to output the first image frame, the first image frame including the first tile; The second output module is further configured to output a second image frame after outputting the first image frame, the second image frame including a second tile and a third tile, wherein the first tile in the first image frame and the second tile in the second image frame are at the same pixel position; The color value of the second tile's pixels depends on the product of the color value of the first tile's pixels and the check mask. The coloring ratio of the second tile is different from that of the third tile.

7. The apparatus according to claim 6, characterized in that, The first image frame also includes a fourth tile, which is at the same pixel position as the third tile of the second image frame; The check mask of the pixels of the first tile is smaller than the check mask of the pixels of the fourth tile, and the granularity of the shading ratio of the second tile is smaller than the granularity of the shading ratio of the third tile.

8. The apparatus according to claim 6 or 7, characterized in that, The shading ratio of the second tile is determined based on the checksum mask of the pixels of the first tile and the rendering parameters; The rendering parameters include at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame.

9. The apparatus according to claim 6 or 7, characterized in that, The shading ratio of the second tile is obtained by correcting the initial shading ratio of the second tile based on the check mask of the pixels of the first tile; The initial shading ratio is determined based on rendering parameters, which include at least one of the following: the brightness gradient of the first tile, the motion vector of the second tile, the motion speed of the second tile, and the frame rate of the second image frame.

10. The apparatus according to any one of claims 6-9, characterized in that, The color ratios of the second tile and the third tile are smoothed.

11. A graphics processor, characterized in that, The graphics processor is used to perform the method as described in any one of claims 1-5.

12. An electronic device, characterized in that, The electronic device includes a memory and a graphics processor as claimed in claim 11, wherein the memory stores instructions that, when executed by the graphics processor, cause the electronic device to perform the method as claimed in any one of claims 1-5.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method described in any one of claims 1-5.

14. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-5.

Images