Image super-division display method and device, terminal, electronic equipment and storage medium

Abstract

The invention relates to an image super-resolution display method and device, a terminal, electronic equipment and a storage medium, and the method comprises the steps: determining known pixels and unknown pixels of a first target frame under a display resolution, and enabling the known pixels to be pixels obtained by rendering the first target frame under a rendering resolution; determining an expansion depth and an expansion speed corresponding to each first target pixel, wherein the first target pixels belong to unknown pixels; determining corresponding historical pixel credibility based on the expansion depth corresponding to the first target pixel; under the condition that the historical pixel credibility and the expansion speed corresponding to the first target pixel meet preset requirements, taking a color information sampling result of a color map of the second target pixel under the rendering resolution of the second target frame as color information of the first target pixel; otherwise, predicting color information of the first target pixel; and displaying the first target frame under the display resolution based on the color information of the known pixel and the first target pixel. According to the invention, image super-resolution display can be accurately carried out.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to an image super-resolution display method and device, a terminal, an electronic device and a storage medium. BACKGROUND

[0002] The image super-resolution display scheme of the related art is inevitably implemented in a color prediction manner. There are many color prediction methods, and the commonly used one is color interpolation. Taking the image super-resolution display scheme of a mobile terminal as an example, it usually uses a display resolution position as the center, and predicts the color of the position according to the colors of the adjacent pixels on the low-resolution rendered image corresponding to the position. The color prediction result is inevitably disturbed by noise to cause image blur. The more the pixels using color prediction, the more serious the image blur problem will be. Therefore, the definition of the image super-resolution display result is significantly affected, resulting in a decline in user experience. This problem is more prominent on high-resolution display devices. Moreover, color prediction also slows down the display speed. The more the pixels using color prediction, the slower the display speed will be. SUMMARY

[0003] The present disclosure provides an image super-resolution display method, device, terminal, electronic device and storage medium to solve the problems in the related art. The technical scheme of the present disclosure is as follows. According to a first aspect of an embodiment of the present disclosure, an image super-resolution display method is provided, and the method comprises: determining known pixels and unknown pixels of a first target frame under a display resolution, the known pixels being pixels obtained by rendering the first target frame under a rendering resolution, and the unknown pixels being other pixels except the known pixels, the display resolution being higher than the rendering resolution; determining an expanded depth and an expanded speed corresponding to each first target pixel based on a depth map and a speed map corresponding to the first target frame under the rendering resolution, the first target pixel belonging to the unknown pixels; for each first target pixel, determining a historical pixel reliability corresponding to the first target pixel based on the expanded depth corresponding to the first target pixel, the historical pixel reliability indicating a reliability degree of using color information of a second target pixel as color information of the first target pixel, the second target pixel being a pixel corresponding to the first target pixel under a display resolution of a second target frame, the second target frame being an image frame before the first target frame; in a case where the historical pixel reliability corresponding to the first target pixel and the expanded speed corresponding to the first target pixel both satisfy a preset requirement, sampling color information of a color map of the second target pixel under the rendering resolution of the second target frame as the color information of the first target pixel. predicting color information of the first target pixel in a case where the preset requirement is not met; rendering the first target frame under the display resolution based on the color information of the known pixels and the first target pixels.

[0004] In an exemplary embodiment, the determining of the dilated depth and the dilated velocity corresponding to each first target pixel based on the depth map and the velocity map corresponding to the first target frame under the rendering resolution comprises: determining, for any first target pixel, a depth value corresponding to each pixel in a first preset neighborhood centered at the first target pixel in a depth map corresponding to the first target frame under the rendering resolution; determining, as a dilated target pixel, a pixel corresponding to a depth value indicating the closest distance to the screen among the depth values; taking, as the dilated depth corresponding to the first target pixel, a depth value sampled from the dilated target pixel in the depth map corresponding to the first target frame under the rendering resolution; taking, as the dilated velocity corresponding to the first target pixel, a velocity sampled from the dilated target pixel in a velocity map corresponding to the first target frame under the rendering resolution.

[0005] In an exemplary embodiment, the determining of the historical pixel reliability corresponding to the first target pixel based on the dilated depth corresponding to the first target pixel comprises: determining, based on the dilated depth corresponding to the first target pixel and position information of the first target pixel, position information of the second target pixel; in a case where it is determined based on the position information that the second target pixel is not located in a rendering range corresponding to the second target frame, determining the historical pixel reliability as a target reliability, the target reliability indicating untrustworthiness.

[0006] In an exemplary embodiment, the determining of the historical pixel reliability corresponding to the first target pixel based on the dilated depth corresponding to the first target pixel comprises: determining, based on the dilated depth corresponding to the first target pixel and position information of the first target pixel, an expected depth corresponding to the second target pixel; extracting an actual depth corresponding to the second target pixel in a depth map corresponding to the second target frame under the rendering resolution; in a case where the actual depth is less than the expected depth and an absolute value of a difference is greater than a first preset threshold, determining the historical pixel reliability as a target reliability, the target reliability indicating untrustworthiness.

[0007] In an exemplary embodiment, the method further comprises: In a case that the historical pixel reliability corresponding to the first target pixel is reliable and a speed value of the expansion speed corresponding to the first target pixel is less than a second preset threshold, it is determined that both the historical pixel reliability corresponding to the first target pixel and the expansion speed corresponding to the first target pixel satisfy the preset requirement.

[0008] In an exemplary embodiment, the predicting the color information of the first target pixel comprises: Within a second preset neighborhood centered on the first target pixel, two associated pixels corresponding to each direction are determined, the two associated pixels being distributed on both sides of the first target pixel along the corresponding direction, and each of the associated pixels belonging to the known pixels; For each direction, a fusion reference color corresponding to the direction is determined by fusing color information rendered by the corresponding two associated pixels at the rendering resolution, respectively; The fusion reference colors corresponding to each direction are fused to obtain a reference basic color of the first target pixel; The color information of the first target pixel is predicted based on the reference basic color of the first target pixel.

[0009] In an exemplary embodiment, the determining a fusion reference color corresponding to each direction by fusing color information rendered by the corresponding two associated pixels at the rendering resolution, respectively, comprises: A fusion reference weight is determined based on color consistency between color information corresponding to the two associated pixels, respectively, the fusion reference weight being positively correlated with the color consistency; In a case that the two associated pixels belong to the same object, the fusion reference color corresponding to the direction is obtained by fusing color information corresponding to the two associated pixels based on the fusion reference weight.

[0010] In an exemplary embodiment, the method further comprises: A probability that the first target pixel is located at an object boundary is determined; In a case that the probability is less than a third preset threshold, it is determined that the two associated elements belong to the same object.

[0011] In an exemplary embodiment, the predicting the color information of the first target pixel based on the reference basic color of the first target pixel comprises: Reference color information corresponding to the second target pixel is determined based on color information of each pixel within a third preset neighborhood sampled from a color map of the second target frame at the display resolution and centered on the second target pixel; determine a fusion weight based on the determined probability that the first target pixel is located at the object boundary and the dilation depth of the first target pixel; determine historical reference color information corresponding to the first target pixel based on the fusion weight and fusion of reference color information corresponding to the second target pixel and color information of the second target pixel sampled from the color map of the second target frame at the display resolution; fuse the historical reference color information and reference base color of the first target pixel to obtain color information of the first target pixel.

[0012] In an exemplary embodiment, the image super-resolution display method is performed at the display resolution and is rendered based on a Tile of a graphics processing unit in a mobile terminal, the probability that the first target pixel is located at the object boundary, the historical pixel credibility corresponding to the first target pixel, and the dilation depth and dilation speed corresponding to each first target pixel are all stored in a Tile video memory; determine the probability that the first target pixel is located at the object boundary using a pixel shader.

[0013] In an exemplary embodiment, in the case that the known pixel is an odd row pixel, the unknown pixel is an even row pixel; In the case that the known pixel is an even row pixel, the unknown pixel is an odd row pixel; In the case that the known pixel is an odd column pixel, the unknown pixel is an even column pixel; In the case that the known pixel is an even column pixel, the unknown pixel is an odd column pixel.

[0014] According to a second aspect of the embodiments of the present disclosure, an image super-resolution display device is provided, and the device comprises: an image processing module configured to determine known pixels and unknown pixels of a first target frame at a display resolution, the known pixels being pixels obtained by rendering the first target frame at a rendering resolution, and the unknown pixels being other pixels except the known pixels, the display resolution being higher than the rendering resolution; a data dilation module configured to determine a dilation depth and a dilation speed corresponding to each first target pixel based on a depth map and a speed map corresponding to the first target frame at the rendering resolution, the first target pixel belonging to the unknown pixels; a historical pixel credibility determination module configured to determine, for each of the first target pixels, a historical pixel credibility corresponding to the first target pixel based on the expansion depth corresponding to the first target pixel, the historical pixel credibility indicating a degree of credibility of color information of a second target pixel as color information of the first target pixel, the second target pixel being a pixel corresponding to the first target pixel in a second target frame at the display resolution, the second target frame being an image frame prior to the first target frame; a pixel processing module configured to perform, in a case where both the historical pixel credibility corresponding to the first target pixel and the expansion speed corresponding to the first target pixel satisfy a preset requirement, sampling of color information of a color map of the second target pixel in the second target frame at the rendering resolution as color information of the first target pixel, and in a case where the preset requirement is not satisfied, predicting color information of the first target pixel; a display module configured to perform, based on the rendered known pixels and the color information of each of the first target pixels, display of the first target frame at the display resolution.

[0015] In an exemplary embodiment, the data expansion module is configured to perform: determining, for any of the first target pixels, a respective depth value in a depth map corresponding to the first target frame at the rendering resolution for each pixel in a first preset neighborhood centered on the first target pixel; determining, as an expansion target pixel, a pixel corresponding to a depth value indicating a closest distance to the screen among the respective depth values; determining, as the expansion depth corresponding to the first target pixel, a depth value sampled in the depth map corresponding to the first target frame at the rendering resolution for the expansion target pixel; determining, as the expansion speed corresponding to the first target pixel, a speed value sampled in a speed map corresponding to the first target frame at the rendering resolution for the expansion target pixel.

[0016] In an exemplary embodiment, the historical pixel credibility determination module is configured to perform: determining, based on the expansion depth corresponding to the first target pixel and position information of the first target pixel, position information of the second target pixel; in a case where it is determined based on the position information that the second target pixel is not located within a rendering range corresponding to the second target frame, determining the historical pixel credibility as a target credibility, the target credibility indicating uncredibility.

[0017] In an exemplary embodiment, the historical pixel credibility determination module is configured to perform: determine an expected depth of the second target pixel based on the dilation depth corresponding to the first target pixel and position information of the first target pixel; extract an actual depth of the second target pixel in a depth map of the second target frame at the rendering resolution; in a case where the actual depth is less than the expected depth and an absolute value of a difference is greater than a first preset threshold, determine the historical pixel credibility as a target credibility, the target credibility indicating untrustworthy.

[0018] In an exemplary embodiment, the pixel processing module is configured to perform: in a case where the historical pixel credibility corresponding to the first target pixel indicates trustworthy and a speed value of the dilation speed corresponding to the first target pixel is less than a second preset threshold, determine that both the historical pixel credibility corresponding to the first target pixel and the corresponding dilation speed satisfy the preset requirement.

[0019] In an exemplary embodiment, the pixel processing module is configured to perform: determine, in a second preset neighborhood centered on the first target pixel, two associated pixels corresponding to each of a plurality of directions, the two associated pixels being distributed on two sides of the first target pixel along the corresponding direction, each of the associated pixels belonging to the known pixels; for each of the directions, determine a fusion reference color corresponding to the direction by fusing color information rendered at the rendering resolution by the two associated pixels respectively; fuse the fusion reference colors corresponding to each of the directions to obtain a reference base color of the first target pixel; predict the color information of the first target pixel based on the reference base color of the first target pixel.

[0020] In an exemplary embodiment, the pixel processing module is configured to perform: determine a fusion reference weight based on color consistency between the color information respectively corresponding to the two associated pixels, the fusion reference weight being positively correlated with the color consistency; in a case where the two associated pixels belong to the same object, fuse the color information respectively corresponding to the two associated pixels based on the fusion reference weight to obtain a corresponding fusion reference color.

[0021] In an exemplary embodiment, the pixel processing module is configured to perform: determine a probability that the first target pixel is located at an object boundary; In a case where the probability is less than a third preset threshold, it is determined that the two associated elements belong to the same object.

[0022] In an exemplary embodiment, the pixel processing module is configured to perform: determine reference color information corresponding to the second target pixel based on color information of each pixel within a third preset neighborhood sampled from a color graph of the second target frame at the display resolution and centered on the second target pixel; determine a fusion weight based on the probability that the first target pixel is located at the object boundary and the dilation depth of the first target pixel; determine historical reference color information corresponding to the first target pixel based on the fusion weight and fusion of the reference color information corresponding to the second target pixel and the color information of the second target pixel sampled from the color graph of the second target frame at the display resolution; fuse the historical reference color information and the reference base color of the first target pixel to obtain color information of the first target pixel.

[0023] In an exemplary embodiment, the image super-resolution display device performs rendering at the display resolution and based on a Tile of a graphics processing unit in a mobile terminal, the probability that the first target pixel is located at the object boundary, the historical pixel reliability corresponding to the first target pixel, and the dilation depth and dilation speed corresponding to each first target pixel are all stored in a Tile video memory; determine the probability that the first target pixel is located at the object boundary using a pixel shader.

[0024] In an exemplary embodiment, in a case where the known pixel is an odd row pixel, the unknown pixel is an even row pixel; In a case where the known pixel is an even row pixel, the unknown pixel is an odd row pixel; In a case where the known pixel is an odd column pixel, the unknown pixel is an even column pixel; In a case where the known pixel is an even column pixel, the unknown pixel is an odd column pixel.

[0025] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image super-resolution display method according to any of the above embodiments.

[0026] According to a fourth aspect of the embodiments of the present disclosure, a computer storage medium is provided. When instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device performs the image super-resolution display method in any of the above embodiments.

[0027] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided. The computer program product comprises a computer program which, when executed by a processor, implements the image super-resolution display method in any of the above embodiments.

[0028] The technical solutions provided by the embodiments of the present disclosure at least have the following beneficial effects: The image super-resolution display method provided by the embodiments of the present disclosure does not perform color information prediction on each pixel corresponding to a display resolution. Instead, the known pixels and unknown pixels are first distinguished. The known pixels are directly rendered by rendering the pixels corresponding to the rendering resolution, without color information prediction. For the unknown pixels, whether the corresponding expansion speed and historical pixel credibility meet the preset requirements is determined. If yes, the unknown pixels are displayed by reusing the previously rendered images, without color information prediction. Only the unknown pixels that do not meet the preset requirements need to be subjected to color information prediction, which greatly reduces the number of pixels that need to be subjected to color information prediction.

[0029] The color information prediction process inevitably needs to use an interpolation algorithm, which introduces noise, thereby reducing the image clarity and accuracy to some extent and slowing down the display speed. The number of pixels that need to be subjected to color information prediction is greatly reduced in the present disclosure, which reduces the clarity loss, accuracy loss and speed reduction caused by color information prediction, thereby improving the clarity, accuracy and display speed of the image super-resolution result.

[0030] Generally, the color information prediction process may produce a jagged effect due to insufficient prediction, thereby affecting the image expressiveness. The number of pixels that need to be subjected to color information prediction is significantly reduced in the present disclosure, thereby having a significant anti-jagged effect.

[0031] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0033] Figure 1This is a flowchart illustrating an image super-resolution display method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating a method for determining expansion depth and expansion speed according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating the expansion effect according to an exemplary embodiment; Figure 4 This is a flowchart illustrating a color prediction method according to an exemplary embodiment; Figure 5 This is a distribution map of known and unknown pixels according to an exemplary embodiment; Figure 6 This is a schematic diagram of the architecture of a super-resolution display scheme according to an exemplary embodiment; Figure 7 This is a block diagram of an image super-resolution display device according to an exemplary embodiment; Figure 8 This is a structural block diagram of a computer device according to an exemplary embodiment. Figure 1 ; Figure 9 This is a structural block diagram of a computer device according to an exemplary embodiment. Figure 2 . Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0035] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] Figure 1This is a flowchart illustrating an image super-resolution display method according to an exemplary embodiment. The image super-resolution display method can be applied to an electronic device, which can be implemented by a server or a terminal alone, or by a terminal and a server working together. The terminal can be, but is not limited to, physical devices such as smartphones, tablets, laptops, desktop computers, smart speakers, smart wearable devices, digital assistants, augmented reality devices, and virtual reality devices, and can also include software such as applications running on the physical device. The server can be, but is not limited to, a standalone server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc. (Refer to...) Figure 1 As shown, the method includes the following steps.

[0037] In S110, known pixels and unknown pixels of the first target frame at the display resolution are determined. The known pixels are pixels obtained by rendering the first target frame at the rendering resolution, and the unknown pixels are other pixels besides the known pixels. The display resolution is higher than the rendering resolution.

[0038] The method for determining known and unknown pixels can be set according to the actual situation. For example, it can be determined based on the parity of the first target frame. If the first target frame is an odd-numbered frame, then the pixels in the odd-numbered rows or columns at the display resolution are known pixels. These pixels can be directly obtained by rendering the first target frame at the rendering resolution. Correspondingly, the pixels in the even-numbered rows or columns are unknown pixels. For example, the first three rows of the first target frame at the rendering resolution are rendered as the first, third, and fifth rows at the display resolution. These pixels are known pixels, while the pixels in the second and fourth rows are unknown pixels. Similarly, if the first target frame is an even-numbered frame, then the pixels in the even-numbered rows or columns at the display resolution are known pixels. These pixels can be directly obtained by rendering the first target frame at the rendering resolution. Correspondingly, the pixels in the odd-numbered rows or columns are unknown pixels.

[0039] In other words, in determining known and unknown pixels based on the parity of the first target frame, if the known pixels are in an odd-numbered row, the unknown pixels are in an even-numbered row; if the known pixels are in an even-numbered row, the unknown pixels are in an odd-numbered row; if the known pixels are in an odd-numbered column, the unknown pixels are in an even-numbered column; and if the known pixels are in an even-numbered column, the unknown pixels are in an odd-numbered column. This method of determining known and unknown pixels has outstanding technical effects when performing image super-resolution on the mobile side because: Typically, image super-resolution on mobile devices requires textures. The rendered textures are then used for super-resolution display to obtain the final result. This process necessitates the use of a parameter called the mipmap level. The mipmap level describes the hierarchical relationship of textures at different resolutions. The mipmap level indicates the resolution of the corresponding texture, decreasing sequentially from high to low resolution. By selecting an appropriate mipmap level, the computational load during rendering can be effectively reduced while maintaining image clarity and detail. Therefore, adjusting the mipmap level is necessary for image super-resolution display, a time-consuming process that can lead to image blurring if not adjusted properly. However, the aforementioned method for determining known and unknown pixels ensures that the rendering resolution is half the display resolution horizontally and the same as the display resolution vertically, or vice versa. Therefore, there exists a direction where the rendering resolution matches the display resolution. In this case, a rasterization-based GPU rendering workflow can achieve rendering without adjusting the mipmap level and without image blurring.

[0040] Of course, other rules or algorithms can also be used to select known pixels, while the rest are marked as unknown pixels. For example, a chessboard rule can be used, where the known pixels are distributed in a chessboard pattern; this disclosure does not limit this approach.

[0041] In S120, based on the depth map and velocity map corresponding to the first target frame at the rendering resolution, the expansion depth and expansion velocity corresponding to each first target pixel are determined, and the first target pixel belongs to the unknown pixel.

[0042] Of course, this step can be performed on every pixel of the first target frame at the display resolution; this disclosure only uses the first target pixel as an example. The depth map and velocity map corresponding to the first target frame at the rendering resolution are known data. Their meanings are as follows: the depth map represents the depth information of each pixel in three-dimensional space, used to determine the distance relationship of objects in the scene; the velocity map represents the motion velocity of each pixel between consecutive frames, used to capture the dynamic changes of objects. In this disclosure, InputDepth represents the depth map corresponding to the first target frame at the rendering resolution, and InputVelocity represents the velocity map corresponding to the first target frame at the rendering resolution.

[0043] This disclosure does not limit the method of obtaining the expansion depth and expansion speed. This is achieved by expanding the depth map and velocity map corresponding to the first target frame at the rendering resolution. For example, an interpolation algorithm can be used to expand the data in the depth map and velocity map to generate more refined expansion depth and expansion speed information. In addition, a machine learning model can be used to train a prediction model based on historical frame data to estimate the expansion depth and expansion speed.

[0044] Please refer to Figure 2 This diagram illustrates a method for determining expansion depth and expansion speed in an exemplary embodiment of this disclosure. The step of determining the expansion depth and expansion speed corresponding to each first target pixel based on the depth map and velocity map corresponding to the first target frame at the rendering resolution includes: S210. For any first target pixel, determine the depth values ​​of each pixel in a first preset neighborhood centered on the first target pixel in the depth map corresponding to the first target frame at the rendering resolution.

[0045] This disclosure does not limit the size of the first preset neighborhood. For example, it can be a 3*3 neighborhood, a 5*5 neighborhood, etc. Taking a 3*3 neighborhood as an example, the depth values ​​of the nine pixels surrounding the first target pixel can be extracted from the depth map corresponding to the first target frame at the rendering resolution. Of course, if the first target pixel is located on the boundary, causing some pixels to be unable to be extracted due to exceeding the boundary, the extraction of that pixel can be directly abandoned, and the extraction result will be empty.

[0046] S220. The pixel corresponding to the depth value closest to the screen among the depth values ​​is determined as the expansion target pixel.

[0047] The depth value closest to the screen is the smallest depth value, and the pixel corresponding to this depth value is the expanded target pixel relative to the first target pixel.

[0048] S230. The depth value sampled by the expanded target pixel in the depth map corresponding to the first target frame at the rendering resolution is taken as the expansion depth corresponding to the first target pixel; the speed sampled by the expanded target pixel in the speed map corresponding to the first target frame at the rendering resolution is taken as the expansion speed corresponding to the first target pixel.

[0049] In simple terms, the depth value of the target pixel in the depth map corresponding to the first target frame at the rendering resolution is used as the dilated depth of the first target pixel, and the depth value of the target pixel in the velocity map corresponding to the first target frame at the rendering resolution is used as the dilated velocity of the first target pixel.

[0050] Please refer to Figure 3 This illustration shows a schematic diagram of the expansion effect in an exemplary embodiment of this disclosure. For a rendering resolution of 960*1080 and a display resolution of 1920*1080, the display resolution is twice the width of the rendering resolution. Figure 3 The left image shows the depth formed by the depth values ​​of relevant pixels without data dilation. Figure 1 , Figure 3 The right figure shows the depth formed by the expansion depth values ​​of relevant pixels after data dilation. Figure 2 It can be seen that the image after data expansion is slightly wider than before data expansion, reflecting its adaptability to the super-resolution width situation, or in other words, its adaptability to the super-resolution direction. Through S210-S230, the depth and velocity of each first target pixel can be expanded in accordance with the super-resolution direction, obtaining the expansion depth and expansion velocity. The expansion depth and expansion velocity are important parameters for achieving image super-resolution display in subsequent steps of this disclosure. Obtaining accurate expansion depth and expansion velocity that are adapted to the super-resolution direction can significantly improve the accuracy and clarity of the image super-resolution display results of this disclosure, and also improve the image super-resolution display speed.

[0051] S130. For each first target pixel, based on the expansion depth corresponding to the first target pixel, determine the historical pixel confidence level corresponding to the first target pixel. The historical pixel confidence level indicates the degree of reliability of using the color information of the second target pixel as the color information of the first target pixel. The second target pixel is the pixel corresponding to the first target pixel in the second target frame at the display resolution. The second target frame is the image frame before the first target frame.

[0052] In this disclosure, the reliability of historical pixels corresponding to the first target pixel can be represented by Reactive. Reactive is the reliability of historical pixels at the display resolution calculated based on the depth occlusion relationship between previous and subsequent frames and the screen coordinate range. The Reactive value is used to record the reliability of historical data. Ideally, using more data already displayed in the previous frame can maximize the advantages of temporal rendering and reduce the impact of interpolation. However, the reliability of historical data from previous frames can decrease in some cases, leading to problems such as blurring and ghosting. For example, in cases where objects have their own material animation (like a scrolling LED screen), pixels from the previous frame may be occluded by other pixels. Other examples include significant changes in lighting and shadows, the presence of semi-transparent overlays with color changes, or pixels from the previous frame not being rendered in screen space. These situations can also render the historical data from the previous frame unusable. Therefore, this disclosure proposes calculating Reactive to determine the usability of historical data from the previous frame. This disclosure does not limit the calculation method of the Reactive value and does not constitute an obstacle to implementation.

[0053] In an exemplary implementation, determining the historical pixel confidence level corresponding to the first target pixel based on the expansion depth corresponding to the first target pixel includes: determining the position information of the second target pixel based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel; if it is determined based on the position information that the second target pixel is not located within the rendering range corresponding to the second target frame, the historical pixel confidence level is determined as the target confidence level, where the target confidence level indicates unreliability.

[0054] Since the second target pixel in this disclosure is the pixel corresponding to the first target pixel in the second target frame at the display resolution, that is, the second target pixel and the first target pixel have a correspondence between consecutive frames, and given that the expansion depth corresponding to the first target pixel has been calculated in this disclosure, the corresponding position of the first target pixel in the previous frame (the second target pixel) can be deduced based on the expansion depth and the position information of the first target pixel, thereby determining the position information ReprojectPos of the second target pixel. If ReprojectPos is within the pixel range [(0,0), (ReprojectPosWidth, ReprojectPosHeight)] of the previous frame, it indicates that there is no screen boundary violation, and in this case, the historical data related to ReprojectPos is usable; otherwise, there is a screen boundary violation, and the historical data related to ReprojectPos is unusable. In this case, the historical pixel reliability is determined as the target reliability, and the target reliability indicates unreliability. ReprojectPosWidth and ReprojectPosHeight are the width and height values ​​of the rendering resolution. In this way, it can be determined whether the position of the second target pixel has a screen boundary violation, thereby judging from the perspective of whether the screen boundary violation has occurred whether the second target pixel is usable when the first target pixel is super-resolution displayed.

[0055] In an exemplary embodiment, determining the historical pixel credibility based on the expansion depth corresponding to the first target pixel includes: determining the expected depth corresponding to the second target pixel based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel; extracting the actual depth of the second target pixel in the depth map of the second target frame at the rendering resolution; and determining the historical pixel credibility as the target credibility when the actual depth is less than the expected depth and the absolute value of the difference is greater than a first preset threshold, wherein the target credibility indicates unreliability.

[0056] Based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel, the expected depth ReprojectDepth corresponding to the second target pixel can be calculated by projection derivation based on a known projection matrix. This expected depth ReprojectDepth is also called the projection depth, which is a value that can be calculated based on the projection relationship. The calculation process is existing technology and will not be elaborated here. In addition, since ReprojectPos can be determined based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel, the depth value (PreDepth) corresponding to ReprojectPos can be sampled in the known depth map (PreInputDepth) of the second target pixel at the rendering resolution of the second target frame. By comparing the relationship between ReprojectDepth and PreDepth, it can be determined whether there is depth occlusion. If PreDepth is close to the screen distance compared to the projection depth ReprojectDepth, and the difference between the two values ​​is large, it indicates that occlusion has occurred. In terms of numerical characteristics, PreDepth is smaller than ReprojectDepth, and the difference between the two is greater than a first preset threshold, indicating that occlusion has occurred. Of course, this disclosure does not limit the first preset threshold, which can be set according to the actual situation and does not constitute an obstacle to implementation. In this case, historical data related to ReprojectPos is unavailable; that is, the historical pixel confidence level is determined as the target confidence level, which indicates unreliability. This method can determine whether the position of the second target pixel is obstructed by the screen, thus determining whether the second target pixel is usable when the first target pixel is super-resolution displayed, based on whether screen obstruction is present.

[0057] S140. If the historical pixel reliability and the corresponding expansion speed corresponding to the first target pixel both meet the preset requirements, the color information sampling result of the color map of the second target pixel in the second target frame at the rendering resolution is used as the color information of the first target pixel.

[0058] This disclosure does not limit the preset requirements. For example, preset requirements may include the historical pixel reliability reaching a specific reliability threshold and the expansion speed being within a certain range. The specific values ​​of these parameters can be adjusted according to the actual application scenario to ensure that historical color information from reliable previous frames is selected for reuse, and to avoid image quality degradation caused by the incorrect reuse of historical color information from unreliable previous frames.

[0059] In an exemplary embodiment, if the historical pixel reliability indicator corresponding to the first target pixel is reliable, and the expansion speed value corresponding to the first target pixel is less than a second preset threshold, it is determined that both the historical pixel reliability and the corresponding expansion speed of the first target pixel meet the preset requirements. This disclosure does not limit the second preset threshold. For example, the second preset threshold can be set according to the actual application scenario to ensure accurate determination of the usability of the second target pixel in different environments. For instance, in high-dynamic scenarios, the second preset threshold can be appropriately lowered to increase sensitivity to expansion speed, thereby more strictly filtering out historical pixel information that meets the requirements. In static or low-dynamic scenarios, the second preset threshold can be appropriately increased to increase tolerance for historical pixel reliability, ensuring that more effective information is reused. This flexible adjustment method can better adapt to diverse display needs while balancing image quality and processing efficiency.

[0060] When the historical pixel confidence indicator corresponding to the first target pixel is reliable, and the expansion speed value corresponding to the first target pixel is less than the second preset threshold, it can be ensured that Reactive is small, the previous frame's historical color data is valid, and the Dilated Depth is also small, ensuring a small color difference between consecutive frames. In this case, color information can be reused, thereby effectively reducing the computational burden in the image processing process and improving overall efficiency. In this case, PreInputColor can be sampled directly using ReprojectPos, without the need for color information prediction. PreInputColor refers to the rendering result of the second target frame at the rendering resolution. In practical applications, by reasonably reusing PreInputColor, the number of pixels that need color information prediction can be reduced. In addition, this method can also reduce the impact of noise interference on image quality, ensuring the stability and consistency of the output results. During dynamic scene switching, by dynamically adjusting the second preset threshold, the system can quickly adapt to environmental changes, maintaining a high level of display effect and user experience.

[0061] S150. If the preset requirement is not met, predict the color information of the first target pixel.

[0062] If the preset requirements are not met, it means that color information prediction cannot be avoided by reusing the color information of ReprojectPos. In this case, it is still necessary to predict the color information of the first target pixel. Of course, this disclosure does not limit the prediction process. For example, color information can be predicted by interpolation calculation based on surrounding pixels.

[0063] S160. Based on the known pixels rendered and the color information of each of the first target pixels, display the first target frame at the display resolution.

[0064] The image super-resolution display method provided in this disclosure does not perform color information prediction for every pixel at each display resolution. Instead, it first distinguishes between known and unknown pixels. Known pixels can be directly rendered using the corresponding pixels at the rendering resolution without color information prediction. For unknown pixels, it determines whether their corresponding expansion speed and historical pixel reliability meet preset requirements. If they do, the pixel can be displayed by reusing previously rendered images without color information prediction. Color information prediction is only required for unknown pixels that do not meet the preset requirements, which significantly reduces the number of pixels requiring color information prediction.

[0065] The color information prediction process inevitably requires the use of interpolation algorithms, which introduces noise, thereby reducing image clarity and accuracy to a certain extent and slowing down the display speed. In this disclosure, the number of pixels that need to be predicted for color information is greatly reduced, which reduces the loss of clarity, accuracy and speed caused by color information prediction, thereby improving the clarity, accuracy and display speed of the image super-resolution results.

[0066] Typically, color information prediction processes are prone to producing jagged edges due to insufficient accuracy, thus affecting image quality. This disclosure significantly reduces the number of pixels that need to be predicted for color information, thereby achieving a significant anti-aliasing effect.

[0067] In one exemplary implementation, please refer to Figure 4 The diagram illustrates a flowchart of a color prediction method in an exemplary embodiment. Predicting the color information of the first target pixel includes: S410. Within a second preset neighborhood centered on the first target pixel, determine two associated pixels corresponding to each of multiple directions. The two associated pixels are distributed on both sides of the first target pixel along the corresponding direction, and each of the associated pixels belongs to the known pixels.

[0068] This disclosure proposes an inventive concept of selecting associated pixels in multiple directions to improve prediction accuracy. Of course, the associated pixels must be known pixels. The selection of directions and associated pixels is related to the distribution of known and unknown pixels.

[0069] Please refer to Figure 5 It shows a distribution map of known and unknown pixels in an exemplary embodiment. Figure 5The left figure shows an example of a chessboard distribution of known and unknown pixels, where the black squares represent known pixels and the gray squares represent unknown pixels. Taking the gray square M as an example, the black squares A and B can be considered as associated pixels in the vertical direction, and the black squares C and D can be considered as associated pixels in the horizontal direction. Figure 5 The right figure shows an example of a situation where known pixels and unknown pixels are distributed in alternating columns. In this case, pixel P is an unknown pixel, and its neighboring known pixels are distributed in the left and right columns. Then, 0, 1, 2, 3, 4, 5 are the three nearest adjacent pixels (3x2), and (0, 5), (1, 4), and (2, 3) are the related pixels in the three directions.

[0070] S420. For each of the directions, determine the corresponding fusion reference color by fusing the color information rendered by the two associated pixels at the rendering resolution.

[0071] This disclosure does not limit the method for determining a corresponding fusion reference color by fusing the color information rendered by two corresponding related pixels at the rendering resolution. For example, weighted averaging, linear interpolation, or other suitable algorithms can be used to achieve color information fusion. To improve prediction accuracy, this disclosure provides a fusion method based on color consistency. Determining the corresponding fusion reference color by fusing the color information rendered by two corresponding related pixels at the rendering resolution includes: determining a fusion reference weight based on the color consistency between the color information corresponding to the two related pixels, wherein the fusion reference weight is positively correlated with the color consistency; and, when the two related pixels belong to the same object, fusing the color information corresponding to the two related pixels based on the fusion reference weight to obtain the corresponding fusion reference color.

[0072] This disclosure does not limit the quantification method for color consistency, which can be inversely correlated with color distance. Color distance is represented by Distance(P0,P1), where P0 and P1 represent two related pixels in the same direction. Distance(P0,P1) is calculated using the absolute value of the color difference between two adjacent pixels in the same direction. In some embodiments, the color difference can be represented using grayscale difference. In an exemplary embodiment, the fusion reference weight DifferentialWeight(P0, P1) = 1.0 / (Distance(P0, P1) + 0.001). Obviously, the smaller the color distance, the higher the color consistency, and the larger the value of DifferentialWeight(P0, P1).

[0073] Furthermore, according to the formula Weight = SameObject(P0, P1) ? DifferentialWeight(P0, P1): Kmin, the weight used for fusion can be determined. This formula means that if P0 and P1 belong to the same object, then the fusion reference weight DifferentialWeight(P0, P1) is used as the fusion reference weight to fuse the color information corresponding to the two related pixels to obtain the corresponding fusion reference color. Otherwise, Kmin is used as the fusion reference weight to fuse the color information corresponding to the two related pixels to obtain the corresponding fusion reference color. Kmin is an empirical weight for different object cases and can be set according to the actual situation.

[0074] This disclosure does not limit the method for determining whether two related elements belong to the same object; for example, it can be determined based on depth similarity, velocity similarity, etc. To improve the accuracy of determining whether two related elements belong to the same object, this disclosure proposes determining the probability that the first target pixel is located on the object boundary; if the probability is less than a third preset threshold, the two related elements are determined to belong to the same object.

[0075] If the probability that the first target pixel is located at the object boundary is relatively small, it means that the first target pixel is likely located inside the object. Therefore, the probability that all associated elements are located inside the same object is also high, and these two associated elements can be determined to belong to the same object. This disclosure does not limit the third preset threshold; it can be set according to the actual situation.

[0076] This disclosure does not limit the method for calculating the probability DepthClip that the first target pixel is located on the object boundary. In an exemplary embodiment, the calculation of the DepthClip value is based on pixels in the neighborhood centered on the first target pixel, and this disclosure does not limit the size of the neighborhood. This disclosure proposes that, based on the design characteristics of GPUs, if the GPU can process a maximum of 4*4 pixels at a time, the neighborhood size can be set to 4*4. The advantage of doing so is that it can fully utilize the parallel computing power of the GPU and improve computational efficiency. It can reduce unnecessary consumption of computational resources while ensuring sufficient accuracy, thereby improving the overall processing speed and accuracy.

[0077] Based on the FSR2 method and the camera's FOV and empirical selection, the depth threshold Ksep and camera parameter Kfov can be determined. For each pixel in a 4x4 pixel array, the following operations can be performed: The WeightDepth corresponding to the pixel is calculated using the following two formulas. The WeightDepth indicates the probability that the corresponding pixel belongs to the boundary.

[0078] Formula 1:

[0079] Formula 2: ; SceneScale can be replaced with the original depth value before expansion. Clamp() is a common mathematical function that limits the value to a specified range. This function is often used in image processing to avoid data overflow or interference from outliers, ensuring the stability and accuracy of the calculation results. DepthDiff refers to the depth difference between the current pixel and the first target pixel. It is a preset constant, and its size is not limited. By using the clamp() function appropriately, the robustness of the algorithm can be effectively improved, while reducing the problem of error accumulation caused by numerical fluctuations.

[0080] Using the WeightDepth corresponding to each pixel, the corresponding DepthClip reference value can be calculated using the formula clamp(1 - WeightDepth, 0,1). The calculated DepthClip reference values ​​are then fused to obtain the final probability DepthClip that the first target pixel is located on the object boundary. This disclosure does not limit the fusion method; for example, a weighted average method, a maximum value selection method, or other suitable fusion strategies can be used. These methods can be flexibly adjusted according to specific application scenarios and requirements, thereby ensuring the accuracy and reliability of the final result.

[0081] S430. Fuse the fusion reference colors corresponding to each of the directions to obtain the reference base color of the first target pixel.

[0082] This disclosure does not limit the fusion reference color corresponding to each of the fusion directions. For example, color fusion can be achieved in various ways, such as by averaging.

[0083] S440. Based on the reference base color of the first target pixel, predict the color information of the first target pixel.

[0084] By fusing colors from multiple directions, the reference base color of the first target pixel can be made more accurate, resulting in a significantly higher accuracy of the predicted color information of the first target pixel compared to conventional interpolation methods. Of course, this disclosure does not limit the prediction of the color information of the first target pixel. For example, in addition to the methods mentioned herein, pixel interpolation methods in color space can also be used, such as the commonly used Lanczos interpolation, which interpolates the corresponding colors using the colors of adjacent 3x3 pixels.

[0085] In one exemplary implementation, predicting the color information of the first target pixel based on a reference base color includes: (1) Based on the color information of each pixel in the third preset neighborhood sampled from the color map of the second target frame at the display resolution with the second target pixel as the center, determine the reference color information corresponding to the second target pixel.

[0086] This disclosure does not limit the range of the third preset neighborhood; for example, it can be a 3x3 or 5x5 neighborhood centered on the second target pixel. The color information of each pixel within this third preset neighborhood can be obtained by sampling PreOutputColor. PreOutputColor refers to the display result of the previous frame at the stated display resolution. Using the color information of each pixel within this third preset neighborhood, the reference color information corresponding to the second target pixel can be determined.

[0087] This disclosure proposes that the color information of the second target pixel can be obtained by directly sampling PreOutputColor using ReprojectPos. However, directly using the color information of the second target pixel will result in blurring and ghosting. Therefore, in this step, this disclosure proposes to form a bounding box using the color information of each pixel in a third preset neighborhood, and to use this bounding box to crop the color information of the second target pixel to obtain the reference color information (ClampedColor) corresponding to the second target pixel. The reference color information corresponding to the second target pixel is also used as the key information for predicting the color information of the first target pixel, thereby solving the blurring and ghosting problems.

[0088] However, ClampedColor may cause pixel flickering when the rendering pixels themselves are unstable, especially at object boundaries. To solve this problem, this disclosure proposes step (2).

[0089] (2) Determine the fusion weight based on the probability that the first target pixel is located at the object boundary and the expansion depth of the first target pixel.

[0090] This disclosure does not limit the method of determining the fusion weight based on the probability that the first target pixel is located on the object boundary and the expansion depth of the first target pixel. For example, the fusion weight Weight can be calculated based on the formula Weight = max(DepthClip, 1 - clamp(Dilated velocity*Kv, 0, 1)), where Kv is a velocity constant and its size is not limited in this disclosure.

[0091] (3) Based on the fusion weight, the reference color information corresponding to the second target pixel and the color information of the second target pixel sampled from the color map of the second target frame at the display resolution are fused to determine the historical reference color information corresponding to the first target pixel.

[0092] This disclosure can perform color information fusion based on the formula HistoryColor = color information obtained by sampling PreOutputColor using ReprojectPos * Weight + ClampedColor * (1-Weight) to obtain the historical reference color information corresponding to the first target pixel. This formula embodies the idea of ​​reasonably mixing the color information of the second target pixel and the reference color information corresponding to the second target pixel, thereby avoiding boundary flickering, blurring and ghosting.

[0093] (4) The historical reference color information and the reference base color of the first target pixel are fused to obtain the color information of the first target pixel.

[0094] This disclosure does not limit the fusion method. For example, temporal antialiasing can be used for fusion, and this disclosure does not limit it. In this disclosure, the color information of the first target pixel is represented as OutputColor. The OutputColor corresponding to each pixel is the updated result of its corresponding PreOutputColor. By reasonably fusing the historical reference color information and the reference base color of the first target pixel, this disclosure can significantly improve the accuracy of the prediction result of the color information of the first target pixel.

[0095] In one exemplary embodiment, the image super-resolution display method is performed at the display resolution and rendered based on the tiles of the graphics processing unit in the mobile terminal. The probability that the first target pixel is located at the object boundary, the historical pixel confidence level corresponding to the first target pixel, and the expansion depth and expansion speed corresponding to each first target pixel are all stored in the tile memory. The probability that the first target pixel is located at the object boundary is determined using a pixel shader.

[0096] The purpose of this design is that the DepthClip calculation process uses pixel shaders (FragmentShaders), which allows this disclosure to completely avoid the use of ComputeShaders, a necessary component in related technologies. This makes this disclosure a solution for image super-resolution display without relying on ComputeShaders. Related technologies require ComputeShaders for super-resolution display on mobile devices, but ComputeShaders are very unfriendly to mobile devices, resulting in significant performance degradation. Furthermore, they prevent the use of mobile device bandwidth compression for cached computations, increasing bandwidth consumption and reducing performance. This disclosure avoids the use of ComputeShaders, thus reducing bandwidth consumption and improving performance.

[0097] Furthermore, related technologies involve a significant amount of intermediate computational results being transferred between GPU memory and main memory, failing to fully utilize the high-speed cache of mobile device on-chip memory. In contrast, this disclosure stores the probability that the first target pixel is located at the object boundary, the historical pixel reliability corresponding to the first target pixel, and the expansion depth and expansion speed corresponding to each first target pixel in Tile memory. This fully utilizes the high-speed cache of mobile device on-chip memory, further reducing data transmission latency and improving computational efficiency. Simultaneously, the high-speed characteristics of Tile memory enable faster access and updating of relevant data during image super-resolution display processing, thereby optimizing overall performance. In addition, the design of this disclosure avoids frequent memory read / write operations, reducing power consumption, which is particularly important for mobile devices. This approach not only improves image processing speed but also ensures device battery life, providing users with a smoother and more efficient user experience.

[0098] This disclosure proposes that current mainstream mobile super-resolution solutions are modified from desktop solutions, thus heavily relying on the less efficient Compute Shaders on mobile devices and requiring significant amounts of main memory to store intermediate results. Support for efficient, low-precision instructions on mobile devices is also insufficient. This disclosure maximizes the use of low-precision GPU instructions; the rendering process avoids dependence on Compute Shaders, relying entirely on Fragment Shaders; and intermediate results, except for the final rendering result, are stored in on-chip video memory. Maximizing the use of mobile GPU capabilities achieves optimization in both efficiency and bandwidth.

[0099] Please refer to Figure 6This diagram illustrates the architecture of the super-resolution display scheme disclosed herein. The architecture of this super-resolution display scheme takes InputDepth, InputVelocity, and PreInputDepth as inputs and finally outputs OutputColor. The first stage involves data expansion, primarily depth and velocity expansion. This stage is implemented in the Fragment Shader and ultimately outputs Dilated Depth and Dilated Velocity. The second stage is the upsampling super-resolution stage. This stage is also implemented in the Fragment Shader. The data required in this stage includes PreInputColor, InputColor, Dilated Depth, Dilated Velocity, Reactive, and PreOutputColor. InputColor refers to the rendering result of the first target frame at the rendering resolution. The meanings of the other parameters have been described above and will not be repeated here.

[0100] Intermediate results between the two stages are stored on ultra-high-speed Tile memory (OnChipMemory), and the final result is not output to main memory. This avoids the performance loss caused by data transfer in main memory.

[0101] The image super-resolution display scheme disclosed herein has the following significant features: 1. Introduce more factors to address the problem of poor image quality in pure color super-resolution images; 2. Improve the prediction accuracy of color information in the current frame and solve the image blurring problem caused by interpolation; 3. Fully leverage the characteristics of mobile hardware and software to optimize bandwidth and performance; 4. It can be applied to mid-range and higher platforms, not just high-end GPU platforms; for display devices with resolutions close to 1080p, this disclosure can greatly optimize rendering performance and bandwidth, and provide higher final image quality. For display devices with resolutions close to 2K and above, this disclosure can provide image output rendered at the original display resolution within the device's performance range, instead of traditional low-resolution output.

[0102] Traditional temporal anti-aliasing has high performance and bandwidth consumption, and is generally only used in high-end devices. This disclosure ensures compatibility in all aspects and avoids performance and bandwidth consumption by taking into account the characteristics of current mobile hardware and software. In addition, it takes advantage of the low-precision computing efficiency of mobile devices and the data access advantages based on Tile caching, so that the whole solution can be adapted to various devices from low-end to high-end. Compared with related technologies, this disclosure can achieve a display resolution effect of nearly 100% restoration, which greatly improves the clarity and quality of the picture.

[0103] Figure 7 This is a block diagram illustrating an image super-resolution display device according to an exemplary embodiment. The device includes: Image processing module 710 is configured to determine known pixels and unknown pixels of a first target frame at a display resolution, wherein the known pixels are pixels obtained by rendering the first target frame at a rendering resolution, and the unknown pixels are other pixels besides the known pixels, and the display resolution is higher than the rendering resolution; The data expansion module 720 is configured to perform an expansion depth and expansion speed for each first target pixel based on the depth map and velocity map corresponding to the first target frame at the rendering resolution, wherein the first target pixel belongs to the unknown pixel. The historical pixel confidence determination module 730 is configured to perform, for each first target pixel, determine the historical pixel confidence corresponding to the first target pixel based on the expansion depth corresponding to the first target pixel. The historical pixel confidence indicates the degree of reliability of using the color information of the second target pixel as the color information of the first target pixel. The second target pixel is the pixel corresponding to the first target pixel in the second target frame at the display resolution. The second target frame is the image frame before the first target frame. The pixel processing module 740 is configured to, when the historical pixel reliability and the corresponding expansion speed corresponding to the first target pixel both meet preset requirements, take the color information sampling result of the color map of the second target pixel at the rendering resolution as the color information of the first target pixel; and when the preset requirements are not met, predict the color information of the first target pixel. Display module 750 is configured to execute, based on the known pixels of the rendering and the color information of each of the first target pixels, to display the first target frame at the display resolution.

[0104] In one exemplary implementation, the data expansion module 720 is configured to perform: For any first target pixel, determine the depth values ​​of each pixel in the first preset neighborhood centered on the first target pixel in the depth map corresponding to the first target frame at the rendering resolution; The pixel corresponding to the depth value closest to the screen among the aforementioned depth values ​​is determined as the expansion target pixel; The depth value sampled by the expanded target pixel in the depth map corresponding to the rendering resolution of the first target frame is used as the expansion depth corresponding to the first target pixel; The speed at which the expanded target pixel is sampled in the speed map corresponding to the rendering resolution of the first target frame is taken as the expansion speed of the first target pixel.

[0105] In one exemplary implementation, the historical pixel confidence determination module 730 is configured to perform: Based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel, the position information of the second target pixel is determined; If, based on the location information, it is determined that the second target pixel is not within the rendering range corresponding to the second target frame, the historical pixel credibility is determined as the target credibility, and the target credibility indicates unreliability.

[0106] In one exemplary implementation, the historical pixel confidence determination module 730 is configured to perform: Based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel, the expected depth corresponding to the second target pixel is determined; Extract the actual depth of the second target pixel in the depth map of the second target frame at the rendering resolution; If the actual depth is less than the expected depth and the absolute value of the difference is greater than a first preset threshold, the historical pixel credibility is determined as the target credibility, and the target credibility indicates unreliability.

[0107] In one exemplary embodiment, the pixel processing module 740 is configured to perform: If the historical pixel confidence level of the first target pixel is reliable and the expansion speed of the first target pixel is less than the second preset threshold, then the historical pixel confidence level and the corresponding expansion speed of the first target pixel both meet the preset requirements.

[0108] In one exemplary embodiment, the pixel processing module 740 is configured to perform: Within a second preset neighborhood centered on the first target pixel, two associated pixels are determined for each of multiple directions. The two associated pixels are distributed on both sides of the first target pixel along the corresponding direction. Each of the associated pixels belongs to the known pixels. For each of the aforementioned directions, a corresponding fusion reference color is determined by fusing the color information rendered by the two associated pixels at the rendering resolution. The reference base color of the first target pixel is obtained by fusing the fusion reference colors corresponding to each of the directions; Based on the reference base color of the first target pixel, predict the color information of the first target pixel.

[0109] In one exemplary embodiment, the pixel processing module 740 is configured to perform: Based on the color consistency between the color information corresponding to the two associated pixels, a fusion reference weight is determined, and the fusion reference weight is positively correlated with the color consistency. When the two associated pixels belong to the same object, the color information corresponding to the two associated pixels is fused based on the fusion reference weight to obtain the corresponding fusion reference color.

[0110] In one exemplary embodiment, the pixel processing module 740 is configured to perform: Determine the probability that the first target pixel is located on the object boundary; If the probability is less than a third preset threshold, the two associated elements are determined to belong to the same object.

[0111] In one exemplary embodiment, the pixel processing module 740 is configured to perform: Based on the color information of each pixel in a third preset neighborhood sampled from the color map of the second target frame at the display resolution with the second target pixel as the center, the reference color information corresponding to the second target pixel is determined. The fusion weights are determined based on the probability that the first target pixel is located at the object boundary and the expansion depth of the first target pixel; Based on the fusion weight, the reference color information corresponding to the second target pixel and the color information of the second target pixel sampled from the color map of the second target frame at the display resolution are fused to determine the historical reference color information corresponding to the first target pixel; The color information of the first target pixel is obtained by fusing the historical reference color information and the reference base color of the first target pixel.

[0112] In an exemplary embodiment, the image super-resolution display device performs rendering at the display resolution and based on the tiles of the graphics processing unit in the mobile terminal. The probability that the first target pixel is located at the object boundary, the confidence level of the historical pixels corresponding to the first target pixel, and the expansion depth and expansion speed corresponding to each first target pixel are all stored in the tile memory. Use a pixel shader to determine the probability that the first target pixel is located on the object boundary.

[0113] In one exemplary implementation, when the known pixels are in odd-numbered rows, the unknown pixels are in even-numbered rows. If the known pixels are in even-numbered rows, the unknown pixels are in odd-numbered rows. If the known pixels are in an odd-numbered column, the unknown pixels are in an even-numbered column. If the known pixels are in an even-numbered column, the unknown pixels are in an odd-numbered column.

[0114] Regarding the apparatus in the above embodiments, the specific manner of each step has been described in detail in the embodiments of the foregoing method, and will not be elaborated here.

[0115] Please refer to Figure 8 It illustrates the structural block of a computer device provided in an exemplary embodiment of this disclosure. Figure 1 The computer device may be a terminal. This computer device is used to implement the image super-resolution display method provided in the above embodiments. Specifically: Typically, computer device 800 includes a processor 801 and a memory 802.

[0116] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In an exemplary embodiment, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In an exemplary embodiment, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0117] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In an exemplary embodiment, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, at least one program, code set, or instruction set, configured to be executed by one or more processors to implement the above-described image super-resolution display method.

[0118] In one exemplary embodiment, the computer device 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a touch display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808, and a power supply 809.

[0119] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the computer device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0120] Please refer to Figure 9 It illustrates the structural block of a computer device provided in another exemplary embodiment of this disclosure. Figure 2 The computer device can be a server for performing the aforementioned image super-resolution display method. Specifically: Computer device 900 includes a Central Processing Unit (CPU) 901, a system memory 904 including Random Access Memory (RAM) 902 and Read Only Memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the CPU 901. Computer device 900 also includes a basic input / output system (I / O system) 906 that facilitates information transfer between various devices within the computer, and a mass storage device 907 for storing the operating system 913, application programs 914, and other program modules 911.

[0121] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 for user input, such as a mouse or keyboard. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 190 connected to the system bus 905. The basic input / output system 906 may also include the input / output controller 190 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 190 also provides output to a display screen, printer, or other types of output devices.

[0122] Mass storage device 907 is connected to central processing unit 901 via a mass storage controller (not shown) connected to system bus 905. Mass storage device 907 and its associated computer-readable media provide non-volatile storage for computer device 900. That is, mass storage device 907 may include computer-readable media (not shown) such as hard disk or CD-ROM (CompactDisc Read-Only Memory) drive.

[0123] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 904 and mass storage device 907 described above can be collectively referred to as memory.

[0124] According to various embodiments of this disclosure, the computer device 900 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to a system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0125] The aforementioned memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the aforementioned image super-resolution display method.

[0126] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is executed by a processor to implement the image super-resolution display method.

[0127] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0128] In an exemplary embodiment, a computer-readable storage medium including program code is also provided, such as a memory including program code, which can be executed by a processor to perform the above-described image super-resolution display method. Optionally, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), compact-disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0129] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described image super-resolution display method.

[0130] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0131] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image super-resolution display method, characterized in that, The method includes: Determine the known and unknown pixels of the first target frame at the display resolution. The known pixels are pixels obtained by rendering the first target frame at the rendering resolution, and the unknown pixels are other pixels besides the known pixels. The display resolution is higher than the rendering resolution. Based on the depth map and velocity map corresponding to the first target frame at the rendering resolution, the expansion depth and expansion velocity corresponding to each first target pixel are determined, and the first target pixel belongs to the unknown pixel; For each first target pixel, based on the expansion depth corresponding to the first target pixel, the historical pixel confidence level corresponding to the first target pixel is determined. The historical pixel confidence level indicates the degree of reliability of using the color information of the second target pixel as the color information of the first target pixel. The second target pixel is the pixel corresponding to the first target pixel in the second target frame at the display resolution. The second target frame is the image frame before the first target frame. If the historical pixel reliability and the corresponding expansion speed of the first target pixel both meet the preset requirements, the color information sampling result of the color map of the second target pixel in the second target frame at the rendering resolution is used as the color information of the first target pixel. If the preset requirement is not met, predict the color information of the first target pixel; Based on the known pixels rendered and the color information of each of the first target pixels, the first target frame is displayed at the display resolution.

2. The method according to claim 1, characterized in that, The step of determining the dilation depth and dilation velocity corresponding to each first target pixel based on the depth map and velocity map corresponding to the first target frame at the rendering resolution includes: For any first target pixel, determine the depth values ​​of each pixel in the first preset neighborhood centered on the first target pixel in the depth map corresponding to the first target frame at the rendering resolution; The pixel corresponding to the depth value closest to the screen among the aforementioned depth values ​​is determined as the expansion target pixel; The depth value sampled by the expanded target pixel in the depth map corresponding to the rendering resolution of the first target frame is used as the expansion depth corresponding to the first target pixel; The speed at which the expanded target pixel is sampled in the speed map corresponding to the rendering resolution of the first target frame is taken as the expansion speed of the first target pixel.

3. The method according to claim 1 or 2, characterized in that, The step of determining the historical pixel confidence level corresponding to the first target pixel based on the expansion depth corresponding to the first target pixel includes: Based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel, the position information of the second target pixel is determined; If, based on the location information, it is determined that the second target pixel is not within the rendering range corresponding to the second target frame, the historical pixel credibility is determined as the target credibility, and the target credibility indicates unreliability.

4. The method according to claim 1 or 2, characterized in that, The step of determining the historical pixel confidence level corresponding to the first target pixel based on the expansion depth corresponding to the first target pixel includes: Based on the expansion depth corresponding to the first target pixel and the position information of the first target pixel, the expected depth corresponding to the second target pixel is determined; Extract the actual depth of the second target pixel in the depth map of the second target frame at the rendering resolution; If the actual depth is less than the expected depth and the absolute value of the difference is greater than a first preset threshold, the historical pixel credibility is determined as the target credibility, and the target credibility indicates unreliability.

5. The method according to claim 1, characterized in that, The method further includes: If the historical pixel confidence level of the first target pixel is reliable and the expansion speed of the first target pixel is less than the second preset threshold, then the historical pixel confidence level and the corresponding expansion speed of the first target pixel both meet the preset requirements.

6. The method according to claim 1, characterized in that, The predicted color information of the first target pixel includes: Within a second preset neighborhood centered on the first target pixel, two associated pixels are determined for each of multiple directions. The two associated pixels are distributed on both sides of the first target pixel along the corresponding direction. Each of the associated pixels belongs to the known pixels. For each of the aforementioned directions, a corresponding fusion reference color is determined by fusing the color information rendered by the two associated pixels at the rendering resolution. The reference base color of the first target pixel is obtained by fusing the fusion reference colors corresponding to each of the directions; Based on the reference base color of the first target pixel, predict the color information of the first target pixel.

7. The method according to claim 6, characterized in that, The step of determining the corresponding fusion reference color by fusing the color information rendered at the rendering resolution by two corresponding related pixels includes: Based on the color consistency between the color information corresponding to the two associated pixels, a fusion reference weight is determined, and the fusion reference weight is positively correlated with the color consistency. When the two associated pixels belong to the same object, the color information corresponding to the two associated pixels is fused based on the fusion reference weight to obtain the corresponding fusion reference color.

8. The method according to claim 7, characterized in that, The method further includes: Determine the probability that the first target pixel is located on the object boundary; If the probability is less than a third preset threshold, the two associated elements are determined to belong to the same object.

9. The method according to claim 6, characterized in that, The step of predicting the color information of the first target pixel based on the reference base color of the first target pixel includes: Based on the color information of each pixel in a third preset neighborhood sampled from the color map of the second target frame at the display resolution with the second target pixel as the center, the reference color information corresponding to the second target pixel is determined. The fusion weights are determined based on the probability that the first target pixel is located at the object boundary and the expansion depth of the first target pixel; Based on the fusion weight, the reference color information corresponding to the second target pixel and the color information of the second target pixel sampled from the color map of the second target frame at the display resolution are fused to determine the historical reference color information corresponding to the first target pixel; The color information of the first target pixel is obtained by fusing the historical reference color information and the reference base color of the first target pixel.

10. The method according to claim 8, characterized in that, The image super-resolution display method is performed at the display resolution and rendered based on the tiles of the graphics processing unit in the mobile terminal. The probability that the first target pixel is located at the object boundary, the confidence level of the historical pixels corresponding to the first target pixel, and the expansion depth and expansion speed corresponding to each first target pixel are all stored in the tile memory. Use a pixel shader to determine the probability that the first target pixel is located on the object boundary.

11. The method according to claim 1, characterized in that, If the known pixels are in odd-numbered rows, the unknown pixels are in even-numbered rows. If the known pixels are in even-numbered rows, the unknown pixels are in odd-numbered rows. If the known pixels are in an odd-numbered column, the unknown pixels are in an even-numbered column. If the known pixels are in an even-numbered column, the unknown pixels are in an odd-numbered column.

12. An image super-resolution display device, characterized in that, The device includes: The image processing module is configured to determine known and unknown pixels of a first target frame at a display resolution, wherein the known pixels are pixels obtained by rendering the first target frame at a rendering resolution, and the unknown pixels are other pixels besides the known pixels, and the display resolution is higher than the rendering resolution; The data expansion module is configured to perform an operation based on the depth map and velocity map corresponding to the first target frame at the rendering resolution to determine the expansion depth and expansion velocity corresponding to each first target pixel, wherein the first target pixel belongs to the unknown pixel. The historical pixel confidence determination module is configured to perform, for each first target pixel, determine the historical pixel confidence corresponding to the first target pixel based on the expansion depth corresponding to the first target pixel. The historical pixel confidence indicates the degree of reliability of using the color information of the second target pixel as the color information of the first target pixel. The second target pixel is the pixel corresponding to the first target pixel in the second target frame at the display resolution. The second target frame is the image frame before the first target frame. The pixel processing module is configured to, when the historical pixel reliability and corresponding expansion speed corresponding to the first target pixel both meet preset requirements, take the color information sampling result of the color map of the second target pixel at the rendering resolution of the second target frame as the color information of the first target pixel; and when the preset requirements are not met, predict the color information of the first target pixel. The display module is configured to execute, based on the known pixels of the rendering and the color information of each of the first target pixels, to display the first target frame at the display resolution.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the image super-resolution display method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device performs the image super-resolution display method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from and executes the computer program, causing the device to perform the image super-resolution display method as described in any one of claims 1 to 11.

Images