Resolution improving method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a resolution improving method and device, electronic equipment and a readable storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a current frame image of a first resolution, a historical frame image of a second resolution adjacent to the current frame image, and an image change parameter; performing sampling operation on the current frame image at a second resolution to obtain a first color parameter of each first pixel point in the current frame image after the sampling operation; a second color parameter of each second pixel point in the historical frame image is corrected based on a color clamping parameter and the image change parameter, a third color parameter of each second pixel point is obtained, and the color clamping parameter is determined based on the first color parameter; and performing color mixing operation on the first color parameter of each first pixel point in the current frame image based on the displacement parameter and the third color parameter of each second pixel point in the historical frame image to obtain the current frame image with the second resolution. The color effect of an image after super-resolution can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a resolution enhancement method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Currently, with the development of computer technology, it is possible to increase the resolution of lower-resolution images, thus achieving image super-resolution. However, the color quality of images obtained after increasing the resolution is currently poor. Summary of the Invention

[0003] This application discloses a resolution enhancement method, apparatus, electronic device, and readable storage medium.

[0004] In a first aspect, embodiments of this application provide a resolution enhancement method, comprising: acquiring a current frame image at a first resolution, a historical frame image at a second resolution adjacent to the current frame image, and image change parameters, wherein the image change parameters include at least one of a depth change parameter and a transparency change parameter between the current frame image and the historical frame image, and the image change parameters further include a displacement parameter between the current frame image and the historical frame image, wherein the first resolution is smaller than the second resolution; performing a sampling operation on the current frame image at the second resolution to obtain a first color parameter for each first pixel in the current frame image after the sampling operation; correcting the second color parameter for each second pixel in the historical frame image based on a color clamping parameter and the image change parameters to obtain a third color parameter for each second pixel, wherein the color clamping parameter is determined based on the first color parameter; and performing a color mixing operation on the first color parameter for each first pixel in the current frame image based on the displacement parameter and the third color parameter for each second pixel in the historical frame image to obtain a current frame image at the second resolution.

[0005] Secondly, embodiments of this application also provide a resolution enhancement device, including: an acquisition unit, a sampling unit, a correction unit, and a mixing unit. The acquisition unit is configured to acquire a current frame image at a first resolution, a historical frame image at a second resolution adjacent to the current frame image, and image change parameters, wherein the image change parameters include at least one of a depth change parameter and a transparency change parameter between the current frame image and the historical frame image, and the image change parameters also include a displacement parameter between the current frame image and the historical frame image, wherein the first resolution is smaller than the second resolution; the sampling unit is configured to perform a sampling operation on the current frame image at the second resolution to obtain a first color parameter for each first pixel in the current frame image after the sampling operation; the correction unit is configured to correct the second color parameter for each second pixel in the historical frame image based on a color clamping parameter and the image change parameters to obtain a third color parameter for each second pixel, wherein the color clamping parameter is determined based on the first color parameter; the mixing unit is configured to perform a color mixing operation on the first color parameter of each first pixel in the current frame image based on the displacement parameter and the third color parameter of each second pixel in the historical frame image to obtain a current frame image at the second resolution.

[0006] Thirdly, embodiments of this application also provide an electronic device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the method described in the first aspect.

[0007] Fourthly, embodiments of this application also provide a computer-readable storage medium storing program code that can be invoked by a processor to execute the method described in the first aspect above.

[0008] The resolution enhancement method, apparatus, electronic device, and readable storage medium provided in this application first acquire a current frame image at a first resolution, a historical frame image at a second resolution adjacent to the current frame image, and image change parameters. Then, the current frame image is sampled at the second resolution to obtain a first color parameter for each first pixel in the current frame image after the sampling operation. Next, the second color parameter for each second pixel in the historical frame image is corrected based on a color clamping parameter and the image change parameters to obtain a third color parameter for each second pixel, wherein the color clamping parameter is determined based on the first color parameter. Finally, a color mixing operation is performed on the first color parameter of each first pixel in the current frame image based on the displacement parameter and the third color parameter of each second pixel in the historical frame image to obtain the current frame image at the second resolution. In the solution of this application, the color clamping parameter is determined based on the first color parameter; therefore, the color clamping parameter can be flexibly changed. Furthermore, based on the color clamping parameters and the image change parameters, the second color parameters of each second pixel in the historical frame image are corrected. Then, based on the displacement parameters and the third color parameters of each second pixel in the historical frame image, a color mixing operation is performed on the first color parameters of each first pixel in the current frame image to obtain the current frame image at the second resolution. It is evident that the colors of the frame image after color clamping are not simply used directly as the colors of the super-resolution image. Instead, a color mixing operation is used to obtain the current frame image at the second resolution, thus avoiding the problem of poor color quality in the generated second-resolution current frame image to a certain extent, resulting in better color performance in the super-resolution image.

[0009] Other features and advantages of the embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the embodiments of this application. The objects and other advantages of the embodiments of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 The diagram illustrates an application scenario of the resolution enhancement method provided in this application embodiment. Figure 2 A flowchart of the resolution enhancement method provided in an embodiment of this application is shown; Figure 3 A flowchart of a resolution enhancement method according to another embodiment of this application is shown; Figure 4 A flowchart of a resolution enhancement method according to another embodiment of this application is shown; Figure 5 A flowchart of a resolution enhancement method according to another embodiment of this application is shown; Figure 6 The comparison chart shows the effects of different resolution enhancement methods; Figure 7 A structural block diagram of the resolution enhancement device provided in an embodiment of this application is shown; Figure 8 A structural block diagram of the electronic device provided in an embodiment of this application is shown; Figure 9 A structural block diagram of a computer-readable storage medium provided in an embodiment of this application is shown. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.

[0013] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0014] Currently, with the development of computer technology, it is possible to increase the resolution of lower-resolution images, thus achieving image super-resolution. However, the color quality of the resulting image after resolution enhancement is currently poor. How to improve the color quality of the super-resolution image is a problem that urgently needs to be solved.

[0015] Currently, in the process of image super-resolution, color clamping techniques can be used to limit the colors of the super-resolution image. For example, by pre-setting clamping parameters, the colors of the frame image are limited based on these parameters, and the colors of the color-clamped frame image are used as the colors of the super-resolution image.

[0016] However, the inventors discovered in their research that, firstly, the clamping parameters used in the super-resolution process are generally preset, which may result in poor color limiting effect when using the same clamping parameters for different frame images; secondly, directly using the color of the frame image after color clamping as the color of the super-resolution image may lead to poor color effect of the super-resolution image.

[0017] Therefore, in order to solve or partially solve the above problems, this application provides a resolution enhancement method, apparatus, electronic device, and readable storage medium.

[0018] Please see Figure 1 , Figure 1 This illustration shows an application scenario of the resolution enhancement method provided in this application, namely resolution enhancement scenario 100. Resolution enhancement scenario 100 includes an electronic device 110 and a user 120. In some embodiments, when the user 120 runs an application through the electronic device 110, the electronic device 110 can be triggered to execute the resolution enhancement method. For example, the user can run a game application through the electronic device 110 and simultaneously trigger the electronic device 110 to execute the resolution enhancement method. Detailed descriptions can be found in subsequent embodiments.

[0019] Please see Figure 2 , Figure 2 A flowchart of a resolution enhancement method provided in an embodiment of this application is shown. This resolution enhancement method can be applied to... Figure 1 In the illustrated resolution enhancement scenario, the electronic device's processor can be used as the execution entity for the resolution enhancement method. This resolution enhancement method may include steps S110 to S140.

[0020] Step S110: Obtain the current frame image at a first resolution, the historical frame image at a second resolution adjacent to the current frame image, and image change parameters, wherein the image change parameters include at least one of the depth change parameters and the transparency change parameters between the current frame image and the historical frame image, and the image change parameters also include the displacement parameters between the current frame image and the historical frame image, wherein the first resolution is smaller than the second resolution.

[0021] As described above, electronic devices can run applications that generate images or image sequences. For example, a game application can generate image sequences at a certain frame rate parameter, thus providing the user with a dynamic display effect. Specifically, this frame rate parameter could be 30 frames per second, 60 frames per second, etc.

[0022] In some implementations, the electronic device can acquire an image sequence based on a running application, which can be used to characterize multiple sequentially arranged frame images, thus constituting a frame image sequence.

[0023] It should be noted that the electronic device acquires image sequences based on the running application, where the running application is used to represent the application currently running in the foreground of the electronic device.

[0024] Therefore, the electronic device can acquire the current frame image based on the frame image sequence. To reduce the power consumption of the electronic device, the frame image sequence rendered by the application of the electronic device can be multiple frame images with lower resolution. In the embodiments of this application, the acquired current frame image can be a frame image with a first resolution, which can be used to represent a lower resolution. The current frame image can be used to represent the frame image corresponding to the current moment.

[0025] It is understandable that, prior to the current moment, the resolution enhancement method may have already been used to super-resolution lower-resolution frame images corresponding to the previous moment. Therefore, the electronic device can also acquire historical frame images adjacent to the current frame image. For example, this historical frame image is used to represent the frame image corresponding to the previous moment that is temporally adjacent to the current frame; this historical frame image can also be used to represent the previous frame image adjacent to the current frame image in the frame image sequence. Here, the historical frame image is the higher-resolution frame image after super-resolution.

[0026] In the embodiments provided in this application, the historical frame image after super-resolution can correspond to a second resolution, which can be used to characterize a higher resolution, wherein the first resolution is smaller than the second resolution. It is understood that the first resolution can be the original resolution rendered by the electronic device, while the second resolution can be the target resolution after super-resolution.

[0027] The electronic device can store historical frame images at a second resolution in a storage module, such as a cache. Therefore, the electronic device can directly retrieve historical frame images at a second resolution from the cache.

[0028] Furthermore, image change parameters can also be obtained. These image change parameters may include at least one of the depth change parameters and transparency change parameters between the current frame image and the historical frame images, and may also include the displacement parameters between the current frame image and the historical frame images.

[0029] A frame image may include multiple pixels; for example, the current frame image may include multiple first pixels, and a historical frame image may include multiple second pixels. In some implementations, a first depth parameter and a first transparency parameter of each first pixel in the current frame image can be obtained separately. Then, a second depth parameter and a second transparency parameter of each second pixel in the historical frame image can be obtained. Further, a depth change parameter is determined based on the first depth parameter of each first pixel in the current frame image and the second depth parameter of each second pixel in the historical frame image; and a transparency change parameter is determined based on the first transparency parameter of each first pixel in the current frame image and the second transparency parameter of each second pixel in the historical frame image.

[0030] Optionally, the first depth parameter and the first transparency parameter of each first pixel in the current frame image can be corrected, and then the corrected first depth parameter and the corrected first transparency parameter can be combined to determine the depth change parameter and the transparency change parameter respectively. For a detailed description, please refer to the following embodiments.

[0031] In addition, when generating the image sequence, the application also generates displacement parameters for each frame image. These displacement parameters can be used to characterize the displacement change of that frame image relative to the previous frame image. Therefore, the displacement parameters corresponding to the current frame image can also be obtained. These displacement parameters are the displacement parameters between the current frame image and historical frame images, that is, they are used to characterize the displacement change of the current frame image relative to historical frame images.

[0032] In some implementations, the displacement parameter may also be referred to as the motion vector parameter.

[0033] It should be noted that the resolution enhancement method provided in this application embodiment can be automatically activated by the electronic device after detecting a specified application, thereby starting to execute step S110 and subsequent steps. For example, the specified application can be a pre-defined application, such as a game application, an online video application, etc. Optionally, the electronic device can also display operation controls corresponding to the super-resolution method. For example, the operation controls can be displayed in a floating manner at a specified position on the display screen of the electronic device. The user can interact with the operation controls to activate the electronic device to execute the resolution enhancement method, thereby starting to execute step S110 and subsequent steps; the user can also interact with the operation controls to stop the electronic device from executing the resolution enhancement method.

[0034] For example, this operation control can be integrated into the frame super-resolution and frame interpolation algorithm control.

[0035] Step S120: Perform a sampling operation on the current frame image at a second resolution to obtain the first color parameter of each first pixel in the current frame image after the sampling operation.

[0036] To improve the display effect after super-resolution of the current frame image, the current frame image can first be sampled at a higher resolution to obtain the first color parameters of each first pixel in the current frame image. For example, the current frame image can be sampled at a second resolution.

[0037] It's understandable that the current frame image corresponds to a first resolution. Therefore, sampling the current frame image at a second resolution, which is higher than the first resolution, is essentially equivalent to stretching the current frame image to the second resolution. However, the first color parameter of each first pixel in the current frame image needs to be combined with the second color parameters of each second pixel in historical frame images for color mixing to obtain a better-looking current frame image. Detailed explanations can be found in the following steps.

[0038] In some implementations, the initial color parameters of each first pixel can be sampled first, and then the color parameters of the first pixel can be adjusted based on at least one sampled pixel corresponding to the first pixel to obtain the first color parameters of each first pixel. For a detailed description, please refer to the following embodiments.

[0039] Step S130: Based on the color clamping parameters and the image change parameters, correct the second color parameters of each second pixel in the historical frame image to obtain the third color parameters of each second pixel, wherein the color clamping parameters are determined based on the first color parameters.

[0040] Furthermore, the second color parameters of each second pixel in the historical frame image can also be obtained. As described above, the historical frame image at the second resolution can be stored in a storage module, and the second color parameters of each second pixel in the historical frame image obtained after super-resolution can also be stored in an electronic device, such as in a storage module. Therefore, in some implementations, the second color parameters of each second pixel in the historical frame image can be directly retrieved from the storage module.

[0041] Additionally, color clamping parameters can be determined to constrain the second color parameters. For example, if the second color parameters of each second pixel in a historical frame image do not satisfy the color clamping parameters, the second color parameters can be adjusted based on the color clamping parameters.

[0042] In some implementations, the color clamping parameters can be determined based on the first color parameters. That is, the corresponding color clamping parameters can be obtained for the current frame image, so that the color clamping parameters can be flexibly changed according to the current frame image, rather than being fixed, thereby improving the effect of obtaining the current frame image at the second resolution.

[0043] For example, the second color parameter of each second pixel in the historical frame image can be corrected according to the color clamping parameters and the image change parameters to obtain the third color parameter of each second pixel. Specifically, for example, the second color parameter of each second pixel in the historical frame image can first be adjusted according to the color clamping parameters, and then the weight ratio of the second color parameters before and after adjustment can be determined according to the image change parameters, and then the third color parameter of the second pixel can be obtained by weighting. For a detailed description, please refer to the following embodiments.

[0044] Step S140: Based on the displacement parameter and the third color parameter of each second pixel in the historical frame image, perform a color mixing operation on the first color parameter of each first pixel in the current frame image to obtain the current frame image with a second resolution.

[0045] Furthermore, the first color parameter of each first pixel in the current frame image can be mixed with the third color parameter of each second pixel in the historical frame image to obtain the current frame image with the second resolution. Specifically, the first color parameter of each first pixel in the current frame image can be mixed with the displacement parameter and the third color parameter of each second pixel in the historical frame image.

[0046] It should be noted that since the current frame image has already been sampled using the second resolution in the aforementioned steps, the number of first pixels in the current frame image is essentially the same as the number of second pixels in the historical frame image. Therefore, during color mixing, the third color parameter of the second pixel is used to perform color mixing on the first color parameter of the corresponding first pixel. For example, a first pixel at a certain position in the current frame image and a second pixel at the same position in a historical frame image can be identified as a matching pixel.

[0047] For example, the weight ratio of the third color parameter of the second pixel and the weight ratio of the first color parameter of the first pixel can be determined according to the displacement parameter. Then, the color parameter of each first pixel after the color mixing operation is obtained by weighting based on the third color parameter of each second pixel and the first color parameter of each first pixel in the current frame image, so as to obtain the current frame image with the second resolution, thereby realizing the super-resolution of the current frame image. For a detailed description, please refer to the following embodiments.

[0048] The resolution enhancement method provided in this application first acquires a current frame image at a first resolution, a historical frame image at a second resolution adjacent to the current frame image, and image change parameters. Then, it performs a sampling operation on the current frame image at the second resolution to obtain a first color parameter for each first pixel in the current frame image after the sampling operation. Next, it corrects the second color parameter for each second pixel in the historical frame image based on a color clamping parameter and the image change parameters to obtain a third color parameter for each second pixel, wherein the color clamping parameter is determined based on the first color parameter. Finally, it performs a color mixing operation on the first color parameter of each first pixel in the current frame image based on the displacement parameter and the third color parameter of each second pixel in the historical frame image to obtain the current frame image at the second resolution. In this application's solution, the color clamping parameter is determined based on the first color parameter; therefore, the color clamping parameter can be flexibly changed. Furthermore, based on the color clamping parameters and the image change parameters, the second color parameters of each second pixel in the historical frame image are corrected. Then, based on the displacement parameters and the third color parameters of each second pixel in the historical frame image, a color mixing operation is performed on the first color parameters of each first pixel in the current frame image to obtain the current frame image at the second resolution. It is evident that the colors of the frame image after color clamping are not simply used directly as the colors of the super-resolution image. Instead, a color mixing operation is used to obtain the current frame image at the second resolution, thus avoiding the problem of poor color quality in the generated second-resolution current frame image to a certain extent, resulting in better color performance in the super-resolution image.

[0049] Please see Figure 3 , Figure 3 A flowchart of a resolution enhancement method provided in an embodiment of this application is shown. This resolution enhancement method can be applied to... Figure 1 In the illustrated resolution enhancement scenario, the electronic device's processor can be used as the execution entity for performing the resolution enhancement method. This resolution enhancement method may include steps S210 to S2100.

[0050] Step S210: Obtain the current frame image at a first resolution, the historical frame image at a second resolution adjacent to the current frame image, and image change parameters, wherein the image change parameters include at least one of the depth change parameters and the transparency change parameters between the current frame image and the historical frame image, and the image change parameters also include the displacement parameters between the current frame image and the historical frame image, and the first resolution is smaller than the second resolution.

[0051] Step S220: Perform a sampling operation on the current frame image using the second resolution to determine the position parameters of each first pixel in the current frame image.

[0052] First, by sampling the current frame image at a second resolution, the position parameters of each first pixel in the current frame image can be determined. It is understood that each first pixel is not a point mass, but can be considered as a rectangular region of a certain size. Therefore, for example, the position parameters of the first pixel can be characterized by its center position.

[0053] It should be noted that the center position of the first pixel can be characterized by its coordinate position in the image coordinate system, for example, the image coordinate system is established based on the current frame image.

[0054] Furthermore, as described in the foregoing embodiments, the image sequence is generated by an application running on the electronic device. When rendering each frame image, the application can set corresponding jitter parameters, which characterize the changes in the camera's field of view within the corresponding frame image. However, directly obtaining the position parameter of the first pixel through sampling results in low accuracy of the position parameter. Therefore, in some implementations, a de-jitter operation can be performed based on the jitter parameters to improve the accuracy of the obtained position parameters. Specifically, step S220 may also include steps S221 to S223.

[0055] Step S221: Obtain the jitter parameters corresponding to the current frame image.

[0056] Step S222: Perform a jitter removal operation on the current frame image based on the jitter parameters.

[0057] Step S223: Sample the current frame image after the jitter removal operation using the second resolution to determine the position parameters of each first pixel in the current frame image.

[0058] First, the jitter parameters corresponding to the current frame image can be obtained. For example, the electronic device can obtain them directly from an application. In some implementations, the application can vary the jitter parameters at fixed intervals, so that each jitter parameter corresponds to a specific frame image within that interval. For example, an interval of 8 frames can be used.

[0059] Therefore, after obtaining the jitter parameters, a de-jitter operation can be performed on the current frame image based on the jitter parameters. For example, a de-jitter algorithm can be used to control the first pixel in the current frame image by subtracting the pixel value corresponding to the jitter parameters, thereby achieving the de-jitter operation on the current frame image.

[0060] Furthermore, the current frame image after the jitter removal operation is sampled using a second resolution to determine the position parameters of each first pixel in the current frame image.

[0061] Step S230: Based on the position parameters of each first pixel, determine at least one sampling pixel corresponding to each first pixel, wherein the sampling pixel corresponding to the first pixel includes the first pixel adjacent to the first pixel.

[0062] Furthermore, in order to obtain more accurate color parameters for the first pixel, the color parameters of the first pixel can be adjusted based on the pixels surrounding it.

[0063] Specifically, at least one sampling pixel can be determined first, corresponding to each first pixel. For example, the corresponding sampling pixel can be determined based on the position parameters of each first pixel. It should be noted that the sampling pixel corresponding to a first pixel includes the first pixels adjacent to that first pixel.

[0064] For example, for a given first pixel, the two adjacent first pixels surrounding that first pixel can be determined as the sampling pixels corresponding to that first pixel. Specifically, for example, the first pixels adjacent to the top left and bottom right of the first pixel can be used as the sampling pixels corresponding to that first pixel.

[0065] If the center point of a pixel is used as the position parameter of the first pixel, then the first pixel points corresponding to the first pixel points can be offset to the lower left by 0.5 times the pixel value and to the upper right by 0.5 times the pixel value, respectively.

[0066] It is understood that the number of sampling pixels mentioned above can also be 1, 3, etc., and this application embodiment does not make specific limitations.

[0067] Step S240: Determine the first color parameter of each first pixel based on the initial color parameter of each first pixel and the initial color parameter of at least one sampled pixel corresponding to the first pixel.

[0068] After determining at least one sampled pixel corresponding to each first pixel, the first color parameter of each first pixel can be determined based on the initial color parameters of each first pixel and the initial color parameters of the at least one sampled pixel corresponding to that first pixel. For example, the initial color parameters of the sampled pixels can be adjusted based on a pre-set specified coefficient, and then the initial color parameters of the first pixel can be further adjusted based on the adjusted color parameters. The initial color parameters can include red, green, and blue (RGB) color parameters; for example, they can include color parameters for the red channel, green channel, and blue channel, respectively. Specifically, step S240 can include steps S241 to S243.

[0069] Step S241: Determine the product of the first specified coefficient and the initial color parameter of at least one sampled pixel corresponding to each first pixel, and obtain the first parameter value corresponding to each first pixel.

[0070] Step S242: The difference between the initial color parameter of the first pixel and the first parameter value corresponding to the first pixel is used as the color gradient value of the first pixel.

[0071] Step S243: Sharpen the initial color parameters of the first pixel based on the color gradient value of the first pixel, and determine the first color parameters of each first pixel.

[0072] The first specified coefficient can be pre-set, for example, by the developer. Then, the product of the first specified coefficient and the initial color parameter of at least one sampled pixel corresponding to each first pixel can be obtained to get the first parameter value corresponding to each first pixel. It should be noted that the first parameter value corresponding to each first pixel includes the product of the initial color parameter of the sampled pixel corresponding to that first pixel and the first specified coefficient.

[0073] Furthermore, the difference between the initial color parameter of the first pixel and the corresponding first parameter value can be obtained as the color gradient value of the first pixel. It should be noted that for each first pixel, its color gradient value is determined by its own initial color parameter and its corresponding sampled pixel.

[0074] Then, the initial color parameters of the first pixel can be sharpened using the color gradient value of the first pixel, thereby determining the first color parameters of each first pixel. Specifically, step S243 may include steps S2431 and S2432.

[0075] Step S2431: Determine the product of the second specified coefficient and the color gradient value of each first pixel to obtain the second parameter value corresponding to each first pixel.

[0076] Step S2432: The sum of the initial color parameter of the first pixel and the second parameter value corresponding to the first pixel is used as the first color parameter of the first pixel.

[0077] First, a second specified coefficient can be determined. This second specified coefficient can be preset, for example, it can be preset by the developer. In some implementations, this second specified coefficient can also be referred to as a sharpening parameter.

[0078] The product of the second specified coefficient and the color gradient value of each first pixel can be determined to obtain the second parameter value corresponding to each first pixel. However, the sum of the initial color parameter of the first pixel and the second parameter value corresponding to that first pixel is then used as the first color parameter of that first pixel, thereby correcting the initial color parameter.

[0079] Optionally, directly using the sum of the initial color parameters of the first pixel and the color gradient value adjusted by the sharpening parameters as the first color parameter may result in poor smoothness of the obtained first color parameter, leading to a more fragmented display effect. Therefore, in some implementations, the first color parameter can also be smoothed using the sampled pixels corresponding to the first pixel. Specifically, step S2432 may also include steps S2433 to S2437.

[0080] Step S2433: Calculate the sum of the initial color parameter of each first pixel and the second parameter value corresponding to that first pixel to obtain the third parameter value corresponding to each first pixel.

[0081] Step S2434: Determine the fifth weight ratio corresponding to the first pixel and the sixth weight ratio of the sampled pixel corresponding to the first pixel, wherein the sum of the fifth weight ratio and the corresponding sixth weight ratios is 1.

[0082] Step S2435: Obtain the product of the fifth weight ratio and the third parameter value corresponding to the first pixel point to obtain the fourth parameter value corresponding to the first pixel point.

[0083] Step S2436: Obtain the product of the sixth weight ratio and the initial color parameter of the sampled pixel corresponding to the first pixel to obtain the fifth parameter value corresponding to the first pixel.

[0084] Step S2437: The sum of the fourth parameter value and the fifth parameter value corresponding to the first pixel is used as the first color parameter of the first pixel.

[0085] First, the sum of the initial color parameter of each first pixel and the second parameter value corresponding to that first pixel can be calculated to obtain the third parameter value corresponding to each first pixel.

[0086] Then, the fifth weight percentage corresponding to the first pixel and the sixth weight percentage of the sampled pixels corresponding to the first pixel can be determined. The sum of the fifth weight percentage and each of the corresponding sixth weight percentages is 1. For example, the fifth and sixth weight percentages can be preset, such as by the developers.

[0087] It should be noted that the first pixel may correspond to one or more sampling pixels. When there is only one corresponding sampling pixel, the sum of the fifth weight and one sixth weight is 1, for example, the fifth weight is 0.6 and the sixth weight is 0.4. When there are multiple corresponding pixels, taking two as an example, the sum of the fifth weight and the two sixth weights is 1, for example, the fifth weight is 0.6 and the sixth weight is 0.2. The above examples are only to illustrate the relationship between the fifth and sixth weights and do not constitute a specific limitation on the embodiments of this application.

[0088] Furthermore, the product of the fifth weight ratio and the third parameter value corresponding to the first pixel is obtained to get the fourth parameter value corresponding to the first pixel; and the product of the sixth weight ratio and the initial color parameter of the sampled pixel corresponding to the first pixel is obtained to get the fifth parameter value corresponding to the first pixel.

[0089] It should be noted that the number of fifth parameter values ​​corresponding to the first pixel is the same as the number of sampled pixels corresponding to that first pixel. In other words, the number of fifth parameter values ​​corresponds to the number of sampled pixels corresponding to the first pixel.

[0090] However, the sum of the fourth and fifth parameter values ​​corresponding to the first pixel can be used as the first color parameter of the first pixel. If there are multiple fifth parameter values, then the sum of the fourth parameter value and the multiple fifth parameter values ​​corresponding to the first pixel can be used as the first color parameter of the first pixel. This achieves smoothing of the color parameter of the first pixel, resulting in a smoother first color parameter and a lower sense of disjointedness in the display effect.

[0091] The method described above for determining the first color parameter of the first pixel by combining the sampled pixel with the first specified coefficient and the second specified coefficient can also be referred to as performing multiple bilinear samplings, such as performing three bilinear samplings.

[0092] Step S250: Adjust the second color parameter of each second pixel in the historical frame image based on the color clamping parameter to obtain the fourth color parameter of each second pixel.

[0093] Step S260: Determine the first weight percentage of the fourth color parameter based on the image change parameters.

[0094] Step S270: Based on the fourth color parameter of each second pixel adjusted by the first weight ratio and the second color parameter of each second pixel adjusted by the second weight ratio, obtain the third color parameter of each second pixel, wherein the sum of the first weight ratio and the second weight ratio is 1.

[0095] Furthermore, the second color parameter of each second pixel in the historical frame image can be adjusted based on the color clamping parameter to obtain the fourth color parameter of each second pixel.

[0096] As described above, the second color parameters of each second pixel in the historical frame image can be retrieved directly from the storage module.

[0097] Optionally, a historical frame image at a first resolution can be obtained based on the current frame image and displacement parameters. Then, a sampling operation can be performed on this historical frame image at a second resolution to obtain the second color parameter of each second pixel in the historical frame image after the sampling operation. Similar to performing a sampling operation on the current frame image at the first resolution, multiple bilinear sampling operations can be performed, such as three bilinear sampling operations. By determining the sampling pixel corresponding to the second pixel, the set first specified coefficient, and the second specified coefficient, the second color parameter of the second pixel can be obtained. For example, the sampling pixel corresponding to the second pixel can be determined within a certain area surrounding the second pixel; specifically, this area can be 4x4 pixels in size.

[0098] It should be noted that the first and second specified coefficients set for the second pixel can be different from the first and second specified coefficients set for the first pixel.

[0099] As described above, the color clamping parameters are determined based on the first color parameters. These color clamping parameters can include parameters corresponding to each first pixel. For example, the corresponding parameters can be adjusted based on the first color parameters of each first pixel to obtain the color clamping parameters.

[0100] Then, the second color parameter of each second pixel in the historical frame image can be adjusted based on the color clamping parameter to obtain the fourth color parameter of each second pixel.

[0101] For example, as described above, each second pixel in a historical frame image can correspond to each first pixel in the current frame image. This correspondence can be determined by the position of each second pixel in the historical frame image and the position of each first pixel in the current frame image. Therefore, the color clamping parameters corresponding to the first pixels also correspond to the second pixels.

[0102] Therefore, the second color parameter of each second pixel in the historical frame image can be compared to see if it satisfies the parameter corresponding to that second pixel in the color clamping parameters. If it does, the color parameter of that second pixel can be retained, i.e., it will not be adjusted; if it does not, the color parameter of that second pixel can be replaced with the corresponding parameter in the color clamping parameters. By sequentially checking whether each second pixel in the historical frame image satisfies the corresponding parameter, the adjustment of the second color parameter of each second pixel in the historical frame image based on the color clamping parameters can be completed.

[0103] Therefore, by using color clamping parameters to limit the second color parameters of each second pixel in the historical frame image, ghosting and artifacts can be prevented to a certain extent in the subsequently generated second-resolution current frame image.

[0104] The color parameters of each second pixel in the historical frame image after adjusting the second color parameters based on the color clamping parameters can be directly used as the third color parameter. However, this may cause artifacts to appear in the second pixels, resulting in poor clarity in the subsequent second-resolution current frame image. Therefore, in some implementations, the color parameters of each second pixel in the historical frame image after adjusting the second color parameters based on the color clamping parameters can also be used as the fourth color parameter. Then, the first weight ratio of the fourth color parameter is determined according to the image change parameters.

[0105] For example, if the image change parameters include displacement parameters, depth change parameters, and transparency change parameters, then the first weight percentage is negatively correlated with the displacement parameters, negatively correlated with the depth change parameters, and negatively correlated with the transparency change parameters, respectively.

[0106] Additionally, the second weighting percentage can be determined based on the first weighting percentage, and this second weighting percentage can be used to adjust the second color parameter. The sum of the second weighting percentage and the first weighting percentage is 1.

[0107] In some implementations, if the first weight percentage is less than or equal to 0.7, then the corresponding second weight percentage can be greater than or equal to 0.3.

[0108] For example, if the displacement parameter, depth change parameter, and transparency change parameter are all very small, the first weight percentage can be determined to be close to 0.7, and the second weight percentage can be close to 0.3; or if the displacement parameter, depth change parameter, and transparency change parameter are all very large, the first weight percentage can be determined to be very small, for example close to 0, and the second weight percentage can be close to 1.

[0109] It should be noted that the above descriptions of the first weight percentage and the second weight percentage being larger or smaller are used to indicate whether the first weight percentage or the second weight percentage is larger or smaller within their respective corresponding ranges, and are not records of absolute numerical values.

[0110] Therefore, the fourth color parameter of each second pixel can be adjusted using the first weighting ratio, and the second color parameter of each second pixel can be adjusted using the second weighting ratio. Then, based on the fourth color parameter of each second pixel adjusted using the first weighting ratio and the second color parameter of each second pixel adjusted using the second weighting ratio, the third color parameter of each second pixel is obtained.

[0111] Specifically, step S270 may include steps S271 to S273.

[0112] Step S271: Multiply the fourth color parameter of each second pixel by the first weight ratio to obtain the first weighted color parameter of each second pixel.

[0113] Step S272: Multiply the second color parameter of each second pixel by the second weight ratio to obtain the second weighted color parameter of each second pixel.

[0114] Step S273: Add the first weighted color parameter of the second pixel to the second weighted color parameter of the second pixel to obtain the third color parameter of each second pixel.

[0115] The fourth color parameter of each second pixel can be multiplied by the first weight ratio to obtain the first weighted color parameter of each second pixel. Then, the second color parameter of each second pixel is multiplied by the second weight ratio to obtain the second weighted color parameter of each second pixel. Finally, the first weighted color parameter of the second pixel is added to the second weighted color parameter of the second pixel to obtain the third color parameter of each second pixel. This improves the sharpness of the current frame image at the subsequent second resolution without introducing significant artifacts.

[0116] Step S280: Determine the third weight ratio of the third color parameter based on the displacement parameter, wherein the third weight ratio is inversely correlated with the displacement parameter.

[0117] Furthermore, in order to perform a color mixing operation on the first color parameter of the first pixel, a third weighting percentage can be determined first. This third weighting percentage is used to adjust the third color parameter. In some implementations, the third weighting percentage of the third color parameter can be determined based on a displacement parameter. The third weighting percentage is inversely correlated with the displacement parameter.

[0118] The displacement parameter is used to characterize the displacement change between the current frame image and historical frame images.

[0119] Additionally, the fourth weighting percentage can be determined based on the third weighting percentage, where the sum of the third and fourth weighting percentages is 1. Therefore, the fourth weighting percentage can be obtained after determining the third weighting percentage. The fourth weighting percentage is used to adjust the first color parameter of each first pixel.

[0120] In some implementations, the third weight percentage can be greater than or equal to 0.7 and less than or equal to 0.9, and the corresponding fourth weight percentage can be greater than or equal to 0.1 and less than or equal to 0.3.

[0121] For example, if the displacement parameter is large, a smaller third weight percentage can be determined, such as close to 0.7, and a larger fourth weight percentage can be obtained, such as close to 0.3.

[0122] It should be noted that the above descriptions of the third and fourth weight percentages being larger or smaller are used to indicate whether the third or fourth weight percentage is larger or smaller within its respective range, and are not records of absolute numerical values.

[0123] Optionally, the third weight percentage can be determined by combining the frame rate parameter of the frame image sequence. Specifically, step S280 may include steps S281 and S282.

[0124] Step S281: Obtain the frame rate parameter of the frame image sequence in which the current frame image is located.

[0125] Step S282: Determine the third weight ratio of the third color parameter based on the frame rate parameter and the displacement parameter, wherein the third weight ratio is also positively correlated with the frame rate parameter.

[0126] As described above, electronic devices can run applications that generate images or image sequences. For example, a game application can generate image sequences at a certain frame rate parameter, providing a dynamic display effect for the user. Specifically, this frame rate parameter could be 30 frames per second (fps), 60 fps, etc. Therefore, the frame rate parameter of the current frame image sequence can also be obtained. This frame rate parameter can be directly provided by the application or calculated by the electronic device based on the number of frame images acquired within a recent time period. For example, if 30 image frames are acquired within the last 0.5 seconds, the frame rate parameter can be determined to be 30 / 0.5 = 60 fps.

[0127] Furthermore, the third weight ratio of the third color parameter is determined based on the frame rate parameter and the displacement parameter, wherein the third weight ratio is also inversely correlated with the frame rate parameter.

[0128] For example, if the displacement parameter is large and the frame rate parameter is small, a smaller third weight percentage can be determined, for example, close to 0.7, and a larger fourth weight percentage can be obtained, for example, close to 0.3. If the displacement parameter is small and the frame rate parameter is large, a larger third weight percentage can be determined, for example, close to 0.9, and a smaller fourth weight percentage can be obtained, for example, close to 0.1.

[0129] Since the frame rate parameter has a significant impact on the rate at which artifacts in the generated second-resolution current frame image are reduced to a barely perceptible level, this embodiment assigns a larger third weight percentage when the frame rate parameter is large, thereby sampling more colors from historical frame images. Conversely, it assigns a smaller third weight percentage when the frame rate parameter is small, thereby sampling fewer colors from historical frame images. This optimizes artifacts in the current frame image and improves image quality.

[0130] Step S290: Based on the third color parameter of each second pixel adjusted by the third weight ratio and the first color parameter of each first pixel adjusted by the fourth weight ratio, obtain the fifth color parameter of each first pixel, wherein the sum of the third weight ratio and the fourth weight ratio is 1.

[0131] Furthermore, the third color parameter of each second pixel can be adjusted by the third weight ratio, and then the first color parameter of each first pixel can be adjusted by the fourth weight ratio. Based on the third color parameter of each second pixel adjusted by the third weight ratio and the first color parameter of each first pixel adjusted by the fourth weight ratio, the fifth color parameter of each first pixel can be obtained.

[0132] Specifically, step S290 may include steps S291 to S293.

[0133] Step S291: Multiply the third color parameter of each second pixel by the third weight ratio to obtain the third weighted color parameter of each second pixel.

[0134] Step S292: Multiply the first color parameter of each first pixel by the fourth weight ratio to obtain the fourth weighted color parameter of each first pixel.

[0135] Step S293: Add the fourth weighted color parameter of the first pixel to the third weighted color parameter of the second pixel corresponding to the first pixel to obtain the fifth color parameter of each first pixel.

[0136] Specifically, the third color parameter of each second pixel can be multiplied by the third weight ratio to obtain the third weighted color parameter of each second pixel. Then, the first color parameter of each first pixel is multiplied by the fourth weight ratio to obtain the fourth weighted color parameter of each first pixel. Finally, the fourth weighted color parameter of the first pixel is added to the third weighted color parameter of the corresponding second pixel to obtain the fifth color parameter of each first pixel.

[0137] Step S2100: Generate the current frame image with a second resolution based on the fifth color parameter of each first pixel.

[0138] Therefore, a second-resolution current frame image can be generated based on the fifth color parameter of each first pixel. It can be understood that the generated second-resolution current frame image is the frame image obtained by super-resolution of the first-resolution current frame image.

[0139] The resolution enhancement method provided in this application can smooth the first color parameter by sampling the pixels corresponding to the first pixel, resulting in a smoother first color parameter and a lower sense of fragmentation in the display effect. Furthermore, since the frame rate parameter has a significant impact on the rate at which artifacts in the current frame image of the generated second resolution are reduced to an imperceptible level, this application embodiment assigns a larger third weight percentage when the frame rate parameter is large, thereby sampling more colors from historical frame images; conversely, it assigns a smaller third weight percentage when the frame rate parameter is small, thereby sampling fewer colors from historical frame images. This optimizes artifacts in the current frame image and improves image quality. Moreover, a jitter reduction operation can be performed based on the jitter parameter to improve the accuracy of the obtained position parameters.

[0140] Please see Figure 4 , Figure 4 A flowchart of a resolution enhancement method provided in an embodiment of this application is shown. This resolution enhancement method can be applied to... Figure 1 In the illustrated resolution enhancement scenario, the electronic device's processor can be used as the execution entity for the resolution enhancement method. This resolution enhancement method may include steps S310 to S370.

[0141] Step S310: Obtain the current frame image at the first resolution and the historical frame image at the second resolution adjacent to the current frame image.

[0142] The method for obtaining the current frame image at a first resolution and the historical frame image at a second resolution adjacent to the current frame image has been described in detail in the foregoing embodiments and will not be repeated here.

[0143] Step S320: Obtain the first depth parameter, the first transparency parameter, the displacement parameter, the second depth parameter, and the second transparency parameter of each second pixel in the current frame image.

[0144] In some implementations, the image change parameters include depth change parameters, transparency change parameters, and displacement parameters, which will be further described as examples.

[0145] The first depth parameter, the first transparency parameter, the displacement parameter, the second depth parameter, and the second transparency parameter of each second pixel in the current frame image can be obtained respectively.

[0146] The electronic device can directly obtain the first depth parameter, the first transparency parameter, the displacement parameter, the second depth parameter, and the second transparency parameter through an application.

[0147] It is understood that the first depth parameter, the first transparency parameter, the displacement parameter, the second depth parameter, and the second transparency parameter all require a certain amount of bandwidth. Therefore, directly obtaining each of these parameters separately may result in a large bandwidth consumption and low bandwidth utilization. Optionally, the first depth parameter, the displacement parameter, and the first transparency parameter can be packaged into data to obtain the first packaged data.

[0148] Optionally, each frame image can also have a corresponding motion vector mask. This motion vector mask can mark target objects in the corresponding frame image, thereby giving higher priority to image regions containing motion vector masked target objects during super-resolution. For example, if a certain region corresponds to only one motion vector mask in both historical and current frame images (e.g., it corresponds to a motion vector mask in a historical frame image but not in the current frame image; or it doesn't correspond to a motion vector mask in a historical frame image but does in the current frame image), then higher weights can be assigned to the color parameters in the frame image corresponding to the motion vector mask.

[0149] Therefore, in some implementations, the first packaged data may also include a motion vector mask corresponding to the current frame image.

[0150] For example, the first packaged data can be 32-bit unsigned integer single-channel (R32UI) format texture data.

[0151] For example, the first depth parameter can be represented by a 24-bit integer, so that the 24-bit integer value of the first depth parameter can be normalized to the range of [0.0, 1.0] by piecewise linear mapping, and then quantized into an 8-bit unsigned normalized value.

[0152] Furthermore, the displacement parameter can be represented by a 16-bit or 20-bit integer, and can be separated into values ​​along the x-channel and the y-channel. The x-channel represents the parameter corresponding to the x-axis in the image coordinate system constructed from the current frame image, and the y-channel represents the parameter corresponding to the y-axis. Then, the values ​​along the x-channel and the y-channel are quantized into 8-bit unsigned normalized values ​​respectively.

[0153] Furthermore, the first transparency parameter can be normalized to the range of [0.0, 1.0] and quantized into a 7-bit unsigned normalized value.

[0154] In addition, the motion vector mask can be represented by an 8-bit integer, which can then be binarized, assigning a value of 1 to the mask value corresponding to a pixel that is greater than 0, and otherwise assigning a value of 0.

[0155] Furthermore, the first packaged data is obtained by packaging the quantized first depth parameter, displacement parameter, first transparency parameter, and binarized motion vector mask into 32-bit unsigned integer single-channel texture data.

[0156] Additionally, it is understood that historical frame images may also correspond to second packaged data, which includes the second depth parameter and the second transparency parameter of each second pixel in the historical frame image. Therefore, the second packaged data can be directly obtained from the storage module.

[0157] Step S330: Based on the first depth parameter of the reference pixel corresponding to the first pixel in the current frame image, the first depth parameter, first transparency parameter and displacement parameter of the first pixel are corrected to obtain the third depth parameter, third transparency parameter and corrected displacement parameter of each first pixel, wherein the reference pixel is the first pixel within a specified range around the first pixel.

[0158] Furthermore, as described above, the first depth parameter and the first transparency parameter can be provided by the application running on the electronic device. However, the edges of objects in each first pixel of the current frame image generally have an anti-aliasing effect, while the first depth parameter and the first transparency parameter directly provided by the application generally do not have an anti-aliasing effect. Therefore, if the obtained first depth parameter and the first transparency parameter are used directly later, the edges of some objects in the first frame image may carry the color of other nearby objects in the color parameters, but the depth parameter, displacement parameter, transparency parameter, etc., may not carry the parameters of other nearby objects. This may lead to lower accuracy of the subsequently determined image change parameters, resulting in a poor effect of the current frame image at the second resolution.

[0159] Therefore, in some implementations, a reference pixel can also be determined for the first pixel. For example, the first pixel within a specified range surrounding it can be determined as the reference pixel corresponding to the first pixel. For instance, the specified range could be a 3x3 pixel range.

[0160] Furthermore, based on the first depth parameter of the reference pixel corresponding to the first pixel in the current frame image, the first depth parameter, first transparency parameter, and displacement parameter of the first pixel can be corrected to obtain the third depth parameter, third transparency parameter, and corrected displacement parameter of each first pixel. The corrected displacement parameter can then be used as one of the aforementioned image change parameters.

[0161] Specifically, step S330 may include steps S331 and S332.

[0162] Step S331: Determine the target reference pixel corresponding to the smallest first depth parameter among the reference pixels corresponding to the first pixel; Step S332: Replace the first depth parameter of the first pixel with the first depth parameter of the corresponding target reference pixel, replace the first transparency parameter of the first pixel with the first transparency parameter of the corresponding target reference pixel, and replace the displacement parameter of the first pixel with the displacement parameter of the corresponding target reference pixel.

[0163] First, we can find the smallest first depth parameter among the first depth parameters of the reference pixels corresponding to the first pixel. Then, we determine the reference pixel corresponding to the smallest first depth parameter as the target reference pixel. Next, we can replace the first depth parameter of the first pixel with the first depth parameter of the corresponding target reference pixel, replace the first transparency parameter of the first pixel with the first transparency parameter of the corresponding target reference pixel, and replace the displacement parameter of the first pixel with the displacement parameter of the corresponding target reference pixel.

[0164] As described above, the first depth parameter, displacement parameter, first transparency parameter, and motion vector mask can be packaged into data to obtain the first packaged data. Therefore, optionally, the first depth parameter, displacement parameter, and first transparency parameter corresponding to the first pixel can be directly corrected based on the first packaged data.

[0165] In some implementations, the process of correcting the first depth parameter, first transparency parameter, and displacement parameter of the first pixel by referring to the first pixel can also be referred to as a data inflation process.

[0166] Step S340: Determine the depth change parameter based on the third depth parameter of each first pixel in the current frame image and the second depth parameter of each second pixel in the historical image frame, and determine the transparency change parameter based on the third transparency parameter of each first pixel in the current frame image and the second transparency parameter of each second pixel in the historical image frame.

[0167] Furthermore, after correcting the first depth parameter and the first transparency parameter, a third depth parameter and a third transparency parameter are obtained. Thus, the depth change parameter can be determined based on the third depth parameter of each first pixel in the current frame image and the second depth parameter of each second pixel in the historical image frame, and the transparency change parameter can be determined based on the third transparency parameter of each first pixel in the current frame image and the second transparency parameter of each second pixel in the historical image frame.

[0168] For example, the depth change parameter can also correspond to each pixel point, so that the difference between the third depth parameter of each first pixel point and the second depth parameter of each second pixel point in the historical image frame can be used as the depth change parameter corresponding to the first pixel point; and the difference between the third transparency parameter of each first pixel point in the current frame image and the second transparency parameter of each second pixel point in the historical image frame can be used as the transparency change parameter corresponding to the first pixel point.

[0169] It should be noted that the first and second pixels for calculating the difference are related. For example, the relationship can be determined based on the position of each second pixel in the historical frame image and the position of each first pixel in the current frame image.

[0170] Step S350: Perform a sampling operation on the current frame image at a second resolution to obtain the first color parameter of each first pixel in the current frame image after the sampling operation.

[0171] Step S360: Based on the color clamping parameters and the image change parameters, correct the second color parameters of each second pixel in the historical frame image to obtain the third color parameters of each second pixel, wherein the color clamping parameters are determined based on the first color parameters.

[0172] Step S370: Based on the displacement parameter and the third color parameter of each second pixel in the historical frame image, perform a color mixing operation on the first color parameter of each first pixel in the current frame image to obtain the current frame image with a second resolution.

[0173] Steps S350 to S370 have been described in detail in the foregoing embodiments and will not be repeated here.

[0174] The resolution enhancement method provided in this application improves the accuracy of image change parameters by referencing a first pixel and correcting its first depth parameter, first transparency parameter, and displacement parameter. This is achieved through data dilation, thereby enhancing the image quality of the current frame at the second resolution. Furthermore, this application also improves bandwidth utilization by performing data packaging processing on the first depth parameter, the displacement parameter, and the first transparency parameter to obtain first packaged data.

[0175] Please see Figure 5 , Figure 5 A flowchart illustrating the resolution enhancement method provided in an embodiment of this application is shown. Figure 5 Steps S401 to S412 are shown in the figure.

[0176] Step S401: Obtain the first depth.

[0177] Step S402: Obtain displacement parameters.

[0178] Step S403: Obtain the motion vector mask.

[0179] Step S404: Obtain the first transparency parameter.

[0180] Step S405: Data preparation.

[0181] Step S406: Data inflation.

[0182] Step S407: Generate the first packaged data.

[0183] Step S408: Obtain the first color parameter.

[0184] Step S409: Super-resolution operation.

[0185] Step S410: Obtain the second packaged data.

[0186] Step S411: Obtain the second color parameters.

[0187] Step S412: Obtain the current frame image at the second resolution.

[0188] The acquisition of the first depth, the displacement parameters, the motion vector mask, and the first transparency parameters can be found in the description of the foregoing embodiments, and will not be repeated here.

[0189] Data preparation may include quantizing the first depth, the acquired displacement parameters, and the first transparency parameters, and binarizing the acquired motion vector mask; for details, please refer to the foregoing embodiments. Then, first packaged data is generated through data dilation.

[0190] Furthermore, a super-resolution operation is performed based on the first color parameter, the first packaged data, the second packaged data, and the second color parameter. The second packaged data and the second color parameter can be obtained by searching the cache for the previous frame image, i.e., by using the corresponding packaged data and color parameters of a historical frame image.

[0191] The super-resolution operation may include correcting the second color parameter of each second pixel in the historical frame image using color clamping parameters, and performing a color blending operation on the first color parameter of each first pixel in the current frame image based on the displacement parameter and the third color parameter of each second pixel in the historical frame image. Finally, a current frame image with a second resolution can be obtained.

[0192] It should be noted that detailed descriptions of the above steps can be found in the foregoing embodiments.

[0193] Furthermore, the inventors demonstrated through experimental testing the effectiveness of the resolution enhancement method provided in this application compared to some existing resolution enhancement methods. Please refer to [link / reference]. Figure 6 , Figure 6 A comparison chart showing the effects of different resolution enhancement methods is presented. Figure 6 The image shows renderings 601, 602, 603, and 604.

[0194] Figure 601 illustrates the effect of combining a higher-resolution original image with native temporal anti-aliasing (TAA); Figure 602 illustrates the effect of combining a lower-resolution original image with native temporal anti-aliasing; Figure 603 illustrates the effect of the resolution enhancement method provided in this application, which can be considered as the generated second-resolution current frame image; Figure 604 illustrates the original image with a lower resolution that has not been processed using any resolution enhancement method. For example, the lower resolution can be 1782x810 pixels, and the higher resolution can be 2376x1080 pixels.

[0195] pass Figure 6 As can be seen, the effect diagram 603 corresponding to the resolution enhancement method provided in this application embodiment has a better effect than the effect diagram 601 based on the original image with native temporal anti-aliasing at a higher resolution, specifically in terms of higher clarity and fewer artifacts.

[0196] In addition, the inventors also conducted power consumption research and testing. Generating a higher resolution image at 60 frames per second resulted in a power consumption of 6.65 watts during a specified test period; while using the same 60 frames per second, combining a lower resolution frame image with the resolution enhancement method provided in this application to obtain a higher resolution image resulted in a power consumption of 5.84 watts during the same test period.

[0197] As can be seen, the resolution enhancement method provided in this application embodiment can achieve low computational overhead and low power consumption while ensuring image quality. It significantly extends the battery life of mobile terminals while guaranteeing an upgraded visual experience.

[0198] Furthermore, currently, half of the methods for resolution enhancement require application developers to integrate the resolution enhancement solution at a specific stage of the native rendering pipeline. This limits the pipeline access points for already released applications, making later adjustments difficult. However, the resolution enhancement method provided in this application obtains the current frame image at a second resolution through image change parameters, the current frame image, and historical frame images. It does not impose specific requirements on the location of the resolution enhancement method within the application's rendering pipeline, thus making it more easily adaptable to different applications, offering greater flexibility, and facilitating deployment to mobile terminals.

[0199] Please see Figure 7 , Figure 7 The diagram shows a structural block diagram of a resolution enhancement device provided in an embodiment of this application. The resolution enhancement device 700 includes: an acquisition unit 710, a collection unit 720, a correction unit 730, and a mixing unit 740.

[0200] The acquisition unit 710 is used to acquire a current frame image at a first resolution, a historical frame image at a second resolution adjacent to the current frame image, and image change parameters, wherein the image change parameters include at least one of a depth change parameter and a transparency change parameter between the current frame image and the historical frame image, and the image change parameters also include a displacement parameter between the current frame image and the historical frame image, and the first resolution is smaller than the second resolution.

[0201] Optionally, the acquisition unit 710 can also be used to acquire a current frame image of a first resolution and a historical frame image of a second resolution adjacent to the current frame image; acquire a first depth parameter, a first transparency parameter, and a displacement parameter of each first pixel in the current frame image, a second depth parameter, and a second transparency parameter of each second pixel in the historical frame image; based on the first depth parameter of a reference pixel corresponding to a first pixel in the current frame image, correct the first depth parameter, the first transparency parameter, and the displacement parameter of the first pixel to obtain a third depth parameter, a third transparency parameter, and a corrected displacement parameter for each first pixel, wherein the reference pixel is a first pixel within a specified range around the first pixel; determine the depth change parameter based on the third depth parameter of each first pixel in the current frame image and the second depth parameter of each second pixel in the historical image frame, and determine the transparency change parameter based on the third transparency parameter of each first pixel in the current frame image and the second transparency parameter of each second pixel in the historical image frame.

[0202] Optionally, the acquisition unit 710 can also be used to determine the target reference pixel corresponding to the smallest first depth parameter among the reference pixels corresponding to the first pixel; replace the first depth parameter of the first pixel with the first depth parameter of the corresponding target reference pixel, replace the first transparency parameter of the first pixel with the first transparency parameter of the corresponding target reference pixel, and replace the displacement parameter of the first pixel with the displacement parameter of the corresponding target reference pixel.

[0203] The sampling unit 720 is used to perform a sampling operation on the current frame image at a second resolution to obtain the first color parameter of each first pixel in the current frame image after the sampling operation.

[0204] Optionally, the sampling unit 720 can also be used to perform a sampling operation on the current frame image using a second resolution to determine the position parameters of each first pixel in the current frame image; based on the position parameters of each first pixel, determine at least one sampling pixel corresponding to each first pixel, wherein the sampling pixel corresponding to the first pixel includes a first pixel adjacent to the first pixel; and based on the initial color parameters of each first pixel and the initial color parameters of the at least one sampling pixel corresponding to the first pixel, determine the first color parameters of each first pixel.

[0205] Optionally, the sampling unit 720 can also be used to determine the product of the first specified coefficient and the initial color parameter of at least one sampled pixel corresponding to each first pixel to obtain the first parameter value corresponding to each first pixel; take the difference between the initial color parameter of the first pixel and the first parameter value corresponding to the first pixel as the color gradient value of the first pixel; and perform sharpening processing on the initial color parameter of the first pixel based on the color gradient value of the first pixel to determine the first color parameter of each first pixel.

[0206] Optionally, the sampling unit 720 can also be used to determine the product of the second specified coefficient and the color gradient value of each first pixel point to obtain the second parameter value corresponding to each first pixel point; and to use the sum of the initial color parameter of the first pixel point and the second parameter value corresponding to the first pixel point as the first color parameter of the first pixel point.

[0207] Optionally, the sampling unit 720 can also be used to calculate the sum of the initial color parameter of each first pixel and the second parameter value corresponding to the first pixel to obtain the third parameter value corresponding to each first pixel; determine the fifth weight ratio corresponding to the first pixel and the sixth weight ratio of the sampled pixel corresponding to the first pixel, wherein the sum of the fifth weight ratio and the corresponding sixth weight ratios is 1; obtain the product of the fifth weight ratio and the third parameter value corresponding to the first pixel to obtain the fourth parameter value corresponding to the first pixel; obtain the product of the sixth weight ratio and the initial color parameter of the sampled pixel corresponding to the first pixel to obtain the fifth parameter value corresponding to the first pixel; and use the sum of the fourth parameter value and the fifth parameter value corresponding to the first pixel as the first color parameter of the first pixel.

[0208] Optionally, the sampling unit 720 can also be used to obtain the jitter parameters corresponding to the current frame image; perform a jitter removal operation on the current frame image based on the jitter parameters; and perform a sampling operation on the current frame image after the jitter removal operation through a second resolution to determine the position parameters of each first pixel in the current frame image.

[0209] The correction unit 730 is used to correct the second color parameter of each second pixel in the historical frame image based on the color clamping parameter and the image change parameter to obtain the third color parameter of each second pixel, wherein the color clamping parameter is determined based on the first color parameter.

[0210] Optionally, the correction unit 730 can also be used to adjust the second color parameter of each second pixel in the historical frame image based on the color clamping parameter to obtain the fourth color parameter of each second pixel; determine the first weight ratio of the fourth color parameter according to the image change parameter; and obtain the third color parameter of each second pixel based on the fourth color parameter of each second pixel adjusted by the first weight ratio and the second color parameter of each second pixel adjusted by the second weight ratio, wherein the sum of the first weight ratio and the second weight ratio is 1.

[0211] Optionally, the correction unit 730 can also be used to multiply the fourth color parameter of each second pixel by the first weight ratio to obtain the first weighted color parameter of each second pixel; multiply the second color parameter of each second pixel by the second weight ratio to obtain the second weighted color parameter of each second pixel; and add the first weighted color parameter of the second pixel to the second weighted color parameter of the second pixel to obtain the third color parameter of each second pixel.

[0212] Optionally, the first weight percentage is negatively correlated with the displacement parameter, negatively correlated with the depth change parameter, and negatively correlated with the transparency change parameter, respectively.

[0213] The mixing unit 740 is used to perform a color mixing operation on the first color parameter of each first pixel in the current frame image based on the displacement parameter and the third color parameter of each second pixel in the historical frame image, so as to obtain the current frame image with a second resolution.

[0214] Optionally, the mixing unit 740 can also be used to determine the third weight ratio of the third color parameter according to the displacement parameter, wherein the third weight ratio is inversely correlated with the displacement parameter; based on the third color parameter of each second pixel adjusted by the third weight ratio and the first color parameter of each first pixel adjusted by the fourth weight ratio, a fifth color parameter of each first pixel is obtained, wherein the sum of the third weight ratio and the fourth weight ratio is 1; and based on the fifth color parameter of each first pixel, a current frame image with a second resolution is generated.

[0215] Optionally, the mixing unit 740 can also be used to multiply the third color parameter of each second pixel by the third weight ratio to obtain the third weighted color parameter of each second pixel; multiply the first color parameter of each first pixel by the fourth weight ratio to obtain the fourth weighted color parameter of each first pixel; and add the fourth weighted color parameter of the first pixel to the third weighted color parameter of the second pixel corresponding to the first pixel to obtain the fifth color parameter of each first pixel.

[0216] Optionally, the mixing unit 740 can also be used to obtain the frame rate parameter of the frame image sequence in which the current frame image is located; and determine the third weight ratio of the third color parameter based on the frame rate parameter and the displacement parameter, wherein the third weight ratio is also positively correlated with the frame rate parameter.

[0217] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0218] In the several embodiments provided in this application, the coupling between the units can be electrical, mechanical, or other forms of coupling. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0219] Please see Figure 8 , Figure 8 This illustration shows a structural block diagram of an electronic device according to an embodiment of this application. The electronic device 110 may be a smartphone, desktop computer, in-vehicle computer, server, or tablet computer, etc. The electronic device 110 in this application may include one or more of the following components: a processor 111, a memory 112, and one or more application programs, wherein the processor 111 is electrically connected to the memory 112, and the one or more programs are configured to perform the methods described in the foregoing embodiments.

[0220] Processor 111 may include one or more processing cores. Processor 111 connects to various parts within the electronic device 110 using various interfaces and lines, and performs various functions and processes data of the electronic device 110 by running or executing instructions, programs, code sets, or instruction sets stored in memory 112, and by calling data stored in memory 112. Optionally, processor 111 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 111 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and computer programs; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 111 and may be implemented separately through a communication chip. Specifically, the methods described in the foregoing embodiments can be executed by one or more processors 111.

[0221] In some implementations, memory 112 may include random access memory (RAM) or read-only memory (ROM). Memory 112 can be used to store instructions, programs, code, code sets, or instruction sets. Memory 112 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the electronic device 110 during use.

[0222] Please see Figure 9 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 900 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0223] The computer-readable storage medium 900 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 910 may, for example, be compressed in a suitable form.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A resolution enhancement method, characterized in that, include: The first resolution current frame image, the second resolution historical frame image adjacent to the current frame image, and image change parameters are obtained respectively. The image change parameters include at least one of the depth change parameters and the transparency change parameters between the current frame image and the historical frame image. The image change parameters also include the displacement parameters between the current frame image and the historical frame image. The first resolution is smaller than the second resolution. The current frame image is sampled at a second resolution to obtain the first color parameter of each first pixel in the current frame image after the sampling operation. The second color parameter of each second pixel in the historical frame image is corrected based on the color clamping parameter and the image change parameter to obtain the third color parameter of each second pixel, wherein the color clamping parameter is determined based on the first color parameter; Based on the displacement parameter and the third color parameter of each second pixel in the historical frame image, a color mixing operation is performed on the first color parameter of each first pixel in the current frame image to obtain the current frame image with a second resolution.

2. The method according to claim 1, characterized in that, The process of correcting the second color parameter of each second pixel in the historical frame image based on the color clamping parameter and the image change parameter to obtain the third color parameter of each second pixel includes: Based on the color clamping parameters, the second color parameters of each second pixel in the historical frame image are adjusted to obtain the fourth color parameters of each second pixel. The first weighting percentage of the fourth color parameter is determined based on the image change parameters; Based on the fourth color parameter of each second pixel adjusted by the first weight ratio and the second color parameter of each second pixel adjusted by the second weight ratio, the third color parameter of each second pixel is obtained, wherein the sum of the first weight ratio and the second weight ratio is 1.

3. The method according to claim 2, characterized in that, The process of obtaining the third color parameter of each second pixel based on the fourth color parameter of each second pixel adjusted by the first weight ratio and the second color parameter of each second pixel adjusted by the second weight ratio includes: Multiply the fourth color parameter of each second pixel by the first weight ratio to obtain the first weighted color parameter of each second pixel; Multiply the second color parameter of each second pixel by the second weight ratio to obtain the second weighted color parameter of each second pixel; The first weighted color parameter of the second pixel is added to the second weighted color parameter of the second pixel to obtain the third color parameter of each second pixel.

4. The method according to claim 2, characterized in that, The image change parameters include displacement parameters, depth change parameters, and transparency change parameters; The first weight percentage is negatively correlated with the displacement parameter, negatively correlated with the depth change parameter, and negatively correlated with the transparency change parameter, respectively.

5. The method according to claim 1, characterized in that, The step of performing a color mixing operation on the first color parameter of each first pixel in the current frame image based on the displacement parameter and the third color parameter of each second pixel in the historical frame image to obtain the current frame image with a second resolution includes: The third weight percentage of the third color parameter is determined based on the displacement parameter, wherein the third weight percentage is inversely correlated with the displacement parameter; Based on the third color parameter of each second pixel adjusted by the third weight ratio and the first color parameter of each first pixel adjusted by the fourth weight ratio, the fifth color parameter of each first pixel is obtained, wherein the sum of the third weight ratio and the fourth weight ratio is 1. Based on the fifth color parameter of each first pixel, generate the current frame image with a second resolution.

6. The method according to claim 5, characterized in that, The process of obtaining the fifth color parameter for each first pixel based on the third color parameter of each second pixel adjusted by the third weighting ratio and the first color parameter of each first pixel adjusted by the fourth weighting ratio includes: Multiply the third color parameter of each second pixel by the third weight ratio to obtain the third weighted color parameter of each second pixel; Multiply the first color parameter of each first pixel by the fourth weight ratio to obtain the fourth weighted color parameter of each first pixel; The fourth weighted color parameter of the first pixel is added to the third weighted color parameter of the second pixel corresponding to the first pixel to obtain the fifth color parameter of each first pixel.

7. The method according to claim 5, characterized in that, The step of determining the third weighting percentage of the third color parameter based on the displacement parameter includes: Obtain the frame rate parameter of the frame image sequence in which the current frame image is located; The third weight ratio of the third color parameter is determined based on the frame rate parameter and the displacement parameter, wherein the third weight ratio is also positively correlated with the frame rate parameter.

8. The method according to claim 1, characterized in that, The step of sampling the current frame image at a second resolution to obtain the first color parameter of each first pixel in the current frame image after the sampling operation includes: The current frame image is sampled using a second resolution to determine the position parameters of each first pixel in the current frame image. Based on the position parameters of each first pixel, at least one sampling pixel corresponding to each first pixel is determined, wherein the sampling pixel corresponding to the first pixel includes the first pixel adjacent to the first pixel; The first color parameter of each first pixel is determined based on the initial color parameter of each first pixel and the initial color parameter of at least one sampled pixel corresponding to that first pixel.

9. The method according to claim 8, characterized in that, The determination of the first color parameter of each first pixel based on the initial color parameter of each first pixel and the initial color parameter of at least one sampled pixel corresponding to that first pixel includes: The product of the first specified coefficient and the initial color parameter of at least one sampled pixel corresponding to each first pixel is determined to obtain the first parameter value corresponding to each first pixel. The difference between the initial color parameter of the first pixel and the first parameter value corresponding to the first pixel is used as the color gradient value of the first pixel. The initial color parameters of the first pixel are sharpened based on the color gradient value of the first pixel to determine the first color parameters of each first pixel.

10. The method according to claim 9, characterized in that, The initial color parameters of the first pixel are sharpened based on the color gradient value of the first pixel to determine the first color parameters of each first pixel, including: The product of the second specified coefficient and the color gradient value of each first pixel is determined to obtain the second parameter value corresponding to each first pixel. The sum of the initial color parameter of the first pixel and the corresponding second parameter value of the first pixel is used as the first color parameter of the first pixel.

11. The method according to claim 10, characterized in that, The step of using the sum of the initial color parameter of the first pixel and the corresponding second parameter value as the first color parameter of the first pixel includes: Calculate the sum of the initial color parameter of each first pixel and the second parameter value corresponding to that first pixel to obtain the third parameter value corresponding to each first pixel; Determine the fifth weight ratio corresponding to the first pixel and the sixth weight ratio of the sampled pixel corresponding to the first pixel, wherein the sum of the fifth weight ratio and the corresponding sixth weight ratios is 1; The product of the fifth weight ratio and the third parameter value corresponding to the first pixel is obtained to get the fourth parameter value corresponding to the first pixel. The product of the sixth weight ratio and the initial color parameter of the sampled pixel corresponding to the first pixel is obtained to get the fifth parameter value corresponding to the first pixel. The sum of the fourth and fifth parameter values ​​corresponding to the first pixel is used as the first color parameter of the first pixel.

12. The method according to claim 8, characterized in that, The step of sampling the current frame image at a second resolution to determine the position parameters of each first pixel in the current frame image includes: Obtain the jitter parameters corresponding to the current frame image; Perform a de-jitter operation on the current frame image based on the jitter parameters; The current frame image after the dejittering operation is performed is sampled using a second resolution to determine the position parameters of each first pixel in the current frame image.

13. The method according to claim 1, characterized in that, The image change parameters include depth change parameters, transparency change parameters, and displacement parameters. The steps of acquiring the current frame image at a first resolution, the historical frame image at a second resolution adjacent to the current frame image, and the image change parameters include: The current frame image at a first resolution and the historical frame image at a second resolution adjacent to the current frame image are acquired respectively. The first depth parameter, the first transparency parameter, the displacement parameter, the second depth parameter, and the second transparency parameter of each second pixel in the current frame image are obtained respectively. Based on the first depth parameter of the reference pixel corresponding to the first pixel in the current frame image, the first depth parameter, first transparency parameter and displacement parameter of the first pixel are corrected to obtain the third depth parameter, third transparency parameter and corrected displacement parameter of each first pixel. The reference pixel is the first pixel within a specified range around the first pixel. The depth change parameter is determined based on the third depth parameter of each first pixel in the current frame image and the second depth parameter of each second pixel in the historical image frame, and the transparency change parameter is determined based on the third transparency parameter of each first pixel in the current frame image and the second transparency parameter of each second pixel in the historical image frame.

14. The method according to claim 13, characterized in that, The step of correcting the first depth parameter, first transparency parameter, and displacement parameter of the first pixel based on the first depth parameter of the reference pixel corresponding to the first pixel in the current frame image to obtain the third depth parameter, third transparency parameter, and corrected displacement parameter of each first pixel includes: Determine the target reference pixel corresponding to the smallest first depth parameter among the reference pixels corresponding to the first pixel; Replace the first depth parameter of the first pixel with the first depth parameter of the corresponding target reference pixel, replace the first transparency parameter of the first pixel with the first transparency parameter of the corresponding target reference pixel, and replace the displacement parameter of the first pixel with the displacement parameter of the corresponding target reference pixel.

15. A resolution enhancement device, characterized in that, include: The acquisition unit is configured to acquire a current frame image at a first resolution, a historical frame image at a second resolution adjacent to the current frame image, and image change parameters, wherein the image change parameters include at least one of a depth change parameter and a transparency change parameter between the current frame image and the historical frame image, and the image change parameters also include a displacement parameter between the current frame image and the historical frame image, wherein the first resolution is smaller than the second resolution; A sampling unit is used to perform a sampling operation on the current frame image at a second resolution to obtain the first color parameter of each first pixel in the current frame image after the sampling operation. The correction unit is used to correct the second color parameter of each second pixel in the historical frame image based on the color clamping parameter and the image change parameter to obtain the third color parameter of each second pixel, wherein the color clamping parameter is determined based on the first color parameter; A mixing unit is used to perform a color mixing operation on the first color parameter of each first pixel in the current frame image based on the displacement parameter and the third color parameter of each second pixel in the historical frame image, so as to obtain the current frame image with a second resolution.

16. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-14.

Images