Image super-resolution model training method and image reconstruction method

Abstract

Embodiments of the present application provide an image super-resolution model training method and an image reconstruction method, which comprises: obtaining a first image and a second image obtained by performing resolution reduction processing on the first image; reconstructing the second image through a super-resolution model to obtain a third image; wherein the resolution of the third image matches the resolution of the second image; performing pixel mapping processing on the third image through a sub-pixel arrangement simulator corresponding to a display screen with a target sub-pixel arrangement structure to obtain a fourth image; determining a perceptual loss function according to the fourth image and the first image; and training the super-resolution model according to the perceptual loss function. The super-resolution model trained by the model training method of the present application has better clarity and texture when reconstructing an image and displaying it on a display screen with a target sub-pixel arrangement structure, thereby improving the visual effect of the image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image super-resolution model training method and an image reconstruction method. Background Technology

[0002] In many electronic image applications, such as image display and video display, people usually expect the images or videos they see to contain more details so that viewers can obtain more information or have a better visual experience.

[0003] Currently, the common practice to achieve the aforementioned image display effects is to increase the image resolution. However, not all images display better on a screen after resolution is increased. Therefore, how to make the image highly adaptable to the screen's parameters, achieving better clarity and texture, and thus enhancing the visual effect, has become a pressing issue. Summary of the Invention

[0004] This invention provides an image super-resolution model training method and an image reconstruction method to improve the visual effect of images during display, resulting in better image clarity and texture.

[0005] In a first aspect, embodiments of the present invention provide an image super-resolution model training method, the method comprising:

[0006] Acquire a first image and a second image obtained by reducing the resolution of the first image;

[0007] The second image is reconstructed using a super-resolution model to obtain a third image; wherein the resolution of the third image matches the resolution of the second image.

[0008] The third image is pixel-mapped to the display screen using a subpixel arrangement simulator corresponding to the display screen with the target subpixel arrangement structure to obtain the fourth image;

[0009] Based on the fourth image and the first image, determine the perceptual loss function;

[0010] The super-resolution model is trained based on the perceptual loss function.

[0011] Secondly, embodiments of the present invention provide an image reconstruction method, the method comprising:

[0012] Acquire a first target image and parameter information of the target display screen corresponding to the first target image;

[0013] Determine the super-resolution model corresponding to the parameter information of the target display screen;

[0014] The first target image is reconstructed using the super-resolution model to obtain a second target image; wherein the resolution of the first target image matches the resolution of the second target image; the super-resolution model is trained using the image super-resolution model training method described in the first aspect;

[0015] The second target image is displayed on the target display screen.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor executes the image super-resolution model training method as described in the first aspect.

[0017] Fourthly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor performs the image reconstruction method as described in the second aspect.

[0018] The technical solution provided in the embodiments of the present invention has the following advantages:

[0019] 1. This invention provides a method for training an image super-resolution model. First, a first image and a second image obtained by reducing the resolution of the first image are acquired. Then, the second image is reconstructed using a super-resolution model to obtain a third image, wherein the resolution of the third image matches that of the second image. Next, the third image is mapped to the pixels of the display screen using a subpixel arrangement simulator corresponding to the display screen with the target subpixel arrangement structure to obtain a fourth image. Finally, a perceptual loss function is determined based on the fourth image and the first image. The super-resolution model is then trained based on the perceptual loss function. In the above image super-resolution model training process, a subpixel arrangement simulator corresponding to the display screen with the target subpixel arrangement structure is introduced. The perceptual loss function is determined by comparing the fourth image obtained from the subpixel arrangement simulator with the first image, and then the image super-resolution model is trained. Therefore, the trained super-resolution model has a higher degree of matching and specificity with the display screen with the target subpixel arrangement structure.

[0020] 2. This invention provides an image reconstruction method. First, a first target image and parameter information of a target display screen corresponding to the first target image are obtained; a super-resolution model corresponding to the parameter information of the target display screen is determined; the first target image is reconstructed using the super-resolution model to obtain a second target image; wherein the resolution of the first target image matches the resolution of the second target image; the super-resolution model is trained using the image super-resolution model training method described in the first aspect; and the second target image is displayed on the target display screen. Based on the solution provided by this invention, after the image is processed by the above image reconstruction method, when displayed on the target display screen, it can highly adapt to the display screen's own parameters, resulting in better clarity and texture, and improving the visual effect of the image. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an image super-resolution model training method provided in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the sub-pixel arrangement structure of the display screen;

[0024] Figure 3 A schematic diagram illustrating an image super-resolution model training method provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of pixel mapping processing when the target sub-pixel arrangement structure is an RGB structure in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of pixel mapping processing when the target sub-pixel arrangement structure is an RGBW structure in an embodiment of the present invention;

[0027] Figure 6 A flowchart of an image reconstruction method provided in an embodiment of the present invention;

[0028] Figure 7 A schematic diagram illustrating an image reconstruction method provided in an embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram of the structure of an image super-resolution model training device provided in an embodiment of the present invention;

[0030] Figure 9 To and Figure 8 A schematic diagram of the electronic device structure corresponding to the image super-resolution model training device provided in the embodiment shown;

[0031] Figure 10 This is a schematic diagram of the structure of an image reconstruction device provided in an embodiment of the present invention;

[0032] Figure 11 To and Figure 10 The illustrated embodiment provides a schematic diagram of the electronic device structure corresponding to the image reconstruction apparatus. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0035] The image super-resolution model training method provided in this embodiment of the invention can be executed by an electronic device, such as a PC, laptop, or smartphone, or a server. The server can be a physical server containing an independent host, a virtual server, a cloud server, or a server cluster.

[0036] Figure 1 A flowchart of an image super-resolution model training method provided in an embodiment of the present invention is shown below. Figure 1 As shown, it may include the following steps:

[0037] 101. Obtain the first image and the second image obtained after reducing the resolution of the first image.

[0038] 102. The second image is reconstructed using a super-resolution model to obtain a third image; wherein the resolution of the third image matches the resolution of the second image.

[0039] 103. By using a subpixel arrangement simulator corresponding to a display screen with a target subpixel arrangement structure, the third image is pixel-mapped to the display screen to obtain the fourth image.

[0040] 104. Determine the perceptual loss function based on the fourth image and the first image.

[0041] 105. Train the super-resolution model based on the perceptual loss function.

[0042] In this embodiment, the purpose of the above-mentioned image super-resolution model training method is to obtain an image super-resolution model that matches a display screen with a certain sub-pixel arrangement structure.

[0043] Currently, there are various types of displays on the market, such as liquid crystal displays (LCDs), LED displays, OLED displays, microLED displays, and raster display devices such as projectors. In these displays, each type of display has a corresponding sub-pixel arrangement structure.

[0044] It is understandable that when an image is displayed on a screen, the smallest unit of light emission on the screen is a pixel. However, when each pixel is magnified, it can be found that a pixel is composed of multiple colors, such as red, green and blue. Each color is called a sub-pixel, representing a color channel. There are many combinations of the arrangement and position of the various colors that make up a pixel. Correspondingly, the multiple sub-pixels that make up a pixel can have many different arrangements, that is, there are multiple sub-pixel arrangement structures. Figure 2 Examples of common subpixel arrangements for displays are given, such as Red Green Blue (RGB), Red Green Blue White (RGBW), and Red Green Blue Green (RGBG). When an image is displayed on a screen, the values ​​of the color channels represented by each subpixel can be adjusted to make the displayed image closer to the actual colors of the image.

[0045] For ease of understanding Figure 1 The image super-resolution model training method shown is as follows: Figure 3 A schematic diagram of the image super-resolution model training method according to an embodiment of the present invention is provided. Combined with... Figure 3 right Figure 1 The image super-resolution model training method shown in the figure is explained.

[0046] In the image super-resolution model training method provided in this embodiment of the invention, a first image and a second image obtained by reducing the resolution of the first image are first acquired. In the actual training process, the first image can be a natural image of various formats, such as GIF, JPG, JPEG, PNG, TIF, etc., or it can be an image sequence of video of various formats, i.e., video frames. In the embodiments of the present invention, it is not limited to these.

[0047] After acquiring the first image, its resolution can be reduced through slight blurring, scaling, or noise addition. The resulting processed image is the second image. The resolution of the second image can be within a preset resolution range, such as [720p, 8k]. Here, 720p represents a resolution of 1280*720, and 8K represents a resolution of 7680*4320. It is understandable that in practical applications, this preset resolution range can include displays with a certain subpixel arrangement structure, such as common resolutions like 1080p (1920*1080), 2k (2048*1080), and 4k (3840*2160).

[0048] like Figure 3 As shown, the second image serves as the input to the super-resolution model to be trained. After reconstruction of the second image by the super-resolution model, the third image is output, where the resolution of the third image matches the resolution of the second image. In specific implementation, matching the resolution of the third image with the resolution of the second image includes: the resolution of the third image is a preset multiple of the resolution of the second image, which is greater than or equal to 1. Setting different values ​​for the preset multiple allows for the training of different super-resolution models (SR). For example, when training super-resolution model SR1, the preset multiple is 1, and the resolution of the third image is 1 times the resolution of the second image, meaning the third image has the same resolution as the second image. When training super-resolution model SRn, the preset multiple is n, and the resolution of the third image is n times the resolution of the second image, where n is greater than 1. In practical applications, n can be 2, 3, 4, etc. When n is 2, the trained super-resolution model is SR2; when n is 3, the trained super-resolution model is SR3, and so on. Training different super-resolution models is to select the appropriate model for the display screen under different conditions, so that the image can better adapt to the screen's parameters and achieve a better display effect. In practical applications, the super-resolution model mentioned above can be a neural network model, such as a convolutional neural network model, but it is not limited to this.

[0049] After obtaining the third image, the third image is pixel-mapped to the display screen using a subpixel arrangement simulator corresponding to the display screen with the target subpixel arrangement structure to obtain the fourth image.

[0050] The aforementioned subpixel arrangement modeler can simulate the arrangement of pixels in a third image based on the target subpixel arrangement structure of the display screen, that is, it performs pixel mapping processing from the third image to the display screen. Different subpixel arrangement simulators can be used for different target subpixel arrangement structures. The subpixel arrangement simulator can simulate the arrangement of pixels in the third image through convolutional and pixel shuffle operations. When performing feature extraction on the third image using convolution, no bias is applied, and other parameters are set to constants. Different constant values ​​are set according to different target subpixel arrangement structures. For example, when the target subpixel arrangement structure corresponding to the subpixel arrangement simulator is... Figure 2 As shown in the RGB example, when performing feature extraction through convolution, the input channel (in channel) is set to 3, the weight matrix (kernel) is set to 1, the stride is set to 1, and the padding is set to 0. If a third image with a resolution of 224*224 is input into this subpixel arrangement simulator, after feature extraction via convolution, a feature image with 27 channels and a size of 224*224 is obtained. This feature image is then reassembled into a fourth image with 3 channels and a size of 672*672, which is a fourth image with a resolution of 672*672 and an RGB subpixel arrangement structure.

[0051] If we consider the multiple pixels that make up an image as a pixel matrix, when performing pixel mapping processing from the third image to the display screen using the aforementioned sub-pixel arrangement simulator, specifically, we can determine the pixel matrix corresponding to the target pixel in the third image based on the target sub-pixel arrangement structure. This pixel matrix consists of multiple pixels, and the target pixel is any pixel in the third image. Based on the values ​​of each sub-pixel in the target pixel, we determine the pixel value of each pixel in the pixel matrix.

[0052] To make it easier to understand, for example, when the target sub-pixel arrangement structure is an RGB structure, such as Figure 4 As shown, based on the RGB values ​​of the target pixel, the first pixel matrix corresponding to any pixel (i.e., the target pixel) in the third image is determined. Using the aforementioned sub-pixel arrangement simulator, after convolution and pixel recombination of the target pixel in the third image, the first pixel matrix and the values ​​of each corresponding pixel in the first pixel matrix are obtained, as shown below. Figure 4As shown, in practical applications, considering the consistency between the shape of the target pixel and the shape of the pixel matrix, the first pixel matrix consists of 3*3 pixels. Since the pixel value of each pixel can be represented by an array of values ​​from its sub-pixels, the target pixel can be represented by an array, for example: (255, 0, 255). This array represents that sub-pixel R has a value of 255, sub-pixel G has a value of 0, and sub-pixel B has a value of 255. The corresponding values ​​of each pixel in the first pixel matrix are determined by referring to... Figure 4 The values ​​of the three pixels in the first column are all (255, 0, 0), the values ​​of the three pixels in the second column are all (0, 0, 0), and the values ​​of the three pixels in the third column are all (0, 0, 255). The pixels in the same column have the same value.

[0053] Figure 5 This example illustrates another target subpixel arrangement structure, which is an RGBW structure, such as... Figure 5 As shown, based on the RGBW values ​​of the target pixel, the second pixel matrix corresponding to the target pixel in the third image is determined. Using the aforementioned subpixel arrangement simulator, after convolution and pixel recombination of the target pixel in the third image, the second pixel matrix and the values ​​of each corresponding pixel in the second pixel matrix are obtained, as shown below. Figure 5 As shown, considering the consistency between the shape of the target pixel and the shape of the pixel matrix, the second pixel matrix consists of 2*2 pixels. An array is used to represent the RGB values ​​of the sub-pixels in the target pixel, for example: (255, 0, 255). This array indicates that the value of sub-pixel R is 255, the value of sub-pixel G is 0, and the value of sub-pixel B is 255. In a target pixel with an RGBW sub-pixel arrangement structure, the value of sub-pixel W is calculated based on the values ​​of sub-pixels R, G, and B. Different display manufacturers design different calculation methods for sub-pixel W according to different display requirements. Figure 5 In this example, assume that the value of sub-pixel W, calculated based on the values ​​of sub-pixels R, G, and B, is n. The values ​​of each pixel in the corresponding second pixel matrix are referenced... Figure 5 The values ​​of the pixels in the first row and first column are (255, 0, 0), the values ​​of the pixels in the first row and second column are (0, 0, 0), the values ​​of the pixels in the second row and first column are (0, 0, 255), and the values ​​of the pixels in the second row and second column are (a, b, c). Here, a, b, and c are calculated according to the values ​​of the sub-pixels R, G, and B in the other three pixels R (RGB), G (RGB), and B (RGB) in the second pixel matrix, based on the settings of the display manufacturer. This is the same as the calculation method for the value n of the sub-pixel W in the target pixel mentioned above.

[0054] Figure 4 and Figure 5This example illustrates the process of mapping a target pixel to a pixel on the display screen using a subpixel arrangement simulator. In fact, after the third image passes through the subpixel arrangement simulator, all pixels in the third image are mapped using the above method. The fourth image is obtained by combining the pixel matrices corresponding to all the mapped target pixels.

[0055] Figure 4 and Figure 5 The scenarios illustrated are merely illustrative. In practical applications, when performing pixel mapping processing from the third image to the display screen using the aforementioned subpixel arrangement simulator, the pixel matrix corresponding to the target pixels of the third image can also be determined based on other target subpixel arrangement structures, without being limited to the two scenarios described above.

[0056] After obtaining the fourth image, a perceptual loss function is determined based on the fourth and first images. The image super-resolution model is then trained using this perceptual loss function. In practice, the perceptual loss function is used to calculate the perceptual loss between the fourth and first images, and the image super-resolution model is trained based on the value of this perceptual loss. By using a subpixel arrangement simulator corresponding to a display screen with the target subpixel arrangement structure during the training process, the final trained image super-resolution model can be matched to a display screen with the target subpixel arrangement structure.

[0057] To facilitate understanding, the training process of the super-resolution model described above is illustrated with an example. Assume the target subpixel arrangement structure corresponding to the current display screen is RGB, and the resolution of the first image is 8k (7680*4320). When the resolution of the first image is reduced, the preset resolution range is [720p, 4k]. Therefore, the resolution of the second image obtained through resolution reduction can be 720p, 1080p, 4k, etc. When training a super-resolution model RGB-SR1 corresponding to a display screen with an RGB subpixel arrangement structure, the preset multiple between the resolution of the third image and the resolution of the second image is 1. For any second image with a resolution of S within the range [720p, 4k] obtained after resolution reduction, a third image with a resolution of S is obtained after reconstruction by the image super-resolution model. After the pixel mapping process from the third image to the display screen is performed by a subpixel arrangement simulator corresponding to the display screen with an RGB subpixel arrangement structure, a fourth image with a resolution of 3S is obtained. The perceptual loss between the fourth image and the first image is calculated. Based on the value of the perceptual loss, the super-resolution model RGB-SR1 is trained. When training a super-resolution model RGB-SR2 corresponding to a display screen with an RGB subpixel arrangement structure, the preset multiple between the resolution of the third image and the resolution of the second image is 2. For any second image with a resolution of S in the interval [720p, 4k], after reconstruction by the image super-resolution model, a third image with a resolution of 2S is obtained. After the pixel mapping process from the third image to the display screen is performed by a subpixel arrangement simulator corresponding to the display screen with an RGB subpixel arrangement structure, a fourth image with a resolution of 6S is obtained. The perceptual loss between the fourth image and the first image is calculated, and the super-resolution model RGB-SR2 is trained based on the value of the perceptual loss. Similarly, RGB-SR3, RGB-SR4, etc., can be trained.

[0058] In another scenario, assuming the target subpixel arrangement structure corresponding to the current display screen is RGBW, and the resolution of the first image is 8k (7680*4320), when the resolution of the first image is reduced, the preset resolution range is [720p, 4k]. Therefore, the resolution of the second image obtained through resolution reduction can be 720p, 1080p, 4k, etc. When training a super-resolution model RGBW-SR1 corresponding to a display screen with an RGBW subpixel arrangement structure, the preset multiple between the resolution of the third image and the resolution of the second image is 1. For any second image with a resolution of S within the range [720p, 4k] obtained after resolution reduction, a third image with a resolution of S is obtained after reconstruction by the image super-resolution model. After the pixel mapping process from the third image to the display screen is performed by a subpixel arrangement simulator corresponding to the display screen with an RGBW subpixel arrangement structure, a fourth image with a resolution of 2S is obtained. The perceptual loss between the fourth image and the first image is calculated, and the super-resolution model RGBW-SR1 is trained based on the value of the perceptual loss. When training a super-resolution model RGBW-SR2 corresponding to a display screen with an RGBW subpixel arrangement structure, the preset multiple between the resolution of the third image and the resolution of the second image is 2. For any second image with a resolution of S in the interval [720p, 4k], after reconstruction by the image super-resolution model, a third image with a resolution of 2S is obtained. After the pixel mapping process from the third image to the display screen is performed by a subpixel arrangement simulator corresponding to the display screen with an RGB subpixel arrangement structure, a fourth image with a resolution of 4S is obtained. The perceptual loss between the fourth image and the first image is calculated, and the super-resolution model RGBW-SR2 is trained based on the value of the perceptual loss. Similarly, RGBW-SR3, RGBW-SR4, etc., can be trained.

[0059] The target sub-pixel arrangement structure and the preset multiple values ​​mentioned above are only for illustrative purposes and are not limited to these in the embodiments of the present invention. In practical applications, an image super-resolution model SRn that matches multiple target sub-pixels can be trained, where n is greater than or equal to 1.

[0060] The above embodiments provide a method for training an image super-resolution model. First, a first image and a second image obtained by reducing the resolution of the first image are acquired. Then, the second image is reconstructed using a super-resolution model to obtain a third image. The resolution of the third image matches the resolution of the second image, and the resolution of the third image is a preset multiple of the resolution of the second image, which is greater than or equal to 1. Next, the third image is pixel-mapped to the display screen using a sub-pixel arrangement simulator corresponding to a display screen with a target sub-pixel arrangement structure to obtain a fourth image. Finally, a perceptual loss function is determined based on the fourth image and the first image. The super-resolution model is then trained based on the perceptual loss function. In the above image super-resolution model training process, a subpixel arrangement simulator corresponding to a display screen with a target subpixel arrangement structure is introduced. The fourth image processed by the subpixel arrangement simulator is compared with the first image to determine the perceptual loss function, and then the image super-resolution model is trained. As a result, the trained super-resolution model has a higher matching degree and specificity with the display screen with the target subpixel arrangement structure. By adjusting the value of the preset multiplier, different super-resolution models corresponding to the display screen with the target subpixel arrangement structure can be trained, increasing the selectivity of the super-resolution model when performing image reconstruction.

[0061] After the image super-resolution rate model is trained, it can be applied to the image display process of a display screen with a target sub-pixel arrangement structure to reconstruct the image that needs to be displayed on the display screen and display the reconstruction result on the display screen.

[0062] Figure 6 A flowchart of an image reconstruction method provided in this embodiment of the invention, wherein the super-resolution model used in image reconstruction is trained using the above-described image super-resolution model training method, such as... Figure 6 As shown, it may include the following steps:

[0063] 601. Obtain the first target image and the parameter information of the target display screen corresponding to the first target image.

[0064] 602. Determine the super-resolution model corresponding to the parameter information of the target display screen.

[0065] 603. The first target image is reconstructed using a super-resolution model to obtain the second target image; wherein the resolution of the first target image matches the resolution of the second target image.

[0066] 604. Display the second target image on the target display screen.

[0067] The parameter information of the target display screen includes the sub-pixel arrangement structure and resolution of the target display screen.

[0068] In this embodiment, the first target image is a natural image or video or other image resource that needs to be displayed on the target display screen. When selecting a model, the corresponding image super-resolution model is selected based on the sub-pixel arrangement structure of the target display screen and the ratio of the resolution of the target display screen to the resolution of the first target image.

[0069] For ease of understanding Figure 6 The image reconstruction method shown, Figure 7 A schematic diagram of an image reconstruction method according to an embodiment of the present invention is provided. Combined with... Figure 7 right Figure 6 The image reconstruction method shown is explained below.

[0070] When performing image reconstruction, a trained image super-resolution model can be implemented as software installed on an electronic device with computing power, such as a PC, laptop, or smartphone, or it can be a server. This server can be a physical server containing a single host, a virtual server, a cloud server, or a server cluster.

[0071] Assuming the software is installed on a server, the process of reconstructing the first target image using the trained super-resolution model and displaying the second target image on the target display screen is as follows:

[0072] First, the server acquires a first target image, along with the sub-pixel arrangement structure and resolution of the target display screen corresponding to the first target image, and calculates the ratio of the resolution of the target display screen to the resolution of the first target image. Then, based on the sub-pixel arrangement structure of the target display screen and the ratio of the resolution of the target display screen to the resolution of the first target image, the server selects a super-resolution model corresponding to the aforementioned resolution ratio and sub-pixel arrangement structure from a pre-stored super-resolution model in the server. After selecting the super-resolution model, the server reconstructs the acquired first target image using the super-resolution model to obtain a second target image. Finally, the server transmits the obtained second target image to the target display screen so that the target display screen can display the second target image.

[0073] To facilitate understanding, let's take an example. Suppose the server acquires a first target image with a resolution of 1920*1080, and the target display screen has an RGB subpixel arrangement structure with a resolution of 3840*2160. Since the target display screen has an RGB subpixel arrangement structure, the ratio of the target display screen's resolution to the first target image's resolution is 2. Therefore, the server can select the super-resolution model RGB-SR2 from the pre-stored super-resolution models. After selecting RGB-SR2, the server inputs the acquired first target image into RGB-SR2 for image reconstruction. The reconstructed second target image has a resolution of 3840*2160, which is twice the resolution of the first target image. The second target image highly matches the target display screen's subpixel arrangement structure and resolution.

[0074] However, it's understandable that during actual image reconstruction, the ratio of the target display's resolution to the first target image's resolution can vary. The server might only pre-store a few high-probability resolution ratios corresponding to super-resolution models, such as RGB-SR1, RGB-SR2, and RGB-SR3. In this case, to ensure the reconstructed image highly matches the target display's subpixel arrangement and resolution, one approach is to select the RGB-SR1 model. To improve the display effect of the first target image after RGB-SR1 reconstruction, it can be smoothly scaled down or up to obtain a third target image matching the target display's resolution. This third target image is then reconstructed using RGB-SR1 to obtain the second target image, which is then transmitted to the target display for display.

[0075] For example, suppose the resolution of the first target image is 2560*1440, and the subpixel arrangement of the target display screen is RGB with a resolution of 3840*2160. The calculated ratio of the resolution of the target display screen to the resolution of the first target image is 1.5. Since the server does not pre-store the super-resolution model RGB-SR1.5, in this case, the server can choose to reconstruct the image using RGB-SR1: The server first performs smooth enlargement processing on the first target image to obtain a third target image that matches the resolution of the target display screen, where the resolution of the third target image is 3840*2160. The third target image is then reconstructed using RGB-SR1 to obtain a second target image with a resolution of 3840*2160. The resolution of the second target image is the same as that of the third target image, and the subpixel arrangement and resolution of the second target image are highly matched with those of the target display screen.

[0076] In the above embodiments, since a server may connect to multiple target displays, it is necessary to obtain the subpixel arrangement structure of the target displays during image reconstruction. Based on this subpixel arrangement structure and the ratio of the target display's resolution to the resolution of the first target image, the corresponding super-resolution model is determined. The above example only illustrates the image reconstruction process performed by the server when displaying an image on a target display with an RBG subpixel arrangement structure. In reality, there are various subpixel arrangement structures for target displays, and the server can store super-resolution models corresponding to displays with multiple subpixel arrangement structures, not limited to the situation in the above example.

[0077] Another way to implement image reconstruction using a trained image super-resolution model is by integrating it as hardware into the screen firmware. Understandably, the super-resolution model integrated into the screen firmware matches the sub-pixel arrangement of the display screen. For example, if the display screen's sub-pixel arrangement is RGB, then the super-resolution model integrated into the screen firmware is RGB-SR, where RGB-SR includes RGB-SR1, RGB-SR2, RGB-SR3, and other frequently used super-resolution models. For a display screen with the aforementioned firmware, after acquiring the first target image, the ratio of the target display screen's resolution to the resolution of the first target image is calculated. Then, the corresponding super-resolution model is selected based on the resolution ratio. Next, the first target image is reconstructed using the selected super-resolution model to obtain the second target image. Finally, the second target image is displayed on the display screen. The process of selecting the super-resolution model and performing image reconstruction is similar to the server processing described above and will not be elaborated further here.

[0078] The above embodiments provide an image reconstruction method. First, a first target image, its resolution, and parameter information of the target display screen corresponding to the first target image are acquired. A super-resolution model corresponding to the parameter information of the target display screen is determined. The first target image is reconstructed using the super-resolution model to obtain a second target image. The resolution of the first target image matches the resolution of the second target image. The super-resolution model is trained using the image super-resolution model training method described in the first aspect. The second target image is then displayed on the target display screen. Based on the solution provided by the embodiments of the present invention, after the image is processed by the above image reconstruction method, when displayed on the target display screen, it can highly adapt to the display screen's own parameters, match the sub-pixel arrangement structure and resolution corresponding to the target display screen, and has better clarity and texture, thus improving the visual effect of the image.

[0079] The image super-resolution model training apparatus and image reconstruction apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these apparatuses can be configured using commercially available hardware components through the steps taught in this solution.

[0080] Figure 8 This is a schematic diagram of the structure of an image super-resolution model training device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the device includes: an acquisition module 11 and a processing module 12.

[0081] The acquisition module 11 is used to acquire a first image and a second image obtained by reducing the resolution of the first image.

[0082] Processing module 12 is used to reconstruct the second image using a super-resolution model to obtain a third image; wherein the resolution of the third image matches the resolution of the second image; the third image is pixel-mapped to the display screen using a sub-pixel arrangement simulator corresponding to the display screen with the target sub-pixel arrangement structure to obtain a fourth image; a perceptual loss function is determined based on the fourth image and the first image; and the super-resolution model is trained based on the perceptual loss function.

[0083] Optionally, the processing module 12 is configured to: determine a pixel matrix corresponding to a target pixel in the third image based on the target sub-pixel arrangement structure, wherein the pixel matrix is ​​composed of multiple pixels and the target pixel is any pixel in the third image; and determine the pixel value of each pixel in the pixel matrix based on the value of each sub-pixel in the target pixel.

[0084] Optionally, the target sub-pixel arrangement structure is an RGB structure, and the processing module 12 is specifically used to: determine a first pixel matrix corresponding to the target pixel in the third image according to the target sub-pixel arrangement structure, wherein the first pixel matrix consists of 3*3 pixels; and determine the pixel value of each column of pixels in the first pixel matrix according to the values ​​of the three sub-pixels R, G, and B in the target pixel, wherein the pixel value of pixels in the same column is the same.

[0085] Optionally, the target sub-pixel arrangement structure is an RGBW structure, and the processing module 12 is specifically used to: determine a second pixel matrix corresponding to the target pixel in the third image according to the target sub-pixel arrangement structure, the second pixel matrix being composed of 2*2 pixels; and determine the pixel value of each pixel in the second pixel matrix according to the values ​​of the four sub-pixels R, G, B, and W in the target pixel.

[0086] Figure 8The device shown can perform the image super-resolution model training method in the foregoing embodiments. For details of the execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0087] In one possible design, the above Figure 8 The structure of the image super-resolution model training device shown can be implemented as an electronic device, such as... Figure 9 As shown, the computing device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores executable code, which, when executed by the processor 22, enables the processor 22 to at least implement the image super-resolution model training method provided in the foregoing embodiments.

[0088] Figure 10 This is a schematic diagram of the structure of an image reconstruction device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the device includes: an acquisition module 31 and a processing module 32.

[0089] The acquisition module 31 is used to acquire a first target image and parameter information of the target display screen corresponding to the first target image.

[0090] Processing module 32 is used to determine a super-resolution model corresponding to the parameter information of the target display screen; reconstruct the first target image using the super-resolution model to obtain a second target image; wherein the resolution of the first target image matches the resolution of the second target image; the super-resolution model is trained using the above-described image super-resolution model training method; and display the second target image on the target display screen.

[0091] Optionally, the parameter information includes the sub-pixel arrangement structure and resolution of the target display screen.

[0092] Specifically, the processing module 32 is used to determine the ratio of the resolution of the target display screen to the resolution of the first target image; determine a super-resolution model corresponding to the ratio and the sub-pixel arrangement structure; reconstruct the first target image using the super-resolution model to obtain a second target image; wherein the resolution of the first target image matches the resolution of the second target image; the super-resolution model is trained using the above-described image super-resolution model training method; and display the second target image on the target display screen.

[0093] Figure 10 The device shown can perform the image reconstruction method described in the foregoing embodiments. For details of the execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0094] In one possible design, the above Figure 10 The structure of the image recognition device shown can be implemented as an electronic device, such as... Figure 11 As shown, the electronic device may include: a memory 41, a processor 42, and a communication interface 43. The memory 41 stores executable code, which, when executed by the processor 42, enables the processor 42 to at least implement the image reconstruction method provided in the foregoing embodiments.

[0095] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code. When the executable code is executed by a processor of an electronic device, the processor can at least implement the image super-resolution model training method or the image reconstruction method provided in the foregoing embodiments.

[0096] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training an image super-resolution model, characterized in that, The method comprises the following steps: obtaining a first image and a second image obtained after performing a resolution reduction process on the first image; reconstructing the second image through a super-resolution model to obtain a third image, wherein the resolution of the third image matches the resolution of the second image; performing pixel mapping processing on the third image to the display screen through a sub-pixel arrangement simulator corresponding to the display screen with a target sub-pixel arrangement structure to obtain a fourth image; determining a perceptual loss function according to the fourth image and the first image; training the super-resolution model according to the perceptual loss function; wherein the pixel mapping processing on the third image to the display screen to obtain a fourth image comprises: performing feature extraction on the third image through convolution by using the sub-pixel arrangement simulator to obtain a feature image, and performing pixel recombination on the feature image to obtain the fourth image of the target sub-pixel arrangement structure; wherein no additional fixed offset is introduced during feature extraction, and the convolution parameter value is set according to the target sub-pixel arrangement structure.

2. The method of claim 1, wherein, The resolution of the third image matches the resolution of the second image, which comprises: The resolution of the third image is a preset multiple of the resolution of the second image, and the preset multiple is greater than or equal to 1.

3. The method of claim 2, wherein, Different values of the preset multiple are used to train different super-resolution models.

4. The method of claim 1, wherein, The pixel mapping processing on the third image to the display screen comprises: determining a pixel matrix corresponding to a target pixel in the third image according to the target sub-pixel arrangement structure, the pixel matrix being composed of multiple pixels, and the target pixel being any pixel in the third image; determining the pixel value of each pixel in the pixel matrix according to the value of each sub-pixel in the target pixel.

5. The method of claim 4, wherein, The target sub-pixel arrangement structure is an RGB structure, and the pixel mapping processing on the third image to the display screen comprises: determining a first pixel matrix corresponding to a target pixel in the third image according to the target sub-pixel arrangement structure, the first pixel matrix being composed of 3*3 pixels; determining the pixel value of each column of pixels in the first pixel matrix according to the value of R, G and B sub-pixels in the target pixel respectively, wherein the pixel values of the same column of pixels are the same.

6. The method of claim 4, wherein, The target sub-pixel arrangement structure is an RGBW structure, and the pixel mapping processing on the third image to the display screen comprises: determining a second pixel matrix corresponding to a target pixel in the third image according to the target sub-pixel arrangement structure, the second pixel matrix being composed of 2*2 pixels; determining the pixel value of each pixel in the second pixel matrix according to the value of R, G, B and W sub-pixels in the target pixel respectively.

7. An image reconstruction method characterized by, The method comprises the following steps: obtaining a first target image and parameter information of a target display screen corresponding to the first target image; determining a super-resolution model corresponding to the parameter information of the target display screen; reconstructing the first target image by the super-resolution model to obtain a second target image; wherein a resolution of the first target image matches a resolution of the second target image; the super-resolution model is trained by the image super-resolution model training method in any one of claims 1-6; displaying the second target image in the target display screen.

8. The method of claim 7, wherein, The parameter information comprises a sub-pixel arrangement structure and a resolution of the target display screen.

9. The method of claim 8, wherein, The super-resolution model corresponding to the parameter information of the target display screen is determined, comprising: determining a ratio of the resolution of the target display screen to the resolution of the first target image; determining a super-resolution model corresponding to the ratio and the sub-pixel arrangement structure.

10. An electronic device, comprising: comprising: a memory, a processor, a communication interface; wherein the memory has stored executable codes, when the executable codes are executed by the processor, the processor executes the image super-resolution model training method in any one of claims 1-6.

11. An electronic device, comprising: comprising: a memory, a processor, a communication interface and a display screen; wherein the memory has stored executable codes, when the executable codes are executed by the processor, the processor executes the image reconstruction method in any one of claims 7-9.

Images