Image super-resolution method and electronic device

Abstract

The present application discloses an image super-resolution method for converting a low-resolution image into a high-resolution image display, which is applicable to various electronic devices with display functions. The method of the present application embodiment not only obtains pixel position offset information, but also extracts texture features of the low-resolution image, obtains a weight matrix that fuses the pixel position offset information and texture features, and obtains a high-resolution image based on the weight matrix. Because the target weight matrix in this solution fuses pixel position offset information and texture features, it can avoid the image quality loss caused by relying solely on pixel position offset information for interpolation processing, thereby improving the image quality of high-resolution images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image super-resolution method and electronic device. Background Art

[0002] With the advancement of electronic display technology, screen resolution continues to increase. Even a few-inch mobile phone screens generally reach a resolution of 1080*1920 (pixel*pixel), while even larger television and computer screens have reached resolutions as high as 4K or even 8K. This increase in screen resolution allows for greater display space for images and videos. However, existing video, image, and other image resources often cannot perfectly adapt to the screen resolution. Network bandwidth limitations during video playback and video calls can also cause devices to receive low-resolution original images.

[0003] Image super-resolution (SR) involves restoring a high-resolution (HR) image or image sequence from a low-resolution (LR) image or image sequence. Learning-based SR methods have been a hot topic in recent years. Their basic idea is to compute the mapping between the LR and HR images in a given training set of images. This mapping is often implemented using deep convolutional models, which can capture high-level, abstract information about images.

[0004] Most existing upsampling methods capable of performing non-integer super-resolution still rely on traditional interpolation algorithms, which involve three steps: extracting a neighborhood, calculating weights based on pixel position offsets, and performing weighted averaging. This interpolation operation inevitably leads to image quality loss during rendering, such as aliasing and blurring. Summary of the Invention

[0005] An embodiment of the present application provides an image super-resolution method for converting a low-resolution image into a high-resolution image for display. Since the weight matrix in this method integrates pixel position offset information and texture features, it can avoid the image quality loss caused by relying solely on pixel position offset information for interpolation processing, and can improve the image quality of high-resolution images.

[0006] A first aspect of an embodiment of the present application provides an image super-resolution method, including: a terminal obtains pixel position offset information based on an image of a first resolution and a target resolution, wherein the target resolution is greater than the first resolution; the terminal performs feature extraction on the image of the first resolution to obtain a feature map containing texture features; the terminal performs information fusion on the pixel position offset information and the feature map to obtain a target weight matrix; and the terminal obtains an image of the target resolution based on the target weight matrix.

[0007] According to the method of the embodiment of the present application, the terminal can obtain an image of the original resolution of the application, which is referred to as the first resolution in this embodiment. The target resolution when the image is displayed can also be obtained. Pixel position offset information can be obtained based on the image of the first resolution and the target resolution. In addition, the terminal also extracts texture features of the image of the first resolution, and obtains a target weight matrix by fusing the pixel position offset information and the texture features. The target weight matrix is used to provide sampling weights for each pixel in the upsampling operation of the super-resolution method. Since the target weight matrix in this scheme fuses the pixel position offset information and the texture features, obtaining a high-resolution image based on the target weight matrix can avoid image quality loss similar to that caused by traditional interpolation operations when relying solely on pixel position offset information to obtain the weight matrix. Therefore, the image quality of the high-resolution image can be improved.

[0008] In a possible implementation of the first aspect, the pixel position offset information includes an offset matrix and a magnification ratio, the offset matrix is used to indicate the sampling offset of each pixel point in the image of the first resolution, and the magnification ratio is the ratio between the target resolution and the first resolution.

[0009] In the method of the embodiment of the present application, the pixel position offset information specifically includes an offset matrix and a magnification ratio, wherein the magnification ratio can be calculated based on the first resolution and the target resolution, and the magnification ratio includes the ratio of the number of pixels in the length direction and the width direction of the image of the target resolution to the image of the first resolution. The offset matrix is composed of the sampling offset of each pixel point in the image of the target resolution. Since the target resolution is the resolution actually required by the image, its value is arbitrary. Therefore, the magnification ratio can be any multiple, thereby obtaining the pixel position offset information, and further fusing the texture features to obtain the target weight matrix, which can be used to achieve super-resolution of any multiple.

[0010] In a possible implementation of the first aspect, the terminal fuses the pixel position offset information and the feature map to obtain a target weight matrix, including: the terminal inputs the pixel position offset information and the feature map into a first neural network model to obtain the target weight matrix.

[0011] The method of the embodiment of the present application can fuse the information of the offset matrix, the magnification ratio and the feature map through a pre-trained first neural network model to obtain a target weight matrix. Unlike the existing super-resolution method in which the weight matrix only carries pixel position offset information, when obtaining a high-resolution image according to the target weight matrix provided by this method, it can avoid image quality loss caused by traditional interpolation operations, thereby improving the image quality of the high-resolution image.

[0012] In a possible implementation of the first aspect, the first neural network model includes a network model composed of a convolutional neural network and a fully connected neural network.

[0013] In the method of the embodiment of the present application, the first neural network model can be composed of a combination of a convolutional neural network and a fully connected neural network, wherein the convolutional neural network is used to convert a feature map containing texture features, and the fully connected neural network is used to process pixel position offset information.

[0014] In a possible implementation of the first aspect, the terminal fuses the pixel position offset information and the feature map to obtain a target weight matrix, including: the terminal obtains an initial weight matrix based on the offset matrix and the magnification ratio; the terminal inputs the initial weight matrix and the feature map into a second neural network model to obtain the target weight matrix.

[0015] The method of the embodiment of the present application provides another information fusion method, wherein the initial weight matrix obtained according to the offset matrix and the magnification ratio is the weight matrix used for upsampling in the prior art. Based on the prior art, this method can input the obtained initial weight matrix and the feature map into the pre-trained second neural network model for information fusion to obtain the target weight matrix used for image upsampling in this scheme, thereby improving the flexibility of the scheme implementation.

[0016] In a possible implementation of the first aspect, the terminal performs feature extraction on the image of the first resolution to obtain a feature map containing texture features, including: the terminal inputs the image of the first resolution into a third neural network model to output the feature map.

[0017] In the method of the embodiment of the present application, the feature map can be obtained based on a neural network model, such as an existing convolutional neural network.

[0018] In a possible implementation of the first aspect, the terminal obtains the image of the target resolution according to the target weight matrix, including: the terminal upsampling the feature map according to the target weight matrix to obtain the image of the target resolution.

[0019] In the method of the embodiment of the present application, the terminal upsamples the feature map according to the target weight matrix to obtain an image of the target resolution, and the quality of the image obtained by upsampling the image of the first resolution is better than that of the image obtained by upsampling.

[0020] A second aspect of an embodiment of the present application provides a model training method, including: obtaining pixel position offset information based on a training image of a first resolution and a target resolution, and obtaining an initial weight map based on the pixel position offset information; performing feature extraction on the image of the first resolution to obtain a feature map containing texture features; inputting the texture features and the initial weight map into a first neural network for training to obtain a first loss value; and updating the weight parameters in the first network according to the first loss value to obtain a target neural network.

[0021] The model training method provided in the embodiment of the present application can be used for image super-resolution. The trained model is based on the existing initial weight map and the pixel position offset information is integrated. The target network trained by this method can be used for image super-resolution, thereby improving the image quality of the obtained high-resolution image.

[0022] A third aspect of an embodiment of the present application provides a model training method, including: obtaining pixel position offset information based on a training image of a first resolution and a target resolution; performing feature extraction on the image of the first resolution to obtain a feature map containing texture features; inputting the pixel position offset information and the initial weight map into a second neural network for training to obtain a first loss value; and updating the weight parameters in the second network according to the first loss value to obtain a target neural network.

[0023] The model training method provided in the embodiment of the present application can be used for image super-resolution by fusing pixel position offset information and feature maps containing texture features. The target network trained by this method can be used for image super-resolution, thereby improving the quality of the obtained high-resolution images.

[0024] A fourth aspect of an embodiment of the present application provides an image super-resolution device, including: an acquisition unit, used to obtain pixel position offset information based on an image of a first resolution and a target resolution, wherein the target resolution is greater than the first resolution; the acquisition unit is also used to perform feature extraction on the image of the first resolution to obtain a feature map containing texture features; a processing unit is used to fuse the pixel position offset information and the feature map to obtain a target weight matrix; the acquisition unit is also used to obtain the image of the target resolution based on the target weight matrix.

[0025] In a possible implementation of the fourth aspect, the pixel position offset information includes an offset matrix and a magnification ratio, the offset matrix is used to indicate the sampling offset of each pixel point in the image of the first resolution, and the magnification ratio is the ratio between the target resolution and the first resolution.

[0026] In a possible implementation of the fourth aspect, the processing unit is specifically used to: input the pixel position offset information and the feature map into a first neural network model to obtain the target weight matrix.

[0027] In a possible implementation of the fourth aspect, the first neural network model includes a network model composed of a convolutional neural network and a fully connected neural network.

[0028] In a possible implementation manner of the fourth aspect, the processing unit is specifically configured to: obtain an initial weight matrix according to the offset matrix and the magnification ratio;

[0029] The initial weight matrix and the feature map are input into a second neural network model to obtain the target weight matrix.

[0030] In a possible implementation manner of the fourth aspect, the acquisition unit is specifically used to: input the image of the first resolution into a third neural network model to output the feature map.

[0031] In a possible implementation manner of the fourth aspect, the acquisition unit is specifically used to: upsample the feature map according to the target weight matrix to obtain an image with the target resolution.

[0032] A fifth aspect of an embodiment of the present application provides a computer program product comprising instructions, characterized in that when the computer program product is run on a computer, the computer is enabled to execute the method as described in the first aspect and any one of various possible implementation methods.

[0033] A sixth aspect of an embodiment of the present application provides a computer-readable storage medium comprising instructions, characterized in that when the instructions are executed on a computer, the computer executes a method as described in any one of the first aspect and various possible implementation methods.

[0034] A seventh aspect of an embodiment of the present application provides a chip comprising one or more processors. Some or all of the processors are used to read and execute a computer program stored in a memory to execute the method in any possible implementation of any of the above aspects. Optionally, the chip includes a memory, and the memory is connected to the processor via a circuit or wire. Further optionally, the chip also includes a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive data and / or information to be processed, and the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing results through the communication interface. The communication interface can be an input and output interface. In some implementations, some of the one or more processors may implement some steps in the above method by means of dedicated hardware. For example, the processing involving a neural network model can be implemented by a dedicated neural network processor or a graphics processor. The method provided in the present application can be implemented by one chip or by multiple chips in collaboration.

[0035] Among them, the technical effects brought about by any implementation method in the fourth aspect, the fifth aspect, the sixth aspect or the seventh aspect can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here.

[0036] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0037] In the method of the embodiment of the present application, the terminal obtains pixel position offset information based on the image of the first resolution and the target resolution. In addition, the terminal also extracts the texture features of the image of the first resolution, and obtains the target weight matrix by fusing the pixel position offset information and the texture features. The target weight matrix is used to provide the sampling weight of each pixel point in the upsampling operation of the super-resolution method. Since the target weight matrix in this scheme fuses the pixel position offset information and the texture features, obtaining a high-resolution image based on the target weight matrix can avoid the image quality loss caused by the traditional interpolation operation when relying solely on the pixel position offset information to obtain the weight matrix. Therefore, the image quality of the high-resolution image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the application scenario of the image super-resolution method;

[0039] Figure 2 Schematic diagram of the system architecture of the image super-resolution method in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of an embodiment of an image super-resolution method in an embodiment of the present application;

[0041] Figure 4aA schematic diagram of an embodiment of a method for extracting a feature map in an embodiment of the present application;

[0042] Figure 4b This is a schematic diagram of an embodiment of a feature fusion module in an embodiment of the present application;

[0043] Figure 5 This is a schematic diagram of another embodiment of the image super-resolution method in the embodiment of the present application;

[0044] Figure 6 Schematic diagram of a product implementation of the image super-resolution method in an embodiment of the present application;

[0045] Figure 7 Schematic diagram of the training process of the super-resolution module in an embodiment of the present application;

[0046] Figure 8a Schematic diagram of the process of collecting training data sets in an embodiment of the present application;

[0047] Figure 8b Schematic diagram of the process of acquiring a high-resolution dataset and a low-resolution dataset in an embodiment of the present application;

[0048] Figure 8c This is a schematic diagram of the structure of the super-resolution model in the embodiment of the present application;

[0049] Figure 8d This is a schematic diagram of the super-resolution model training process in an embodiment of the present application;

[0050] Figure 8e This is a schematic diagram of an application scenario in which a super-resolution model is deployed in a terminal in an embodiment of the present application;

[0051] Figure 9 This is a schematic diagram of an embodiment of an image super-resolution device in an embodiment of the present application;

[0052] Figure 10 This is a schematic diagram of an embodiment of a terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0053] An embodiment of the present application provides an image super-resolution method for converting a low-resolution image into a high-resolution image for display. Since the target weight matrix integrates pixel position offset information and texture features, it can avoid the image quality loss caused by relying solely on pixel position offset information for interpolation processing, thereby improving the image quality of the high-resolution image.

[0054] For ease of understanding, some technical terms involved in the embodiments of this application are briefly introduced below:

[0055] 1. To avoid confusion, several terms related to resolution are introduced and distinguished below.

[0056] Image resolution: refers to the amount of information stored in the image, which is the number of pixels per inch of the image. It is usually expressed as "horizontal pixels × vertical pixels", and can also be expressed by a specification code. For example, an image resolution of 640*480 means that the number of horizontal pixels is 640 and the number of vertical pixels is 480, so the resolution is 307,200 pixels, which is often referred to as 300,000 pixels. The specification code P means progressive scanning. The resolution corresponding to 720P is 1280*720. Similarly, the resolution corresponding to 1080P is 1920*1080. The specification code K can also be used to describe the resolution. Due to the huge number of pixels that make up a digital image, K is usually used to represent the horizontal pixel value, regardless of the aspect ratio, where 1K=2 10 =1024, 2K=2 11 =2048, 4K=2 12 =4096. 4K resolution means that the number of pixels per horizontal row reaches or approaches 4096.

[0057] It is understandable that the higher the image resolution, the more data it contains and the richer the details it can show, but it also requires more computer storage resources.

[0058] Display resolution: Display resolution refers to the physical resolution of a computer monitor. For CRT monitors, this refers to the phosphor dots on the screen; for LCD monitors, it refers to the pixels on the display. Display resolution is determined during the monitor's manufacturing process and describes the number of pixels on the monitor itself. It is a fixed, unchangeable value. Display resolution is typically expressed as "horizontal pixels x vertical pixels," such as 800×600, 1024×768, 1280×1024, etc. It can also be expressed by a specification code. Display resolution is crucial for display devices. For the same screen size, a higher resolution means a more detailed screen, which translates to clearer image details and a significantly enhanced visual experience.

[0059] Screen resolution refers to the resolution used when displaying images. You can set the screen resolution based on your needs. The upper limit of screen resolution is limited by the display resolution size.

[0060] 2. Image super-resolution (SR) refers to the restoration of a high-resolution (HR) image or image sequence from a low-resolution (LR) image or image sequence, often referred to as super-resolution.

[0061] 3. The following is an introduction to the terms involved in the image upsampling process:

[0062] 1) Upsampling: It is a process of interpolation applied to digital signal processing. When a sequence of digital signals undergoes upsampling, the output result is approximately equal to the sequence obtained by sampling the original analog signal at a higher sampling rate.

[0063] 2) Image upsampling: Similar to upsampling, it is a process of interpolation. An image with a first resolution can be regarded as a two-dimensional digital matrix. When this matrix undergoes upsampling, the output result is approximately equal to the digital matrix obtained by sampling the analog signal image in the real world at a higher sampling rate, that is, the image with the target resolution. For the convenience of describing the solution of this application, the image upsampling process is described as: The pixel points on the target resolution image sample and weighted-average the pixel points of the first resolution image. The detailed process is as follows: Calculate the sampling position of the corresponding first resolution image according to the pixel coordinates of the high-resolution image, and further calculate the sampling center and offset. Take the pixel points within a fixed-size neighborhood around the sampling center as the pixel points to be processed, then calculate the sampling weights according to the offset, and use the sampling weights to weighted-average the pixel points to be processed within the neighborhood to obtain the upsampling result. It should be noted that the number of pixel points in the neighborhood near the selected sampling center can be preset, and no specific value is limited.

[0064] 3) Pixel coordinates: In an image, the coordinates of the position where a pixel point is located, and the pixel coordinates must be integers. Taking the image with the target resolution as an example: x_out and y_out respectively represent the coordinates of a pixel point on the target resolution image in the length direction and the width direction, satisfying 0 ≤ x_out < w_out, 0 ≤ y_out < h_out. Where w_out represents the width of the target resolution, and h_out represents the length of the target resolution.

[0065] 4) Sampling position: In the process of obtaining an image with the target resolution by upsampling an image with a first resolution, the output result of each pixel point in the image with the target resolution needs to be sampled from the first resolution image. The position of each pixel point in the image with the target resolution undergoes a calculation process related to the magnification factor, corresponding to the position in the image with the first resolution, which is the sampling position. Taking a pixel point (x_out, y_out) on the target resolution image as an example, the calculation process of the sampling position of this pixel point is introduced as follows:

[0066]

[0067]

[0068] Among them, sample_pos x is the coordinate of the sampling position in the length direction; sample_pos yis the coordinate of the sampling position in the width direction; w_in is the pixel width of the first resolution image; h_in is the pixel length of the first resolution image; w_out represents the pixel width of the target resolution image, and h_out represents the length of the target resolution.

[0069] 5) Sampling center: The sampling position is rounded to the integer, and the pixel coordinates of the sampling center can be obtained.

[0070] The following is an example of a rounding operation:

[0071]

[0072]

[0073] Among them, center x is the coordinate of the sampling center in the width direction, center y is the coordinate of the sampling center in the length direction, Represents rounding n down.

[0074] 6) Offset: Also known as sampling offset, the distance between the sampling position and the sampling center. Offset is an important basis for generating sampling weights. For example, the offset calculation formula is as follows:

[0075] offset x =sample_pos x -center x

[0076] offset y =sample_pos y -center y

[0077] Among them, offset x Represents the offset distance of the sampling position relative to the sampling center in the width direction, offset y Represents the offset distance of the sampling position relative to the sampling center in the length direction.

[0078] The matrix composed of the offset of each pixel in the target resolution image is called the offset matrix.

[0079] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0080] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0081] The image super-resolution method provided in the embodiments of the present application can be applied to various electronic devices with display functions, such as mobile terminals, tablets, notebooks, computers, televisions, all-in-one machines or projectors, etc., and there is no limitation on the specific type of electronic device. The output image content includes various media information or real-time operation screens, etc., which can be static images or dynamic images, i.e., videos. The content of the image to be displayed at super-resolution is not limited in this application. The following is an introduction using terminal devices and images as examples.

[0082] First, combine Figure 1 This paper introduces the application scenarios of image super-resolution methods.

[0083] With the development of electronic display technology, screen resolution continues to improve. When images are displayed on electronic devices, the screen resolution is high, but the original resolution of the image resource is usually low and often cannot be perfectly adapted to the screen resolution. Figure 1 As shown, the original image resolution is 180P. Even if it is enlarged to 540P through 3x super-resolution, the screen resolution of the mobile phone is 1080P when it is in landscape mode and 720P when it is in portrait mode. If you want to match the screen resolution, you still need to interpolate and stretch the 540P image, which may cause jagged, blurry and other negative effects caused by interpolation.

[0084] The image super-resolution method provided in the embodiment of the present application is used to achieve super-resolution of any multiple, adapt to the terminal device display, and can also reduce the image quality loss caused by interpolation in traditional super-resolution methods, thereby improving image quality.

[0085] The following describes the system architecture of the image super-resolution method in the embodiment of the present application. Figure 2 .

[0086] Figure 2 This paper presents an example of a super-resolution application scenario for mobile terminals, consisting of an application, a rendering pipeline, and a display module. The mobile terminal application feeds the image or video frame to be displayed into the rendering pipeline, which then uses the super-resolution module to upscale and render the image. Ultimately, the display module displays a high-resolution, high-quality image on the display device.

[0087] The image super-resolution method provided in the embodiment of the present application realizes image super-resolution through an improved super-resolution module.

[0088] According to the actual application scenario, the low-resolution image is super-resolved to the target resolution, and the super-resolution module can be specifically implemented through the super-resolution model.

[0089] The formal description of the super-resolution process is as follows:

[0090] image_out=SR_model(LR_image,(height LR ,width LR ),(height dst ,width dst ))

[0091] Among them, LR_image: refers to the low-resolution image to be super-resolved; image_out: the high-resolution image generated by the super-resolved model; (height dst ,width dst ): target resolution; (height LR ,width LR ): original resolution; SR_model: super-resolution model.

[0092] The following is a detailed description of the image super-resolution method provided in the embodiment of the present application. Figure 3 .

[0093] 301. The terminal obtains pixel position offset information according to an image with a first resolution and a target resolution;

[0094] The image or video frame in the terminal application needs to be displayed at the target resolution on the display screen or display window of the terminal. The target resolution can be a preset resolution in the terminal application, or it can be determined according to the usage scenario determined by the user operation. The specific value is not limited. It can be understood that the target resolution is generally higher than the first resolution. The terminal obtains the original image of the application. The resolution of the original image is the first resolution. Since the first resolution is less than the target resolution, the terminal needs to perform super-resolution processing on the image to obtain an image or video frame of the target resolution. Since the video frame is composed of a sequence of images, the image super-resolution method provided in the embodiment of the present application and subsequent embodiments is introduced by taking an image as an example.

[0095] The pixel position offset information includes an offset matrix and a magnification scale, wherein the offset matrix is used to indicate the position offset of each pixel in the image of the first resolution when it is magnified to the target resolution, and the magnification scale is the ratio between the target resolution and the first resolution, including the ratio of the pixel values of the image of the target resolution in the length direction to the pixel values of the image of the first resolution in the length direction, and the ratio of the pixel values of the image of the target resolution in the width direction to the pixel values of the image of the first resolution in the width direction. Considering the arbitrary numerical values of the first resolution and the target resolution, the magnification scale required for super-resolution must also be an arbitrary multiple.

[0096] The terminal can obtain the offset matrix and the magnification ratio based on the image of the first resolution and the target resolution according to the existing algorithm. The calculation method of the offset matrix can refer to the aforementioned terminology introduction about the image upsampling process.

[0097] The formal description of this step is:

[0098] offset,scale=GET_POSITION_INFO((height LR ,width LR ),(height HR ,width HR ))

[0099] Wherein, offset: the position offset information of the target resolution image mapped on the low resolution image; scale: the ratio of the target resolution to the first resolution; (height LR ,width LR ): The resolution of the low-resolution image input to the super-resolution model; (height HR ,width HR):The resolution of the high-resolution image corresponding to the low-resolution image, that is, the super-resolution target resolution; GET_POSITION_INFO: Calculates the position offset information offset and magnification scale based on the target resolution and original resolution.

[0100] 302. The terminal performs feature extraction on the image at the first resolution to obtain a feature map including texture features.

[0101] The terminal extracts texture features of the image of the first resolution according to a preset neural network model to obtain a feature map, which may specifically be a multi-channel feature map.

[0102] Optionally, the preset neural network model is a pre-trained neural network, or an existing neural network, which is not specifically limited here. Optionally, the type of the neural network is a convolutional neural network (CNN).

[0103] The formal description of this step is:

[0104] feature maps = CNN(LR_image)

[0105] Among them, LR_image: low-resolution image input to the super-resolution model; CNN: convolutional neural network for feature extraction; feature maps: multi-channel feature maps output by the convolutional neural network for feature extraction.

[0106] The process of feature map usually does not change the resolution of the image, but can change the number of channels of the image. Figure 4a As shown in the figure, after processing by the convolutional neural network, the image with the original resolution of (height, width) and the number of channels is channel (if the image is in RGB format, the channel value is 3; if the image is a grayscale image, the channel value is 1) is converted into a feature map with a shape of (height, width, channel′). The value of channel′ is related to the design of the CNN network.

[0107] It should be noted that the execution order of step 301 and step 302 is not specifically limited.

[0108] 303. The terminal fuses the pixel position offset information and the feature map to obtain a target weight matrix.

[0109] The target weight matrix is used to provide the weight information required for the upsampling step in the super-resolution process. There are multiple methods for the terminal to obtain the target weight matrix by fusing the pixel position offset information and the feature map.

[0110] Optionally, the terminal inputs the offset matrix and magnification ratio obtained in step 301 and the feature map obtained in step 302 into a preset first neural network model to output a target weight matrix.

[0111] Optionally, the terminal obtains an initial weight matrix based on the offset matrix and the magnification ratio obtained in step 301; then, the terminal performs information fusion on the initial weight matrix and the feature map obtained in step 302 to obtain the target weight matrix. Information fusion can be implemented through a neural network, optionally including common channel fusion methods in neural networks: concatenation operator (concat), concatenation and superposition (concat+add), or attention mechanism (attention). The formal description of this step is:

[0112] weight maps=OFFSET_TRANSFORM(offset,scale)

[0113] weight matrix=FUSION(TRANSFORM(feature maps),weight maps)

[0114] Among them, weight maps: weight values generated based on pixel position information; OFFSET_TRANSFORM: algorithm for position information processing, which can be implemented using a fully connected neural network or other nonlinear mapping algorithm; feature maps: multi-channel feature maps extracted by the feature extraction step; TRANSFORM: feature transformation of multi-channel feature maps; FUSION: fusion of texture feature information and pixel position offset information to generate a weight matrix; weight matrix: target weight matrix.

[0115] like Figure 4b As shown in the figure, in the texture feature processing flow, the main operation is to perform feature conversion on the extracted feature map so that the feature information can adapt to the pixel position information, and the feature conversion does not change the resolution of the feature map; in the processing flow of pixel position offset information, the position information can be represented by the offset matrix and the magnification ratio, and a preliminary weight map is generated after the position information conversion; finally, the weight map containing the pixel position offset information and the feature map containing the pixel texture feature information are fused to generate the final target weight matrix.

[0116] 304. The terminal obtains an image of the target resolution according to the target weight matrix;

[0117] The feature map is upsampled using the weight matrix to obtain a super-resolution image at the target resolution. Since the target weight matrix incorporates the required magnification ratio information, it can achieve any multiple super-resolution from the first resolution image to the target resolution image according to the magnification ratio.

[0118] Formal description of this step:

[0119] SR image =Upsample_Transform(feature maps,wieght matrix)

[0120] Among them, SR_image: fuses feature information and pixel position information to generate a weight matrix; Upsample_Transform: uses the weight matrix to upsample and convert the feature map to output an image of the target resolution.

[0121] See also Figure 5 , is a schematic diagram of a system for deploying an image super-resolution method in an embodiment of the present application;

[0122] like Figure 5 The figure shows a technology and system deployed on a terminal device that uses a super-resolution module capable of achieving arbitrary super-resolution to adapt to screen resolution. The system primarily includes: a pixel offset information extraction module (or offset generation module) 501, a texture feature extraction module 502, a feature fusion module (also referred to as an arbitrary scale super-resolution (SASR) module 503 in this embodiment of the application), and an upsampling module 504. Figure 5 An example of an arbitrary multiple super-resolution scenario for a terminal device is given. Optionally, the training of the feature extraction network and the information fusion network is completed on the PC side. After the training is completed, the pixel position offset information extraction module 501, the texture feature extraction network 502, the SASR module 503, and the upsampling module 504 are deployed on the computing chip of the mobile side.

[0123] The terminal device is the main body of the system. The application hosted by the terminal device provides an image or video frame that needs to be displayed on a screen or window. Generally, the resolution of the screen and window is greater than the resolution of the image or video. At this time, the pixel position offset information extraction module 501 obtains the resolution of the original image (i.e., the first resolution) and the resolution of the target window or screen (i.e., the target resolution). After mapping the high and low resolution pixels, an offset matrix is generated. The scale of the image length and width is calculated respectively. The scale and offset are calculated through a neural network to obtain a preliminary weight matrix (i.e., the initial weight matrix) weight map. The texture feature extraction module 502 obtains the image at the first resolution and performs feature extraction. The extracted multi-channel feature map is converted to generate a new feature map (feature map). The feature map and weight map are fused through the information network of the SASR module 503 to generate the target weight matrix. The upsampling module 504 uses the weight matrix to upsample the multi-channel feature map extracted by the feature extraction network to obtain a high-quality image at the target resolution.

[0124] According to the introduction to the image super-resolution method in the above embodiment, the super-resolution model involved in the image super-resolution method needs to be obtained through pre-training before being deployed on the terminal device. Specifically, in the embodiment of the present application, the super-resolution model mainly includes four parts: a texture feature extraction module, a pixel position offset information extraction module, a SASA module, and an upsampling module. Among them: the texture feature extraction module can be implemented based on a convolutional neural network model; the SASA module can be implemented based on a convolutional neural network and a fully connected neural network model. Optionally, the SASR module is composed of a neural network, and its parameters participate in the update iteration; the texture feature extraction module is composed of an existing neural network, and the offset generation module and the upsampling module are completed by ordinary mathematical calculations, that is, the training of the super-resolution model only involves the SASR module; optionally, the texture feature extraction module and the SASR module are composed of neural networks, and their parameters participate in the update iteration; and the offset generation module and the upsampling module are completed by ordinary mathematical calculations, that is, the training of the super-resolution model involves the texture feature extraction module and the SASR module.

[0125] In the embodiment of the present application, the texture feature extraction module and the SASR module are composed of neural networks, and their parameters are updated and iterated; while the offset generation module and the upsampling module are completed by ordinary mathematical calculations.

[0126] See also Figure 6, which is a schematic diagram of the product implementation form of the image super-resolution method in the embodiment of the present application; wherein, the training process of the neural network involved in this method is completed on the first electronic device side. The first electronic device can be, for example, a server or a desktop computer or other electronic device with more computing resources and storage resources. In this embodiment, a desktop computer (PC) is used as an example for introduction. The trained neural network is deployed on the second electronic device side to implement image super-resolution. The second electronic device can be various electronic devices with display functions, including mobile terminals, tablet computers, wearable devices or smart screens, etc. In this embodiment, a mobile terminal is used as an example for introduction. Figure 6 As shown, first, the collected high-definition data set is used to create a training data set, and the super-resolution model (including the texture feature extraction module and the SASR module) is trained on the PC. The conversion program (ShaderNN converter) based on the rendering pipeline converts the model into a form that can be run on the mobile terminal. The application of the mobile terminal sends the image to be displayed to the rendering pipeline, and the rendering pipeline calls the converted super-resolution model for forward reasoning and rendering. Finally, a high-definition high-resolution image with high picture quality is displayed through the display module.

[0127] The following is an introduction to the training process of the super-resolution model in the embodiment of this application. Figure 7 .

[0128] 701. Obtain a training data set;

[0129] First, we need to create image pairs with different resolution multiples as training datasets.

[0130] The training dataset includes a high-resolution dataset and a low-resolution dataset.

[0131] First, we need to collect high-definition images. These images can be sourced from public datasets, web crawling, or our own collection. These images include high-quality images of buildings, landscapes, and people. Low-resolution datasets are generated from high-resolution images. The high-definition images are cropped and then downsampled at different scales to reduce the resolution.

[0132] For this task, we need to construct a high-definition dataset containing various textures and lines. We collect high-definition images with different texture features from the internet and publicly available datasets, including buildings, people, and landscapes. The high-definition data serves as the label, and the downsampled data serves as the data.

[0133] Formal description of this step:

[0134] labelset={buildings|person|landscape…}

[0135] labelset ↓=Downsample(labelset)

[0136] Dataset={labelset,dataset|dataset∈labelset ↓}

[0137] Among them, Dataset: training data set; Labelset: collected high-definition images; Dataset: low-resolution images produced by downsampling high-definition images; Downsample: downsampling, the operation of reducing the image, the downsampling multiple is not fixed.

[0138] Specifically, if Figure 8a The figure shows the implementation process of collecting datasets in step 1. First, high-definition images are collected, including public datasets, datasets searched using search engines, and manually collected datasets. Next, several cropping sizes of image patches are generated, i.e., resolution sizes. Optionally, the size range is within the range of (90, 400) (pixel * pixel). For example, several cropping sizes such as (128, 128), (192, 192), (256, 384), and (384, 384) can be generated. The collected high-definition images are evenly cropped into the resolution sizes of the cropping sizes to obtain patches of different cropping sizes. These patches of different cropping sizes are the image data in the high-resolution image dataset; then a low-resolution size generator is used to determine a series of corresponding low-resolution sizes for each cropping size patch. For example, for a cropping size of (128, 128), (96, 96), (64, 64), (48, 48), (64, 48), (48, 64), and (32, 32) can be generated. The patches obtained at each cropping resolution are downsampled to the corresponding low-resolution size to obtain low-resolution images with different reduction ratios as the low-resolution image dataset.

[0139] Figure 8b This figure shows how patches of different sizes are generated from original HD images to form an HR dataset. Each patch size is then downsampled to generate a series of low-resolution images to form an LR dataset. The leftmost image shows the collected HD images, including categories such as portraits, buildings, and landscapes. The images following them represent the high-resolution dataset (HR dataset) and the low-resolution dataset (LR dataset), respectively.

[0140] 702. Build a super-resolution model;

[0141] In this embodiment, the process of super-resolutioning a low-resolution image to obtain a high-resolution image primarily consists of four parts: a feature extraction module, an offset generation module, a SASR module, and an upsampling module. The feature extraction module uses a convolutional neural network-based model to extract pixel texture features from the low-resolution image. Optionally, this pixel texture feature extraction is implemented using a convolutional neural network-based model to extract multi-channel features from the low-resolution image. The model architecture can be designed and configured based on the platform's computing power. The feature extraction module generally does not change the image resolution, but can change the number of channels. The offset generation module generates an offset matrix containing pixel position information and determines the image magnification ratio. The SASR module fuses the pixel texture features with the pixel position information. Specifically, information is fused based on the multi-channel features, the offset matrix, and the magnification ratio to generate a weight matrix. Optionally, the information fusion method includes concat, concat+add, or attention. The upsampling module upsamples the feature maps corresponding to the multi-channel features based on the target weight matrix generated by the SASR module to obtain a super-resolved image at the target resolution.

[0142] like Figure 8c As shown, a model is built, the model is connected to form a trainable super-resolution network, the model is trained based on the training data set obtained in step 701, and finally the model is optimized using an optimizer. Optionally, the optimizer uses an adaptive moment estimation (Adam) optimizer. The process of model training is an existing technology and will not be described in detail here. It should be noted that in the multiple iterative processes of model training, the training data image pairs selected in each iterative process have the same magnification ratio, and the training data image pairs used in different iterative training in multiple iterative processes can have different magnification ratios. Thus, the super-resolution magnification ratio range applicable to the super-resolution model can be improved.

[0143] The specific model structure of the super-resolution model can be referred to Figure 8d . According to the previous introduction, build the texture feature extraction module and SASR module, and then based on Figure 8d The given process of generating the offset module 8002 and the upsampling module 8003 completes the connection between the network data streams and completes a trainable super-resolution network model.

[0144] 703. Perform training according to the training data set to obtain a trained super-resolution model;

[0145] Finally, if Figure 8dAs shown, low-resolution images 8001 of different resolutions are input into the super-resolution network, the model processes the low-resolution image, outputs the super-resolution image 8004, calculates the loss 8005 between the super-resolution image and the high-resolution image, such as the regression loss (huber loss), and uses the optimizer 8006 to optimize the loss function, such as the Adam optimizer, to update the model weights until the loss function converges to obtain a trained super-resolution model.

[0146] like Figure 8e As shown, after the super-resolution model is trained, the trained super-resolution model is deployed on the mobile terminal and implemented in the glsl language. The original rendering pipeline is that the image 8012 in the application (APP) 8011 needs to be displayed, and the image is enlarged or reduced to the target resolution through bilinear interpolation in the rendering pipeline 8013, and then displayed on the screen. The pipeline adapted to the algorithm of arbitrary multiple super-resolution uses an arbitrary multiple super-resolution model instead of the bilinear interpolation algorithm. The texture feature extraction module and SASR module in the trained model are converted into the glsl language through the conversion program (shaderNN converter) 8014, while the offset generation module and upsampling module in the super-resolution model are manually converted into the glsl language to adapt to the entire super-resolution model. After the super-resolution model is deployed, the image enters the rendering pipeline. The rendering pipeline relies on the OpenGL library (OpenGL libs) and calls various modules that have been converted into the shading language (OpenGL Shading Language, glsl) to process the image 8012. At the same time, the pipeline sends the resolution of the target window or screen to the super-resolution model to calculate information such as offset. The image resolution generated by the final model is the same as the target window or screen resolution.

[0147] An existing class of learning-based image super-resolution methods achieves super-resolution through a sub-pixel convolutional layer. Due to the limitations of the network structure, a network model can only achieve fixed integer multiples of super-resolution. If there is a need to change the magnification factor, the network needs to be redesigned and retrained. The image super-resolution method provided in the embodiment of the present application is based on an innovative upsampling process and uses a target weight matrix that integrates pixel position offset information and texture features for upsampling, which can achieve arbitrary multiples of super-resolution. In the process of obtaining the target weight matrix, this solution inputs a pre-trained neural network model (i.e., the feature fusion module in this application) based on the actual required magnification ratio, and the output target weight matrix is used for upsampling to obtain a magnification ratio that meets the actual needs (i.e., an arbitrary multiple). At the same time, the target weight matrix in this solution integrates pixel position offset information and texture features, avoiding the image quality loss problem of similar interpolation super-resolution technology caused by upsampling based only on pixel position offset information.

[0148] The above describes the image super-resolution method provided by this application. The following describes the image super-resolution device that implements the image super-resolution method. Figure 9 , is a schematic diagram of an embodiment of an image super-resolution device in an embodiment of the present application.

[0149] Figure 9 One or more of the modules in the system may be implemented in software, hardware, firmware, or a combination thereof. The software or firmware includes, but is not limited to, computer program instructions or codes, and may be executed by a hardware processor. The hardware includes, but is not limited to, various integrated circuits, such as a central processing unit (CPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0150] The image super-resolution device comprises:

[0151] An acquiring unit 901 is configured to acquire pixel position offset information according to an image of a first resolution and a target resolution, where the target resolution is greater than the first resolution;

[0152] The acquisition unit 901 is further configured to perform feature extraction on the image at the first resolution to obtain a feature map containing texture features;

[0153] A processing unit 902 is configured to fuse the pixel position offset information with the feature map to obtain a target weight matrix;

[0154] The acquisition unit 901 is further configured to acquire the image of the target resolution according to the target weight matrix.

[0155] Optionally, the pixel position offset information includes an offset matrix and a magnification ratio, the offset matrix is used to indicate a sampling offset of each pixel point in the image of the first resolution, and the magnification ratio is a ratio between the target resolution and the first resolution.

[0156] Optionally, the processing unit 902 is specifically used to: input the pixel position offset information and the feature map into a first neural network model to obtain the target weight matrix.

[0157] Optionally, the first neural network model includes a network model composed of a convolutional neural network and a fully connected neural network.

[0158] Optionally, the processing unit 902 is specifically used to: obtain an initial weight matrix based on the offset matrix and the magnification ratio; input the initial weight matrix and the feature map into a second neural network model to obtain the target weight matrix.

[0159] Optionally, the acquisition unit 901 is specifically used to: input the image of the first resolution into a third neural network model to output the feature map.

[0160] Optionally, the acquisition unit 901 is specifically configured to: upsample the feature map according to the target weight matrix to acquire an image of the target resolution.

[0161] See also Figure 10 , is a schematic diagram of an embodiment of a terminal in the embodiments of the present application;

[0162] The terminal 1000 provided in this embodiment can be any type of terminal with a display function, such as a mobile phone, a tablet computer, a desktop computer, a smart screen or a wearable device, etc. The specific device form is not limited in the embodiments of the present application.

[0163] The terminal 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors 1001 and a memory 1002 , in which programs or data are stored.

[0164] The memory 1002 may be a volatile memory or a non-volatile memory. Optionally, the processor 1001 is one or more central processing units (CPUs, which may be single-core CPUs or multi-core CPUs. The processor 1001 may communicate with the memory 1002 and execute a series of instructions in the memory 1002 on the terminal 1000.

[0165] The terminal 1000 further includes one or more wired or wireless network interfaces 1003 , such as an Ethernet interface.

[0166] Alternatively, although Figure 10 Not shown, the terminal 1000 may also include one or more power supplies; one or more input and output interfaces, which can be used to connect a display, a mouse, a keyboard, a touch screen device or a sensor device, etc. The input and output interfaces are optional components and may or may not exist, and are not limited here.

[0167] The process executed by the processor 1001 in the terminal 1000 in this embodiment can refer to the method process described in the aforementioned method embodiment, and will not be repeated here.

[0168] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0173] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image super-resolution method, characterized in that: include: The terminal obtains pixel position offset information based on an image with a first resolution and a target resolution, where the target resolution is greater than the first resolution, the pixel position offset information including an offset matrix and a magnification ratio, the offset matrix being used to indicate a sampling offset for each pixel in the image with the first resolution, and the magnification ratio being a ratio between the target resolution and the first resolution; The terminal performs feature extraction on the image with the first resolution to obtain a feature map including texture features; The terminal fuses the pixel position offset information with the feature map to obtain a target weight matrix; The terminal obtains an image of the target resolution according to the target weight matrix.

2. The method according to claim 1, characterized in that The terminal fuses the pixel position offset information with the feature map to obtain a target weight matrix, including: The terminal inputs the pixel position offset information and the feature map into a first neural network model to obtain the target weight matrix.

3. The method according to claim 2, characterized in that The first neural network model includes a network model composed of a convolutional neural network and a fully connected neural network.

4. The method according to claim 1, wherein The terminal fuses the pixel position offset information with the feature map to obtain a target weight matrix, including: The terminal obtains an initial weight matrix according to the offset matrix and the amplification ratio; The terminal inputs the initial weight matrix and the feature map into a second neural network model to obtain the target weight matrix.

5. The method according to any one of claims 1 to 4, characterized in that The terminal extracts features from the image at the first resolution to obtain a feature map containing texture features, including: The terminal inputs the image of the first resolution into a third neural network model to output the feature map.

6. The method according to any one of claims 1 to 4, characterized in that The terminal acquiring the image of the target resolution according to the target weight matrix includes: The terminal upsamples the feature map according to the target weight matrix to obtain an image of the target resolution.

7. An image super-resolution device, characterized in that: include: an acquiring unit, configured to acquire pixel position offset information based on an image with a first resolution and a target resolution, wherein the target resolution is greater than the first resolution, the pixel position offset information including an offset matrix and a magnification ratio, the offset matrix being used to indicate a sampling offset for each pixel point in the image with the first resolution, and the magnification ratio being a ratio between the target resolution and the first resolution; The acquisition unit is further configured to perform feature extraction on the image with the first resolution to obtain a feature map containing texture features; a processing unit, configured to fuse the pixel position offset information with the feature map to obtain a target weight matrix; The acquisition unit is further configured to acquire the image of the target resolution according to the target weight matrix.

8. The device according to claim 7, characterized in that The processing unit is specifically configured to: The pixel position offset information and the feature map are input into a first neural network model to obtain the target weight matrix.

9. The device according to claim 8, characterized in that The first neural network model includes a network model composed of a convolutional neural network and a fully connected neural network.

10. The device according to claim 7, characterized in that The processing unit is specifically configured to: Obtaining an initial weight matrix according to the offset matrix and the magnification ratio; The initial weight matrix and the feature map are input into a second neural network model to obtain the target weight matrix.

11. The device according to any one of claims 7 to 10, characterized in that The acquisition unit is specifically configured to: The image with the first resolution is input into a third neural network model to output the feature map.

12. The device according to any one of claims 7 to 10, characterized in that The acquisition unit is specifically configured to: The feature map is upsampled according to the target weight matrix to obtain an image of the target resolution.

13. A terminal, characterized in that: include: One or more processors and memory; wherein, The memory stores computer-readable instructions; The one or more processors read the computer-readable instructions to cause the terminal to implement the method according to any one of claims 1 to 6.

14. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that The method comprises computer-readable instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 6.

Images