Image super-resolution method and device, computer equipment and readable storage medium

Abstract

The invention relates to an image super-resolution method and device, computer equipment and a readable storage medium. The method comprises the following steps: inputting a plurality of sample image pairs into a neural network model before training; under the condition that the current training round is smaller than a preset number of times, updating model parameters of the first encoder, the second encoder and the decoder, and obtaining a feature mapping relationship between the first sample image and the second sample image; under the condition that the current training round is not less than a preset number of times, continuously updating the coding table, and updating all model parameters in the neural network model based on an updating result of the coding table and the feature mapping relationship; and under the condition that the training of the neural network model is finished, inputting the to-be-processed image and a preset amplification factor into the trained neural network model, and outputting a super-resolution image. According to the method provided by the invention, the model does not need to be trained for multiple times, and computing resources and time resources are saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image super-resolution method and device, computer equipment and readable storage medium. BACKGROUND

[0002] Image super-resolution is an important technical means in the field of image processing. The current mainstream method is based on a deep learning architecture, which directly learns the mapping relationship from a low-resolution image to a high-resolution image by constructing an end-to-end neural network model.

[0003] However, the existing method usually binds the super-resolution multiple with the up-sampling layer parameters in the network structure. When different multiple super-resolution tasks need to be processed, the network structure needs to be redesigned and a complete training process needs to be performed, which consumes a large amount of computing resources and time resources. SUMMARY

[0004] Therefore, it is necessary to provide an image super-resolution method, device, computer equipment and readable storage medium capable of saving computing resources and time resources in view of the above technical problems.

[0005] In a first aspect, the present application provides an image super-resolution method, comprising:

[0006] obtaining a plurality of pairs of sample image pairs and inputting the plurality of pairs of sample image pairs into a neural network model before training; wherein for any one pair of sample image pairs, the sample image pair comprises a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content; the neural network model comprises a first encoder, a second encoder and a decoder;

[0007] for a current training round in a training process, if the current training round is less than a preset number of times, updating the model parameters of the first encoder, the second encoder and the decoder, and obtaining a feature mapping relationship between the first sample image and the second sample image;

[0008] if the current training round is not less than the preset number of times, continuously updating an encoding table based on a preset iteration period, and updating all model parameters in the neural network model based on the update result of the encoding table and the feature mapping relationship; wherein the encoding table comprises a plurality of feature vectors, and the feature vectors are used to represent the semantic features of image regions in sample images;

[0009] In a case where the training of the neural network model is completed, a to-be-processed image and a preset magnification factor are acquired, and the to-be-processed image and the preset magnification factor are input into the trained neural network model, and an super-resolution image is output.

[0010] In one of the embodiments, the codebook includes a plurality of cluster center vectors, which are generated after clustering the first feature vectors and the second feature vectors; the first feature vectors are extracted from the first continuous codes, and the second feature vectors are extracted from the second continuous codes; the first continuous codes are extracted from the first sample images by the first encoder, and the second continuous codes are extracted from the second sample images by the second encoder; the first feature vectors correspond to the image regions in the first sample images one by one, and the second feature vectors correspond to the image regions in the second sample images one by one; and the first feature vectors correspond to the second feature vectors one by one.

[0011] In one of the embodiments, the updating the model parameters of the first encoder and the second encoder includes acquiring a feature mapping relationship between the first sample images and the second sample images.

[0012] For the corresponding first feature vectors and second feature vectors generated in the current training round, a first similarity between the first feature vectors and the second feature vectors is acquired, and a similarity integration value of all the first similarities is acquired.

[0013] Based on the similarity integration value, a first loss value of the neural network model in the current training round is acquired.

[0014] Based on the first loss value, the model parameters of the first encoder and the second encoder are updated, and the feature mapping relationship is acquired.

[0015] In one of the embodiments, the updating all the model parameters in the neural network model includes:

[0016] For any one feature vector generated in the current training round, a second similarity between the feature vector and any one cluster center vector in the codebook is acquired.

[0017] Based on the cluster center vector corresponding to the minimum second similarity, a discrete code corresponding to the continuous code to which the feature vector belongs is constructed, a discrete code corresponding to the first continuous code is determined as the first discrete code, and a discrete code corresponding to the second continuous code is determined as the second discrete code.

[0018] obtaining a reconstruction loss value based on the second discrete code and the first sample image, and obtaining an encoding table loss value and a commitment loss value based on the first continuous code and the first discrete code; obtaining a second loss value of the neural network model in the current training round based on the reconstruction loss, the encoding table loss value and the commitment loss value;

[0019] updating all model parameters in the neural network model based on the second loss value.

[0020] In one of the embodiments, the reconstruction loss value is generated based on the first sample image and a corresponding reconstructed image, and the reconstructed image is generated by the decoder based on a weighted discrete code, which is obtained by weighting the second discrete code based on an attention weight matrix.

[0021] In one of the embodiments, the method further comprises:

[0022] In the case that the current training round and a first preset number of consecutive historical training rounds before the current training round both satisfy a preset training termination condition, ending the training process of the neural network model, and determining the neural network model obtained in the current training round as the trained neural network model; wherein the preset training termination condition is that the improvement degree of the peak signal-to-noise ratio between the first sample image and the corresponding reconstructed image is less than a preset threshold, and the first loss value is within a preset loss value range.

[0023] In one of the embodiments, the obtaining a plurality of pairs of sample images comprises:

[0024] obtaining a plurality of first sample images, and performing convolution operation on the first sample image by using a preset Gaussian kernel for any first sample image;

[0025] dividing the first sample image into a second preset number of pixel blocks based on a preset reduction factor for the first sample image after the convolution operation, and obtaining the positional relationship of different pixel blocks in the first sample image;

[0026] obtaining the pixel value of any one pixel point in the pixel block, and obtaining the pixel value integration value of the pixel values of all pixel points in the pixel block for any one pixel block;

[0027] generating a second sample image comprising a second preset number of pixel points and satisfying the positional relationship between different pixel points;

[0028] For any one pixel point in the second sample image, based on the position relationship, a pixel value integration value of a pixel block corresponding to the pixel point is determined as a pixel value of the pixel point; wherein a position of the pixel point in the second sample image is same as a position of the corresponding pixel block in the first sample image;

[0029] The first sample image and the second sample image are determined as a pair of sample image pairs.

[0030] In a second aspect, the present application further provides an image super-resolution device, the device comprising:

[0031] A first obtaining module is configured to obtain a plurality of pairs of sample image pairs and input the plurality of pairs of sample image pairs into a neural network model before training; wherein for any one pair of sample image pairs, the sample image pair comprises a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content; the neural network model comprises a first encoder, a second encoder and a decoder;

[0032] A first updating module is configured to, for a current training round in a training process, update model parameters of the first encoder, the second encoder and the decoder, obtain a feature mapping relationship between the first sample image and the second sample image, and in a case that the current training round is less than a preset number of times.

[0033] A second updating module is configured to, in a case that the current training round is not less than the preset number of times, continuously update an encoding table based on a preset iteration period, and update all model parameters in the neural network model based on an update result of the encoding table and the feature mapping relationship; wherein the encoding table comprises a plurality of feature vectors, and the feature vectors are used to represent semantic features of image regions in sample images.

[0034] A second obtaining module is configured to, in a case that the training of the neural network model is completed, obtain a to-be-processed image and a preset magnification factor, input the to-be-processed image and the preset magnification factor into the trained neural network model, and output a super-resolution image.

[0035] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0036] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the method in any one of the above embodiments.

[0037] The image super-resolution method, device, computer device and readable storage medium described above obtain a plurality of pairs of sample image pairs, and input the plurality of pairs of sample image pairs to a neural network model before training. For any one pair of sample image pairs, the sample image pair includes a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content. The neural network model includes a first encoder, a second encoder and a decoder. For a current training round in a training process, if the current training round is less than a preset number of times, the model parameters of the first encoder, the second encoder and the decoder are updated, and a feature mapping relationship between the first sample image and the second sample image is obtained. If the current training round is not less than the preset number of times, the encoding table is continuously updated based on a preset iteration period, and all model parameters in the neural network model are updated based on the update result of the encoding table and the feature mapping relationship. If the training of the neural network model ends, a to-be-processed image and a preset magnification factor are obtained, and the to-be-processed image and the preset magnification factor are input to the trained neural network model, and a super-resolution image is output. The method provided in the present application trains the neural network model through a plurality of pairs of sample image pairs, updates the model parameters in the neural network model, so that the trained neural network model can be used to perform super-resolution processing on the to-be-processed image with different magnification factors, without the need for multiple training of the model, thereby saving computing resources and time resources. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0039] Figure 1 A flowchart of an image super-resolution method in an embodiment;

[0040] Figure 2 A flowchart of a feature mapping relationship obtaining method in an embodiment;

[0041] Figure 3 A structural diagram of a neural network model in another embodiment;

[0042] Figure 4A structural block diagram of an image super-resolution device in one embodiment;

[0043] Figure 5 An internal structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] For the purpose, technical solutions and advantages of the present application to be more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0045] It should be noted that the terms "first", "second" and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options, or any combination of a plurality of options.

[0046] In one embodiment, as shown in Figure 1 An image super-resolution method is provided, and the present embodiment is exemplified by the method applied to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be realized through the interaction of the terminal and the server. In the present embodiment, the method includes the following steps:

[0047] S102, a plurality of pairs of sample image pairs are acquired, and the plurality of pairs of sample image pairs are input to a neural network model before training; wherein for any one pair of sample image pairs, the sample image pair includes a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content; the neural network model includes a first encoder, a second encoder and a decoder.

[0048] Optionally, the sample image input into the first encoder is the first sample image, and the sample image input into the second encoder is the second sample image.

[0049] Optionally, the first encoder and the second encoder each comprise five convolutional layers; wherein the convolutional kernels of the five convolutional layers each have a size of 3x3; the convolutional kernels of the first and fifth convolutional layers slide on the input feature map with a step size of 1 pixel, and the convolutional kernels of the second to fourth convolutional layers slide on the input feature map with a step size of 2 pixels; the input data received by the first convolutional layer has three channels, and the feature map channel numbers output by the first to fifth convolutional layers after convolutional operation are 64, 128, 256, 512 and 256 respectively; the first to fourth convolutional layers apply a ReLU (Rectified Linear Unit) activation function to the output feature map after convolutional operation, and the fifth convolutional layer does not apply any activation function after completing convolutional calculation.

[0050] Optionally, the decoder comprises five transposed convolutional layers; wherein the convolutional kernels of the five transposed convolutional layers each have a size of 3x3; the convolutional kernels of the first to fourth transposed convolutional layers slide on the input feature map with a step size of 2 pixels, and the convolutional kernel of the fifth transposed convolutional layer slides on the input feature map with a step size of 1 pixel; the input feature map channel numbers of the first to fourth transposed convolutional layers are all 256, the feature map channel numbers output by the first to fourth transposed convolutional layers after convolutional operation are 512, 256, 128 and 64 respectively, and the output feature map of the fifth transposed convolutional layer contains three channels; the first to fourth transposed convolutional layers apply a ReLU (Rectified Linear Unit) activation function to the output feature map after convolutional operation, and the fifth transposed convolutional layer applies a Sigmoid activation function to the output feature map after convolutional operation, wherein the Sigmoid activation function is a kind of nonlinear activation function.

[0051] S104, for a current training round in the training process, updating the model parameters of the first encoder, the second encoder and the decoder to obtain a feature mapping relationship between the first sample image and the second sample image if the current training round is less than a preset number of times.

[0052] Optionally, before training the neural network model, the encoder and the decoder are first initialized by He (HeInitialization) and 0 initialization, wherein the He initialization is used to assign an initial value to the weight matrix of the convolutional layer, and the 0 initialization means assigning an initial value of 0 to the bias vector of all layers of the encoder and the decoder; and the total training rounds (epoch), the batch size (batch size) and the initial learning rate are preset, for example, epoch = 100, batch size = 32, and the initial learning rate = 1e-4.

[0053] Optionally, in the case that the current training round is less than the preset number of times, in order to avoid the initial noise causing interference to the training process of the model, the encoding table is frozen, that is, the encoding table is not updated, for example, the preset number of times can be 10 times.

[0054] S106, in the case that the current training round is not less than the preset number of times, based on the preset iteration period, the encoding table is continuously updated, and based on the update result of the encoding table and the feature mapping relationship, all model parameters in the neural network model are updated; wherein the encoding table includes a plurality of feature vectors, and the feature vectors are used to represent the semantic features of the image regions in the sample images.

[0055] Optionally, in the case that the current training round is not less than the preset number of times, in order to optimize the synergy between the encoding table and the neural network model, the encoding table is unfrozen, and the encoding table is updated once every preset iteration period of iteration process, and the preset iteration period can be but not limited to 1000 iterations.

[0056] S108, in the case that the training of the neural network model is completed, the image to be processed and the preset magnification factor are obtained, and the image to be processed and the preset magnification factor are input into the trained neural network model, and the super-resolution image is output.

[0057] Wherein, the preset magnification factor refers to the multiple of the resolution of the image to be processed.

[0058] Optionally, after the image to be processed and the preset magnification factor are input into the trained neural network model, the second encoder first extracts features from the low-resolution image to be processed, compresses the image information into high-dimensional feature vectors, and then the decoder gradually upsamples the features based on these feature vectors and in combination with the requirement of the preset magnification factor, and finally generates a super-resolution image meeting the magnification multiple.

[0059] In the image super-resolution method, a plurality of pairs of sample image pairs are obtained, and the plurality of pairs of sample image pairs are input into the neural network model before training; wherein, for any one pair of sample image pairs, the sample image pair includes a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content; the neural network model includes a first encoder, a second encoder and a decoder; for a current training round in a training process, if the current training round is less than a preset number of times, the model parameters of the first encoder, the second encoder and the decoder are updated, and a feature mapping relationship between the first sample image and the second sample image is obtained; if the current training round is not less than the preset number of times, the code table is continuously updated based on a preset iteration period, and all model parameters in the neural network model are updated based on the update result of the code table and the feature mapping relationship; if the neural network model training is completed, a to-be-processed image and a preset magnification factor are obtained, and the to-be-processed image and the preset magnification factor are input into the trained neural network model, and a super-resolution image is output. The method provided in the application trains the neural network model through a plurality of pairs of sample image pairs, updates the model parameters in the neural network model, so that the trained neural network model can be used to perform super-resolution processing on the to-be-processed image with different magnification factors, without the need for multiple training of the model, thereby saving computing resources and time resources.

[0060] In some embodiments, the code table includes a plurality of cluster center vectors, the cluster center vectors being generated after clustering the first feature vectors and the second feature vectors; the first feature vectors are extracted from the first continuous codes, and the second feature vectors are extracted from the second continuous codes; the first continuous codes are extracted from the first sample image by the first encoder, and the second continuous codes are extracted from the second sample image by the second encoder; the first feature vectors correspond to the image regions in the first sample image one by one, the second feature vectors correspond to the image regions in the second sample image one by one, and the first feature vectors correspond to the second feature vectors one by one.

[0061] Optionally, after the first sample image is input into the first encoder, the first encoder obtains the first continuous code by performing a compression process with a step of 2 on the first sample image, wherein the image size of the first sample image is n x n x 3, the size of the first continuous code is m x m x 256, and m = n / 32; after the second sample image is input into the second encoder, the second encoder obtains the second continuous code by performing a compression process with a step of 2 on the second sample image, wherein the image size of the second sample image is n / 4 x n / 4 x 3, the size of the second continuous code is m x m x 256, and m = n / 32.

[0062] Optionally, the spatial position information of the picture in the input sample image is preserved in both the first continuous encoding and the second continuous encoding; and the m x m grid in the continuous encoding corresponds to a local image region in the input sample image one by one.

[0063] Optionally, during the training of the neural network model, the encoding table is updated once every preset number of iteration processes, for example, the preset number of iteration processes is 1000 iterations. Specifically, all the first continuous encodings and the second continuous encodings generated in the preset number of iteration processes are obtained, and all the first feature vectors are extracted from the first continuous encodings, and all the second feature vectors are extracted from the second continuous encodings, wherein the feature vectors are used to represent the spatial positions in the sample images, and the dimensions of the feature vectors are 256 dimensions; the K-means (K=512) clustering is used to cluster all the extracted first feature vectors and second feature vectors, 2K=1024 cluster centers are obtained, and the cluster centers are combined to form the encoding table , wherein the encoding table includes 512 first feature vectors and 512 second feature vectors, e i represents a cluster center; the encoding table E is subjected to L2 normalization to ensure that the vector lengths are consistent, and the L2 normalization is a feature scaling method used to normalize the L2 norm (i.e., the length of the vector) of the vector to 1.

[0064] In this embodiment, by converting the image spatial information into compact feature encodings, a basis can be provided for subsequent image reconstruction; by generating the encoding table based on the extracted feature vectors, the continuous feature space can be discretized, and the common patterns of high and low resolution features can be preserved.

[0065] In some embodiments, as Figure 2 shown, the model parameters of the first encoder and the second encoder are updated, and the feature mapping relationship between the first sample image and the second sample image is obtained, including:

[0066] S202, for the corresponding first feature vector and second feature vector generated in the current training round, a first similarity between the first feature vector and the second feature vector is obtained, and a similarity integration value of all the first similarities is obtained.

[0067] S204, based on the similarity integration value, a first loss value of the neural network model in the current training round is obtained.

[0068] S206, based on the first loss value, the model parameters of the first encoder and the second encoder are updated, and the feature mapping relationship is obtained.

[0069] Optionally, the first loss value is used to represent semantic consistency between the corresponding first feature vector and the second feature vector; the first similarity can be but is not limited to represented by a cosine distance, an Euclidean distance or a Mahalanobis distance, and the similarity integration value can be but is not limited to an average value or a median value of all the first similarities; for example, the first loss value can be calculated based on an average value of the cosine distance between the first feature vector and the second feature vector, and is specifically as shown in the following formula:

[0070]

[0071] In the formula, L cos is the first similarity, m is a dimension m of the continuous encoding, is a cosine distance between the first feature vector Z_H(i, j) and the second feature vector Z_L(i, j).

[0072] Optionally, in the back propagation process in the model training process, the first loss value gradient synchronously optimizes the first encoder and the second encoder, and forces the first continuous encoding and the second continuous encoding to be semantically aligned at the feature vectors in the same spatial position.

[0073] In the embodiment, the model parameters of the first encoder and the second encoder are updated based on the first loss value and the second loss value, so that the super-resolution precision of the model after parameter updating is higher.

[0074] In some embodiments, updating all model parameters in the neural network model includes: for any one feature vector generated in the current training round, obtaining a second similarity between the feature vector and any one cluster center vector in the encoding table; based on the cluster center vector corresponding to the minimum second similarity, constructing a discrete encoding corresponding to the continuous encoding to which the feature vector belongs, and determining a discrete encoding corresponding to the first continuous encoding as the first discrete encoding, and determining a discrete encoding corresponding to the second continuous encoding as the second discrete encoding; based on the second discrete encoding and the first sample image, obtaining a reconstruction loss value, and based on the first continuous encoding and the first discrete encoding, obtaining an encoding table loss value and a commitment loss value; based on the reconstruction loss, the encoding table loss value and the commitment loss value, obtaining a second loss value of the neural network model in the current training round; and based on the second loss value, updating all model parameters in the neural network model.

[0075] Optionally, the second similarity between the feature vector and the cluster center vector can be calculated as shown in the following formula:

[0076]

[0077] In the formula, k is the second similarity, Z(i, j) is the feature vector, e j is the cluster center vector.

[0078] Optionally, after the first discrete code and the second discrete code are constructed, the first discrete code and the second discrete code are converted into an m×m integer matrix, wherein the elements of the integer matrix are indexes of 1-1024, facilitating storage.

[0079] Optionally, the reconstruction loss value is generated based on the first sample image and a corresponding reconstruction image, the reconstruction image being generated by the decoder based on a weighted discrete code, the weighted discrete code being obtained by weighting the second discrete code based on an attention weight matrix. The weighted discrete code can highlight key region features such as edges and textures in the sample image.

[0080] Optionally, for any one second discrete code, an attention weight matrix W (m×m) is generated based on the feature energy of the second feature vector in the corresponding second continuous code of the second discrete code, wherein the higher the feature energy, the greater the weight value of the corresponding second feature vector; the product between any one feature vector in the second discrete code and the corresponding weight value is obtained, and all the corresponding products of the feature vectors are determined as the feature vectors in the corresponding weighted discrete code of the second discrete code; the decoder performs up-sampling on the weighted discrete code to generate a reconstruction image, wherein the size of the reconstruction image is the same as that of the first sample image, and the reconstruction loss value represents the similarity between the reconstruction image and the first sample image.

[0081] Optionally, the second loss value is as follows:

[0082]

[0083] In the formula, L total is the second loss value, L code is the code table loss value, L rec is the reconstruction loss value, L commit is the commitment loss value, L cos is the first loss value, and λ=0.1.

[0084] Optionally, the code table loss value L code can be calculated as follows:

[0085]

[0086] In the formula, β=0.25, which is used to constrain the code table to be close to the continuous code, Z H is the first continuous code, is the first discrete code.

[0087] Optionally, the commitment loss value L commit can be calculated as follows:

[0088]

[0089] In the formula, , for constraining continuous coding to approach the coding table.

[0090] Optionally, the coding table is updated by sliding translation with a momentum of 0.99, which can avoid codebook conflicts, and the coding table updating process is as follows:

[0091]

[0092] In the formula, represents the k-th cluster center vector in the coding table, is the set of all , is the number of in , and “←” represents the update operation.

[0093] In this embodiment, by converting the continuous coding into discrete coding, the complexity of the feature space can be reduced, and the generalization ability and reconstruction efficiency of the model can be improved; based on the reconstruction loss value, the coding table loss value and the commitment loss value, the second loss value is obtained, which can effectively improve the performance of the trained image.

[0094] In some embodiments, the method further comprises: in the case that the current training round and the first preset number of consecutive historical training rounds before the current training round both satisfy the preset training termination condition, ending the training process of the neural network model, and determining the neural network model obtained in the current training round as the trained neural network model; wherein the preset training termination condition is that the improvement degree of the peak signal-to-noise ratio between the first sample image and the corresponding reconstructed image is less than a preset threshold, and the first loss value is within a preset loss value range.

[0095] Optionally, if the peak signal-to-noise ratio of any one of the five consecutive training rounds is less than 0.1 dB, and the first loss value is in a stable state, the model training process is ended, and the model obtained in the current training is determined as the optimal model.

[0096] In this embodiment, in the case that the current training round and the first preset number of consecutive historical training rounds before the current training round both satisfy the preset training termination condition, the training process of the neural network model is ended, which can not only ensure the quality of model training, but also improve the training efficiency.

[0097] In some embodiments, obtaining a plurality of pairs of sample images comprises: obtaining a plurality of first sample images, and performing convolution operation on each first sample image using a preset Gaussian kernel; dividing each first sample image into a second preset number of pixel blocks based on a preset downsizing factor, and obtaining the positional relationship of different pixel blocks in the first sample image; obtaining the pixel value of any pixel point in each pixel block, and obtaining the pixel value integration value of all pixel points in the pixel block; generating a second sample image comprising a second preset number of pixel points and satisfying the positional relationship between different pixel points; determining the pixel value integration value of the pixel block corresponding to each pixel point in the second sample image as the pixel value of the pixel point based on the positional relationship; wherein the position of the pixel point in the second sample image is the same as the position of the corresponding pixel block in the first sample image; and determining the first sample image and the second sample image as a pair of sample images.

[0098] Optionally, after the first sample image and the second sample image are generated, the image can be further processed by standardization, random cropping, horizontal flipping, etc.

[0099] In this embodiment, the corresponding second sample image is obtained by performing Gaussian blur and average pooling on the first sample image, which can provide diversified training data and prevent overfitting risk.

[0100] In one exemplary embodiment, another image super-resolution method is provided, which comprises the following contents:

[0101] Step S1: data preparation and preprocessing.

[0102] Trigger condition: the user starts the model training task.

[0103] Processing action: collect original high-resolution image data, obtain low-resolution image LR and high-resolution image HR through different downsampling methods, and a set of high and low resolution images as a pair of input and output of the model. Wherein, the model is as shown in Figure 3 Figure 3 301 is a low-resolution encoder, 302 is a high-resolution encoder, 303 is a decoder, 304 is a model parameter optimization module, LR is a low-resolution image, HR is a high-resolution image, and codebook is an encoding table.

[0104] Result: obtain the required high and low resolution image pairs.

[0105] Result use: as input data and reconstructed high-resolution image in the model encoding stage.

[0106] ​Step S2: Encoder constructs a continuous encoding generation.

[0107] Trigger condition: The pre-processed low-resolution image is input to the low-resolution encoder, and the high-resolution image is input to the high-resolution encoder.

[0108] Processing action: The high-low resolution encoder is an encoder composed of multiple layers of convolution. The low-resolution image is transmitted into the encoder, and the features are extracted through convolution operation to generate a continuous encoding vector zL. The structure of zL is an m x m x d matrix (m is much smaller than n), which retains the position information of the input image, and each element is a d-dimensional vector. Similarly, the high-resolution image is transmitted into the encoder, and the features are extracted through convolution operation to generate a continuous encoding vector zH. The structure of zH is an m x m x d matrix (m is much smaller than n), which retains the position information of the input image, and each element is a d-dimensional vector.

[0109] Result: Get continuous high-resolution encoding vector zH and low-resolution encoding vector zL.

[0110] Result use: Provide basic encoding for subsequent vector quantization.

[0111] Step S3: High-low resolution encoding loss calculation.

[0112] Trigger condition: Generate high-low resolution encoding.

[0113] Processing action: After generating high-low resolution encoding, calculate the cosine distance of high-low resolution encoding, which is the shortest cosine distance of high-low resolution encoding.

[0114] Result: Get the feature information of the low-resolution image and the scaling information before the high-low resolution image.

[0115] Step S4: Construction of high-low resolution vector encoding table.

[0116] Trigger condition: Continuous high-resolution encoding vector zH and low-resolution encoding vector zL are generated.

[0117] Processing action: Continuous high-resolution encoding vector zH and low-resolution encoding vector zL, construct an encoding table containing 2K d-dimensional vectors .

[0118] Result: Get an encoding table containing 2K-1 encodings.

[0119] Result use: Get an encoding table as a codebook for discrete encoding generation.

[0120] Step S5: Discrete encoding vector generation.

[0121] Trigger condition: Generation of high-low resolution feature codebook.

[0122] Process action: If the image is enlarged 4 times, for each m x m dimension vector in z, find the nearest vector in the codebook by nearest neighbor search (calculate the L2 distance with the vectors in the codebook) , as the quantized discrete code vector zh_q, that is , where .

[0123] Result: Get the discrete code vector z_q with the same structure as z, which is equivalent to a 4m x 4m integer matrix (elements are integers of 1-K).

[0124] Result usage: As the input of the decoder, used for image reconstruction.

[0125] Step S6: Discrete code vector weighting.

[0126] Trigger condition: Discrete code vector z_q generation is completed.

[0127] Process action: Design a feature with the same shape as the discrete code vector z_q, and weight it to the z_q vector by multiplication.

[0128] Result: Get the input of the decoder.

[0129] Result usage: Used for diversified codebook features.

[0130] Step S7: Reconstruct the image.

[0131] Trigger condition: Discrete code vector z_q weighting generation is completed.

[0132] Process action: Design a decoder corresponding to the encoder (such as multi-layer transpose convolution), input z_q into the decoder, and reconstruct the image by upsampling and other operations to generate the reconstructed image .

[0133] Result: Get the reconstructed image .

[0134] Result usage: Used for calculating reconstruction loss and evaluating reconstruction effect.

[0135] Step S8: Loss function definition and model training.

[0136] Trigger condition: Reconstructed image generation is completed.

[0137] Process action: Use the Straight-Through Estimator technique to design the loss function, and the total loss is composed of three parts:

[0138] Reconstruction loss: Forward propagation is equivalent to Backward propagation is equivalent to No gradient is provided, equivalent to computing gradient with z as input.

[0139] Encoding table loss: (β is a hyper-parameter), fix z, make encoding table vector close to z.

[0140] Commitment loss: (γ=0.25β), fix encoding table vector, make z close to z_q.

[0141] Use an optimizer (such as Adam) to update the encoder, decoder parameters and encoding table based on the total loss (the encoding table can be updated in a sliding average manner).

[0142] Result: get the trained encoder, decoder and encoding table.

[0143] Result usage: used for subsequent image generation and encoding distribution fitting.

[0144] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise stated herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0145] Based on the same inventive concept, the embodiments of the present application also provide an image super-resolution device for implementing the above-mentioned image super-resolution method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more image super-resolution device embodiments provided below can refer to the limitations of the image super-resolution method in the foregoing, which will not be repeated here.

[0146] In an exemplary embodiment, as shown in Figure 4 An image super-resolution device 400 is provided, comprising a first acquisition module 401, a first update module 402, a second update module 403 and a second acquisition module 404, wherein:

[0147] The first obtaining module 401 is configured to obtain a plurality of pairs of sample image pairs and input the plurality of pairs of sample image pairs to the neural network model before training; wherein, for any pair of sample image pairs, the sample image pair comprises a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content; the neural network model comprises a first encoder, a second encoder and a decoder.

[0148] The first updating module 402 is configured to, for a current training round in a training process, update model parameters of the first encoder, the second encoder and the decoder, obtain a feature mapping relationship between the first sample image and the second sample image, and the current training round is less than a preset number of times.

[0149] The second updating module 403 is configured to, when the current training round is not less than the preset number of times, continuously update an encoding table based on a preset iteration period, and update all model parameters in the neural network model based on an update result of the encoding table and the feature mapping relationship; wherein, the encoding table comprises a plurality of feature vectors, and the feature vectors are used to represent semantic features of image regions in sample images.

[0150] The second obtaining module 404 is configured to, when the training of the neural network model is completed, obtain a to-be-processed image and a preset magnification factor, input the to-be-processed image and the preset magnification factor to the trained neural network model, and output a super-resolution image.

[0151] In some embodiments, the image super-resolution device 400 is specifically configured to, the encoding table comprises a plurality of cluster center vectors, the cluster center vectors are generated after clustering processing of first feature vectors and second feature vectors; the first feature vectors are extracted from first continuous encodings, the second feature vectors are extracted from second continuous encodings, the first continuous encodings are extracted from the first sample image by the first encoder, and the second continuous encodings are extracted from the second sample image by the second encoder; the first feature vectors correspond to image regions in the first sample image one by one, the second feature vectors correspond to image regions in the second sample image one by one, and the first feature vectors correspond to the second feature vectors one by one.

[0152] In some embodiments, the first updating module 402 is further configured to, for any one of the corresponding first feature vectors and second feature vectors generated in the current training round, obtain a first similarity between the first feature vector and the second feature vector, and obtain a similarity integration value of all the first similarities; based on the similarity integration value, obtain a first loss value of the neural network model in the current training round; based on the first loss value, update the model parameters of the first encoder and the second encoder, and obtain the feature mapping relationship.

[0153] In some embodiments, the second updating module 403 is further configured to, for any one of the feature vectors generated in the current training round, obtain a second similarity between the feature vector and any one of the cluster center vectors in the encoding table; based on the cluster center vector corresponding to the minimum second similarity, construct the discrete code corresponding to the continuous code to which the feature vector belongs, and determine the discrete code corresponding to the first continuous code as the first discrete code, and determine the discrete code corresponding to the second continuous code as the second discrete code; based on the second discrete code and the first sample image, obtain a reconstruction loss value, and based on the first continuous code and the first discrete code, obtain an encoding table loss value and a commitment loss value; based on the reconstruction loss, the encoding table loss value and the commitment loss value, obtain a second loss value of the neural network model in the current training round; based on the second loss value, update all the model parameters in the neural network model.

[0154] In some embodiments, the second updating module 403 is further configured to, the reconstruction loss value is generated based on the first sample image and the corresponding reconstructed image, the reconstructed image is generated by the decoder based on the weighted discrete code, and the weighted discrete code is obtained by weighting the second discrete code based on the attention weight matrix.

[0155] In some embodiments, the image super-resolution apparatus 400 is further configured to, in a case where the current training round and a first preset number of consecutive historical training rounds before the current training round all satisfy a preset training termination condition, end the training process of the neural network model, and determine the neural network model obtained in the current training round as the trained neural network model; wherein the preset training termination condition is that the improvement degree of the peak signal-to-noise ratio between the first sample image and the corresponding reconstructed image is less than a preset threshold, and the first loss value is within a preset loss value range.

[0156] In some embodiments, the image super-resolution device 400 is further configured to acquire multiple first sample images, and perform a convolution operation on any one of the first sample images using a preset Gaussian kernel; for the first sample image after the convolution operation, based on a preset reduction factor, divide the first sample image into a second preset number of pixel blocks, and acquire the positional relationship between different pixel blocks in the first sample image; for any pixel block, acquire the pixel value of any pixel in the pixel block, and acquire the pixel value integration value of all pixel values ​​in the pixel block; generate a second sample image including the second preset number of pixels, and satisfying the positional relationship between different pixels; for any pixel in the second sample image, based on the positional relationship, determine the pixel value of the pixel block corresponding to the pixel as the pixel value of the pixel; wherein the position of the pixel in the second sample image is the same as the position of the corresponding pixel block in the first sample image; and determine the first sample image and the second sample image as a pair of sample images.

[0157] Each module in the aforementioned image super-resolution device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0158] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an image super-resolution method.

[0159] Those skilled in the art will understand thatFigure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0160] In an embodiment, a computer device is also provided, including a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the steps in the above method embodiments.

[0161] In an embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0164] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0165] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An image super-resolution method, characterized by, The method comprises: a plurality of pairs of sample image pairs are acquired, and the plurality of pairs of sample image pairs are input to a neural network model before training; wherein, for any one pair of sample image pairs, the sample image pair comprises a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content; the neural network model comprises a first encoder, a second encoder and a decoder; for a current training round in a training process, if the current training round is less than a preset number of times, model parameters of the first encoder, the second encoder and the decoder are updated, and a feature mapping relationship between the first sample image and the second sample image is acquired; if the current training round is not less than the preset number of times, an encoding table is continuously updated based on a preset iteration period, and all model parameters in the neural network model are updated based on an update result of the encoding table and the feature mapping relationship; wherein, the encoding table comprises a plurality of feature vectors, and the feature vectors are used to represent semantic features of image regions in sample images; if the neural network model training is completed, a to-be-processed image and a preset magnification factor are acquired, the to-be-processed image and the preset magnification factor are input to the trained neural network model, and a super-resolution image is output.

2. The method of claim 1, wherein, The encoding table comprises a plurality of cluster center vectors, the cluster center vectors are generated after clustering processing of a first feature vector and a second feature vector; the first feature vector is extracted from a first continuous code, the second feature vector is extracted from a second continuous code, the first continuous code is extracted from the first sample image by the first encoder, and the second continuous code is extracted from the second sample image by the second encoder; the first feature vector corresponds to an image region in the first sample image one by one, the second feature vector corresponds to an image region in the second sample image one by one, and the first feature vector corresponds to the second feature vector one by one.

3. The method of claim 2, wherein, The updating of the model parameters of the first encoder and the second encoder and the acquisition of the feature mapping relationship between the first sample image and the second sample image comprise: for corresponding first feature vectors and second feature vectors generated in the current training round, a first similarity between the first feature vector and the second feature vector is acquired, and a similarity integration value of all first similarities is acquired; based on the similarity integration value, a first loss value of the neural network model in the current training round is acquired; based on the first loss value, the model parameters of the first encoder and the second encoder are updated, and the feature mapping relationship is acquired.

4. The method of claim 2, wherein, The updating of all model parameters in the neural network model comprises: for any one feature vector generated in the current training round, a second similarity between the feature vector and any one cluster center vector in the encoding table is acquired; construct a discrete code corresponding to the continuous code to which the feature vector belongs based on the clustering center vector corresponding to the minimum second similarity, and determine a discrete code corresponding to the first continuous code as the first discrete code and a discrete code corresponding to the second continuous code as the second discrete code; obtain a reconstruction loss value based on the second discrete code and the first sample image, and obtain an encoding table loss value and a commitment loss value based on the first continuous code and the first discrete code; obtain a second loss value of the neural network model in the current training round based on the reconstruction loss, the encoding table loss value, and the commitment loss value; update all model parameters in the neural network model based on the second loss value.

5. The method of claim 4, wherein, The reconstruction loss value is generated based on the first sample image and a corresponding reconstructed image, and the reconstructed image is generated by the decoder based on a weighted discrete code obtained by weighting the second discrete code based on an attention weight matrix.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: In a case where the current training round and a first preset number of consecutive historical training rounds before the current training round both satisfy a preset training termination condition, ending the training process of the neural network model, and determining the neural network model obtained in the current training round as the trained neural network model; wherein the preset training termination condition is that the improvement degree of the peak signal-to-noise ratio between the first sample image and the corresponding reconstructed image is less than a preset threshold, and the first loss value is within a preset loss value range.

7. The method of claim 1, wherein, The method further comprises: obtaining a plurality of first sample images, and performing convolution operation on the first sample image by using a preset Gaussian kernel for any first sample image; dividing the first sample image into a second preset number of pixel blocks based on a preset reduction factor for the first sample image after the convolution operation, and obtaining the positional relationship of different pixel blocks in the first sample image; obtaining the pixel value of any pixel point in the pixel block, and obtaining the pixel value integration value of the pixel values of all pixel points in the pixel block for any pixel block; generating a second sample image including a second preset number of pixel points and satisfying the positional relationship between different pixel points; determining the pixel value integration value of the pixel block corresponding to the pixel point as the pixel value of the pixel point based on the positional relationship for any pixel point in the second sample image; wherein the position of the pixel point in the second sample image is the same as the position of the corresponding pixel block in the first sample image; determining the first sample image and the second sample image as a pair of sample images.

8. An image super-resolution apparatus, characterized by comprising: The apparatus comprises: The first obtaining module is configured to obtain a plurality of pairs of sample image pairs and input the plurality of pairs of sample image pairs to a neural network model before training; wherein, for any pair of sample image pairs, the sample image pair comprises a first sample image and a second sample image, the resolution of the first sample image is greater than the resolution of the second sample image, and the first sample image and the second sample image display the same content; the neural network model comprises a first encoder, a second encoder and a decoder; The first updating module is configured to, for a current training round in a training process, update model parameters of the first encoder, the second encoder and the decoder when the current training round is less than a preset number of times, and obtain a feature mapping relationship between the first sample image and the second sample image; The second updating module is configured to, when the current training round is not less than the preset number of times, continuously update an encoding table based on a preset iteration period, and update all model parameters in the neural network model based on an update result of the encoding table and the feature mapping relationship; wherein, the encoding table comprises a plurality of feature vectors, and the feature vectors are used to represent semantic features of image regions in sample images; The second obtaining module is configured to, when the training of the neural network model is completed, obtain a to-be-processed image and a preset magnification factor, input the to-be-processed image and the preset magnification factor to the trained neural network model, and output a super-resolution image. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images