Image super-resolution processing method, device, equipment and storage medium

Abstract

The present application proposes an image super-resolution processing method, apparatus, device and storage medium. The image super-resolution processing includes: obtaining an image to be processed; performing super-resolution processing on the image to be processed through a target neural network model, wherein the target neural network model is obtained by iteratively training a pyramid neural network model with a target image sample set, and the target image samples are obtained by filtering original image samples according to the sample information content interval, and the number of image samples in each sample information content interval is the same.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image super-resolution processing method, apparatus, device, and storage medium. Background Art

[0002] Super-resolution reconstruction technology based on deep learning has shown significant advantages over traditional methods in reconstruction effect comparison. The mainstream of current super-resolution reconstruction methods is beginning to move towards deep learning.

[0003] Current deep neural network-based super-resolution methods fall into two broad categories: those based on generative adversarial networks (GANs) and supervised fully convolutional neural networks (FCNs). The former uses a perceptron cost function within the GAN framework to position the output in the high-order space of the ground truth (GT), with the goal of making it impossible for the discriminator to distinguish between the generated image and the real image. Supervised methods generate low-resolution and high-resolution pairs (LHPs) using a defined degradation model and then identify these LHPs using a robust network structure model.

[0004] Both approaches generally randomly crop each dataset into fixed-size blocks, then combine these blocks into a sample set. This sample set serves as input for forward and backward propagation in the network to update network parameters. This approach suffers from significant imbalance in natural images, primarily due to the statistically significant proportion of smooth regions compared to various texture types. This means that undisturbed random sampling results in more smooth regions being fed into the network, causing the network to favor LHP in smooth regions. Summary of the Invention

[0005] The present application provides an image super-resolution processing method, apparatus, device and storage medium, which can solve the problem of imbalance in the ratio of sampling output samples in the prior art.

[0006] In a first aspect, an embodiment of the present application provides an image super-resolution processing method, comprising: acquiring an image to be processed;

[0007] The image to be processed is subjected to super-resolution processing by a target neural network model, wherein the target neural network model is obtained by iteratively training a pyramid neural network model with a target image sample set, and the target image samples are obtained by filtering original image samples according to a sample information content interval, and the number of image samples in each sample information content interval is the same.

[0008] In a second aspect, an embodiment of the present application provides an image super-resolution processing device, comprising: an acquisition module, configured to acquire an image to be processed;

[0009] A processing module is used to perform super-resolution processing on the image to be processed through a target neural network model, wherein the target neural network model is obtained by iteratively training a pyramid neural network model with a target image sample set, and the target image samples are obtained by filtering the original image samples according to the sample information content interval, and the number of image samples in each sample information content interval is the same.

[0010] In a third aspect, an embodiment of the present application provides a device comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the methods in the embodiments of the present application.

[0011] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods in the embodiments of the present application is implemented.

[0012] With respect to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of an information determination method provided in this application;

[0014] Figure 1a An architectural diagram of a deep neural network provided in this application;

[0015] Figure 1b A training flow chart of the neural network model provided for this application;

[0016] Figure 2 A schematic diagram of the structure of an information determination device provided in this application;

[0017] Figure 3 A schematic diagram of the structure of a terminal provided in this application. DETAILED DESCRIPTION

[0018] To make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any way.

[0019] The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. Also, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be performed in an order different from that shown here.

[0020] In an exemplary embodiment, Figure 1 This is a flow chart of an image super-resolution processing method provided in this application. This method is applicable to super-resolution processing of low-resolution images to be processed. This method can be performed by the image super-resolution processing device provided in this application, which can be implemented in software and / or hardware and integrated into a device.

[0021] like Figure 1 As shown, the present application provides an image super-resolution processing method, including S110-S120.

[0022] S110: Acquire an image to be processed.

[0023] Wherein, the image to be processed is a low-resolution image.

[0024] Specifically, the image to be processed may be acquired by capturing the image to be processed through a camera, or may be acquired by intercepting the image to be processed. The embodiment of the present invention does not limit the method of acquiring the image to be processed.

[0025] S120. Super-resolution processing is performed on the image to be processed through a target neural network model, wherein the target neural network model is obtained by iteratively training a pyramid neural network model with a target image sample set, and the target image samples are obtained by filtering the original image samples according to the sample information content interval, and the number of image samples in each sample information content interval is the same.

[0026] Among them, the pyramid neural network model, for the basic feature extraction module, realizes the acquisition from shallow features to high-level features through multi-layer cascades. It should be noted that the adoption of the pyramid network structure, on the one hand, retains the bottom-level features, middle-level features and high-level features can participate in the upsampling process, and on the other hand, the stability of the network can be improved due to the multi-layer skip connection. For the basic feature extraction module φ, through multi-layer cascades, it is possible to obtain features from shallow features to high-level features. The basic feature extraction module can choose a residual network or a U-Net structural model. The features extracted by each module are used as the input of the next level feature extraction, and finally the features of each layer are used as the input of the upsampling module, as shown in the following formula:

[0027] Input:[φ0,φ1,...,φ n ],

[0028] Among them, φ is the basic feature extraction module.

[0029] Specifically, such as Figure 1a As shown in the figure, it is an architecture diagram of the deep neural network in the embodiment of the present invention. A low-resolution image is input into the neural network model to obtain a high-resolution image. In the neural network model, shallow feature extraction is first performed. The basic feature extraction module includes: basic feature extraction module φ0, basic feature extraction module φ1... basic feature extraction module φ n The features extracted by each module are used as the input of the next level feature extraction, and finally the features of each layer are used as the input of the upsampling module. The features extracted by each module are used as the input of the next level feature extraction, and finally the features of each layer are used as the input of the upsampling module.

[0030] In the embodiment of the present invention, the output of super-resolution reconstruction is not limited to a limited number of classification targets, which is different from deep neural networks that are used for classification and recognition. From the input-output dimension, super-resolution is a many-to-many mapping process, as shown in the following formula:

[0031] Γ:X→q

[0032] K:X→Y

[0033] Here, Γ represents the classification recognition network, X represents the input image, which can be two-dimensional (grayscale), three-dimensional (for example, containing color information), or even four-dimensional (for example, a video containing color information). q represents a scalar. For target recognition, q may be a Boolean variable, and for multi-classification problems, q may be a finite-sized scalar. K represents the super-resolution mapping relationship, where Y represents the output high-resolution result. In most cases, Y should maintain the same dimension as X, and the amplitude in each dimension should not be less than X. This can be seen as a process of information expansion. In terms of information dimension and effect improvement, K requires a broader consideration of all input information.

[0034] The traditional super-resolution input sample extraction is to randomly select batch samples {a0, a1, ..., a n Then for each sample a i Random cropping is performed, and finally, data augmentation is performed as needed before inputting into the network for training. This approach simply transfers the classification recognition network and fails to consider the similarity of X's underlying information. This means that most images contain large smooth areas (this phenomenon is more pronounced in high-resolution images), and smooth areas have higher similarity. This similarity manifests itself during network training as a statistically higher weighting of textured areas at the input compared to other textured areas. This imbalance causes the super-resolution neural network to prefer smooth outputs. This result contradicts the true background image information, resulting in the loss of detail in previously clear areas.

[0035] In the embodiment of the present invention, the target image sample is obtained by filtering the original image sample according to the sample information content interval, that is, the filtering method is used to achieve balanced control of the sample. The specific implementation method can be: using gradient information to characterize the texture richness of the sample, calculating the Sobel gradient information of each image sample, and summing the gradient information, and taking the summation result as the sample information content. Or it can be: using variance, because variance can also characterize the degree of change of the sample, calculating the variance in each randomly obtained sample, and then filling it into the corresponding interval according to the value of the variance until the samples in each interval are satisfied. Since each sample is obtained in a random manner, shuffle is used between each round of training, which can also ensure the randomness of the sample.

[0036] Specifically, the entire sample information content space corresponding to the target image sample set is divided to obtain at least one sample information content interval, each interval having a starting point and an end point of the sample information content, and the number of samples within each interval is set to obtain the sample information content of the original image sample, and the sample information content interval corresponding to the sample information content is determined according to the sample information content. After finding the corresponding interval, the number of samples in the space is obtained. If the number of samples does not reach the set number of samples, the space is selected to be entered; if the number of samples reaches the set number of samples, the space is not selected.

[0037] Specifically, after filtering the original image samples, target image samples are obtained, a target neural network model is obtained by iteratively training a pyramid neural network model through the target image sample set, and super-resolution processing is performed on the image to be processed through the trained target neural network model.

[0038] like Figure 1b As shown in the figure, it is a training flow chart of the neural network model. First, obtain the initial samples from the training set, select the samples, train the network model based on the selected samples, calculate the loss, update the neural network model, and iterate the above steps until the final model is obtained.

[0039] This application provides an image super-resolution processing method. It obtains a target image and performs super-resolution processing on it using a target neural network model. The target neural network model is obtained by iteratively training a pyramid neural network model with a target image sample set. The target image samples are obtained by filtering original image samples according to sample information content intervals. The number of image samples in each sample information content interval is the same. This method solves the problem of unbalanced sample output ratios in the prior art and effectively alleviates network bias.

[0040] Based on the above embodiment, a modified embodiment of the above embodiment is proposed. It should be noted that, in order to simplify the description, only the differences from the above embodiment are described in the modified embodiment.

[0041] In one embodiment, iteratively training the pyramid neural network model using the target image sample set includes:

[0042] Establish a pyramid neural network model;

[0043] Inputting the low-resolution image in the target sample set into the pyramid neural network to obtain a predicted image;

[0044] Training parameters of the pyramid neural network according to an objective function formed by the predicted image and a high-resolution image corresponding to the low-resolution image, wherein the objective function includes one or more of a function L2+αL1, a function L1+θLP, and a function L2+μL1+βLP, wherein L2 is an L2 norm, L1 is an L1 norm, LP is an LP norm, and α, θ, μ, and β are weights of regularization factors;

[0045] Return to executing the operation of inputting the low-resolution image in the target sample set into the pyramid neural network to obtain a predicted image until the target neural network model is obtained.

[0046] Among them, the L1 norm is The L2 norm is The LP norm is Among them, x i is the difference between the predicted image and the high-resolution image corresponding to the low-resolution image.

[0047] Specifically, when p is greater than 0 and less than 1, x i There will be a greater response under the Lp norm. Therefore, using the Lp norm as the objective function can make smaller differences output larger losses, improving the network's sensitivity to weak differences.

[0048] Specifically, the parameters of the pyramid neural network training based on the objective function formed by the predicted image and the high-resolution image corresponding to the low-resolution image can be: the objective function is the function L2+αL1, L2 is the L2 norm, the L1 is the L1 norm, and the L1 norm is The L2 norm is x i is the difference between the predicted image and the high-resolution image corresponding to the low-resolution image; or it can be: the initial objective function is the function L2+αL1, after the objective function does not decrease significantly, the objective function is the function L1+θLP, the LP is the LP norm, the α and θ are the weights of the regularization factor, and the LP norm is x i is the difference between the predicted image and the high-resolution image corresponding to the low-resolution image; it can also be: the objective function is the function L2+μL1+βLP, μ and β are the weights of the regularization factors, and the approximation to the global optimum is achieved by adjusting the weights.

[0049] In one embodiment, training parameters of the pyramid neural network according to an objective function formed by the predicted image and the high-resolution image corresponding to the low-resolution image includes:

[0050] Training parameters of the pyramid neural network based on the objective function L2+αL1 and the learning rate set to a first value;

[0051] After the decrease value of the output value of the objective function is less than the second data, the parameters of the pyramid neural network are continued to be trained based on the objective function L1+θLP and the learning rate set to the second value, wherein the second value is less than the first value, and the α and θ increase as the learning rate decreases.

[0052] The first value may be a set larger learning rate, and the second value may be a set relatively smaller learning rate.

[0053] Specifically, in the initial stage of training, the function L2+αL1 is used as the objective function, and a large learning rate is used for training. After the objective function does not decrease significantly, L1+θLP is used as the objective function, and training is carried out at a relatively small learning rate.

[0054] In one embodiment, training parameters of the pyramid neural network according to an objective function formed by the predicted image and the high-resolution image corresponding to the low-resolution image includes:

[0055] Training the parameters of the pyramid neural network according to the objective function L2+μL1+βLP formed by the predicted image and the high-resolution image corresponding to the low-resolution image;

[0056] In each round of training, or between two rounds of training, μ and β are adjusted to achieve an approximation to the global optimum.

[0057] Specifically, the L2+μL1+βLP norm constraint is selected globally as the objective function, and the approximation to the global optimum is achieved by dynamically adjusting the weight of the regularization factor in each round of training or between two rounds of training.

[0058] Embodiments of the present invention relate to setting objective functions and adjusting parameters during training. A multi-norm regularization method is used during training to enhance the contribution to non-salient texture components. Furthermore, by adjusting the weights of the regularization factors based on network convergence during training, dynamic adjustments improve network convergence efficiency while preventing the network from falling into a minimum and more efficiently updating network hyperparameters. Multi-norm regularization is employed. Existing analytical methods suffer from low stability and occasionally introduce components or textures not included in the original image, resulting in low objective metrics (such as PSNR and SSIM). The L2 or L1 norm is typically used as the objective function. Network training results constrained by these functions typically exhibit high objective quality. Pursuing quantitative metrics often leads to a significant increase in network complexity, resulting in limited improvement in performance with increased computing power requirements, making deployment difficult. Furthermore, the emphasis on increasing depth often overlooks the impact of network structure and data. Compared to the currently commonly used L2 or L1 norms as objective functions, embodiments of the present invention simultaneously incorporate the Lp norm and employ dynamic adjustments during training to improve training convergence efficiency and focus on small objects.

[0059] In one embodiment, the sample information content includes: gradient information of the image sample, or variance within the image sample.

[0060] Among them, gradient information represents the texture richness of image samples.

[0061] Specifically, the Sobel operator gradient information of each sample is calculated, and the gradient information is summed up, and the summation result is used as the representation of the sample information content.

[0062] The variance within an image sample represents the degree of variation of the image sample.

[0063] Specifically, the variance within each randomly obtained sample is calculated and used as a representation of the information content of the sample.

[0064] In one embodiment, the sample information content is:

[0065] PIC=||max(abs(Sobel(a i [x, y, w, h])))||1, where PIC represents the sample information content, x and y represent the horizontal and vertical coordinates of the starting point of the selected image sample, w and h represent the width and height of the selected image sample, and a i is an image sample.

[0066] In one embodiment, the sample information content is:

[0067] PIC=||max(abs(Var(a i[x, y, w, h])))||1, where PIC represents the sample information content, x and y represent the horizontal and vertical coordinates of the starting point of the selected image sample, w and h represent the width and height of the selected image sample, and a i is an image sample.

[0068] In one embodiment, the target image samples are obtained by filtering the original image samples according to the sample information content intervals, and the number of image samples in each sample information content interval is the same, including:

[0069] Dividing the sample information content space corresponding to the target image sample set to obtain at least two sample information content intervals, each sample information content interval corresponding to a different information content range, wherein each sample information content interval contains a target number of image samples;

[0070] Obtaining the information content of image samples;

[0071] Determining a corresponding sample information content interval according to the information content of the image sample;

[0072] If the number of samples within the sample information content interval is less than the target number, the image sample is determined as a target image sample.

[0073] The sample information content space corresponding to the target image sample set is divided to obtain at least two sample information content intervals. For example, the sample information content space corresponding to the target image sample set can be divided to obtain four sample information content intervals, as shown in the following formula:

[0074] PIC=[[0,pic0],[pic0,pic1],[pic1,pic2],[pic2,∞]];

[0075] Among them, [0,pic0] represents the starting point and ending point of pic in the first interval, [pic0,pic1] represents the starting point and ending point of pic in the second interval, [pic1,pic2] represents the starting point and ending point of pic in the third interval, and [pic2,∞] represents the starting point and ending point of pic in the fourth interval.

[0076] The number of samples corresponding to the target image sample set is set to N, and the number of samples corresponding to the target image sample set is averaged to four intervals. The number of image samples in each interval is N / 4, as shown in the following formula:

[0077] Interval=[N / 4,N / 4,N / 4,N / 4];

[0078] Where N is the number of samples corresponding to the target image sample set, and Internal represents the number of samples allocated to each interval after the overall PIC space is divided.

[0079] Specifically, the sample information content space corresponding to the target image sample set is divided into 4 valid intervals. According to the batch number N, the samples in each interval are selected as N / 4. For example, the number of samples allocated in the first interval is N / 4. When the PIC of the patch is in the interval [0, pic0] and the number of samples in the interval does not reach N / 4, the sample is selected to enter the batch to be processed. At the same time, when it exceeds N / 4, the samples in the area are filled, and the patches in the interval are no longer selected. At this time, if there is still a shortage in other intervals, then random sampling is performed again, and the information content of each sample is measured until the data in all intervals is satisfied.

[0080] This application provides an image super-resolution processing device, Figure 2 This is a schematic diagram of the structure of an image super-resolution processing device provided by this application. Figure 2 As shown, an image super-resolution processing device in an embodiment of the present application can be integrated into a device. The device includes: an acquisition module 21 for acquiring an image to be processed; a processing module 22 for performing super-resolution processing on the image to be processed using a target neural network model. The target neural network model is obtained by iteratively training a pyramid neural network model using a target image sample set. The target image samples are obtained by filtering original image samples according to sample information content intervals, and the number of image samples in each sample information content interval is the same.

[0081] The device provided in this embodiment is used to implement Figure 1 The method of the embodiment shown in the figure, the device provided in this embodiment realizes the principle and technical effect similar to Figure 1 The method of the illustrated embodiment is similar and will not be described again here.

[0082] Based on the above embodiment, a modified embodiment of the above embodiment is proposed. It should be noted that, in order to simplify the description, only the differences from the above embodiment are described in the modified embodiment.

[0083] In one embodiment, the processing module 22 is specifically configured to:

[0084] Establish a pyramid neural network model;

[0085] Inputting the low-resolution image in the target sample into the pyramid neural network to obtain a predicted image;

[0086] Training parameters of the pyramid neural network according to an objective function formed by the predicted image and a high-resolution image corresponding to the low-resolution image, wherein the objective function includes one or more of a function L2+αL1, a function L1+θLP, and a function L2+μL1+βLP, wherein L2 is an L2 norm, L1 is an L1 norm, LP is an LP norm, and α, θ, μ, and β are weights of regularization factors;

[0087] Return to executing the operation of inputting the low-resolution image in the target sample set into the pyramid neural network to obtain a predicted image until the target neural network model is obtained.

[0088] In one embodiment, the processing module 22 is specifically configured to:

[0089] Training parameters of the pyramid neural network based on the objective function L2+αL1 and the learning rate set to a first value;

[0090] After the decrease value of the output value of the objective function is less than the second data, the parameters of the pyramid neural network are continued to be trained based on the objective function L1+θLP and the learning rate set to the second value, wherein the second value is less than the first value, and the α and θ increase as the learning rate decreases.

[0091] In one embodiment, the processing module 22 is specifically configured to:

[0092] Training the parameters of the pyramid neural network according to the objective function L2+μL1+βLP formed by the predicted image and the high-resolution image corresponding to the low-resolution image;

[0093] In each round of training, or between two rounds of training, μ and β are adjusted to achieve an approximation to the global optimum.

[0094] In one embodiment, the sample information content includes: gradient information of the image sample, or variance within the image sample.

[0095] In one embodiment, the sample information content is:

[0096] PIC=||max(abs(Sobel(a i [x, y, w, h])))||1, where PIC represents the sample information content, x and y represent the horizontal and vertical coordinates of the starting point of the selected image sample, w and h represent the width and height of the selected image sample, and a i is an image sample.

[0097] In one embodiment, the sample information content is:

[0098] PIC=||max(abs(Var(a i [x, y, w, h])))||1, where PIC represents the sample information content, x and y represent the horizontal and vertical coordinates of the starting point of the selected image sample, w and h represent the width and height of the selected image sample, and a i is an image sample.

[0099] In one embodiment, the processing module 22 is specifically configured to:

[0100] Dividing the sample information content space corresponding to the target image sample set to obtain at least two sample information content intervals, each sample information content interval corresponding to a different information content range, wherein each sample information content interval contains a target number of image samples;

[0101] Obtaining the information content of image samples;

[0102] Determining a corresponding sample information content interval according to the information content of the image sample;

[0103] If the number of samples within the sample information content interval is less than the target number, the image sample is determined as a target image sample.

[0104] This application provides an image super-resolution processing device for obtaining an image to be processed and performing super-resolution processing on the image using a target neural network model. The target neural network model is obtained by iteratively training a pyramid neural network model with a target image sample set. The target image samples are obtained by filtering original image samples according to sample information content intervals, and the number of image samples in each sample information content interval is the same. This solves the problem of unbalanced sample output ratios in the prior art and effectively alleviates network bias.

[0105] Specifically, during the training sample collection process, embodiments of the present invention filter samples by adding sample information content to patches, improving their statistical balance when entering network training. First, a statistical metric is established to evaluate the sample information content. Patches randomly cropped from the dataset are calibrated using this metric. The statistical metric is segmented to determine the number of samples required for each segment. Then, a second selection is performed on the randomly cropped patches to ensure that the final batch remains balanced throughout the training process. In terms of network architecture design, embodiments of the present invention employ a pyramid structure as the core structure of the network. Semantic information from the bottom, middle, and top layers is simultaneously introduced into the upsampling module. Given that semantic information tends to be higher layers as convolution deepens, discarding bottom and middle layer information would reduce the focus on subtle image details in the reconstruction. Therefore, the pyramid structure is used to condense the extracted information, reducing computational cost, while also integrating multiple layers of information to improve reconstruction quality. The objective function is set and hyperparameters are adjusted during training. Multi-norm regularization is employed during training to enhance the contribution of non-salient texture components. At the same time, by adjusting the weight of the regularization factor according to the network convergence during the training process, dynamic adjustment can improve the convergence efficiency of the network on the one hand, and prevent the network from falling into the minimum value on the other hand, and update the network hyperparameters more effectively.

[0106] The embodiments of the present invention address the problem of unbalanced sample output ratios in existing super-resolution methods. The similarity of smooth areas is statistically significant, and the proportion of smooth areas will be significantly higher than that of textured areas after other data segmentation. The deepening of the network structure leads to the loss of shallow information, causing imbalance of shallow, middle, and high-level information in the reconstruction process. The solution of this patent can effectively alleviate the bias of the network, improve the problem of missing details in the super-resolution reconstruction process, and improve the defect of disordered direction of the reconstructed texture part.

[0107] The embodiments of the present invention fully utilize the various feature information in the samples. The effectiveness of supervised deep learning depends largely on the richness of the samples. Although the absolute number of sample data provided by the target has greatly increased, the relative number of data is uneven. This is especially true for high-resolution samples, which often contain large areas of sky, glass, or other slowly varying areas. The embodiments of the present invention reduce data imbalance by analyzing the information entropy of the samples, providing higher-quality information input for the neural network model. The multi-scale network architecture uses a pyramid method to input multi-level information into the upsampling module. This reduces the bias of subsequent features towards high-level features with increasing depth, which leads to a decrease in the proportion of shallow and mid-level features. The present invention feeds the output of each layer of the pyramid into the upsampling module, improving the data richness of the upsampling module. Compared to the L2 norm or L1 norm commonly used as the objective function in current supervised methods, the present invention proposes the simultaneous use of the Lp norm and dynamic adjustment during training to improve training convergence efficiency and focus on small objects.

[0108] This application provides a device, Figure 3 The present application provides a schematic diagram of a device structure, such as Figure 3 As shown, the device provided by this application includes one or more processors 31 and a storage device 32; the processor 31 in the device can be one or more, Figure 3 In the embodiment, a processor 31 is used as an example; the storage device 32 is used to store one or more programs; the one or more programs are executed by the one or more processors 31, so that the one or more processors 31 implement the embodiment of the present application. Figure 1 The method described.

[0109] The device further comprises: a communication device 33 , an input device 34 and an output device 35 .

[0110] The processor 31, storage device 32, communication device 33, input device 34 and output device 35 in the device can be connected through a bus or other means. Figure 3 The bus connection is taken as an example.

[0111] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0112] The communication device 33 may include a receiver and a transmitter. The communication device 33 is configured to transmit and receive information according to the control of the processor 31. The information includes but is not limited to uplink authorization information.

[0113] The storage device 32 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, and modules, such as the embodiment of the present application. Figure 1 The program instructions / modules corresponding to the image super-resolution processing method (for example, the acquisition module 21 and the processing module 22 in the image super-resolution processing device). The storage device 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the storage device 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 32 may further include a memory remotely located relative to the processor 31, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The present application also provides a storage medium storing a computer program. When the computer program is executed by a processor, the image super-resolution processing method described in the present application is implemented. The method includes:

[0115] Get the image to be processed;

[0116] The image to be processed is subjected to super-resolution processing by a target neural network model, wherein the target neural network model is obtained by iteratively training a pyramid neural network model with a target image sample set, and the target image samples are obtained by filtering original image samples according to a sample information content interval, and the number of image samples in each sample information content interval is the same.

[0117] The computer storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable media.Computer-readable media can be computer-readable signal media or computer-readable storage media.Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof.More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, portable CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof.Computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0118] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0119] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0120] The computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The above description is merely an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application.

[0122] It will be appreciated by those skilled in the art that the term user equipment encompasses any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device, a portable web browser or a car-mounted mobile station.

[0123] In general, various embodiments of the present application may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although the present application is not limited thereto.

[0124] Embodiments of the present application may be implemented by executing computer program instructions by a data processor of a mobile device, for example, in a processor entity, or by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0125] The block diagram of any logical flow in the drawings of the present application may represent program steps, or may represent interconnected logical circuits, modules and functions, or may represent a combination of program steps and logical circuits, modules and functions. A computer program may be stored on a memory. The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as, but not limited to, a read-only memory (ROM), a random access memory (RAM), an optical storage device and system (a digital versatile disc (DVD) or a compact disk (CD)). Computer-readable media may include non-transient storage media. A data processor may be of any type suitable for the local technical environment, such as, but not limited to, a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a processor based on a multi-core processor architecture.

[0126] The above description of exemplary embodiments of the present application has been provided by way of exemplary and non-limiting examples. However, various modifications and adaptations of the above embodiments will be apparent to those skilled in the art, when considered in conjunction with the accompanying drawings and the appended claims, without departing from the scope of the present application. Therefore, the proper scope of the present application will be determined by reference to the appended claims.

Claims

1. A method for image super-resolution processing, characterized in that: include: Get the image to be processed; Performing super-resolution processing on the image to be processed by a target neural network model, wherein the target neural network model is obtained by iteratively training a pyramid neural network with a target image sample set, wherein the target image samples are obtained by filtering original image samples according to sample information content intervals, and the number of image samples in each sample information content interval is the same; The sample information content includes: gradient information of the image sample or variance within the image sample; The gradient information represents the texture richness of the image sample, and the variance within the image sample represents the degree of variation of the image sample.

2. The method according to claim 1, characterized in that Iteratively training a pyramid neural network through a target image sample set includes: Build a pyramid neural network; Inputting the low-resolution image in the target sample set into the pyramid neural network to obtain a predicted image; Training parameters of the pyramid neural network according to an objective function formed by the predicted image and a high-resolution image corresponding to the low-resolution image, wherein the objective function includes one or more of a function L2+αL1, a function L1+θLP, and a function L2+μL1+βLP, wherein L2 is an L2 norm, L1 is an L1 norm, LP is an LP norm, and α, θ, μ, and β are weights of regularization factors; Return to executing the operation of inputting the low-resolution image in the target sample set into the pyramid neural network to obtain a predicted image until the target neural network model is obtained.

3. The method according to claim 2, characterized in that Training the parameters of the pyramid neural network according to the objective function formed by the predicted image and the high-resolution image corresponding to the low-resolution image includes: Training parameters of the pyramid neural network based on the objective function L2+αL1 and the learning rate set to a first value; After the decrease value of the output value of the objective function is less than the second data, the parameters of the pyramid neural network are continued to be trained based on the objective function L1+θLP and the learning rate set to the second value, wherein the second value is less than the first value, and α and θ increase as the learning rate decreases.

4. The method according to claim 2, characterized in that Training the parameters of the pyramid neural network according to the objective function formed by the predicted image and the high-resolution image corresponding to the low-resolution image includes: Training the parameters of the pyramid neural network according to the objective function L2+μL1+βLP formed by the predicted image and the high-resolution image corresponding to the low-resolution image; In each round of training, or between two rounds of training, μ and β are adjusted to achieve an approximation to the global optimum.

5. The method according to claim 1, wherein When the sample information content includes gradient information of the image sample, the sample content information is expressed as: PIC=||max(abs(Sobel(a i [x,y,w,h])))||1, where PIC represents the sample information content, x and y represent the horizontal and vertical coordinates of the starting point of the selected image sample, w and h represent the width and height of the selected image sample, and a i is an image sample.

6. The method according to claim 1, characterized in that When the sample information content includes the variance within the image sample, the sample content information is expressed as: PIC=||max(abs(Var(a i [x,y,w,h])))||1, where PIC represents the sample information content, x and y represent the horizontal and vertical coordinates of the starting point of the selected image sample, w and h represent the width and height of the selected image sample, and a i is an image sample.

7. The method according to claim 1, characterized in that The target image samples are obtained by filtering the original image samples according to the sample information content interval, and the number of image samples in each sample information content interval is the same, including: Dividing the sample information content space corresponding to the target image sample set to obtain at least two sample information content intervals, each sample information content interval corresponding to a different information content range, wherein each sample information content interval contains a target number of image samples; Obtaining the information content of image samples; Determining a corresponding sample information content interval according to the information content of the image sample; If the number of samples within the sample information content interval is less than the target number, the image sample is determined as a target image sample.

8. An image super-resolution processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; a processing module, configured to perform super-resolution processing on the image to be processed using a target neural network model, wherein the target neural network model is obtained by iteratively training a pyramid neural network with a target image sample set, wherein the target image samples are obtained by filtering original image samples according to sample information content intervals, and each sample information content interval has the same number of image samples; The sample information content includes: gradient information of the image sample or variance within the image sample; The gradient information represents the texture richness of the image sample, and the variance within the image sample represents the degree of variation of the image sample.

9. A device, characterized in that include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image super-resolution processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the image super-resolution processing method according to any one of claims 1 to 7 is implemented.

Images