Random-scale feature solution correlation image super-resolution reconstruction method and system

Abstract

The invention discloses a super-resolution reconstruction method and system for a feature solution correlation image of any scale. Performing data enhancement on the real super-resolution image ISR to obtain a low-resolution image ILR; inputting the ILR into a neural network g (.) for extracting features to obtain a feature map Z '= g (ILR); performing mean dimension reduction and random Fourier feature decorrelation on the Z'to obtain a decorrelation feature map Z; establishing a position projection module for super-resolution reconstruction and a weight prediction network # imgabs0 # to construct a loss function, performing optimization training on g (.) and the weight prediction network # imgabs1 # to obtain trained # imgabs2 # and weight prediction network # imgabs3 #, inputting a low-resolution image to be processed into a neural network # imgabs4 # by a user, and performing super-resolution reconstruction on the low-resolution image; and a super-resolution image is reconstructed by using a weight prediction network # imgabs5 # and a position projection module. According to the method, any-scale super-resolution reconstruction facing the real world can be realized, the requirement of any-scale super-resolution reconstruction is met, and essential features of the image are theoretically learned by adopting feature decorrelation.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and system for super-resolution reconstruction of feature decorrelation images of any scale. Background Art

[0002] In the fields of intelligent monitoring, medical diagnosis, industrial inspection, earth observation, deep space exploration, etc., obtaining high-resolution images is of great significance for subsequent work. The optical system, image sensor and shooting process of the camera may cause image degradation. Super-resolution technology is a technology for estimating high-resolution images from low-resolution images (or image sequences). Its main purpose is to improve the spatial resolution of the image.

[0003] The definition of image super-resolution is as follows: Given a low-resolution image I LR ∈R H×W×3 , the single image super-resolution algorithm aims to reconstruct the corresponding high-resolution image I SR ∈R sH×sW×3 , where H and W represent the length and width of the input image, and s represents the magnification. The purpose of the algorithm is to estimate the original high-resolution image. This process can be expressed as:

[0004] I SR =F(I LR )=F(D(I SR )),

[0005] Where f(·) represents the super-resolution algorithm, and D(·) represents the degradation process from the super-resolution image (SR) to the low-resolution image (LR), which usually includes blurring and downsampling.

[0006] Deep generative models have been successful in learning complex empirical distributions of images. Autoregressive models, variational autoencoders, normalized flows, and GANs show convincing image generation results and are beneficial for conditional tasks such as image super-resolution. However, existing techniques often have various limitations; autoregressive models are too expensive for high-resolution image generation, normalized flows and variational autoencoders often produce suboptimal sample quality, and GANs require carefully designed regularization and optimization techniques to tame optimization instability and mode collapse. Most current super-resolution reconstruction research mainly relies on downsampling at low resolution to obtain ultra-low resolution images for supervised training. For single image super-resolution, although supervised methods are impressive, they lack generalization ability on out-of-distribution (OOD) data, that is, the super-resolution reconstruction effect of natural images with unknown degradation is poor. This is due to the deviation between the training data scene and the real data distribution, that is, the training data and the test data do not meet the assumption of independent and identical distribution. In addition, due to the integer multiple downsampling at low resolution, the trained model can only perform integer factor SR reconstruction, and cannot perform arbitrary scale factor reconstruction, which cannot meet the actual needs.

[0007] One of the current existing technologies is the improved image restoration model SwinIR based on the Swin Transformer model in the paper "Swinir: Image restoration using swintransformer". SwinIR consists of three modules: shallow feature extraction, deep feature extraction, and high-quality image reconstruction module. The shallow feature extraction module uses a convolutional layer to extract shallow features and directly transmits them to the reconstruction module to retain low-frequency information. The deep feature extraction module is mainly composed of residual Swin Transformer blocks (RSTBs), each of which uses multiple Swin Transformer layers for local attention and cross-window interaction. In addition, a convolutional layer is added at the end of the block for feature enhancement, and a residual connection is used to provide a shortcut for feature aggregation. Finally, the shallow and deep features are fused in the reconstruction module for high-quality image reconstruction. The disadvantages of this technology are: 1. The work of SwinIR mainly explores the fixed factor scaling problem of image super-resolution based on the Swin Transformer model, using the standard L1 or L2 loss as the learning target to deterministically map the LR image to the HR image. SwinIR aims to learn the mapping function from a given LR image to a fixed-scale HR image. Although it performs well in terms of peak signal-to-noise ratio, the super-resolution reconstruction of fixed factors is difficult to meet practical needs. 2. The causes of degradation of low-resolution images in the real world are often complex. Pictures in real scenes often suffer from a variety of blurry conditions or noise of varying degrees. Therefore, when designing an image super-resolution network, it is necessary not only to consider the problem of arbitrary scaling factors, but also to consider different blur degradations and noise. 3. For single image super-resolution, although supervised methods are impressive, they lack generalization capabilities on out-of-distribution (OOD) data, that is, the super-resolution reconstruction of natural images with unknown degradation levels is not effective. This is due to the deviation between the training data scene and the real data distribution, that is, the training data and the test data do not meet the assumption of independent and identical distribution. When faced with practical applications, these methods lack scalability. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the existing methods and propose a method and system for super-resolution reconstruction of arbitrary-scale feature decorrelation images. The main problems solved by the present invention are: first, how to use a single model to achieve super-resolution reconstruction of arbitrary scale factors (including non-integer scale factors); second, how to take both blur and noise into account in the super-resolution task; and third, how to use random Fourier features for decorrelation.

[0009] In order to solve the above problems, the present invention proposes a method for super-resolution reconstruction of feature decorrelation images of any scale, the method comprising:

[0010] On the real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , as the basis for model training;

[0011] The low-resolution image dataset I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR );

[0012] Performing mean dimension reduction and random Fourier feature decorrelation on the feature map dataset Z′ to obtain a decorrelation feature map dataset Z;

[0013] Establishing a position projection module and weight prediction network for super-resolution reconstruction The position projection module is responsible for feature amplification of the decorrelation feature map, that is, the pixel value of the pixel point (i, j) on the super-resolution image is determined by the decorrelation feature vector Z(i′, j′) of the pixel point (i′, j′) on the low-resolution image; the weight prediction network Responsible for predicting the weight W(i, j) of the filter corresponding to the point (i, j) on the super-resolution image; the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image;

[0014] The decorrelation feature map data set Z is used to perform feature amplification according to the position projection module, and combined with the weight prediction network The calculated filter weights generate a reconstructed super-resolution image dataset, and then based on the reconstructed super-resolution image dataset and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training to obtain the trained neural network for feature extraction With weight prediction network

[0015] User input of low-resolution image to be processed To the trained feature extraction neural network And after the mean dimension reduction and random Fourier feature decorrelation operation, the decorrelation feature map Z is obtained. target , and then use the trained weight prediction network Calculate the filter weight of each point; decorrelation feature map Z targetThe position projection module is used to amplify the features and reconstruct the super-resolution image by combining the filter weights of each point.

[0016] Preferably, the real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , specifically:

[0017] Real Super-Resolution Image Dataset I SR , after scaling, we get a low-resolution image dataset Afterwards Degradation treatment:

[0018]

[0019] Among them, the blur kernel Blur is The fuzzy processing is performed, and the noise is added to the fuzzy result. The new low-resolution image dataset I is obtained by image degradation operation. LR .

[0020] Preferably, the low-resolution image dataset I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR ), where Z′∈R b×c×h×w , b, c, h, w are the batch size, number of channels, height and width respectively.

[0021] Preferably, the feature map dataset Z′ is subjected to mean dimension reduction and random Fourier feature decorrelation to obtain the decorrelated feature map dataset Z, specifically:

[0022] Each channel of the feature map dataset Z′ represents a homogeneous feature. Each channel is regarded as a random variable. In order to remove the correlation between any two variables in the representation space and make the variables independent, the mean of each channel in Z′ is first taken to obtain the reduced dimension feature map dataset Z″, whose dimension becomes b×c, that is, Z″∈R b×c ;

[0023] After obtaining the reduced-dimensional feature graph dataset Z″, in order to make any two random variables Z″ in Z″ ：，i and Z″ ：，j Independently, the high-dimensional nonlinear feature information in Z″ is represented by a low-dimensional explicit linear representation using random Fourier feature transform RFF, so that the linear statistical correlation can be calculated; statistically, the random variable Z″ ：，i and Z″ ：，j Relevance The calculation formula is:

[0024]

[0025] Among them, Cov(Z″ ：，i , Z″ ：，j ) is the covariance, is the random variable Z″ ：，i The standard deviation of is the random variable Z″ ：，j The standard deviation of When it is equal to 0, the two variables are unrelated; since the standard deviation will not be less than zero, only the denominator covariance needs to be considered;

[0026] The formula for covariance is:

[0027] Cov(Z″ ：，i , Z″ ：，j )=E{[Z″ ：，i -E(Z″ ：，i )][Z″ ：，j -E(Z″ ：，j )]},

[0028] Where E represents the expectation; according to the Hilbert-Schmidt independence criterion, When , the variables are uncorrelated. is the Frobenius norm; let u and v be the random Fourier feature mapping function RFF respectively, then the above formula can be rewritten as:

[0029] Cov(u(Z″ ：，i ), v(Z″ ：，j ))=E{[u(Z″ ：，i )-E(u(Z″ ：，i ))][v(Z″ ：，j )-E(v(Z″ ：，j ))]},

[0030] Since the Gaussian kernel is a positive definite kernel, The value of is greater than or equal to 0; Recorded as when When it gradually decreases to zero, the random variable Z″ ：，i and Z″ ：，j The correlation is removed, and a decorrelated feature map dataset Z is obtained.

[0031] Preferably, the position projection module further includes:

[0032] Position projection uses the following projection operator to map the pixel point (i, j) on the super-resolution image and the pixel point (i′, j′) on the low-resolution image:

[0033]

[0034] Where T represents the mapping function, Represents the rounding down operation, s represents the scaling factor; by using position projection and convolution operation, feature map amplification with any scaling factor can be achieved.

[0035] Preferably, the weight prediction network Also includes:

[0036] Through the neural network, the corresponding number of filter weights is predicted for any scaling factor, expressed as:

[0037]

[0038] Where W(i, j) is the filter weight corresponding to the pixel (i, j) on the super-resolution image. is the weight prediction network, v ijs is the vector associated with i, j, s, and is also the weight prediction network The input of θ1 is the parameter of the weight prediction network; the weight prediction network It consists of a fully connected layer, a ReLU layer, and a fully connected layer;

[0039] The weight of each pixel is related to the coordinates of the point and the scaling factor, so the input v of the weight prediction network is ijs Related to the coordinates and scaling factors, in addition, in order to train multiple scaling factors s at the same time, the scaling factor is added to v ijs To distinguish the weights of different scaling factors, v ijs Designed for:

[0040]

[0041] Preferably, the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image, specifically:

[0042] The super-resolution reconstruction process is formulated as:

[0043]

[0044] in, represents the pixel value of pixel point (i, j) on the reconstructed super-resolution image, Φ represents the mapping function, Z(i′, j′) is the feature vector at position point (i′, j′) on the feature map, and W(i, j) is the filter weight corresponding to pixel point (i, j).

[0045] Preferably, the super-resolution image dataset based on the reconstruction is consistent with the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training, specifically:

[0046] The loss function is:

[0047]

[0048] Where n is the image dataset I SR The amount of data, θ2 is the parameter of the network g(·), and are the predicted super-resolution image and the actual super-resolution image of the i-th model, and α is the decorrelation weight.

[0049] Accordingly, the present invention also provides a system for super-resolution reconstruction of feature decorrelation images of any scale, comprising:

[0050] Data enhancement unit, used to enhance the real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , as the basis for model training;

[0051] A decorrelation feature map generating unit is used to convert the low-resolution image data set I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR ); performing mean dimension reduction and random Fourier feature decorrelation on the feature map dataset Z′ to obtain a decorrelation feature map dataset Z;

[0052] Position projection and weight prediction unit, used to establish position projection module and weight prediction network for super-resolution reconstruction The position projection module is responsible for feature amplification of the decorrelation feature map, that is, the pixel value of the pixel point (i, j) on the super-resolution image is determined by the decorrelation feature vector Z(i′, j′) of the pixel point (i′, j′) on the low-resolution image; the weight prediction network Responsible for predicting the weight W(i, j) of the filter corresponding to the point (i, j) on the super-resolution image; the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image;

[0053] A network training unit is used to use the decorrelation feature map data set Z to perform feature amplification according to the position projection module, and combine it with the weight prediction network The calculated filter weights generate a reconstructed super-resolution image dataset, and then based on the reconstructed super-resolution image dataset and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training to obtain the trained neural network for feature extraction With weight prediction network

[0054] Image reconstruction unit, used for user input of low-resolution images to be processed To the trained feature extraction neural network And after the mean dimension reduction and random Fourier feature decorrelation operation, the decorrelation feature map Z is obtained. target , and then use the trained weight prediction network Calculate the filter weight of each point; decorrelation feature map Z target The position projection module is used to amplify the features and reconstruct the super-resolution image by combining the filter weights of each point.

[0055] The implementation of the present invention has the following beneficial effects:

[0056] The present invention can realize super-resolution reconstruction of any scale in the real world, and meets the requirements of super-resolution reconstruction of any scale. The present invention adopts feature decorrelation, which theoretically guarantees that the essential features of the image are learned, thereby achieving robustness across data domains, so the present invention is suitable for different application scenarios. The present invention has high intelligence and adopts a dimensionality reduction method to achieve high computational efficiency, convenience and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 1 is a schematic diagram of the overall framework of a method for super-resolution reconstruction of feature decorrelation images of any scale according to an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of feature map magnification by position projection according to an embodiment of the present invention;

[0059] Figure 3 It is a structural diagram of a system for super-resolution reconstruction of feature decorrelation images of any scale according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Figure 1 is a schematic diagram of the overall framework of a method for super-resolution reconstruction of feature decorrelation images of any scale according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0062] S1, for real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , as the basis for model training;

[0063] S2, the low-resolution image dataset I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR );

[0064] S3, performing mean dimension reduction and random Fourier feature decorrelation on the feature map dataset Z′ to obtain a decorrelation feature map dataset Z;

[0065] S4, establish the position projection module and weight prediction network for super-resolution reconstruction The position projection module is responsible for feature amplification of the decorrelation feature map, that is, the pixel value of the pixel point (i, j) on the super-resolution image is determined by the decorrelation feature vector Z(i′, j′) of the pixel point (i′, j′) on the low-resolution image; the weight prediction network Responsible for predicting the weight W(i, j) of the filter corresponding to the point (i, j) on the super-resolution image; the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image;

[0066] S5, using the decorrelation feature map data set Z to perform feature amplification according to the position projection module, and combining it with the weight prediction network The calculated filter weights generate a reconstructed super-resolution image dataset, and then based on the reconstructed super-resolution image dataset and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training to obtain the trained neural network for feature extraction With weight prediction network

[0067] S6, the user inputs the low-resolution image to be processed To the trained feature extraction neural network And after the mean dimension reduction and random Fourier feature decorrelation operation, the decorrelation feature map Z is obtained. target , and then use the trained weight prediction network Calculate the filter weight of each point; decorrelation feature map Z target The position projection module is used to amplify the features and reconstruct the super-resolution image by combining the filter weights of each point.

[0068] Step S1 is as follows:

[0069] Real Super-Resolution Image Dataset I SR , after scaling, we get a low-resolution image dataset Afterwards Degradation treatment:

[0070]

[0071] Among them, the blur kernel Blur is The fuzzy processing is performed, and the noise is added to the fuzzy result. The new low-resolution image dataset I is obtained by image degradation operation. LR The Blur used in this embodiment is a Gaussian blur kernel, and the added noise is Gaussian noise.

[0072] Step S2 is as follows:

[0073] The low-resolution image dataset I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR ), where Z′∈R b×c×h×w , b, c, h, w are the batch size, number of channels, height and width respectively.

[0074] Step S3 is as follows:

[0075] Recent out-of-distribution generalization studies have shown that subtle statistical correlations between features are the main reason for model collapse under distribution drift. The correlations between these features may confuse the analytical ability of deep models, so feature decorrelation is needed. However, the features extracted by neural networks are highly nonlinear, and Z′∈R b×c×h×wIt is a four-dimensional feature map and cannot be operated according to the traditional statistical decorrelation method. Therefore, the present invention uses mean dimensionality reduction and random Fourier to perform feature decorrelation.

[0076] S3-1, each channel of the feature map dataset Z′ represents a homogeneous feature. Each channel is regarded as a random variable. In order to remove the correlation between any two variables in the representation space and make the variables independent, the mean of each channel in Z′ is first taken to obtain the reduced dimension feature map dataset Z″, whose dimension becomes b×c, that is, Z″∈R b×c .

[0077] S3-2, after obtaining the reduced-dimensional feature graph dataset Z″, in order to make any two random variables Z″ in Z″ ：，i and Z″ ：，j Independently, the high-dimensional nonlinear feature information in Z″ is represented by a low-dimensional explicit linear representation using random Fourier feature transform RFF, so that the linear statistical correlation can be calculated; statistically, the random variable Z″ ：，i and Z″ ：，j Relevance The calculation formula is:

[0078]

[0079] Among them, Cov(Z″ ：，i , Z″ ：，j ) is the covariance, is the random variable Z″ ：，i The standard deviation of is the random variable Z″ ：，j The standard deviation of When it is equal to 0, the two variables are unrelated; since the standard deviation will not be less than zero, only the denominator covariance needs to be considered;

[0080] The formula for covariance is:

[0081] Cov(Z″ ：，i , Z″ ：，j )=E{[Z″ ：，i -E(Z″ ：，i )][Z″ ：，j -E(Z″ ：，j )]},

[0082] Where E represents the expectation; according to the Hilbert-Schmidt independence criterion, When , the variables are uncorrelated. is the Frobenius norm; let u and v be the random Fourier feature mapping function RFF respectively, then the above formula can be rewritten as:

[0083] Cov(u(Z″ ：，i), v(Z″ ：，j ))=E{[u(Z″ ：，i )-E(u(z″ ：，i ))][v(Z″ ：，j )-E(v(Z″ ：，j ))]},

[0084] Since the Gaussian kernel is a positive definite kernel, The value of is greater than or equal to 0; Recorded as when When it gradually decreases to zero, the random variable Z″ ：，i and Z″ ：，j The correlation is removed, and a decorrelated feature map dataset Z is obtained.

[0085] In specific implementation, the specific calculation of covariance is as follows:

[0086] From the random Fourier feature map u(Z″) ：，i ), v(Z″ ：，j ) distribution, and we get

[0087]

[0088]

[0089] In this embodiment, m=5 is set;

[0090] Therefore, the specific calculation formula of covariance is as follows:

[0091]

[0092] Where n is the image dataset I SR The amount of data.

[0093] Step S4 is as follows:

[0094] S4-1, the position projection module is responsible for feature amplification of the decorrelation feature map, that is, the pixel value of the pixel point (i, j) on the super-resolution image is determined by the decorrelation feature vector Z(i′, j′) of the pixel point (i′, j′) on the low-resolution image;

[0095] Position projection uses the following projection operator to map the pixel point (i, j) on the super-resolution image and the pixel point (i′, j′) on the low-resolution image:

[0096]

[0097] Where T represents the mapping function, represents the rounding down operation, s represents the scaling factor; by using position projection and convolution operation, feature map amplification with any scaling factor can be achieved;

[0098] Figure 2 The use of the position projection module to magnify the feature map is demonstrated. In this embodiment, a one-dimensional situation is drawn as a demonstration, and the scaling factor s in the example is 1.5.

[0099] S4-2, weight prediction network Responsible for predicting the weight W(i, j) of the filter corresponding to the point (i, j) on the super-resolution image;

[0100] Through the neural network, the corresponding number of filter weights is predicted for any scaling factor, expressed as:

[0101]

[0102] Where W(i, j) is the filter weight corresponding to the pixel (i, j) on the super-resolution image. is the weight prediction network, v ijs is the vector associated with i, j, s, and is also the weight prediction network The input of θ1 is the parameter of the weight prediction network; the weight prediction network It consists of a fully connected layer, a ReLU layer, and a fully connected layer;

[0103] The weight of each pixel is related to the coordinates of the point and the scaling factor, so the input v of the weight prediction network is ijs Related to the coordinates and scaling factors, in addition, in order to train multiple scaling factors s at the same time, the scaling factor is added to v ijs To distinguish the weights of different scaling factors, v ijs Designed for:

[0104]

[0105] S4-3, the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image;

[0106] The super-resolution reconstruction process is formulated as:

[0107]

[0108] in, represents the pixel value of pixel point (i, j) on the reconstructed super-resolution image, Φ represents the mapping function, Z(i′, j′) is the feature vector at position point (i′, j′) on the feature map, and W(i, j) is the filter weight corresponding to pixel point (i, j).

[0109] Step S5 is as follows:

[0110] Based on the reconstructed super-resolution image dataset and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network For optimization training, the loss function is:

[0111]

[0112] Where n is the image dataset I SR The amount of data, θ2 is the parameter of the network g(·), and are respectively the predicted super-resolution image and the actual super-resolution image of the ith model, and α is the decorrelation weight, which is set to 0.5 in this embodiment.

[0113] Accordingly, the present invention also provides a super-resolution reconstruction system for feature decorrelation images of any scale, such as Figure 3 As shown, including:

[0114] Data enhancement unit 1 is used to enhance the real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , as the basis for model training;

[0115] The decorrelation feature map generating unit 2 is used to convert the low-resolution image data set I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR ); performing mean dimension reduction and random Fourier feature decorrelation on the feature map dataset Z′ to obtain a decorrelation feature map dataset Z:

[0116] Position projection and weight prediction unit 3, used to establish a position projection module and weight prediction network for super-resolution reconstruction The position projection module is responsible for feature amplification of the decorrelation feature map, that is, the pixel value of the pixel point (i, j) on the super-resolution image is determined by the decorrelation feature vector Z(i′, j′) of the pixel point (i′, j′) on the low-resolution image; the weight prediction network Responsible for predicting the weight W(i, j) of the filter corresponding to the point (i, j) on the super-resolution image; the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image;

[0117] The network training unit 4 is used to use the decorrelation feature map data set Z to perform feature amplification according to the position projection module, and combine it with the weight prediction network The calculated filter weights generate a reconstructed super-resolution image dataset, and then based on the reconstructed super-resolution image dataset and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training to obtain the trained neural network for feature extraction With weight prediction network

[0118] Image reconstruction unit 5, used for user input of low-resolution image to be processed To the trained feature extraction neural network And after the mean dimension reduction and random Fourier feature decorrelation operation, the decorrelation feature map Z is obtained. target , and then use the trained weight prediction network Calculate the filter weight of each point; decorrelation feature map Z target The position projection module is used to amplify the features and reconstruct the super-resolution image by combining the filter weights of each point.

[0119] Therefore, the present invention can realize super-resolution reconstruction of any scale in the real world, and meet the requirements of super-resolution reconstruction of any scale. The present invention adopts feature decorrelation, which theoretically guarantees that the essential features of the image are learned, thereby achieving robustness across data domains, so the present invention is suitable for different application scenarios. The present invention has high intelligence and adopts a dimensionality reduction method to achieve high computational efficiency, convenience and efficiency.

[0120] The above is a detailed introduction to a method and system for super-resolution reconstruction of feature decorrelation images of any scale provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for super-resolution reconstruction of feature decorrelation images of any scale, characterized in that: The method comprises: On the real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , as the basis for model training; The low-resolution image dataset I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR ); Performing mean dimension reduction and random Fourier feature decorrelation on the feature map dataset Z′ to obtain a decorrelation feature map dataset Z; Establishing a position projection module and weight prediction network for super-resolution reconstruction The position projection module is responsible for feature amplification of the decorrelation feature map, that is, the pixel value of the pixel point (i, j) on the super-resolution image is determined by the decorrelation feature vector Z(i′, j′) of the pixel point (i′, j′) on the low-resolution image; the weight prediction network Responsible for predicting the weight W(i, j) of the filter corresponding to the point (i, j) on the super-resolution image; the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image; The decorrelation feature map data set Z is used to perform feature amplification according to the position projection module, and combined with the weight prediction network The calculated filter weights generate a reconstructed super-resolution image dataset, and then based on the reconstructed super-resolution image dataset and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training to obtain the trained neural network for feature extraction With weight prediction network User input of low-resolution image to be processed To the trained feature extraction neural network And after the mean dimension reduction and random Fourier feature decorrelation operation, the decorrelation feature map Z is obtained. target , and then use the trained weight prediction network Calculate the filter weight of each point; decorrelation feature map Z target The position projection module is used to amplify the features and reconstruct the super-resolution image by combining the filter weights of each point.

2. The method for super-resolution reconstruction of feature decorrelation images of any scale as claimed in claim 1, characterized in that: The real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , specifically: Real Super-Resolution Image Dataset I SR , after scaling, we get a low-resolution image dataset Afterwards Degradation treatment: Among them, the blur kernel Blur is The fuzzy processing is performed, and the noise is added to the fuzzy result. The new low-resolution image dataset I is obtained by image degradation operation. LR .

3. The method for super-resolution reconstruction of feature decorrelation images of any scale as claimed in claim 1, characterized in that: The low-resolution image dataset I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR ), where Z′∈R b×c×h×w , b, c, h, w are the batch size, number of channels, height and width respectively.

4. The method for super-resolution reconstruction of feature decorrelation images of any scale as claimed in claim 1, characterized in that: The feature map dataset Z′ is subjected to mean dimension reduction and random Fourier feature decorrelation to obtain the decorrelated feature map dataset Z, specifically: Each channel of the feature map dataset Z′ represents a homogeneous feature. Each channel is regarded as a random variable. In order to remove the correlation between any two variables in the representation space and make the variables independent, the mean of each channel in Z′ is first taken to obtain the reduced dimension feature map dataset Z″, whose dimension becomes b×c, that is, Z″∈R b×c ; After obtaining the reduced-dimensional feature graph dataset Z″, in order to make any two random variables Z″ in Z″ ：，i and Z″ ：，j Independently, the high-dimensional nonlinear feature information in Z″ is represented by a low-dimensional explicit linear representation using random Fourier feature transform RFF, so that the linear statistical correlation can be calculated; statistically, the random variable Z″ ：，i and Z″ ：，j Correlation ρ Z″：，i，Z″：，j The calculation formula is: Among them, Cov(Z″ ：，i , Z″ ：，j ) is the covariance, is the random variable Z″ ：，i The standard deviation of is the random variable Z″ ：，j The standard deviation of When it is equal to 0, the two variables are unrelated; since the standard deviation will not be less than zero, only the denominator covariance needs to be considered; The formula for covariance is: Cov(Z″ ：，i ,Z″ ：，j )=E{[Z″ ：，i -E(Z″ ：，i )][Z″ ：，j -E(Z″ ：，j )]}, Where E represents the expectation; according to the Hilbert-Schmidt independence criterion, When , the variables are uncorrelated. is the Frobenius norm; let u and v be the random Fourier feature mapping function RFF respectively, then the above formula can be rewritten as: Cov(u(Z″ ：，i ),v(Z″ ：，j ))=E{[u(Z″ ：，i )-E(u(Z″ ：，i ))][v(Z″ ：，j )-E(v(Z″ ：，j ))]}, Since the Gaussian kernel is a positive definite kernel, The value of is greater than or equal to 0; Recorded as when When it gradually decreases to zero, the random variable Z″ ：，i and Z″ ：，j The correlation is removed, and a decorrelated feature map dataset Z is obtained.

5. The method for super-resolution reconstruction of feature decorrelation images of any scale as claimed in claim 1, characterized in that: The position projection module further includes: Position projection uses the following projection operator to map the pixel point (i, j) on the super-resolution image and the pixel point (i′, j′) on the low-resolution image: Where T represents the mapping function, Represents the rounding down operation, s represents the scaling factor; by using position projection and convolution operation, feature map amplification with any scaling factor can be achieved.

6. The method for super-resolution reconstruction of feature decorrelation images of any scale as claimed in claim 5, characterized in that: The weight prediction network Also includes: Through the neural network, the corresponding number of filter weights is predicted for any scaling factor, expressed as: Where W(i, j) is the filter weight corresponding to the pixel (i, j) on the super-resolution image. is the weight prediction network, v ijs is the vector associated with i, j, s, and is also the weight prediction network The input of θ1 is the parameter of the weight prediction network; the weight prediction network It consists of a fully connected layer, a ReLU layer, and a fully connected layer; The weight of each pixel is related to the coordinates of the point and the scaling factor, so the input v of the weight prediction network is ijs Related to the coordinates and scaling factors, in addition, in order to train multiple scaling factors s at the same time, the scaling factor is added to v ijs To distinguish the weights of different scaling factors, v ijs Designed for:

7. The method for super-resolution reconstruction of feature decorrelation images of any scale as claimed in claim 1, characterized in that: The pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image, specifically: The super-resolution reconstruction process is formulated as: in, represents the pixel value of pixel point (i, j) on the reconstructed super-resolution image, Φ represents the mapping function, Z(i′, j′) is the feature vector at position point (i′, j′) on the feature map, and W(i, j) is the filter weight corresponding to pixel point (i, j).

8. The method for super-resolution reconstruction of feature decorrelation images of any scale according to claim 4 or 7, characterized in that: The super-resolution image dataset based on the reconstruction and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training, specifically: The loss function is: Where n is the image dataset I SR The amount of data, θ2 is the parameter of the network g(·), and are the predicted super-resolution image and the actual super-resolution image of the i-th model, and α is the decorrelation weight.

9. A super-resolution reconstruction system for feature decorrelation images of any scale, characterized in that: The system comprises: Data enhancement unit, used to enhance the real super-resolution image dataset I SR Perform data enhancement by adding blur and noise to obtain a low-resolution image dataset I LR , as the basis for model training; A decorrelation feature map generating unit is used to convert the low-resolution image data set I LR Input it into the neural network g(·) for feature extraction, and obtain the feature map dataset Z′=g(I LR ); performing mean dimension reduction and random Fourier feature decorrelation on the feature map dataset Z′ to obtain a decorrelation feature map dataset Z; Position projection and weight prediction unit, used to establish a position projection module and weight prediction network for super-resolution reconstruction The position projection module is responsible for feature amplification of the decorrelation feature map, that is, the pixel value of the pixel point (i, j) on the super-resolution image is determined by the decorrelation feature vector Z(i′, j′) of the pixel point (i′, j′) on the low-resolution image; the weight prediction network Responsible for predicting the weight W(i, j) of the filter corresponding to the point (i, j) on the super-resolution image; the pixel point (i, j) on the super-resolution image to be predicted is determined by the filter weight W(i, j) corresponding to the point and the decorrelation feature vector Z(i′, j′) of the pixel point on the low-resolution image; A network training unit is used to use the decorrelation feature map data set Z to perform feature amplification according to the position projection module, and combine it with the weight prediction network The calculated filter weights generate a reconstructed super-resolution image dataset, and then based on the reconstructed super-resolution image dataset and the real super-resolution image dataset I SR By comparing and constructing a loss function, the neural network g(·) for extracting features and the weight prediction network Perform optimization training to obtain the trained neural network for feature extraction With weight prediction network Image reconstruction unit, used for user input of low-resolution images to be processed To the trained feature extraction neural network And after the mean dimension reduction and random Fourier feature decorrelation operation, the decorrelation feature map Z is obtained. target , and then use the trained weight prediction network Calculate the filter weight of each point; decorrelation feature map Z target The position projection module is used to amplify the features and reconstruct the super-resolution image by combining the filter weights of each point.

Images