Blind image super-resolution model training method, blind image super-resolution method, electronic equipment, storage medium and program product

Abstract

The invention provides a blind image super-resolution model training method, a blind image super-resolution method, electronic equipment, a storage medium and a program product. The blind image super-resolution model comprises a degradation feature estimator used for determining degradation features of a blind image, an image feature correction module used for correcting image features of the blind image according to the degradation features, and an image reconstruction module used for reconstructing a high-definition image based on the corrected image features. The training method comprises the following steps: performing first-stage training on a to-be-trained degradation feature estimator by adopting a contrast learning method based on degradation prior constraint to obtain a first degradation feature estimator; performing second-stage training on the blind image super-resolution model containing the first degradation feature estimator to obtain a teacher model; and taking the teacher model as a learning object, and carrying out third-stage training by adopting a distillation learning method to obtain a final blind image super-resolution model. According to the method, a low-complexity model can be obtained, and the method is suitable for low-computing-power equipment.

Description

Technical Field

[0001] The present disclosure generally relates to the field of image processing technology, and more specifically, to a training method for a blind image super-resolution model, a blind image super-resolution method, an electronic device, a storage medium, and a program product. Background Art

[0002] Blind Super-Resolution (BSR) is an important image processing task. It mainly refers to the process of restoring a low-resolution image (LR) to a high-resolution image (HR) without knowing or being sure about the specific image degradation process (such as blur kernel, noise level, downsampling operation, etc.). It has been widely used in remote sensing, video surveillance, and medical imaging. At the same time, blind image super-resolution can also be used as a pre-operation for many advanced visual tasks, such as image retrieval, object detection, scene analysis, etc. The challenge faced by the BSR task is the unknown nature of the degradation process, because the image degradation in the real world is often complex and diverse. Therefore, how to accurately estimate the degradation information and how to use the degradation information to guide the reconstruction process are key issues in the BSR field.

[0003] Early BSR methods used an explicit degradation parameter estimation strategy, which used a complex explicit degradation estimation network to accurately estimate the degradation information of the LR image and used it as the guidance information of the BSR network. However, this explicit degradation parameter estimation strategy still cannot handle the complex degradation in real scenes well, including noise interference and multiple degradation interactions. Therefore, the BSR field has gradually begun to study implicit degradation parameter estimation strategies and how to use implicit degradation features to guide the reconstruction of LR images to HR images.

[0004] However, the existing BSR methods based on implicit degradation feature guidance have increasingly higher model complexity, often exceeding 10M, and the amount of computation is also large, often exceeding 600 GFLOPs (Giga Floating-Point Operations per Second), which greatly limits the application of these methods on low-computing power devices. Summary of the invention

[0005] The present disclosure provides a training method for a blind image super-resolution model, a blind image super-resolution method, an electronic device, a storage medium and a program product, which are used to solve at least one of the above problems.

[0006] According to a first aspect of an embodiment of the present disclosure, a training method for a blind image super-resolution model is provided, wherein the blind image super-resolution model comprises a degradation feature estimator for determining degradation features of a blind image, an image feature correction module for correcting the image features of the blind image according to the degradation features, and an image reconstruction module for reconstructing a high-definition image based on the corrected image features, wherein the training method comprises: determining a first-stage training sample based on a first sample blind image, performing a first-stage training on the degradation feature estimator to be trained using a contrast learning method based on degradation prior constraints to obtain a first degradation feature estimator; determining a second-stage training sample based on a second sample blind image, performing a second-stage training on the blind image super-resolution model including the first degradation feature estimator to obtain a teacher model; determining a third-stage training sample based on a third sample blind image, taking the teacher model as a learning object, performing a third-stage training using a distillation learning method to obtain a final blind image super-resolution model.

[0007] Optionally, the first-stage training samples are determined based on the first sample blind image, and a contrastive learning method based on degradation prior constraints is used to perform the first-stage training on the degradation feature estimator to be trained to obtain a first degradation feature estimator, including: constructing a trunk branch and a momentum branch, wherein the trunk branch and the momentum branch both include degradation feature estimators; performing degradation prior constraint processing on the first sample blind image to obtain a training blind image; using the training blind image as a training sample, and using a contrastive learning method to train the trunk branch and the momentum branch, and using the degradation feature estimator in the trunk branch obtained through training as the first degradation feature estimator.

[0008] Optionally, performing degradation prior constraint processing on the first sample blind image to obtain a training blind image includes: performing dimension extension processing on the blur kernel and noise information in the first sample blind image to obtain a degradation prior having the same spatial dimension as the first sample blind image; and splicing the degradation prior and the first sample blind image along the channel dimension to obtain the training blind image.

[0009] Optionally, the method uses the training blind image as a training sample, adopts a contrastive learning method, trains the trunk branch and the momentum branch, and uses the degradation feature estimator in the trunk branch obtained through training as the first degradation feature estimator, including: performing the following iterative calculation steps on the trunk branch and the momentum branch until a stop condition is met, and using the degradation feature estimator in the latest trunk branch as the first degradation feature estimator, wherein the number of the training blind images is multiple, and the training blind images used in each round of calculation are different: inputting the training blind images calculated in this round into the trunk branch and the momentum branch respectively to obtain trunk features and momentum features; obtaining reference features, wherein the reference features are features obtained by inputting images different from the training blind images calculated in this round into the trunk branch and / or the momentum branch; determining a contrast loss value based on the trunk features, the momentum features and the reference features; adjusting the parameters of the trunk branch based on the contrast loss value, and adjusting the parameters of the momentum branch based on the parameters of the trunk branch using a momentum update strategy of contrastive learning.

[0010] Optionally, the image feature correction module includes a degradation feature converter and a degradation feature adaptation module, wherein the degradation feature converter includes a channel domain conversion branch and a space domain conversion branch, the channel domain conversion branch is used to convert the degradation feature into a channel domain degradation feature, and the space domain conversion branch is used to convert the degradation feature into a space domain degradation feature. The degradation feature adaptation module includes multiple degradation feature adaptation groups, each degradation feature adaptation group includes multiple degradation feature adaptation units, and each degradation feature adaptation unit includes a feature segmenter, a channel domain adaptation branch, a space domain adaptation branch, and a feature fusion device. The feature segmenter is used to segment the image feature input therein into a first image feature and a second image feature, the channel domain adaptation branch is used to integrate the channel domain degradation feature into the first image feature to obtain a first corrected image feature, the space domain adaptation branch is used to integrate the space domain degradation feature into the second image feature to obtain a second corrected image feature, and the feature fusion device is used to fuse the first corrected image feature and the second corrected image feature into a new image feature.

[0011] Optionally, the third-stage training samples are determined based on the third sample blind image, the teacher model is used as a learning object, and a distillation learning method is used to perform the third-stage training to obtain a final blind image super-resolution model, including: copying the teacher model to obtain a student model; using the third sample blind image as a training sample, and using a distillation learning method to perform model learning with the purpose of aligning the output of the degraded feature converter in the teacher model and the student model, and using the learned student model as the final blind image super-resolution model.

[0012] According to a second aspect of an embodiment of the present disclosure, a blind image super-resolution method is provided, comprising: using a degradation feature estimator in a blind image super-resolution model to process a blind image to obtain degradation features of the blind image, wherein the blind image super-resolution model also includes an image feature correction module and an image reconstruction module; using the image feature correction module to correct the image features of the blind image according to the degradation features to obtain corrected image features; using the image reconstruction module to reconstruct a high-definition image corresponding to the blind image based on the corrected image features, wherein the blind image super-resolution model is trained by the training method of the blind image super-resolution model according to the exemplary embodiment of the present disclosure.

[0013] According to a third aspect of an embodiment of the present disclosure, a training device for a blind image super-resolution model is provided, wherein the blind image super-resolution model includes a degradation feature estimator for determining degradation features of a blind image, an image feature correction module for correcting the image features of the blind image according to the degradation features, and an image reconstruction module for reconstructing a high-definition image based on the corrected image features, wherein the training device includes: a first training unit, configured to determine a first-stage training sample based on a first sample blind image, and perform a first-stage training on the degradation feature estimator to be trained using a contrast learning method based on degradation prior constraints to obtain a first degradation feature estimator; a second training unit, configured to determine a second-stage training sample based on a second sample blind image, and perform a second-stage training on the blind image super-resolution model including the first degradation feature estimator to obtain a teacher model; and a third training unit, configured to determine a third-stage training sample based on a third sample blind image, and use the teacher model as a learning object to perform a third-stage training using a distillation learning method to obtain a final blind image super-resolution model.

[0014] According to a fourth aspect of an embodiment of the present disclosure, a blind image super-resolution device is provided, comprising: a degradation estimation unit, configured to use a degradation feature estimator in a blind image super-resolution model to process a blind image to obtain a degradation feature of the blind image, wherein the blind image super-resolution model also includes an image feature correction module and an image reconstruction module; a correction unit, configured to use the image feature correction module to correct the image feature of the blind image according to the degradation feature to obtain a corrected image feature; and a reconstruction unit, configured to use the image reconstruction module to reconstruct a high-definition image corresponding to the blind image based on the corrected image feature, wherein the blind image super-resolution model is trained by the training method of the blind image super-resolution model according to the exemplary embodiment of the present disclosure.

[0015] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute a training method for a blind image super-resolution model or a blind image super-resolution method according to an exemplary embodiment of the present disclosure.

[0016] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute a training method for a blind image super-resolution model or a blind image super-resolution method according to an exemplary embodiment of the present disclosure.

[0017] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute a training method for a blind image super-resolution model or a blind image super-resolution method according to an exemplary embodiment of the present disclosure.

[0018] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: According to the training method of the blind image super-resolution model, the blind image super-resolution method, the electronic device, the storage medium and the program product of the present disclosure, a new implicit degradation feature learning scheme is proposed. By adopting the degradation prior constraint on the basis of the contrastive learning method, the degradation prior obtained by advance analysis can be used to make the degradation feature estimator pay more attention to the degradation-related information in the process of contrastive learning, which can effectively improve the discriminability of the learned implicit degradation features. There is no need to design a very complex blind image super-resolution model to improve the super-resolution performance. Combined with the distillation learning method to further reduce the computational amount of the model, a low-complexity blind image super-resolution model can be used to achieve excellent image reconstruction quality, which is suitable for application on low-computing power devices and end-side devices. In addition, the degradation feature estimator is used to determine the degradation features. By first applying the contrastive learning method based on the degradation prior constraints in the first stage to the degradation feature estimator alone instead of directly applying it to the entire model, the ability to extract degradation features can be improved in a targeted manner. Then, the degradation feature estimator trained in the first stage is embedded into the entire model. The entire model is trained, which helps to improve the overall training efficiency.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0021] Figure 1 is a flowchart of a training method of a blind image super-resolution model according to an exemplary embodiment of the present disclosure.

[0022] Figure 2 is a schematic diagram of the structure of a degradation feature converter according to a specific embodiment of the present disclosure.

[0023] Figure 3 It is a structural diagram of a blind image super-resolution model according to a specific embodiment of the present disclosure.

[0024] Figure 4 It is a schematic diagram of the structure and training logic of the teacher network according to a specific embodiment of the present disclosure.

[0025] Figure 5 It is a schematic diagram of the training logic of the first and second stages according to a specific embodiment of the present disclosure.

[0026] Figure 6 It is a structural schematic diagram of a degradation feature adaptation unit according to a specific embodiment of the present disclosure.

[0027] Figure 7 It is a schematic diagram of the structure and training logic of the student model according to a specific embodiment of the present disclosure.

[0028] Figure 8 It is a training logic diagram of the third stage according to a specific embodiment of the present disclosure.

[0029] Fig. 9 is a flowchart of a blind image super-resolution method according to an exemplary embodiment of the present disclosure.

[0030] Fig.10 is a block diagram of a training apparatus for a blind image super-resolution model according to an exemplary embodiment of the present disclosure.

[0031] Fig.11 is a block diagram of a blind image super-resolution apparatus according to an exemplary embodiment of the present disclosure.

[0032] Fig.12 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0035] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0036] Hereinafter, a training method of a blind image super-resolution model, a blind image super-resolution method, an electronic device, a storage medium, and a program product according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0037] Figure 1 1 is a flow chart of a method for training a blind image super-resolution model according to an exemplary embodiment of the present disclosure. The method can be executed on an electronic device with sufficient computing power.

[0038] The blind image super-resolution model includes a degradation feature estimator, an image feature correction module and an image reconstruction module. The degradation feature estimator is used to determine the degradation features of the blind image, the image feature correction module is used to correct the image features of the blind image according to the degradation features, and the image reconstruction module is used to reconstruct the high-definition image based on the corrected image features. It should be understood that a blind image is a low-resolution image whose specific degradation process is unknown or uncertain, and a high-definition image is a high-resolution image restored from a low-resolution image. Here, low resolution and high resolution are relative concepts. Super-resolution means that the original image (i.e., the low-resolution image) is restored to improve its resolution (i.e., obtain a high-resolution image). It should be understood that the model may also include other reasonable structures, such as, but not limited to, an image feature extractor for extracting image features of a blind image.

[0039] Reference Figure 1In step S101, a first-stage training sample is determined based on a first sample blind image, and a first-stage training is performed on the degradation feature estimator to be trained by using a contrastive learning method based on degradation prior constraints to obtain a first degradation feature estimator.

[0040] Specifically, the first sample blind image can be an original blind image directly obtained, or a plurality of blocks sampled from the original blind image directly obtained according to a specified size, and the present disclosure does not limit this. The second sample blind image and the third sample blind image in the following text are similar, and will not be described one by one. The first sample blind image, the second sample blind image, and the third sample blind image can be completely the same, partially the same, or completely different, and the present disclosure does not limit this.

[0041] Optionally, step S101 includes: constructing a trunk branch and a momentum branch, wherein both the trunk branch and the momentum branch include a degradation feature estimator; performing degradation prior constraint processing on the first sample blind image to obtain a training blind image; using the training blind image as a training sample, using a contrast learning method to train the trunk branch and the momentum branch, and using the degradation feature estimator in the trunk branch obtained through training as the first degradation feature estimator. By constructing a trunk branch and a momentum branch both including a degradation feature estimator, contrast learning can be achieved by comparing the output results of the two branches, and then the degradation feature estimator obtained through training in the trunk branch is used as the training result of the first stage. By using the result of the degradation prior constraint processing on the first sample blind image as the training sample of this stage, the degradation prior can be present in the training sample, and the degradation feature estimator can be guided to pay more attention to the degradation-related information during the training process, thereby effectively improving the discriminability of the learned degradation features. It should be understood that since the degradation prior is integrated into the training samples in advance and can be used as a learning reference for the degradation feature estimator, the degradation prior can also be called a degradation reference prior. It should be noted that the structures of the degradation feature estimators in the trunk branch and the momentum branch are the same as the structure of the degradation feature estimator in the blind image super-resolution model, but the methods of adjusting the parameters of the trunk branch and the momentum branch are different, so that the degradation feature estimators in the two branches finally trained are different, wherein the degradation feature estimator in the trunk branch is directly applied to the blind image super-resolution model, and the degradation feature estimator in the momentum branch is only used to assist in training the degradation feature estimator in the trunk branch. As an example, the trunk branch and the momentum branch further include a projector connected after the degradation feature estimator, which is used to perform nonlinear transformation of features, which can improve the representation ability of features.

[0042] Further optionally, in step S101, the operation of performing degradation prior constraint processing on the first sample blind image to obtain the training blind image includes: performing dimension extension processing on the blur kernel and noise information in the first sample blind image to obtain a degradation prior having the same spatial dimension as the first sample blind image; splicing the degradation prior and the first sample blind image along the channel dimension to obtain the training blind image. Blur kernel is a mathematical model used to describe the process of image blurring, which can be regarded as a convolution kernel that convolves a high-resolution image to generate a low-resolution image. In the blind image super-resolution task, the blur kernel is usually unknown and needs to be estimated from the low-resolution image. For specific estimation methods, reference can be made to relevant technologies in the field, such as the literature "Li Gongping, Lu Yao, Wang Zijian, Wu Ziwei, Wang Shunzhou. Image blind super-resolution neural network based on blur kernel estimation. Acta Automatica Sinica, 2023, 49(10): 2109-2121 doi: 10.16383 / j.aas.c200987", which will not be elaborated here. Noise information includes, for example, noise level, which refers to the intensity or amplitude of noise in an image, and can be estimated by a variety of methods, such as by analyzing the statistical characteristics of the image or using a deep learning model, which is also a prior art in the field and will not be elaborated on. By using the prior art to extract the blur kernel and noise information in the first sample blind image, it is possible to obtain degradation-related information in advance, and then expand this information into a degradation prior through a dimensional extension strategy, so that the spatial dimension of the degradation prior is consistent with that of the first sample blind image, and then the degradation prior and the first sample blind image can be spliced ​​along the channel dimension, thereby obtaining a blind image incorporating the degradation prior as a training blind image, thereby realizing constraints based on the degradation prior.

[0043] For an embodiment in which the first sample blind image is a sampled block, as an example, when performing the degradation prior constraint processing, specifically, randomly sampling in each original blind image indivual Blocks of size form a blind block sequence , as the first sample blind image sequence. Each pixel of each blind image block is represented by three components (red, green, blue) in the RGB color space, so each blind image block can be represented as a third-order matrix Then, using the dimension extension strategy, the corresponding degradation prior is generated according to the degradation information (including blur kernel and noise level) of each blind image block. Specifically, for each blind image block, the first step is to convert The blur kernel size is vectorized to size, and project it into size; the second step is to Noise level Replicate three times and concatenate to the aforementioned blur kernel vector to form The third step is to copy the above vector times, that is, forming a degenerate prior Finally, the obtained degradation prior is integrated into the corresponding blind image block, specifically by splicing the two along the channel dimension to form a training blind image sequence , each training blind image can be represented as a third-order matrix .

[0044] Further optionally, in step S101, the training blind image is used as a training sample, and the contrast learning method is adopted to train the trunk branch and the momentum branch, and the operation of using the degradation feature estimator in the trunk branch obtained by training as the first degradation feature estimator includes: performing the following iterative calculation steps on the trunk branch and the momentum branch until the stop condition is met, and using the degradation feature estimator in the latest trunk branch as the first degradation feature estimator, wherein the number of training blind images is multiple, and the training blind images used in each round of calculation are different: inputting the training blind images calculated in this round into the trunk branch and the momentum branch respectively to obtain trunk features and momentum features; obtaining reference features, wherein the reference features are features obtained by inputting images different from the training blind images calculated in this round into the trunk branch and / or the momentum branch; determining the contrast loss value according to the trunk features, the momentum features and the reference features; adjusting the parameters of the trunk branch according to the contrast loss value, and adjusting the parameters of the momentum branch based on the parameters of the trunk branch using the momentum update strategy of contrast learning. The principle of contrastive learning is to shorten the distance between the features of the same or similar images and to increase the distance between the features of different images. Since both the trunk branch and the momentum branch contain degraded feature estimators, and the same training blind images are input to the two in each round of calculation, it is necessary to shorten the outputs of the two, that is, shorten the trunk features and the momentum features. By introducing the above-mentioned reference features, a reference for shortening can be provided, that is, shortening the distance between the trunk features and the reference features, so that contrastive learning can be achieved based on the contrast loss value composed of the three features. In addition, it has been found in practice that the training efficiency can be improved by adjusting the parameters of the trunk branch only based on the contrast loss value and adjusting the parameters of the momentum branch using the momentum update strategy.

[0045] Still taking the embodiment in which the first sample blind image is a sampled image block as an example, in each round of calculation, a training blind image sequence corresponding to an original blind image may be As the training blind image for this round of calculation, the set of trunk features and / or momentum features output by the previous rounds of calculation are used as reference features. At this time, the number of reference features will gradually increase as the iterative calculation proceeds. It should be noted that the "trunk features and / or momentum features" here means that only the trunk branch can be used, only the momentum branch can be used, or both the trunk branch and the momentum branch can be used, and the present disclosure does not limit this. In addition, for the first round of calculation, since it has not been calculated before, the reference features can be determined randomly, or an original blind image can be selected additionally, and its corresponding training blind image sequence can be determined and the untrained trunk branches and / or momentum branches can be input to obtain the reference features. Of course, the reference features can also be determined by other reasonable methods, and the present disclosure does not limit this either.

[0046] Continue to refer to Figure 1 In step S102, a second-stage training sample is determined based on the second sample blind image, and a second-stage training is performed on the blind image super-resolution model including the first degradation feature estimator to obtain a teacher model.

[0047] The second stage of training can adopt any reasonable training method, such as supervised training method. In this case, the second sample blind image is also accompanied by a corresponding real high-definition image for calculating the loss function value to realize the training of the overall model. The present disclosure does not impose any restrictions on this.

[0048] It should be noted that, in this stage, when training the entire model, the degradation feature estimator trained is specifically the first degradation feature estimator, that is, the degradation feature estimator of the trunk branch. As for the degradation feature estimator of the momentum branch, since it is no longer used, it does not need to be trained. Of course, it can also be trained, and the present disclosure does not limit this. For the embodiment of training the degradation feature estimator in the momentum branch at the same time, its parameter adjustment method can follow the method adopted in the first stage.

[0049] In step S103, the training samples of the third stage are determined based on the third sample blind image, the teacher model is taken as the learning object, and the distillation learning method is used to perform the third stage training to obtain the final blind image super-resolution model.

[0050] In the third stage, the distillation learning method is used to transfer the degradation-related knowledge learned by the teacher model to a student model with a simple structure. The student model is used as the final blind image super-resolution model, which helps to reduce the computational complexity of the model and realize lightweight model deployment in the inference stage.

[0051] Before specifically introducing step S103, the image feature correction module is further introduced.

[0052] Optionally, the image feature correction module includes a degradation feature converter and a degradation feature adaptation module, wherein the degradation feature converter includes a channel domain conversion branch and a space domain conversion branch, the channel domain conversion branch is used to convert the degradation feature into a channel domain degradation feature, the space domain conversion branch is used to convert the degradation feature into a space domain degradation feature, the degradation feature adaptation module includes multiple degradation feature adaptation groups, each degradation feature adaptation group includes multiple degradation feature adaptation units, each degradation feature adaptation unit includes a feature segmenter, a channel domain adaptation branch, a space domain adaptation branch, and a feature fusion device, the feature segmenter is used to segment the image feature input therein into a first image feature and a second image feature, the channel domain adaptation branch is used to integrate the channel domain degradation feature into the first image feature to obtain a first corrected image feature, the space domain adaptation branch is used to integrate the space domain degradation feature into the second image feature to obtain a second corrected image feature, and the feature fusion device is used to fuse the first corrected image feature and the second corrected image feature into a new image feature. By using a degradation feature converter to convert the degradation features into channel domain degradation features and spatial domain degradation features, and using a degradation feature adaptation module to repeatedly perform feature adaptation fusion, that is, first segmenting the image features, and then performing feature fusion in the channel domain and spatial domain respectively, and merging and fusion of the two corrected image features after fusion, it is possible to simultaneously consider the characteristics of the channel domain and the spatial domain, achieve a more efficient fusion of the degradation features and the image features, and thus achieve efficient correction of the image features. It should be understood that the structure of the channel domain conversion branch and the spatial domain conversion branch in the degradation feature converter can be reasonably designed as needed. As an example, Figure 2 As shown, the channel domain conversion branch may include a global average pooling layer and a fully connected layer, and the spatial domain conversion branch may include a pixel reorganization layer and a convolution layer, which is not limited in the present disclosure. In addition to multiple degradation feature adaptation groups, the degradation feature adaptation module may also include other structural layers, and each degradation feature adaptation group may also include other structural layers in addition to multiple degradation feature adaptation units. For example, Figure 3 As shown, the degradation feature adaptation module also includes a convolutional layer connected after multiple degradation feature adaptation groups, and each degradation feature adaptation group also includes a ConvNeXt (AConvNet for the 2020s, convolutional neural network for the 2020s) block and a convolutional layer connected after multiple degradation feature adaptation units, and the present disclosure does not limit this.

[0053] On this basis, optionally, step S103 includes: copying the teacher model to obtain a student model; using the third sample blind image as a training sample, using a distillation learning method, with the purpose of aligning the output of the degradation feature converter in the teacher model and the student model, performing model learning, and using the learned student model as the final blind image super-resolution model. By first copying the teacher model as the student model, it can be ensured that the student model has the same model structure as the teacher model, and the model structure is unchanged. In addition, the operations related to the degradation features are mainly concentrated in the degradation feature estimator and the degradation feature converter, and thereafter, it is only necessary to adapt and fuse the converted channel domain degradation features and spatial domain degradation features with the image features extracted from the training samples. Based on this, by aligning the output of the degradation feature converter in the teacher model and the student model when performing distillation learning, the degradation-related knowledge learned in the teacher model can be concentratedly transferred to the student model, thereby improving the efficiency of distillation learning. It should be understood that the purpose of aligning the output of the degradation feature converter in the teacher model and the student model means that the output of the degradation feature converter in the two models is used as the basis for calculating the loss value, and then the parameters of the student model are adjusted according to the loss value.

[0054] Next, we introduce a blind image super-resolution model and its training method according to a specific embodiment of the present disclosure. In this specific embodiment, given a set of A set of low-resolution images (i.e., the original blind images mentioned above) for training is divided into three groups, which are used for three stages of training respectively. The training process is as follows, wherein step S1 is a basic model building step, steps S2 to S6 correspond to the first stage of training, steps S7 to S12 correspond to the second stage of training, and steps S13 to S16 correspond to the third stage of training.

[0055] Step S1: Construct Figure 3 The blind image super-resolution model based on implicit degradation feature guidance is shown in FIG. The model includes an image feature extractor, a degradation feature estimator, a degradation feature converter, a degradation feature adaptation module and an image reconstruction module, wherein the degradation feature converter and the degradation feature adaptation module constitute an image feature correction module.

[0056] The image feature extractor uses a 3×3 convolutional layer.

[0057] The degradation feature estimator uses a six-layer convolutional network.

[0058] The degenerate feature converter contains one channel domain conversion branch and one spatial domain conversion branch, such as Figure 2 As shown in the figure, the former contains 1 global average pooling layer and 1 fully connected layer, and the latter contains 1 pixel reorganization layer and 1 3×3 convolution layer.

[0059] The structure of the degradation feature adaptation module is as follows Figure 3 As shown in , specifically, the basic component of the degradation feature adaptation module is the degradation feature adaptation unit, which is composed of a series of degradation feature adaptation units and convolution blocks to form a degradation feature adaptation group, and a series of degradation feature adaptation groups and convolution blocks to form a degradation feature adaptation module. The structure of the degradation feature adaptation unit will be introduced below.

[0060] The image reconstruction module can adopt an upsampling module based on a convolutional neural network.

[0061] Step S2: In the first stage of training, a teacher network structure based on contrastive learning with degradation prior constraints is first constructed. It should be noted that the teacher network is modified based on the aforementioned blind image super-resolution model according to the requirements of the “contrastive learning method with degradation prior constraints”. Figure 4 As shown in Figure 1, the teacher network consists of two branches: the backbone branch and the momentum branch, both of which consist of a degraded feature estimator and a projector. The degraded feature estimator is a six-layer convolutional network with a GELU (Gaussian Error Linear Unit) activation function, and the projector is a two-layer fully connected network with a GELU activation function.

[0062] It should be noted that the degraded feature estimator in the backbone branch of the teacher network is Figure 3 The degraded feature estimator in the blind image super-resolution model is physically the same. Figure 4 The bottom branch does not represent inputting the output of the degradation feature estimator into the blind image super-resolution model, but rather uses the trained first degradation feature estimator as part of the blind image super-resolution model, and then performs the second stage of training the entire blind image super-resolution model based on the image reconstruction loss.

[0063] The degradation feature estimator in the momentum branch of the teacher network is consistent with the degradation feature estimator in the aforementioned blind image super-resolution model in structure, but the parameters are different because the way of adjusting its parameters is different from that of the trunk branch. The degradation feature estimator in the momentum branch is only used to assist in training the degradation feature estimator in the trunk branch. Figure 5 As shown in the process flow in the upper middle row, the first stage of training requires multiple rounds of iterative calculations. In each round of calculations, the parameters of the trunk branch and momentum branch in the teacher network need to be adjusted, but the adjustment methods are different.

[0064] Step S3: In the data preprocessing stage, first, for each low-definition image in the first group of low-definition images, randomly sample indivual The size of the block forms a low-definition block sequence corresponding to the low-definition image. Then, the dimension extension strategy can be used to extract the degradation prior (also called degradation reference prior) of each low-definition tile. Finally, the degradation prior is integrated into the low-definition tile to form a training sample sequence. .

[0065] Step S4: Sequence the training samples The outputs of the degradation feature estimators of the two branches are and ,in This is because there are two downsamplings in the degraded feature estimator, and 256 represents the feature dimension. The final outputs of the two branches are and , where 128 represents the feature dimension.

[0066] Step S5: In the teacher network, each round of parameter update of the trunk branch and momentum branch is divided into two steps. The first step is to use the contrast loss between the output of the trunk branch and the momentum branch, and combine the negative sample queue to update the parameters of the trunk branch, where the negative sample queue refers to the queue composed of the output features obtained by inputting other low-definition blocks different from the low-definition block sequence calculated in this round into the trunk branch and / or momentum branch. Specifically, the trunk features output by the trunk branch in previous rounds of calculation and / or the momentum features output by the momentum branch can be used. The loss function is defined as follows.

[0067]

[0068] in, Indicates the index of the trunk branch output, Represents the index of the momentum branch output, similarity function The InfoNCE (Information Noise-Contrastive Estimation, mutual information unsupervised feature learning) loss is used and is defined as follows.

[0069]

[0070] in, represents the temperature coefficient, represents the negative sample queue, express Length, Indicates negative samples, here It is a superscript rather than a representation The calculation of the power is not marked as a subscript because it belongs to a different marking level from the subscripts i and j, so a different marking position is used to distinguish them.

[0071] In the first step, the stochastic gradient descent algorithm is used as the optimizer to optimize the parameters of the trunk branch, where the learning rate is set to 1e-3 until the loss value is less than or equal to the preset value.

[0072] Step S6: The second step is to update the parameters of the momentum branch using the momentum update strategy of contrastive learning, that is, after updating the parameters of the trunk branch, the parameters of the momentum branch are updated based on the parameters of the trunk branch, as follows.

[0073]

[0074] in, and are the parameters of the main branch and momentum branch, respectively. is the momentum coefficient.

[0075] After the first stage of training, the degradation feature estimator in the trunk branch can be obtained as the first degradation feature estimator, which is embedded in the blind image super-resolution model as the training object of the second stage.

[0076] Step S7: In the second stage of training, for the second set of low-definition images, the same method as step S3 is used to randomly sample the low-definition block sequence corresponding to each low-definition image. , as a training sample sequence. Figure 5 As shown, in each round of calculation, a training sample sequence is sent to the blind image super-resolution model obtained in the first stage. First, the image feature extractor is used to extract the image features of each low-definition block (hereinafter referred to as low-definition block features) .

[0077] Step S8: Extract degradation features using the first degradation feature estimator trained in the first stage , and use the degradation feature converter to convert the output of the first degradation feature estimator Perform channel-level and space-level expansion. Specifically, Figure 2 As shown, the channel domain conversion branch Perform a global average pooling operation, and then send the pooling result to the fully connected layer for nonlinear transformation to obtain the channel domain degradation feature ; The spatial domain transformation branch uses the pixel reconstruction layer to transform Upsample to the same size as the low-definition tile, and then send the upsampled feature map to a 3×3 convolutional layer to obtain the spatial domain degradation feature .

[0078] Step S9: using the degradation feature adaptation module to integrate the degradation features into the low-definition block features.

[0079] Specifically, for the degenerate feature adaptation unit, such as Figure 6 As shown, firstly, the input low-definition block features The channel dimension is divided into two parts in a ratio of 1:3. In the spatial domain adaptation branch, first, along the channel dimension, and Then, they are modulated by two 3×3 convolutional layers and a Sigmoid function. Then, the modulated spatial domain degradation features are compared with the original Multiply them together, and then connect a 5×5 convolutional layer for further fusion to obtain the low-definition block features that have been adapted through spatial domain degradation. . Adapt the branch in the channel domain and Perform a GAP (Global Average Pooling) operation and then combine the pooling result with Then, they are modulated by two fully connected layers and a Sigmoid function. Next, the modulated channel domain degradation features are compared with the original Multiply them together to obtain the low-definition block features that have been adapted through channel domain degradation Finally, and The low-resolution image block features are concatenated along the channel dimension and fed into a ConvNeXt block for further fusion. Finally, the skip connection is used again to merge the low-resolution image block features. Introduced to restore high-frequency details. This process can be expressed as:

[0080] in, It is a ConvNeXt block that contains one 7×7 depthwise separable convolutional layer and two 1×1 convolutional layers. Represents the image features processed by the current degraded feature adaptation unit.

[0081] For the degraded feature adaptation group, it consists of 4 degraded feature adaptation units, 1 ConvNeXt block and 1 3×3 convolution layer. In the degraded feature adaptation group, the low-definition block features and the degraded features are sequentially sent to each component for processing, and a jump link is added between the input and output of the degraded feature adaptation group.

[0082] The degradation feature adaptation module is composed of three degradation feature adaptation groups and one 3×3 convolutional layer in series. In the degradation feature adaptation module, the low-definition block features and degradation features are sequentially sent to each component for processing, and a jump link is added between the input and output of the degradation feature adaptation module. After being processed by the degradation feature adaptation module, the low-definition block features that fully integrate the degradation information can be obtained. , that is, the corrected image features.

[0083] Step S10: Using the image reconstruction module, As input, we reconstruct the HD image block (or high-resolution image block) corresponding to the aforementioned low-definition image block. The image reconstruction process can be expressed as:

[0084] in, represents the super-resolution module based on convolutional neural network, Represents the reconstructed high-definition image.

[0085] Step S11: Calculate the real HD image block and reconstruct high-definition tiles The L1 loss between them is as follows:

[0086] Step S12: Use the stochastic gradient descent algorithm as an optimizer to optimize the parameters of the entire blind image super-resolution model, wherein the learning rate is gradually adjusted from 2e-4 to 1e-6 using a cosine annealing adjustment strategy, and the number of iterations is set to 600. The trained blind image super-resolution model is used as a teacher model, and it should be understood that the teacher model here and the aforementioned teacher network are different concepts.

[0087] Step S13: In the third stage of student model training, first construct Figure 7 The student model is shown in Figure 2. Note that the student model can be obtained by copying the teacher model. Figure 7 As shown, the student model mainly trains a degraded feature estimator and a degraded feature converter.

[0088] Step S14: For each low-definition image in the third group of low-definition images, the method described in step S3 is still used to obtain the corresponding low-definition block sequence, which is sent to the student model as a training sample sequence for training the degradation feature estimator and the degradation feature converter. For the sake of distinction, the degradation feature converter in the student model is called the student degradation feature converter, and the degradation feature converter in the teacher model is called the teacher degradation feature converter. As mentioned above, the outputs of the spatial domain conversion branch and the channel domain conversion branch in the teacher degradation feature converter are defined as and , accordingly, the outputs of the spatial domain conversion branch and the channel domain conversion branch in the student-degraded feature transformer are defined as and .

[0089] Step S15: Figure 8 As shown in Figure 2, knowledge distillation is achieved by aligning the outputs of the teacher degraded feature converter and the student degraded feature converter. For the distillation of spatial domain degraded features, L2 loss is used to perform arrive Knowledge transfer:

[0090] For the distillation of channel-domain degenerate features, L1 loss is used to minimize and The absolute difference is:

[0091] The total loss of knowledge distillation is:

[0092] The stochastic gradient descent algorithm is used as the optimizer to optimize the parameters of the degraded feature estimator and the degraded feature converter of the student model, where the learning rate is set to 1e-3 until the loss value is less than or equal to the preset value.

[0093] Step S16: Save the model parameters at the end of training to obtain the expected lightweight blind image super-resolution model guided by implicit degradation features, whose parameter amount and calculation amount are reduced by more than 50% compared with the most advanced model.

[0094] Fig. 9 1 is a flow chart of a blind image super-resolution method according to an exemplary embodiment of the present disclosure. The method can be executed on an electronic device with sufficient computing power.

[0095] Reference Fig. 9 In step S901, the blind image is processed using the degradation feature estimator in the blind image super-resolution model to obtain the degradation features of the blind image. The blind image super-resolution model also includes an image feature correction module and an image reconstruction module.

[0096] In step S902, an image feature correction module is used to correct the image features of the blind image according to the degradation features to obtain corrected image features.

[0097] In step S903, an image reconstruction module is used to reconstruct a high-definition image corresponding to the blind image based on the corrected image features.

[0098] In the blind image super-resolution method according to the exemplary embodiment of the present disclosure, the blind image super-resolution model is trained by the training method of the blind image super-resolution model according to the exemplary embodiment of the present disclosure, and thus has all the beneficial technical effects of the training method, which will not be repeated here.

[0099] Fig.10 is a block diagram of a blind image super-resolution apparatus according to an exemplary embodiment of the present disclosure.

[0100] The blind image super-resolution model includes a degradation feature estimator for determining the degradation features of the blind image, an image feature correction module for correcting the image features of the blind image according to the degradation features, and an image reconstruction module for reconstructing a high-definition image based on the corrected image features.

[0101] Reference Fig.10 The training device 1000 of the blind image super-resolution model includes a first training unit 1001, a second training unit 1002, and a third training unit 1003.

[0102] The first training unit 1001 may determine a first-stage training sample based on the first sample blind image, and perform a first-stage training on the degradation feature estimator to be trained by using a contrastive learning method based on degradation prior constraints to obtain a first degradation feature estimator.

[0103] The second training unit 1002 may determine the second-stage training samples based on the second sample blind image, and perform the second-stage training on the blind image super-resolution model including the first degradation feature estimator to obtain a teacher model.

[0104] The third training unit 1003 can determine the training samples of the third stage based on the third sample blind image, take the teacher model as the learning object, adopt the distillation learning method to perform the third stage training, and obtain the final blind image super-resolution model.

[0105] Optionally, the first training unit 1001 can also: construct a trunk branch and a momentum branch, wherein the trunk branch and the momentum branch both include a degradation feature estimator; perform degradation prior constraint processing on the first sample blind image to obtain a training blind image; use the training blind image as a training sample, adopt a contrastive learning method, train the trunk branch and the momentum branch, and use the degradation feature estimator in the trunk branch obtained through training as the first degradation feature estimator.

[0106] Optionally, the first training unit 1001 may also: perform dimension extension processing on the blur kernel and noise information in the first sample blind image to obtain a degradation prior having the same spatial dimension as the first sample blind image; and splice the degradation prior and the first sample blind image along the channel dimension to obtain a training blind image.

[0107] Optionally, the first training unit 1001 may also: perform the following iterative calculation steps on the trunk branch and the momentum branch until the stopping condition is met, and use the degradation feature estimator in the latest trunk branch as the first degradation feature estimator, wherein the number of training blind images is multiple, and the training blind images used in each round of calculation are different: input the training blind images calculated in this round into the trunk branch and the momentum branch respectively to obtain trunk features and momentum features; obtain reference features, wherein the reference features are features obtained by inputting images different from the training blind images calculated in this round into the trunk branch and / or the momentum branch; determine the contrast loss value based on the trunk features, momentum features and reference features; adjust the parameters of the trunk branch based on the contrast loss value, and use the momentum update strategy of contrast learning to adjust the parameters of the momentum branch based on the parameters of the trunk branch.

[0108] Optionally, the image feature correction module includes a degradation feature converter and a degradation feature adaptation module, wherein the degradation feature converter includes a channel domain conversion branch and a space domain conversion branch, the channel domain conversion branch is used to convert the degradation feature into a channel domain degradation feature, the space domain conversion branch is used to convert the degradation feature into a space domain degradation feature, the degradation feature adaptation module includes multiple degradation feature adaptation groups, each degradation feature adaptation group includes multiple degradation feature adaptation units, each degradation feature adaptation unit includes a feature segmenter, a channel domain adaptation branch, a space domain adaptation branch, and a feature fusion device, the feature segmenter is used to segment the image feature input therein into a first image feature and a second image feature, the channel domain adaptation branch is used to integrate the channel domain degradation feature into the first image feature to obtain a first corrected image feature, the space domain adaptation branch is used to integrate the space domain degradation feature into the second image feature to obtain a second corrected image feature, and the feature fusion device is used to fuse the first corrected image feature and the second corrected image feature into a new image feature.

[0109] Optionally, the third training unit 1003 can also: copy the teacher model to obtain a student model; use the third sample blind image as a training sample, adopt a distillation learning method, perform model learning with the purpose of aligning the output of the degraded feature converter in the teacher model and the student model, and use the learned student model as the final blind image super-resolution model.

[0110] Fig.11 is a block diagram of a blind image super-resolution apparatus according to an exemplary embodiment of the present disclosure. Fig.11 The blind image super-resolution device 1100 includes a degradation estimation unit 1101, a correction unit 1102, and a reconstruction unit 1103.

[0111] The degradation estimation unit 1101 may use the degradation feature estimator in the blind image super-resolution model to process the blind image to obtain the degradation feature of the blind image. The blind image super-resolution model also includes an image feature correction module and an image reconstruction module.

[0112] The correction unit 1102 may be configured to use an image feature correction module to correct the image feature of the blind image according to the degradation feature to obtain a corrected image feature.

[0113] The reconstruction unit 1103 may be configured to use an image reconstruction module to reconstruct a high-definition image corresponding to the blind image based on the corrected image features. The blind image super-resolution model is trained by the training method of the blind image super-resolution model according to an exemplary embodiment of the present disclosure.

[0114] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0115] Fig.12 A structural block diagram of an electronic device 1200 according to an exemplary embodiment of the present disclosure is shown.

[0116] Reference Fig.12 The electronic device 1200 includes: at least one memory 1201 and at least one processor 1202, wherein the at least one memory 1201 stores computer executable instructions, and when the computer executable instructions are executed by the at least one processor 1202, the at least one processor is prompted to execute the target corresponding method as described in the above exemplary embodiment.

[0117] As an example, the electronic device 1200 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 1200 is not necessarily a single electronic device 1200, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 1200 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device 1200 that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0118] In the electronic device 1200, the processor 1202 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 1202 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0119] The processor 1202 may execute instructions or codes stored in the memory 1201, wherein the memory 1201 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0120] The memory 1201 may be integrated with the processor 1202, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1201 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1201 and the processor 1202 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1202 can read files stored in the memory.

[0121] In addition, the electronic device 1200 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1200 may be connected to each other via a bus and / or a network.

[0122] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, cause the at least one processor to execute the target corresponding method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0123] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including computer instructions, which, when executed by at least one processor, execute the target corresponding method as described in the above exemplary embodiment.

[0124] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0125] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a blind image super-resolution model, characterized in that: The blind image super-resolution model includes a degradation feature estimator for determining degradation features of a blind image, an image feature correction module for correcting image features of the blind image according to the degradation features, and an image reconstruction module for reconstructing a high-definition image based on the corrected image features, wherein the training method includes: Determine the first stage training sample based on the first sample blind image, adopt the contrast learning method based on degradation prior constraint, perform the first stage training on the degradation feature estimator to be trained, and obtain the first degradation feature estimator; Determining a second-stage training sample based on the second sample blind image, performing second-stage training on the blind image super-resolution model including the first degradation feature estimator, and obtaining a teacher model; Based on the third sample blind image, the training samples of the third stage are determined, the teacher model is used as a learning object, and the distillation learning method is used to perform the third stage training to obtain the final blind image super-resolution model.

2. The training method according to claim 1, characterized in that: The method of determining the first-stage training sample based on the first sample blind image, and using a contrastive learning method based on degradation prior constraints to perform the first-stage training on the degradation feature estimator to be trained, to obtain the first degradation feature estimator, includes: Constructing a trunk branch and a momentum branch, wherein the trunk branch and the momentum branch both include a degradation feature estimator; Performing degradation priori constraint processing on the first sample blind image to obtain a training blind image; The training blind image is used as a training sample, and a contrastive learning method is adopted to train the trunk branch and the momentum branch, and the degradation feature estimator in the trunk branch obtained by training is used as the first degradation feature estimator.

3. The training method according to claim 2, characterized in that: The step of performing degradation priori constraint processing on the first sample blind image to obtain a training blind image includes: Performing dimension extension processing on blur kernel and noise information in the first sample blind image to obtain a degradation prior having the same spatial dimension as the first sample blind image; The degradation prior and the first sample blind image are concatenated along the channel dimension to obtain the training blind image.

4. The training method according to claim 2, characterized in that: The method of using the training blind image as a training sample, adopting a contrastive learning method, training the trunk branch and the momentum branch, and using the degradation feature estimator in the trunk branch obtained through training as the first degradation feature estimator includes: The following iterative calculation steps are performed on the trunk branch and the momentum branch until a stop condition is met, and the degradation feature estimator in the latest trunk branch is used as the first degradation feature estimator, wherein the number of the training blind images is multiple, and the training blind images used in each round of calculation are different: Input the training blind image calculated in this round into the trunk branch and the momentum branch respectively to obtain the trunk feature and the momentum feature; Acquire reference features, wherein the reference features are features obtained by inputting an image different from the training blind image calculated in this round into the trunk branch and / or the momentum branch; Determining a contrast loss value according to the backbone feature, the momentum feature and the reference feature; According to the contrast loss value, the parameters of the trunk branch are adjusted, and the parameters of the momentum branch are adjusted based on the parameters of the trunk branch using a momentum update strategy of contrastive learning.

5. The training method according to any one of claims 1 to 4, characterized in that: The image feature correction module includes a degradation feature converter and a degradation feature adaptation module. The degradation feature converter includes a channel domain conversion branch and a space domain conversion branch, wherein the channel domain conversion branch is used to convert the degradation feature into a channel domain degradation feature, and the space domain conversion branch is used to convert the degradation feature into a space domain degradation feature. The degradation feature adaptation module includes multiple degradation feature adaptation groups, each degradation feature adaptation group includes multiple degradation feature adaptation units, each degradation feature adaptation unit includes a feature segmenter, a channel domain adaptation branch, a spatial domain adaptation branch, and a feature fusion device. The feature segmenter is used to segment the image features input therein into a first image feature and a second image feature. The channel domain adaptation branch is used to integrate the channel domain degradation feature into the first image feature to obtain a first corrected image feature. The spatial domain adaptation branch is used to integrate the spatial domain degradation feature into the second image feature to obtain a second corrected image feature. The feature fusion device is used to fuse the first corrected image feature and the second corrected image feature into a new image feature.

6. The training method according to claim 5, characterized in that: The method of determining the training sample of the third stage based on the third sample blind image, taking the teacher model as the learning object, and performing the third stage training by adopting the distillation learning method to obtain the final blind image super-resolution model includes: Copy the teacher model to obtain a student model; The third sample blind image is used as a training sample, and a distillation learning method is adopted to perform model learning with the purpose of aligning the outputs of the degraded feature converters in the teacher model and the student model, and the learned student model is used as the final blind image super-resolution model.

7. A blind image super-resolution method, characterized in that: include: Using a degradation feature estimator in a blind image super-resolution model, the blind image is processed to obtain degradation features of the blind image, wherein the blind image super-resolution model further includes an image feature correction module and an image reconstruction module; Using the image feature correction module, correct the image feature of the blind image according to the degradation feature to obtain a corrected image feature; Using the image reconstruction module, based on the corrected image features, a high-definition image corresponding to the blind image is reconstructed, The blind image super-resolution model is obtained by training the blind image super-resolution model according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the training method of the blind image super-resolution model as described in any one of claims 1 to 6 or the blind image super-resolution method as described in claim 7.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the blind image super-resolution model as described in any one of claims 1 to 6 or the blind image super-resolution method as described in claim 7.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the at least one processor is prompted to execute the training method of the blind image super-resolution model as described in any one of claims 1 to 6 or the blind image super-resolution method as described in claim 7.

Images