An implicit image degradation estimation method based on decoupled contrastive learning

Abstract

The present invention discloses an implicit image degradation estimation method based on decoupled contrast learning, which belongs to the field of image blind super-resolution technology. The method samples blocks from different LR images with the same degradation information to construct a positive sample set, which ensures that task-irrelevant information such as content and texture will not affect degradation feature learning at the data level. At the same time, data amplification is achieved by continuously cyclically pairing sampled blocks. In the training process, a feature enhancement strategy is introduced to increase the number of positive samples of the same degradation type, thereby further expanding the sample diversity of each degradation category during the training process to ensure that the network learns degradation information more fully. Applying the implicit image degradation estimator trained by the present invention to a blind image super-resolution network guided by implicit degradation features can effectively improve the degradation information modeling quality of LR images, thereby improving the reconstruction effect of LR images to HR images.

Description

Technical Field

[0001] The present invention belongs to the technical field of blind image super-resolution, and in particular relates to an implicit image degradation estimation method based on decoupled contrast learning. Background Art

[0002] Blind Super-Resolution (‌BSR) is a subfield of image super-resolution (SR). The research goal is to recover a high-resolution image (HR) from a low-resolution image (LR) when the image degradation process (such as blur kernel, noise level, etc.) is unknown. Compared with traditional SR techniques that require prior knowledge of the degradation process, BSR techniques have wider applicability, because the degradation in real Internet scenarios is usually complex and unknown. Therefore, BSR has gradually become a research hotspot and attracted the attention of a large number of scholars and engineers. Although the model framework and calculation process of BSR and SR are similar, the BSR task faces almost infinite degradation forms, so how to estimate the degradation information in the image to help the super-resolution network limit the solution space is the key. Early BSR methods adopted an explicit degradation parameter estimation strategy, in which a learnable degradation estimator needs to be constructed to calculate the degradation parameters from the LR image. However, the scope of application of explicit estimation is still limited. The inevitable noise interference in real scenes and the complex interactions between different degradations make it more difficult to construct an accurate mathematical model of the degradation process, resulting in methods based on explicit estimation being able to only handle some simple, low-order degradation problems. In addition, accurate numerical estimation often requires a larger estimator or an iterative approach, which is very time-consuming. Therefore, research in the BSR field has gradually turned to the use of implicit degradation parameter estimation strategies.

[0003] Among various implicit image degradation estimation related studies, the framework based on contrastive learning has attracted a lot of attention from researchers due to its simple model structure, convenient training process and good degradation information estimation effect, and has gradually become a mainstream framework. This framework assumes that the degradation information between patches in the same image is consistent, while the degradation information between patches in different images is different, and constructs positive and negative sample pairs on this basis. Then, by forcing positive samples to move closer to each other and forcing negative samples to move away from each other, the implicit features of degradation are learned. However, the existing implicit degradation estimation methods based on contrastive learning lack systematic constraints, making the learning process of implicit degradation features easily interfered by irrelevant information such as content and texture, resulting in insufficient purity and lack of sufficient discriminability of the degradation features extracted from the image, which in turn affects the effect of subsequent image super-resolution. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an implicit image degradation estimation method based on decoupled contrastive learning, which improves the purity and discriminability of the learned degradation features by reducing the interference of task-irrelevant information such as content and texture in the implicit degradation feature modeling process and increasing the sample diversity of each degradation category. Applying the implicit image degradation estimator trained by the present invention to the blind image super-resolution network guided by implicit degradation features can effectively improve the degradation information modeling quality of LR images, thereby improving the reconstruction effect of LR images to HR images.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] An implicit image degradation estimation method based on decoupled contrastive learning, the method comprising the following steps:

[0007] Step 1: Construct a feature-enhanced contrastive learning network model;

[0008] Step 2: Collect high-definition image sets and perform preprocessing;

[0009] Step 3: Use the preprocessed high-definition image set as training data and input it into the feature enhancement contrast learning network model for training; update the parameters of the feature enhancement contrast learning network model based on the contrast loss function and momentum update strategy;

[0010] Step 4: Use the stochastic gradient descent algorithm to optimize the parameters of the feature enhancement contrastive learning network model until the training termination condition is reached, and save the optimized feature enhancement contrastive learning network model.

[0011] The beneficial effects of the present invention are:

[0012] On the one hand, the present invention designs a circular shift sampling module, which samples blocks from different LR images with the same degradation information to construct a set of positive samples, ensuring that task-irrelevant information such as content and texture will not affect degradation feature learning at the data level. At the same time, data amplification is achieved by continuously cyclically pairing sampled blocks. On the other hand, the present invention designs a feature-enhanced contrast learning network, which introduces a feature enhancement strategy during the training process to increase the number of positive samples of the same degradation type, thereby further expanding the sample diversity of each degradation category during the training process to ensure that the network learns degradation information more fully. Combined with the above key designs, the present invention can learn a more discriminative and generalized implicit degradation feature space, and the trained implicit degradation estimator can be flexibly adapted to various blind image super-resolution models guided by implicit degradation features, which can significantly improve the reconstruction effect of LR images to HR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of an implicit image degradation estimation method based on decoupled contrast learning of the present invention;

[0014] Figure 2 This is a diagram of the network architecture of feature-enhanced contrastive learning of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0016] like Figure 1 As shown, the present invention proposes an implicit image degradation estimation method based on decoupled contrastive learning. Specifically, a new contrastive learning framework for implicit degradation estimator training is constructed, including two parts: data preprocessing stage and parameter training stage. After inputting a set of high-definition graphic samples, randomly sampled tiles to construct a high-definition tile sequence, and a high-definition tile matrix is ​​constructed by cyclic shift to perform degradation on the high-definition tile matrix in units of columns. A positive sample set is constructed by sampling tiles from different LR images with the same degradation information, thereby avoiding pairing between tiles with similar contents at the data level, and realizing data amplification by continuously cyclically pairing these tiles. This strategy can ensure the decoupling between degradation information and content texture and other information during the degradation feature training process at the data level. As Figure 2 As shown in the figure, the feature enhancement contrastive learning network is selected in the parameter training stage. The feature enhancement contrastive learning network adopts the classic dual-branch network structure, one is defined as the main branch and the other is defined as the auxiliary branch. The main branch is used to extract the enhanced features of the low-definition blocks, and the auxiliary branch is used to extract the features of the low-definition blocks. The main branch consists of an enhanced estimator, a projector and two converters, and the auxiliary branch consists of a naive estimator and a projector. The naive estimator has the same basic structure as the enhanced estimator in the main branch, except that the feature enhancement processing operation is removed. The projector has the same basic structure and the same calculation process as the projector in the main branch. In the feature extraction stage of training, the feature enhancement contrastive learning network only takes the positive sample set as input, and adopts the feature enhancement strategy to further expand the sample diversity of each degradation category to ensure that the network can learn the degradation information more fully. The contrast loss is calculated to update the main branch parameters, and the momentum update strategy is used to update the auxiliary branch parameters. A highly discriminative implicit degradation feature space is gradually constructed to determine whether the termination condition is met: the number of iterations or the loss value reaches the preset value. After termination, during inference, the naive estimator in the output feature enhancement contrast learning network is the expected implicit graph degradation estimator.

[0017] Example

[0018] Given a set A collection of high-resolution images for training , an implicit image degradation estimation method based on decoupled contrastive learning includes the following steps:

[0019] Step S1: Construct a feature-enhanced contrastive learning network, which contains two branches. Figure 2 As shown in the figure, the main branch consists of an enhanced estimator, a first projector, and two converters (the first converter and the second converter are shown in the figure). Among them, the enhanced estimator is a six-layer convolutional network with feature enhancement processing operations. The first projector and the two converters are both two-layer fully connected networks with batch normalization and GELU activation functions. The auxiliary branch consists of a naive estimator and a second projector, among which the naive estimator has the same basic structure as the enhanced estimator in the main branch, except that the feature enhancement processing operation is removed. The second projector has the same basic structure as the first projector in the main branch and the same calculation process. Note that the trained naive estimator is the expected implicit degradation estimator.

[0020] Step S2: In the data preprocessing stage, randomly sample a block in each HD image to form a HD block sequence .

[0021] Step S3: Sequence the high-definition tiles Cyclic shift with a step size of 1 to obtain a new high-definition tile sequence The circular shift operation is performed continuously. times, get New HD tile sequences.

[0022] Step S4: Sequence the high-definition tiles Together with all new HD tile sequences obtained by cyclic shift, a OK High-definition tile matrix of columns .Notice, and The choice depends on the actual task situation.

[0023] Step S5: Calculate the HD tile matrix in columns Different degradation parameters are applied to each column in to obtain the low-definition version of the HD block. In other words, the degradation of the HD blocks in each column is consistent, while the degradation of HD blocks in different columns is different. In this way, we get groups of positive samples, each of which includes low-resolution tile samples and have the same degradation information.

[0024] Step S6: In the training phase, the sample set (i.e. The positive samples are sent to the feature enhancement contrast learning network constructed in step S1. For the main branch, first, each low-definition block sample is sent to the enhancement estimator for feature encoding and enhancement. Specifically, each low-definition block sample is first divided into Non-overlapping local blocks are sent to the six-layer convolutional network for feature encoding. The encoded features are then reorganized into Different feature pairs are generated, and the features of each feature pair are averaged. Then, all the averaged features are sequentially sent to the first projector and two converters for continuous feature transformation to obtain the output of the main branch. , where 256 represents the feature dimension.

[0025] Step S7: For the auxiliary branch, first, each low-resolution block sample is sent to the naive estimator for encoding. Note that the feature enhancement operation based on segmentation and recombination is no longer performed at this time. Then, all the features encoded by the naive estimator are sent to the second projector for transformation to obtain the output of the target branch. .

[0026] Step S8: The parameter update of the feature enhanced contrastive learning network is divided into two steps. The first step is to update the parameters of the main branch using the contrast loss between the outputs of the main branch and the auxiliary branch. The loss function is defined as follows:

[0027]

[0028] in, represents the contrast loss function, D represents the number of low-definition tiles, i, k, j represent the main branch low-definition tile index, the main branch low-definition tile reorganization feature index, the auxiliary branch low-definition tile feature index, and the similarity function Using InfoNCE, the definition is as follows:

[0029]

[0030] Among them, B represents the number of positive sample sets (each set contains D samples and has the same degradation information), n, m represent the index of the positive sample set, represents temperature coefficient, exp represents exponential function;

[0031] The second step is to use the momentum update strategy to update the parameters of the auxiliary branch as follows:

[0032]

[0033] in, and denote the parameters of the enhanced estimator and the first projector of the main branch, respectively, and denote the parameters of the naive estimator and the second projector of the auxiliary branch, respectively, represents the momentum coefficient.

[0034] Step S9: Use the stochastic gradient descent algorithm as the optimizer to optimize the main branch parameters, where the learning rate is set to 1 , until the loss value is less than or equal to the preset value or the number of iterations reaches the preset value.

[0035] Step S10: Save the model parameters at the end of training, where the naive estimator is the expected implicit image degradation estimator based on decoupled contrastive learning, which can be used in conjunction with a variety of different blind image super-resolution methods guided by implicit degradation features, and can effectively improve the degradation modeling quality of these networks.

[0036] The present invention is a universal implicit degradation estimator training method, which can be used in conjunction with a variety of different blind image super-resolution methods based on implicit degradation feature guidance to solve image super-resolution problems in complex and unknown degradation scenes, and can effectively improve the degradation information modeling quality of these methods, thereby significantly improving the reconstruction effect of LR images to HR images. Specifically, the user can first use the solution of the present invention to pre-train the implicit degradation estimator on a specified data set, and then embed the implicit degradation estimator into the user's own or classic blind image super-resolution network and perform overall fine-tuning training, which can be used for LR image super-resolution tasks. It should be noted that the present invention focuses on solving the implicit degradation estimation problem in real complex scenes, aiming to model the degradation process as a vector in a learnable latent space, which can better capture the essential characteristics of the degradation process, not only avoiding the complex calculation of explicit degradation estimation, but also making the degradation estimator have strong generalization and better fault tolerance.

[0037] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An implicit image degradation estimation method based on decoupled contrastive learning, characterized in that: The method comprises the following steps: Step 1: Construct a feature-enhanced contrastive learning network model; Step 2: Collect high-definition image sets and perform preprocessing; including: Randomly sample a tile from each HD image in the HD image set to form a HD tile sequence; The high-definition image block sequence is cyclically shifted with a step size of 1 to obtain a first new high-definition image block sequence, and the cyclic shift operation is performed several times continuously to obtain several second new high-definition image block sequences; The first high-definition tile sequence and a plurality of second new high-definition tile sequences obtained by cyclic shift form a high-definition tile matrix; Different degradation parameters are applied to each column in the HD image matrix in units of columns to obtain low-definition image samples corresponding to the HD images, which are used as positive samples for training the feature enhancement contrast learning network model. Step 3: Use the preprocessed high-definition image set as training data and input it into the feature enhancement contrast learning network model for training; update the parameters of the feature enhancement contrast learning network model based on the contrast loss function and momentum update strategy; Step 4: Use the stochastic gradient descent algorithm to optimize the parameters of the feature enhancement contrastive learning network model until the training termination condition is reached, and save the optimized feature enhancement contrastive learning network model.

2. The implicit image degradation estimation method based on decoupled contrastive learning according to claim 1, characterized in that: The feature enhancement contrast learning network model in step 1 includes a main branch and an auxiliary branch, wherein the main branch is used to extract enhanced features of low-definition blocks, and the auxiliary branch is used to extract features of low-definition blocks.

3. The implicit image degradation estimation method based on decoupled contrastive learning according to claim 2, characterized in that: The main branch includes an enhanced estimator, a first projector and two converters which are sequentially communicatively connected, and the auxiliary branch includes a naive estimator and a second projector, wherein the first projector and the second projector have the same structure and the same calculation process.

4. The implicit image degradation estimation method based on decoupled contrastive learning according to claim 3, characterized in that: The step 3 comprises: The positive sample is sent to the feature enhancement contrast learning network. For the main branch, each low-definition block sample is sent to the enhancement estimator for feature encoding and enhancement. All the enhanced features are sequentially sent to the first projector and two converters for continuous feature transformation to obtain the output of the main branch. For the auxiliary branch, each low-definition block sample is sent to the naive estimator for encoding, and all features encoded by the naive estimator are sent to the second projector for transformation to obtain the output of the target branch; The contrast loss between the output of the main branch and the output of the auxiliary branch is used to update the parameters of the main branch. The loss function is defined as follows: in, represents the contrast loss function, D represents the number of low-definition tiles, i, k, j represent the main branch low-definition tile index, the main branch low-definition tile reorganization feature index, the auxiliary branch low-definition tile feature index, and the similarity function Using InfoNCE, the definition is as follows: Among them, B represents the number of positive sample sets, n and m represent the indexes of positive sample sets, represents temperature coefficient, exp represents exponential function; Update the parameters of the auxiliary branch using the momentum update strategy: in, and denote the parameters of the enhanced estimator and the first projector of the main branch, respectively, and denote the parameters of the naive estimator and the second projector of the auxiliary branch, respectively, represents the momentum coefficient.

5. The implicit image degradation estimation method based on decoupled contrastive learning according to claim 1, characterized in that: The step 4 comprises: The stochastic gradient descent algorithm is used as the optimizer to optimize the main branch parameters, where the learning rate is set to 1 , until the loss value is less than or equal to the preset value or the number of iterations reaches the preset value, the training is terminated; Save the parameters of the feature-enhanced contrastive learning network model at the end of training. At this point, the naive estimator is the expected implicit image degradation estimator based on decoupled contrastive learning.

Images