Unsupervised image restoration method applied to complex weather conditions

Abstract

The invention provides an unsupervised image restoration method applied to complex weather conditions, which is based on a framework of comparative learning, and a constructed network model learns degradation representation from a degraded input image so as to restore a clear image. Specifically, through a multi-category comparative learning strategy, the model can further improve feature constraint representation from three levels, namely a non-matching image level, a pixel level and a detail level. Besides, the network uses a self-calibration feature extraction module as the core of model feature representation, so that the feature dependency relationship between a long-distance space and a channel can be effectively captured, and a degraded image can be better recovered. According to the method, unsupervised training can be carried out on a non-matching data set, and the applicability of the algorithm is improved to a great extent.

Description

Technical Field

[0001] The present invention mainly relates to technical fields such as image processing, image restoration, deep learning and contrastive learning, and proposes an unsupervised image restoration method for use under complex weather conditions. Background Art

[0002] Optical images are a reflection of the scenes in the objective real world. In recent years, with the continuous advancement of optical image sensor technology, video and photography equipment containing such sensors have been widely used in people's production and life. However, under complex and severe weather conditions, the images obtained by shooting often have obvious quality degradation phenomena, such as image blur, color distortion and loss of details. The above degradation problems will seriously affect people's visual observation and further damage the performance of computer vision systems with clear images as input. To this end, researchers have proposed a large number of image restoration algorithms to solve such image degradation problems. Traditional optical image restoration methods include direct inverse filtering, Wiener filtering and wavelet transform. Most of the above methods are based on the prior knowledge of the image to construct a suitable degradation model, and then use various inverse degradation processing methods to restore the image. However, this type of restoration method based on prior knowledge estimation is usually limited by parameter settings and manual experience, and it is difficult to have good generalization performance. In recent years, with the continuous development of convolutional neural networks, image restoration algorithms based on deep learning have gradually become technology-dominated. By designing different network structures, the nonlinear mapping relationship between the degraded image and the clear image can be directly represented, so as to restore the clear image end-to-end. Commonly used structures include codec structure, pyramid structure and multi-scale structure. However, most of the designed network structures are based on supervised algorithms, which require matching data sets for network training. Generally speaking, it is difficult to obtain degraded and clear image pairs in real life. The lack of real data sets for training will make it difficult for deep learning-based methods to obtain further performance improvements.

[0003] In order to solve the problem of poor algorithm performance caused by the lack of real matching datasets for network training under complex weather conditions, an unsupervised image restoration method for complex weather conditions is proposed. Based on multi-type contrastive learning strategies, different levels of network constraints are obtained, and there is no need to match paired datasets for network training, thereby achieving unsupervised image restoration tasks and improving the visual quality of optical images and the performance of subsequent computer vision systems. Summary of the invention

[0004] In order to solve the problems existing in the above technologies, the present invention proposes an unsupervised image restoration method applied to complex weather conditions. The main idea of ​​the method is to obtain network constraints of different levels based on multi-type contrast learning strategies. In model training, there is no need to match paired data sets for network training, thereby achieving the purpose of unsupervised image restoration.

[0005] The technical solution adopted by the present invention is an unsupervised image restoration method applied to complex weather conditions. The method is implemented in the following steps: Step 1: Under various complex weather conditions (rain, fog and snow), collect unmatched data sets and build training data sets; Step 2: Design a multi-category contrastive learning strategy and use contrastive learning constraints from different image perspectives. In addition, construct a corresponding feature encoder to map positive and negative samples into an appropriate feature space for distinction; Step 3: Design and construct the image restoration network structure and design the corresponding optimization function. Train the collected data set so that the model can generate the corresponding clear image through inference.

[0006] The step 1 includes the following steps: Step 1.1: Data collection of degraded clear images under various weather conditions; Step 1.2: Preprocess the non-matching dataset using data augmentation techniques; Step 1.3: Normalization and processing of non-matching training data sets; The step 2 includes the following steps: Step 2.1: Construct pixel-level contrast constraints based on the non-matching degraded clear image; Step 2.2: Based on the same degraded image, construct pixel-level contrast constraints at corresponding positions; Step 2.3: Based on the same degraded image, construct detail level contrast constraints at corresponding positions; Step 2.4: Analyze different positive and negative samples and design feature encoders for different categories; The step 3 includes the following steps: Step 3.1: Design of overall network structure and construction of loss function; Step 3.2: Self-calibration feature extraction model design; Step 3.3: Use the collected data set to train the model; Step 3.4: Model reasoning verification; In summary, the characteristics and beneficial effects of the present invention are as follows: Based on various types of contrastive learning strategies, the present invention proposes an unsupervised image restoration method for complex weather conditions. The invention can achieve excellent image restoration effects under non-matching data sets, and has stronger algorithm generalization ability and environmental adaptability compared to traditional supervised image restoration algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of the overall steps of an unsupervised image restoration method applied to complex weather conditions proposed by the present invention; Figure 2 This is a schematic diagram of the structure of the self-calibration feature extraction model proposed in the present invention. DETAILED DESCRIPTION

[0008] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the described embodiments are intended only to facilitate the understanding of the present invention and do not have any limiting effect on it. The accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. The structures shown in the accompanying drawings are part of the actual structure. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0009] In general, the proposed unsupervised image restoration method for complex weather conditions is Figure 1 As shown. It contains the following three main parts: Step 1: Under various complex weather conditions (rain, fog and snow), collect unmatched data sets and build training data sets; The step 1 comprises the following steps: Step 1.1: Data collection of degraded clear images under various weather conditions. In order to meet the algorithm's adaptation to various complex weather conditions, random and fixed shooting equipment is used to shoot samples of various scenes in the real world under various weather conditions. Since the image restoration method used is an unsupervised algorithm, paired clear images are not required. Only a matching number of clear images need to be collected randomly. After shooting, a data set containing M pairs of degraded images and clear images is formed; Step 1.2: Preprocess the non-matching dataset using data augmentation technology. In order to enrich the collected image dataset samples, the collected M pairs of images are rotated, scaled, and cropped to form fixed-size image blocks. Note that the image block size used is 512×512. All image blocks are processed in a unified format to ensure the consistency of the collected data; Step 1.3: Normalization and processing of non-matching training data sets. Rename the expanded images, starting from 00001.jpg. If the number of images exceeds this level, a larger image naming range is used. Note that the image data format is unified as JPG image format. On this basis, non-matching data set image pairs are formed for subsequent network training; Step 2: Design a multi-category contrastive learning strategy and use contrastive learning constraints from different image perspectives. In addition, construct a corresponding feature encoder to map positive and negative samples into an appropriate feature space for distinction; The step 2 comprises the following steps: Step 2.1: Construct pixel-level contrast constraints based on non-matching degraded clear images. As we all know, the construction of positive and negative sample patterns is the key to the success of contrastive learning. First, a non-matching contrast constraint is introduced. The main idea is to regard the degraded image input as a negative sample, and the non-matching clear image as a positive sample. The purpose of the algorithm is to make the generated image closer to the positive sample, that is, the clear image state, and away from the degraded image, that is, away from the degraded image. Mathematically, it can be expressed as: in, represents the scaling coefficient between different layers in different feature encoders, O represents the restored image, B represents the non-matching clear image, and H represents the degraded image. In addition, It indicates that the proposed feature encoder adopts a discriminator with classification effect; Step 2.2: Based on the same degraded image. Construct pixel-level contrast constraints for corresponding positions. Generally speaking, when the degree of degradation is not severe, the difference between the clear image restored by the algorithm and the original image is not large, or there is a certain similarity between the two. Based on this prior assumption, a pixel-level contrast constraint for corresponding positions is designed and proposed. Specifically, the image blocks at the same position in the restored image and the input degraded image are regarded as a pair of positive samples, that is, the image blocks in the restored image are regarded as anchor points, and the image blocks at the same position in the degraded image are regarded as positive samples. In the selection of negative samples, image blocks of the same size at other positions are used as various types of negative samples. In order to improve the contrastive learning strategy, the number of negative samples should be increased as much as possible during the random sampling process to improve the strong constraints in contrastive learning. Mathematically, this constraint can be expressed as: in, represents the amount of encoding obtained by the anchor point through the feature encoder, represents the amount of encoding of the positive sample after feature encoding, represents the amount of coding after negative sample encoding, represents vector similarity calculation, represents the temperature coefficient; Step 2.3: Based on the same degraded image, construct detail level contrast constraints at corresponding positions. For image restoration tasks, in addition to maintaining pixel consistency, the detail correspondence of the image also needs to be considered. At the same pixel position, the details of the degraded image and the restored image should also have certain similarities. Therefore, an additional detail level contrast constraint at corresponding positions is introduced. Specifically, the image blocks at corresponding positions in the restored image and the degraded image are also regarded as a pair of positive samples, and the image blocks at other pixel positions are regarded as negative samples. The image detail feature extractor is used to extract features, thereby constructing contrast constraints under image detail conditions. This process can be described as: in, Represents the detail feature extraction of anchor point samples, Represents the detail feature extraction of positive samples, Represents the detailed feature extraction of negative samples; Step 2.4: Analyze different positive and negative samples and design different categories of feature encoders. In contrastive learning, in addition to the construction and design of positive and negative samples, the selection of feature encoders is equally important. In general, the feature encoder is responsible for mapping different sample inputs to a low-dimensional feature representation space to describe the spatial distance between positive and negative samples. For the above three different categories of contrastive learning constraints, different corresponding feature encoders are established, which focus on image classification differences, image pixel semantic differences, and image detail differences, respectively.

[0010] Among them, the contrast constraint based on non-matching images is mainly used to distinguish whether the image contains degradation information. In other words, the feature encoder used needs to have the ability to distinguish different image categories. Therefore, we use a discriminator-like architecture, that is, a binary classification network that analyzes whether the image has degradation. Different positive and negative samples are input into the discriminator network in turn to obtain the corresponding feature vector size, which is used to calculate the loss amount representing the contrast constraint.

[0011] For the pixel difference constraints of the same degraded image, the selected feature encoder should pay more attention to the semantic information of the image, that is, it should have deep feature extraction capabilities. Based on this assumption, the decoder of the overall restoration network structure is used as a feature extractor. As the number of layers of the network increases, the deep structure will be easier to capture semantic information to characterize the differences at the pixel level of the image.

[0012] For the detail difference constraints of the same degraded image, the selected feature encoder should pay more attention to the detail information of the image, that is, it should have the ability to extract the features of the image detail information. Based on this assumption, the encoder of the overall restoration network is used as the feature extractor. In the shallow network of the encoder, more contours, boundaries and detail information can be included to characterize the differences in the image detail level; Step 3: Design and construct the image restoration network structure and design the corresponding optimization function. Train the collected data set so that the model can generate the corresponding clear image through inference.

[0013] Step 3.1: Design of overall network structure and construction of loss function. The goal of the constructed image restoration model is to input a degraded image and output a clear image. The present invention divides the degraded image into a clear part and a degraded part, and regresses them through different prediction networks. Given a degraded image B, which consists of a clear image part C and a degradation factor R: In the proposed method, C and R are predicted separately, and the optimization function can be expressed as: in, and represents the proportional coefficient between different constraints, and P represents different normalized self-constraints. In addition, three different types of contrast constraints are added to improve the overall performance of the model. Therefore, the overall optimization goal of the model can be expressed as: in, , , , Represent different constraint loss functions, , , Represents the proportional coefficient between different constraint loss functions; Step 3.2: Self-calibration feature extraction model design. As we all know, the self-calibration feature convolution module is an improved CNN architecture that can capture long-range spatial and channel dependencies around each spatial position. Therefore, it can effectively expand the receptive field of each convolution layer and help the neural network to produce richer features. Therefore, the self-calibration convolution module is used as a multi-scale feature extraction module to improve the feature mining ability of the network model. Specifically, Figure 2As shown in the figure, given a specific feature input, first, it is divided into two feature maps X and Y according to the channel. Then, the proposed self-calibrated convolution module uses four different convolution operations (i.e., K1, K2, K3, and K4) to extract and fuse multi-scale features X and Y, further enriching the feature representation. Finally, the feature extraction module can obtain the output features of the two branches and concatenate and fuse them as the final feature output result; Step 3.3: Use the acquired dataset to train the model. First, process the constructed dataset, where the degraded image is passed to the constructed degradation recovery model as the input of the model. Secondly, input the non-matching clear image as a positive sample of the contrast constraint for model training. Finally, calculate the total loss value and back-propagate the updated parameters to obtain the final training weights; Step 3.4: Model inference verification. Load the trained model weights and perform model inference verification. Experimental verification shows that the proposed method can achieve good restoration effect on degraded images, and it does not require a matching dataset for model training, which shows its superiority. In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An unsupervised image restoration method for complex weather conditions, characterized in that: Step 1: Under various complex weather conditions, including rain, fog and snow, collect unmatched data sets and build training data sets; Step 2: Design a multi-category contrastive learning strategy, use contrastive learning constraints from different image perspectives, and construct a corresponding feature encoder to map positive and negative samples into an appropriate feature space for distinction; Step 3: Design and construct the image restoration network structure, and design the corresponding optimization function to train the collected data set so that the model can infer and generate the corresponding clear image.

2. The unsupervised image restoration method for complex weather conditions as claimed in claim 1, wherein the non-matching dataset construction in step 1 is characterized by: The collected degraded image training data set does not need to match the corresponding clear true value, which reduces the difficulty of image data collection.

3. The unsupervised image restoration method for complex weather conditions as claimed in claim 1, wherein the multi-class contrast learning strategy in step 2 is characterized by: Based on the non-matching degraded clear image, a pixel-level contrast constraint is constructed, the degraded image is regarded as a negative sample, and the non-matching image is regarded as a positive sample. The discriminator is used as a feature encoder to perform feature mapping on different samples.

4. The unsupervised image restoration method for complex weather conditions as claimed in claim 1, wherein the multi-class contrast learning strategy in step 2 is characterized in that: Based on the same degraded image, pixel-level contrast constraints at corresponding positions are constructed. The pixel areas at the same position of the degraded image and the restored image are regarded as positive sample pairs, and the pixel areas at different positions are regarded as negative samples. The decoder of the model is used as the feature representation space to extract the difference in semantic information.

5. The unsupervised image restoration method for complex weather conditions as claimed in claim 1, wherein the multi-class contrast learning strategy in step 2 is characterized by: Based on the same degraded image, detail level contrast constraints at corresponding positions are constructed. The detail information at the same position of the degraded image and the restored image is regarded as a positive sample pair, and the detail information at different positions is regarded as a negative sample. The encoder of the model is used as a feature representation space for extracting shallow detail differences.

6. The unsupervised image restoration method for complex weather conditions as claimed in claim 1, wherein the image restoration network structure construction in step 3 is characterized by: In addition to the basic pixel constraints, the overall model adds three different categories of network constraints based on contrastive learning to improve the robustness of the algorithm.

7. The unsupervised image restoration method for complex weather conditions as claimed in claim 1, wherein the image restoration network structure construction in step 3 is characterized by: The self-calibrated feature extraction module is designed to effectively expand the receptive field of each convolutional layer by capturing long-range spatial and channel dependencies around each spatial position, helping the model to generate richer features.

Images