Domain generalization method for turbulence degraded image restoration under wide spectrum distribution offset

Abstract

The invention provides a domain generalization method for turbulence degraded image restoration under wide-spectrum distribution offset, which is used for solving the technical problem that an existing digital adaptive optical system based on computational imaging has a liftable space in the aspect of light field modeling or a turbulence degraded image restoration algorithm based on data driving is poor in generalization. The invention provides a domain generalization method for turbulent degraded image restoration under wide spectrum band distribution offset, and the method comprises the steps: employing a turbulent degraded image restoration model which is obtained through training on a visible light wave band, is based on a UNet architecture and is provided with frozen parameters to generate a "deadlock" network, taking the "deadlock" network as a generation branch, carrying out the feature extraction of a source domain sample, and carrying out the feature extraction of the source domain sample; and taking a trainable copy of the'deadlock 'network as a guide branch, carrying out feature extraction on a target domain positive sample or a target domain negative sample, and finally fusing visible light wave band features and short wave wave band features of a first target, so that turbulence degraded image restoration under wide spectrum band distribution offset can be completed.

Description

Technical Field

[0001] The present invention relates to a domain generalization method for restoring turbulent degraded images, and particularly to a domain generalization method for restoring turbulent degraded images under wide spectral band distribution shift. Background Art

[0002] Ground-based exoplanet direct imaging observation equipment is expected to achieve high-contrast imaging within a wide spectral band. The emergence of adaptive optics systems can significantly improve the best resolution of ground-based exoplanet direct imaging observation equipment. Although adaptive optics systems can largely compensate for the effects of atmospheric turbulence, their compensation or correction is only partial and insufficient, and there will still be relatively serious loss of high-frequency information of the imaging target.

[0003] In recent years, digital adaptive optics systems based on computational imaging have made great progress in solving spatially non-uniform optical aberrations under atmospheric turbulence. Through an integrated meta-imaging chip architecture, ultra-fine perception and fusion of incoherent complex light fields can be achieved. Even after passing through imperfect optical lenses and complex imaging environments, perfect three-dimensional optical imaging can still be realized. Although digital adaptive optics systems based on computational imaging have achieved certain results in correcting atmospheric disturbance aberrations, due to the limitation of the pixel resolution of imaging chips, there is still room for improvement in such methods for enhancing the acquisition of dense spatial dimensions and performing light field modeling based on sparse / undersampled data. In addition, there are also data-driven turbulent degraded image restoration algorithms. However, considering the different reflection and radiation characteristics of the microstructures and their components on the surface of ground-based exoplanets in different spectral bands, for example, their imaging characteristics have certain differences within a wide spectral range, it will cause data-driven turbulent degraded image restoration algorithms learned in a single band to be unable to effectively perceive obvious distribution shifts, resulting in a significant decline in the performance of turbulent restoration algorithms, that is, there is a problem of poor algorithm generalization. Therefore, the coordinated development of higher-performance adaptive optics systems, computational imaging sensors, and more advanced image restoration algorithms has become the key technology for improving the performance of ground-based optical observation systems. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problems that existing digital adaptive optics systems based on computational imaging have room for improvement in light field modeling, or data-driven turbulent degraded image restoration algorithms have poor generalization, and to provide a domain generalization method for restoring turbulent degraded images under wide spectral band distribution shift.

[0005] To achieve the above purpose, the technical solution provided by the present invention is as follows: A domain generalization method for restoring turbulent degraded images under wide spectral band distribution shift, characterized in that it includes the following steps: Step 1, construct a training set and a test set; the training set includes one-to-one corresponding visible light turbulence degraded images D1, visible light clear images V1, short-wave turbulence degraded images D2, short-wave clear images V2 based on the first target, and short-wave turbulence degraded images D3 based on the second target; the test set includes one-to-one corresponding visible light turbulence degraded images D1 and short-wave turbulence degraded images D2 based on the first target; Step 2, use the visible light turbulence degraded image D1, short-wave turbulence degraded image D2, and short-wave turbulence degraded image D3 in the training set as the source domain sample, target domain positive sample, and target domain negative sample respectively; Step 3, construct a turbulence degradation restoration domain generalization model; the turbulence degradation restoration domain generalization model includes a generation branch, a guidance branch, and a fusion module; the generation branch is a "frozen" network, which is a turbulence degradation image restoration model based on the UNet architecture and with frozen parameters trained in the visible light band; the guidance branch includes a zero convolution layer Z1, a trainable copy, and a zero convolution layer Z2 arranged in sequence according to the input and output order; The input end of the "frozen" network is used to receive the source domain sample, extract features from the source domain sample, and reconstruct the extracted features to obtain visible light band features of three different depths; the input end of the zero convolution layer Z1 is used to receive the target domain positive sample or target domain negative sample and expand it into three channels; the trainable copy is a trainable copy of the "frozen" network, which communicates bidirectionally with the "frozen" network, and the trainable copy and the zero convolution layer Z2 are used to extract features from the three-channel target domain positive sample or target domain negative sample to obtain first target short-wave band features or second target short-wave band features of three different depths; the two input ends of the fusion module are respectively connected to the output ends of the "frozen" network and the zero convolution layer Z2, and are used to fuse the visible light band features of three different depths with the corresponding first target short-wave band features to obtain fusion features of two different depths; Step 4, input the source domain sample into the "frozen" network, and at the same time input the target domain positive sample or target domain negative sample into the zero convolution layer Z1, and combine the visible light clear image V1, short-wave clear image V2 in the training set, and the visible light band features, first target short-wave band features, second target short-wave band features, and fusion features generated during the training process to train the turbulence degradation restoration domain generalization model to obtain a trained turbulence degradation restoration domain generalization model; Step 5: Respectively input the visible light turbulence degraded image D1 and the short-wave turbulence degraded image D2 in the test set into the "locked" network and the zero convolution layer Z1 in the trained turbulence degradation restoration domain generalization model. The fusion module fuses the visible light band features of three different depths and the short-wave band features of the corresponding first target, and outputs the restored clear image of the short-wave band turbulence degraded image D2, completing the restoration of the turbulence degraded image under the wide spectral band distribution offset.

[0006] Further, in Step 3, the "locked" network includes an encoder and a decoder of the UNet architecture; At least three cascaded convolution blocks arranged in the input-output order are set in the encoder of the "locked" network for feature extraction of the source domain samples, and at least three cascaded convolution blocks arranged in the input-output order are set in its decoder for reconstructing the extracted features; Among them, the last three cascaded convolution blocks in the decoder of the "locked" network are also respectively used to output visible light band features of three different depths.

[0007] Further, in Step 3, the number of cascaded convolution blocks in the encoder of the "locked" network is five; There is two-way communication between the five cascaded convolution blocks in the encoder of the "locked" network and the corresponding cascaded convolution blocks in the encoder of the trainable copy.

[0008] Further, in Step 3, the fusion module is a residual pyramid convolution module, which includes three cascaded convolution blocks arranged in the input-output order; The second cascaded convolution block of the fusion module is also used to separately output the fusion features of its corresponding depth.

[0009] Further, in Step 4, when training the turbulence degradation restoration domain generalization model, the turbulence degradation restoration domain generalization model is jointly optimized by a contrast loss function, a content loss function, and a structure loss function.

[0010] Further, in Step 4, jointly optimizing the turbulence degradation restoration domain generalization model by a contrast loss function, a content loss function, and a structure loss function is specifically as follows: Step a1, the contrast loss function The expression of is: ; Among them, is the loss function of the positive sample in the target domain, , i = 1, 2, 3; is the loss function of the negative sample in the target domain, , i = 1, 2, 3; are the visible light band features at three different depths, are the first target shortwave band features at three different depths, are the second target shortwave band features at three different depths; and are respectively and weighting coefficients; Step b1, denote the output of the second cascaded convolutional block in the fusion module as the sub-fusion feature , and denote the output of the third cascaded convolutional block as the sub-fusion feature ; adopt a content loss function to minimize and optimize the distances between the sub-fusion feature and the shortwave clear image V2 in the training set, and between the sub-fusion feature and the visible light clear image V1 in the training set respectively; Step c1, adopt a structure loss function to minimize and optimize the distances between the sub-fusion feature and the shortwave clear image V2 in the training set, and between the sub-fusion feature and the visible light clear image V1 in the training set respectively.

[0011] Furthermore, in step 4, the content loss function is the pre-trained VGG19 network; The structure loss function is the multi-scale structure function MS-SSIM.

[0012] The beneficial effects of the present invention compared with the prior art are as follows: Compared with the prior art, the beneficial effects of the present invention are as follows: 1. A domain generalization method for restoring turbulent degradation images under wide spectral band distribution offset provided by the present invention uses a turbulent degradation image restoration model based on the UNet architecture and with frozen parameters trained in the visible light band to generate a "locked" network, and uses the "locked" network as a generation branch to extract features from source domain samples, uses a trainable copy of the "locked" network as a guiding branch to extract features from target domain positive samples or target domain negative samples, and finally fuses the visible light band features and the shortwave band features of the first target, thus completing the restoration of turbulent degradation images under wide spectral band distribution offset; in the present invention, the "locked" network and the trainable copy interact in a "bidirectional, high-frequency, large bandwidth" interaction manner, which not only improves the feature interaction efficiency, but also improves the accuracy of the extracted features.

[0013] 2. A domain generalization method for restoring turbulent degraded images under wide spectral band distribution offset provided by the present invention proposes to use the learning idea of the contrast loss function to constrain the features of wide spectral band source domain samples and target domain positive samples or target domain negative samples, which can better express cross-domain similarity features and discriminative features.

[0014] 3. The present invention uses the short-wave turbulent degraded image D2 of the first target and the short-wave turbulent degraded image D3 of the second target as the fine control conditions of the turbulent degraded restoration domain generalization model, which can guide and improve the generalization of the turbulent degraded image restoration algorithm learned in the visible light band in the short-wave band. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the principle of the turbulent degraded restoration domain generalization model in step 3 of the embodiment of a domain generalization method for restoring turbulent degraded images under wide spectral band distribution offset of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] To make the advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Considering the different reflection and radiation characteristics of the microscopic structure and its composition on the surface of exoplanets in different bands, and the imaging characteristics in the wide spectral range have certain differences, the present invention proposes a domain generalization method for restoring turbulent degraded images under wide spectral band distribution offset, which is used to improve the performance of restoring turbulent degraded images in the target band. The specific steps are as follows: Step 1, construct a training set and a test set.

[0018] The training set includes the corresponding visible light turbulent degraded image D1, visible light clear image V1, short-wave turbulent degraded image D2, short-wave clear image V2 based on the first target, and the short-wave turbulent degraded image D3 based on the second target; the test set includes the corresponding visible light turbulent degraded image D1 and short-wave turbulent degraded image D2 based on the first target.

[0019] Step 2, take the visible light turbulent degraded image D1, short-wave turbulent degraded image D2, and short-wave turbulent degraded image D3 in the training set as source domain samples, target domain positive samples, and target domain negative samples respectively.

[0020] Step 3, construct a turbulent degraded restoration domain generalization model.

[0021] As Figure 1 shown, the turbulent degraded restoration domain generalization model includes a generation branch, a guidance branch, and a fusion module. The generation branch is the "locked" network , which uses an existing turbulence-degraded image restoration model based on the UNet architecture trained in the visible light band, and the parameters (weights and deflections of all training layers) of the turbulence-degraded image restoration model based on the UNet architecture are frozen.

[0022] "Locked" network Includes the encoder and decoder of the UNet architecture. In this embodiment, the "locked" network has five cascaded convolutional blocks in its encoder. The input end of the first cascaded convolutional block is used to receive source domain samples, and the five cascaded convolutional blocks sequentially extract features from the source domain samples; the "locked" network also has five cascaded convolutional blocks in its decoder, which are used to sequentially reconstruct the extracted features to obtain visible light band features. Among them, the features extracted by the first cascaded convolutional block in the decoder are sent to the second cascaded convolutional block, the features extracted by the second cascaded convolutional block are sent to the third cascaded convolutional block, the features extracted by the third cascaded convolutional block are sent to the fourth cascaded convolutional block on one hand and used as the output of the third cascaded convolutional block on the other hand , the features extracted by the fourth cascaded convolutional block are sent to the fifth cascaded convolutional block on one hand and used as the output of the fourth cascaded convolutional block on the other hand , the features extracted by the fifth cascaded convolutional block are the output of the "locked" network .

[0023] The guiding branch includes a zero convolutional layer Z1, a trainable copy and a zero convolutional layer Z2 arranged in sequence according to the input and output order. Among them, the trainable copy is a trainable copy obtained by copying the "locked" network . The input end of the zero convolutional layer Z1 is used to receive target domain positive samples or target domain negative samples, and expands the received target domain positive samples or target domain negative samples into three channels; the trainable copy and the zero convolutional layer Z2 are used to extract features from the three-channel target domain positive samples or target domain negative samples to obtain first target shortwave band features or second target shortwave band features with three different depths. The five cascaded convolutional blocks in the encoder of the "locked" network and the corresponding cascaded convolutional blocks in the encoder of the trainable copy communicate bidirectionally.

[0024] Correspondingly, the features extracted by the third cascaded convolutional block in the decoder of the trainable copy are sent to the corresponding fourth cascaded convolutional block on one hand and used as the output of the third cascaded convolutional block on the other hand. When corresponding to the target domain positive sample, the output is denoted as , when corresponding to the target domain negative sample, the output is denoted as ; Trainable copy The features extracted by the fourth cascaded convolutional block in the [specific context] are sent all the way to the corresponding fifth cascaded convolutional block and are also used as the output of the fourth cascaded convolutional block. When it corresponds to a positive sample in the target domain, the output is denoted as , and when it corresponds to a negative sample in the target domain, the output is denoted as ; When the features extracted by the fifth cascaded convolutional block are output and correspond to a positive sample in the target domain, it is denoted as , and when it corresponds to a negative sample in the target domain, it is denoted as .

[0025] The fusion module is a residual pyramid convolutional module , which includes three cascaded convolutional blocks arranged in the order of input and output. The two input ends of the first cascaded convolutional block in the fusion module are respectively connected to the output ends of the "[locked]" network and the zero convolutional layer Z2, and are used to fuse the visible light band features of three different depths with the corresponding first target shortwave band features to obtain two fusion features of different depths. Specifically, the features output by the first cascaded convolutional block in the fusion module are sent to the second cascaded convolutional block. The features extracted by the second cascaded convolutional block are sent all the way to the third cascaded convolutional block and are also used as the fusion feature output corresponding to the depth of the second cascaded convolutional block, denoted as sub-fusion feature , and the features extracted by the third cascaded convolutional block are used as the output of the entire fusion module, denoted as sub-fusion feature .

[0026] Step 4, Training of the turbulence degradation restoration domain generalization model.

[0027] Input the source domain samples into the "[locked]" network , and at the same time input the target domain positive samples or target domain negative samples into the zero convolutional layer Z1 to train the turbulence degradation restoration domain generalization model and obtain a trained turbulence degradation restoration domain generalization model.

[0028] When training the turbulence degradation restoration domain generalization model, the turbulence degradation restoration domain generalization model is optimized jointly through a contrast loss function, a content loss function, and a structure loss function, specifically as follows: (1) Minimize the distance between the visible light band features of three different depths and the corresponding first target shortwave band features respectively through the second norm to obtain the loss function of the target domain positive samples , and its expression is as follows: , i = 1, 2, 3; Maximize the distance between the visible light band features of three different depths and the corresponding second target shortwave band features respectively through the second norm to obtain the loss function of the target domain negative samples , and its expression is as follows: , i = 1, 2, 3; Among them, are the second target short-wave band features at three different depths.

[0029] The contrast loss function is used to learn the similar features of the same target in the visible light band and the short-wave band on the positive samples in the target domain, and to distinguish the discriminative features of different targets in the visible light band and the short-wave band on the negative samples in the target domain.

[0030] Contrast loss function is calculated by the following formula: ; Among them, and are respectively and weighted coefficients of.

[0031] (2) Adopt the content loss function to minimize the distances between the sub-fusion feature and the short-wave clear image V2 in the training set, and between the sub-fusion feature and the visible light clear image V1 in the training set respectively; in this embodiment, the content loss function is the pre-trained VGG19 network.

[0032] (3) Adopt the structural loss function to minimize the distances between the sub-fusion feature and the short-wave clear image V2 in the training set, and between the sub-fusion feature and the visible light clear image V1 in the training set respectively; in this embodiment, the structural loss function is the multi-scale structure function MS-SSIM.

[0033] The weights and biases of the zero convolutional layer Z1 and the zero convolutional layer Z2 are both 0 at the initial moment of training. As the training progresses, the weights and biases will gradually increase from zero, that is, the interaction weights gradually increase, which is used to ensure that the turbulence degradation restoration domain generalization model trained in the visible light band will not cause a sudden change in the expression of the similar features of the same target in the visible light band and the short-wave band to the model performance.

[0034] Step 5, input the visible light turbulence degradation image D1 and the short-wave turbulence degradation image D2 in the test set into the "locked" network and the zero convolutional layer Z1 in the trained turbulence degradation restoration domain generalization model respectively. The fusion module fuses the visible light band features and the short-wave band features of the first target, and outputs the restored clear image of the visible light turbulence degradation image D1, completing the restoration of the turbulence degradation image under the wide spectral band distribution offset.

[0035] The "locked" network that generates branches in the present invention and the trainable copy that guides the branch interact repeatedly during the feature extraction process, that is, during the encoding and decoding processes, the "locked" network not only obtains the feedback control signal of the trainable copy after traversing multiple cascaded convolutional blocks, the "locked" network the output of each cascaded convolutional block in can be individually affected by the trainable copy as shown by the red arrow; the trainable copy also needs to obtain the feature output of the "locked" network in time as feedback regulation, as shown by the green arrow. In the present invention, the "locked" network and the trainable copy interact in a "bidirectional, high-frequency, large bandwidth" interaction manner, which not only improves the feature interaction efficiency but also improves the accuracy of the extracted features.

[0036] As described above, it is only used to illustrate the technical solution of the present invention and is not a limitation thereof. For those of ordinary skill in the art, the specific technical solution recorded in the above embodiments can be modified, or some of the technical features can be equivalently replaced, and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution protected by the present invention.

Claims

1. A domain generalization method for restoring turbulent degraded images under wide spectral band distribution offset, characterized in that It includes the following steps: Step 1: Construct a training set and a test set; the training set includes one-to-one corresponding visible light turbulence degraded images D1, visible light clear images V1, short-wave turbulence degraded images D2, short-wave clear images V2 based on a first target, and short-wave turbulence degraded images D3 based on a second target; the test set includes one-to-one corresponding visible light turbulence degraded images D1 and short-wave turbulence degraded images D2 based on the first target; Step 2: Respectively use the visible light turbulence degraded image D1, short-wave turbulence degraded image D2, and short-wave turbulence degraded image D3 in the training set as the source domain sample, target domain positive sample, and target domain negative sample; Step 3: Construct a turbulence degradation restoration domain generalization model; the turbulence degradation restoration domain generalization model includes a generation branch, a guidance branch, and a fusion module; the generation branch is a "frozen" network, which is a turbulence degradation image restoration model based on the UNet architecture and with frozen parameters trained in the visible light band; the guidance branch includes a zero convolution layer Z1, a trainable copy, and a zero convolution layer Z2 arranged in sequence according to the input and output order; The input end of the "frozen" network is used to receive the source domain sample, extract features from the source domain sample, and reconstruct the extracted features to obtain visible light band features of three different depths; The input end of the zero convolution layer Z1 is used to receive the target domain positive sample or target domain negative sample and expand it into three channels; the trainable copy is a trainable copy of the "frozen" network, which communicates bidirectionally with the "frozen" network. The trainable copy and the zero convolution layer Z2 are used to extract features from the three-channel target domain positive sample or target domain negative sample to obtain first target short-wave band features or second target short-wave band features of three different depths; Two input ends of the fusion module are respectively connected to the output ends of the "frozen" network and the zero convolution layer Z2, and are used to fuse the visible light band features of three different depths with the corresponding first target short-wave band features to obtain fusion features of two different depths; Step 4: Input the source domain sample into the "frozen" network, and at the same time input the target domain positive sample or target domain negative sample into the zero convolution layer Z1. Combine the visible light clear image V1, short-wave clear image V2 in the training set, and the visible light band features, first target short-wave band features, second target short-wave band features, and fusion features generated during the training process to train the turbulence degradation restoration domain generalization model to obtain a trained turbulence degradation restoration domain generalization model; Step 5: Respectively input the visible light turbulence degraded image D1 and short-wave turbulence degraded image D2 in the test set into the "frozen" network and the zero convolution layer Z1 in the trained turbulence degradation restoration domain generalization model. The fusion module fuses the visible light band features of three different depths with the corresponding short-wave band features of the first target, and outputs the restored clear image of the short-wave band turbulence degraded image D2, completing the restoration of the turbulence degraded image under the wide spectral band distribution offset.

2. The domain generalization method for restoring a turbulence degraded image under a wide spectral band distribution offset according to claim 1, wherein: In step 3, the "locking" network includes an encoder and a decoder of the UNet architecture; In the encoder of the "locking" network, at least three cascaded convolutional blocks arranged in the input-output order are provided for feature extraction of source domain samples, and in its decoder, at least three cascaded convolutional blocks arranged in the input-output order are provided for reconstructing the extracted features; Among them, the last three cascaded convolutional blocks in the decoder of the "locking" network are also respectively used to output visible light band features of three different depths.

3. The domain generalization method for turbulent degradation image restoration under wide spectral band distribution offset according to claim 2, wherein: In step 3, the number of cascaded convolutional blocks in the encoder of the "locking" network is five; Bidirectional communication is carried out between the five cascaded convolutional blocks in the encoder of the "locking" network and the corresponding cascaded convolutional blocks in the encoder of the trainable copy.

4. The domain generalization method for turbulent degradation image restoration under wide spectral band distribution offset according to any one of claims 1-3, wherein: In step 3, the fusion module is a residual pyramid convolution module, which includes three cascaded convolution blocks arranged in the order of input and output; the two input ends of the first cascaded convolution block in the fusion module are respectively connected to the "lock" network and the output ends of the zero convolution layer Z2; The second cascaded convolutional block of the fusion module is also used to separately output the fusion feature of its corresponding depth.

5. The domain generalization method for turbulent degradation image restoration under wide spectral band distribution offset according to claim 4, wherein: In step 4, when training the turbulent degradation restoration domain generalization model, the turbulent degradation restoration domain generalization model is optimized jointly by a contrast loss function, a content loss function, and a structure loss function.

6. The domain generalization method for turbulent degradation image restoration under wide spectral band distribution offset according to claim 5, wherein: In step 4, the specific method of jointly optimizing the turbulent degradation restoration domain generalization model by a contrast loss function, a content loss function, and a structure loss function is: Step a1, the contrastive loss function has the following expression: ； Among them, is the loss function of the positive sample in the target domain, , i = 1, 2, 3; is the loss function of the negative sample in the target domain, , i = 1, 2, 3; are the visible light band features at three different depths, are the first target shortwave band features at three different depths, are the second target shortwave band features at three different depths; and are respectively and 's weighting coefficients; Step b1, denote the output of the second cascaded convolutional block in the fusion module as sub-fusion feature , and denote the output of the third cascaded convolutional block as sub-fusion feature ; adopt a content loss function to minimize the distances between the sub-fusion feature and the short-wave clear image V2 in the training set, and between the sub-fusion feature and the visible light clear image V1 in the training set respectively; Step c1, using a structural loss function, minimize the distances between the sub-fusion feature and the short-wave clear image V2 in the training set, and between the sub-fusion feature and the visible-light clear image V1 in the training set respectively.

7. The domain generalization method for turbulent degradation image restoration under wide spectral band distribution offset according to claim 6, wherein: In step 4, the content loss function is the pre-trained VGG19 network; The structure loss function is the multi-scale structure function MS-SSIM.

Images