A domain generalization method for turbulence-degraded image restoration under wide spectral distribution migration

Abstract

The present invention provides a domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift, which is used to solve the technical problems that existing digital adaptive optics systems based on computational imaging have room for improvement in light field modeling, or that data-driven turbulence-degraded image restoration algorithms have poor generalization. The present invention provides a domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift, which uses a turbulence-degraded image restoration model based on a UNet architecture and with frozen parameters trained on the visible light band to generate a "locked" network, and uses the "locked" network as a generation branch to extract features from source domain samples. The trainable copy of the "locked" network is used as a guiding branch to extract features from target domain positive samples or target domain negative samples. Finally, the visible light band features and the shortwave band features of the first target are fused to complete turbulence-degraded image restoration under wide spectral distribution shift.

Description

Technical Field

[0001] The present invention relates to a domain generalization method for turbulence-degraded image restoration, and in particular to a domain generalization method for turbulence-degraded image restoration under wide spectral distribution offset. Background Art

[0002] Ground-based direct imaging of exoplanets is expected to achieve high-contrast imaging across a wide spectral range. The emergence of adaptive optics systems has the potential to significantly improve the optimal resolution of ground-based direct imaging of exoplanets. While adaptive optics can largely compensate for the effects of atmospheric turbulence, their compensation or correction is only partial and inadequate, resulting in significant loss of high-frequency information in the imaged target.

[0003] In recent years, computational imaging-based digital adaptive optics (DAO) systems have made significant progress in addressing spatially non-uniform optical aberrations in atmospheric turbulence. Through an integrated meta-imaging chip architecture, they achieve ultra-fine perception and fusion of incoherent, complex light fields, enabling perfect three-dimensional optical imaging even through imperfect optical lenses and complex imaging environments. While DAO systems based on computational imaging have achieved some success in correcting atmospheric disturbance aberrations, due to the limitations of imaging chip pixel resolution, these approaches still have room for improvement in acquiring dense spatial dimensions and modeling light fields based on sparse / undersampled data. Furthermore, data-driven turbulence-degraded image restoration algorithms exist. However, given the varying reflective and radiative properties of the surface microstructure and components of Earth-based exoplanets across different spectral bands, such as the varying imaging characteristics across a wide spectral range, data-driven turbulence-degraded image restoration algorithms learned in a single band are unable to effectively detect significant distribution shifts, significantly degrading their performance and resulting in poor generalization. Therefore, the coordinated development of higher-performance adaptive optical systems, computational imaging sensors, and more advanced image restoration algorithms has become a key technology to improve the performance of ground-based optical observation systems. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problems that there is room for improvement in light field modeling of existing digital adaptive optics systems based on computational imaging, or that data-driven turbulence-degraded image restoration algorithms have poor generalization, and to provide a domain generalization method for turbulence-degraded image restoration under wide spectral distribution offset.

[0005] In order to achieve the above object, the technical solution provided by the present invention is as follows:

[0006] A domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift is proposed, which has the following characteristics:

[0007] Step 1: Construct a training set and a test set; the training set includes a one-to-one correspondence between a visible light turbulence-degraded image D1, a visible light clear image V1, a shortwave turbulence-degraded image D2, and a shortwave clear image V2 based on a first target, and a shortwave turbulence-degraded image D3 based on a second target; the test set includes a one-to-one correspondence between a visible light turbulence-degraded image D1 and a shortwave turbulence-degraded image D2 based on the first target;

[0008] Step 2: Use the visible light turbulence-degraded image D1, shortwave turbulence-degraded image D2, and shortwave turbulence-degraded image D3 in the training set as source domain samples, target domain positive samples, and target domain negative samples, respectively;

[0009] Step 3: Construct a generalized model for turbulence-degraded restoration domain; the generalized model for turbulence-degraded restoration domain includes a generation branch, a guidance branch, and a fusion module; the generation branch is a "locked" network, which is a turbulence-degraded image restoration model based on the UNet architecture and with frozen parameters trained on the visible light band; the guidance branch includes a zero convolution layer Z1, a trainable copy, and a zero convolution layer Z2, which are arranged in the order of input and output;

[0010] The input end of the "locked" network is used to receive source domain samples to extract features from the source domain samples and reconstruct the extracted features to obtain visible light band features of three different depths; the input end of the zero convolution layer Z1 is used to receive target domain positive samples or target domain negative samples and expand them into three channels; the trainable copy is a trainable copy of the "locked" network, which communicates bidirectionally with the "locked" network, and the trainable copy and the zero convolution layer Z2 are used to extract features from the three-channel target domain positive samples or target domain negative samples to obtain first target shortwave band features or second target shortwave band features of three different depths; the two input ends of the fusion module are respectively connected to the output ends of the "locked" network and the zero convolution layer Z2, and are used to fuse the visible light band features of three different depths with the corresponding first target shortwave band features to obtain two fused features of different depths;

[0011] Step 4: Input the source domain samples into the "locked" network, and input the target domain positive samples or target domain negative samples into the zero convolution layer Z1. Combined with the visible light clear image V1 and shortwave clear image V2 in the training set, as well as the visible light band features of three different depths generated during the training process, the first target shortwave band features, the second target shortwave band features, and the fusion features, the turbulence degradation restoration domain generalization model is trained to obtain the trained turbulence degradation restoration domain generalization model.

[0012] Step 5: The visible light turbulence-degraded image D1 and the shortwave turbulence-degraded image D2 in the test set are respectively input into the "locked" network and the zero convolution layer Z1 in the trained turbulence-degraded restoration domain generalization model. The fusion module fuses the visible light band features of three different depths with the corresponding shortwave band features of the first target, and outputs a restored clear image of the shortwave band turbulence-degraded image D2, completing the restoration of the turbulence-degraded image under wide spectrum distribution offset.

[0013] Furthermore, in step 3, the “locked” network includes an encoder and a decoder of a UNet architecture;

[0014] The encoder of the "locked" network is provided with at least three cascaded convolution blocks arranged in input-output order for extracting features from source domain samples, and the decoder is provided with at least three cascaded convolution blocks arranged in input-output order for reconstructing the extracted features;

[0015] Among them, the last three cascaded convolution blocks in the decoder of the "locked" network are also used to output visible light band features of three different depths.

[0016] Furthermore, in step 3, the number of cascaded convolutional blocks in the encoder of the "locked" network is five;

[0017] There is bidirectional communication between the five cascaded convolutional blocks in the encoder of the “locked” network and the corresponding cascaded convolutional blocks in the encoder of the trainable counterpart.

[0018] Furthermore, in step 3, the fusion module is a residual pyramid convolution module, which includes three cascaded convolution blocks arranged in input-output order;

[0019] The second cascade convolution block of the fusion module is also used to separately output the fusion features of its corresponding depth.

[0020] Furthermore, in step 4, when training the turbulence degradation restoration domain generalization model, the turbulence degradation restoration domain generalization model is optimized jointly by the contrast loss function, the content loss function and the structure loss function.

[0021] Furthermore, in step 4, the turbulence degradation restoration domain generalization model is optimized by using the contrast loss function, content loss function, and structure loss function as follows:

[0022] Step a1, the contrast loss function The expression is:

[0023] ;

[0024] in, is the loss function of the positive sample in the target domain, , i=1,2,3; is the loss function of negative samples in the target domain, , i=1,2,3; are the visible light band characteristics at three different depths, The shortwave band characteristics of the first target at three different depths, The shortwave band characteristics of the second target at three different depths; and They are and The weighting coefficient of

[0025] Step b1: record the output of the second cascade convolution block in the fusion module as the sub-fusion feature , the output of the third cascade convolution block is recorded as the sub-fusion feature ; Using content loss function, pair fusion features The distance and sub-fusion features between the shortwave clear image V2 in the training set The distance between the visible light clear image V1 in the training set is minimized and optimized respectively;

[0026] Step c1, using the structural loss function, the sub-fusion features The distance and sub-fusion features between the shortwave clear image V2 in the training set The distance between the visible light clear image V1 in the training set is minimized and optimized respectively.

[0027] Furthermore, in step 4, the content loss function is a pre-trained VGG19 network;

[0028] The structural loss function is the multi-scale structure function MS-SSIM.

[0029] The beneficial effects of the present invention compared to the prior art are as follows:

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The present invention provides a domain generalization method for turbulence-degraded image restoration under wide spectral band distribution offset. A turbulence-degraded image restoration model based on the UNet architecture and with frozen parameters obtained by training on the visible light band is used to generate a "locked" network, and the "locked" network is used as a generation branch to extract features of source domain samples. The trainable copy of the "locked" network is used as a guiding branch to extract features of target domain positive samples or target domain negative samples. Finally, the visible light band features and the shortwave band features of the first target are fused to complete the turbulence-degraded image restoration under wide spectral band distribution offset. In the present invention, the "locked" network and the trainable copy interact in a "bidirectional, high-frequency, and large-bandwidth" interactive manner, which not only improves the feature interaction efficiency, but also improves the accuracy of feature extraction.

[0032] 2. The present invention provides a domain generalization method for turbulence-degraded image restoration under wide-spectrum distribution offset, and proposes to use the learning idea of ​​contrast loss function to constrain the features of wide-spectrum source domain samples and target domain positive samples or target domain negative samples, which can better express cross-domain similarity features and discriminative features.

[0033] 3. The present invention uses the shortwave turbulence-degraded image D2 of the first target and the shortwave turbulence-degraded image D3 of the second target as fine control conditions for the generalization model of the turbulence-degraded restoration domain, which can guide and improve the generalization of the turbulence-degraded image restoration algorithm learned in the visible light band in the shortwave band. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the principle of the turbulence degradation restoration domain generalization model in step 3 of an embodiment of a domain generalization method for turbulence degradation image restoration under wide spectral distribution offset of the present invention. DETAILED DESCRIPTION

[0035] In order to make the advantages and features of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Considering the differences in the reflectance and radiation characteristics of the surface microstructure and components of exoplanets in different bands, their imaging characteristics over a wide spectral range have certain differences. This paper proposes a domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift to improve the performance of turbulence-degraded image restoration in the target band. The specific steps are as follows:

[0037] Step 1: Construct training and test sets.

[0038] The training set includes one-to-one correspondences of visible light turbulence-degraded image D1, visible light clear image V1, shortwave turbulence-degraded image D2, shortwave clear image V2 based on the first target, and shortwave turbulence-degraded image D3 based on the second target; the test set includes one-to-one correspondences of visible light turbulence-degraded image D1 and shortwave turbulence-degraded image D2 based on the first target.

[0039] Step 2: The visible light turbulence-degraded image D1, shortwave turbulence-degraded image D2, and shortwave turbulence-degraded image D3 in the training set are used as source domain samples, target domain positive samples, and target domain negative samples, respectively.

[0040] Step 3: Construct a generalized model of the turbulence degradation restoration domain.

[0041] like Figure 1 As shown in Figure 1, the turbulence degradation restoration domain generalization model includes a generation branch, a guidance branch, and a fusion module. The generation branch is the “locked” network. , which adopts the existing turbulence-degraded image restoration model based on the UNet architecture trained in the visible light band, and the parameters of the turbulence-degraded image restoration model based on the UNet architecture (weights and deflections of all training layers) are frozen.

[0042] "Locked" network Including the encoder and decoder of the UNet architecture, in this embodiment, the network is "locked" There are five cascade convolution blocks in the encoder, the input end of the first cascade convolution block is used to receive the source domain sample, and the five cascade convolution blocks extract features of the source domain samples in turn; "locking" the network The decoder is also equipped with five cascade convolution blocks, which are used to reconstruct the extracted features in sequence to obtain the visible light band features. Among them, the features extracted by the first cascade convolution block in the decoder are fed into the second cascade convolution block, the features extracted by the second cascade convolution block are fed into the third cascade convolution block, and the features extracted by the third cascade convolution block are fed into the fourth cascade convolution block and used as the output of the third cascade convolution block. , the features extracted by the fourth cascade convolution block are fed into the fifth cascade convolution block and used as the output of the fourth cascade convolution block , the features extracted by the fifth cascade convolutional block This is called a "locked" network Output.

[0043] The guide branch includes the zero convolution layer Z1, the trainable copy and zero convolutional layers Z2, where the trainable copies That is to "lock" the network The input end of the zero convolution layer Z1 is used to receive the target domain positive sample or the target domain negative sample, and expand the received target domain positive sample or target domain negative sample into three channels; the trainable copy The zero convolution layer Z2 is used to extract features from the three-channel target domain positive samples or target domain negative samples to obtain the first target shortwave band features or the second target shortwave band features at three different depths. The encoder consists of five cascaded convolutional blocks and trainable replicas There is bidirectional communication between the corresponding cascaded convolutional blocks in the encoder.

[0044] Correspondingly, the trainable replica The features extracted by the third cascade convolution block in the decoder are fed into the corresponding fourth cascade convolution block, and one cascade convolution block is used as the output of the third cascade convolution block. When the corresponding target domain positive sample is , corresponding to the negative sample of the target domain, the output is recorded as ; Trainable replica The features extracted by the fourth cascade convolution block are fed into the corresponding fifth cascade convolution block, and one channel is used as the output of the fourth cascade convolution block. When it corresponds to a positive sample in the target domain, the output is recorded as , corresponding to the negative sample of the target domain, the output is recorded as ; The feature output extracted by the fifth cascade convolution block is recorded as , corresponding to the negative sample of the target domain, is recorded as .

[0045] The fusion module is the residual pyramid convolution module , which consists of three cascaded convolution blocks arranged in input-output order. The two input ends of the first cascaded convolution block in the fusion module are connected to the "locked" network The output end of the zero convolution layer Z2 is used to fuse the visible light band features of three different depths with the corresponding first target shortwave band features to obtain two fusion features of different depths. Specifically, the features output by the first cascade convolution block in the fusion module are fed into the second cascade convolution block, and the features extracted by the second cascade convolution block are fed into the third cascade convolution block, and all the way is used as the fusion feature output of the corresponding depth of the second cascade convolution block, which is recorded as the sub-fusion feature. , the features extracted by the third cascade convolution block are used as the output of the entire fusion module and are recorded as sub-fusion features .

[0046] Step 4: Training of the generalized model in the turbulence degradation restoration domain.

[0047] Input source domain samples into the "locked" network In the process, the positive samples or negative samples of the target domain are input into the zero convolution layer Z1 at the same time to train the generalization model of the turbulence degradation restoration domain and obtain the trained generalization model of the turbulence degradation restoration domain.

[0048] When training the turbulence degradation restoration domain generalization model, the contrast loss function, content loss function, and structure loss function are used to optimize the turbulence degradation restoration domain generalization model. The details are as follows:

[0049] (1) By minimizing the distance between the visible light band features of three different depths and the corresponding first target shortwave band features through the two-norm, the loss function of the positive sample in the target domain is obtained. , which is expressed as follows:

[0050] , i=1,2,3;

[0051] By maximizing the distance between the visible light band features of three different depths and the corresponding second target shortwave band features through the two-norm, the loss function of the target domain negative sample is obtained. , which is expressed as follows:

[0052] , i=1,2,3;

[0053] in, The shortwave band characteristics of the second target at three different depths.

[0054] The contrastive loss function is used to learn the similar features of the same target in the visible light band and the shortwave band on the positive samples in the target domain, and to distinguish the discriminative features of different targets in the visible light band and the shortwave band on the negative samples in the target domain.

[0055] Contrastive loss function Calculated by the following formula:

[0056] ;

[0057] in, and They are and The weighting coefficient of .

[0058] (2) Using content loss function, pair fusion features The distance and sub-fusion features between the shortwave clear image V2 in the training set The distance between the image V1 and the visible light clear image V1 in the training set is minimized and optimized respectively; in this embodiment, the content loss function is the pre-trained VGG19 network.

[0059] (3) Using structural loss function, sub-fusion features The distance and sub-fusion features between the shortwave clear image V2 in the training set The distance between the image and the visible light clear image V1 in the training set is minimized and optimized respectively; in this embodiment, the structural loss function is the multi-scale structure function MS-SSIM.

[0060] The weights and biases of the zero-convolution layer Z1 and the zero-convolution layer Z2 are both 0 at the initial moment of training. As the training progresses, the weights and biases will gradually increase from zero, that is, the interaction weights will gradually increase, which is used to ensure that the turbulence-degraded restoration domain generalization model trained in the visible light band can express the similarity characteristics of the same target in the visible light band and the shortwave band without causing sudden changes in the model performance.

[0061] Step 5: Input the visible light turbulence-degraded image D1 and the shortwave turbulence-degraded image D2 in the test set into the “locked” network in the trained turbulence-degraded restoration domain generalization model. In the zero convolution layer Z1, the fusion module fuses the visible light band features with the shortwave band features of the first target, outputs a restored clear image of the visible light turbulence-degraded image D1, and completes the restoration of the turbulence-degraded image under wide spectrum distribution offset.

[0062] The "locked" network that generates branches in the present invention and a trainable copy of the bootstrap branch Repeated interaction during feature extraction, i.e., "locking" the network during encoding and decoding Not only does it obtain trainable copies after traversing multiple cascaded convolutional blocks The feedback control signal “locks” the network The output of each cascaded convolutional block in Individual influence, as indicated by the red arrow; trainable replicas It is also necessary to obtain the "locked" network in time The characteristic output of is used as feedback regulation, as shown by the green arrow. With trainable copies The two systems interact with each other in a "two-way, high-frequency, and large-bandwidth" manner, which not only improves the efficiency of feature interaction, but also improves the accuracy of feature extraction.

[0063] The above description is only used to illustrate the technical solution of the present invention, rather than to limit it. For ordinary professional and technical personnel in this field, the specific technical solutions recorded in the above embodiments can be modified, or some of the technical features therein can be replaced by equivalents. These modifications or replacements do not cause the essence of the corresponding technical solution to deviate from the scope of the technical solution protected by the present invention.

Claims

1. A domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift, characterized by: The following steps are involved: Step 1: Construct a training set and a test set; the training set includes a one-to-one correspondence between a visible light turbulence-degraded image D1, a visible light clear image V1, a shortwave turbulence-degraded image D2, and a shortwave clear image V2 based on a first target, and a shortwave turbulence-degraded image D3 based on a second target; the test set includes a one-to-one correspondence between a visible light turbulence-degraded image D1 and a shortwave turbulence-degraded image D2 based on the first target; Step 2: Use the visible light turbulence-degraded image D1, shortwave turbulence-degraded image D2, and shortwave turbulence-degraded image D3 in the training set as source domain samples, target domain positive samples, and target domain negative samples, respectively; Step 3: Construct a generalized model for turbulence-degraded restoration domain; the generalized model for turbulence-degraded restoration domain includes a generation branch, a guidance branch, and a fusion module; the generation branch is a "locked" network, which is a turbulence-degraded image restoration model based on the UNet architecture and with frozen parameters trained on the visible light band; the guidance branch includes a zero convolution layer Z1, a trainable copy, and a zero convolution layer Z2, which are arranged in the order of input and output; The input end of the "locked" network is used to receive source domain samples to extract features from the source domain samples and reconstruct the extracted features to obtain visible light band features at three different depths; The input end of the zero convolution layer Z1 is used to receive the target domain positive sample or the target domain negative sample and expand it into three channels; the trainable copy is the trainable copy of the "locked" network, which communicates bidirectionally with the "locked" network. The trainable copy and the zero convolution layer Z2 are used to extract features of the three-channel target domain positive sample or target domain negative sample to obtain the first target shortwave band features or the second target shortwave band features at three different depths; The two input ends of the fusion module are respectively connected to the output end of the "locked" network and the zero convolution layer Z2, and are used to fuse the visible light band features of three different depths with the corresponding first target shortwave band features to obtain two fusion features of different depths; Step 4: Input the source domain samples into the "locked" network, and input the target domain positive samples or target domain negative samples into the zero convolution layer Z1. Combined with the visible light clear image V1 and shortwave clear image V2 in the training set, the visible light band features of three different depths generated during the training process, the first target shortwave band features, the second target shortwave band features, and the fusion features, the turbulence degradation restoration domain generalization model is trained to obtain the trained turbulence degradation restoration domain generalization model. In step 5, the visible light turbulence-degraded image D1 and the shortwave turbulence-degraded image D2 in the test set are respectively input into the "locked" network and the zero convolution layer Z1 in the trained turbulence-degraded restoration domain generalization model. The fusion module fuses the visible light band features of three different depths with the corresponding shortwave band features of the first target, and outputs a restored clear image of the shortwave band turbulence-degraded image D2, completing the restoration of the turbulence-degraded image under the wide spectrum distribution offset.

2. The domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift according to claim 1, characterized in that: In step 3, the "locked" network includes the encoder and decoder of the UNet architecture; The encoder of the "locked" network is provided with at least three cascaded convolution blocks arranged in input-output order for extracting features from source domain samples, and the decoder is provided with at least three cascaded convolution blocks arranged in input-output order for reconstructing the extracted features; Among them, the last three cascaded convolution blocks in the decoder of the "locked" network are also used to output visible light band features of three different depths.

3. The domain generalization method for turbulence-degraded image restoration under wide-band distribution shift according to claim 2, characterized in that: In step 3, the number of cascaded convolutional blocks in the encoder of the "locked" network is five; There is bidirectional communication between the five cascaded convolutional blocks in the encoder of the “deadlocked” network and the corresponding cascaded convolutional blocks in the encoder of the trainable counterpart.

4. The domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift according to any one of claims 1 to 3, characterized in that: In step 3, the fusion module is a residual pyramid convolution module, which includes three cascade convolution blocks arranged in input and output order; the two input ends of the first cascade convolution block in the fusion module are respectively connected to the "locked" network and the output of the zero convolutional layer Z2; The second cascade convolution block of the fusion module is also used to separately output the fusion features of its corresponding depth.

5. The domain generalization method for turbulence-degraded image restoration under wide-band distribution shift according to claim 4, characterized in that: In step 4, when training the turbulence degradation restoration domain generalization model, the turbulence degradation restoration domain generalization model is optimized jointly by the contrast loss function, content loss function and structure loss function.

6. The domain generalization method for turbulence-degraded image restoration under wide-band distribution shift according to claim 5, characterized in that: In step 4, the turbulence degradation restoration domain generalization model is optimized by comparing the loss function, content loss function and structure loss function. Specifically: Step a1, the contrast loss function The expression is: ； in, is the loss function of the positive sample in the target domain, , i=1,2,3; is the loss function of negative samples in the target domain, , i=1,2,3; are the visible light band characteristics at three different depths, The shortwave band characteristics of the first target at three different depths, The shortwave band characteristics of the second target at three different depths; and They are and The weighting coefficient of Step b1: record the output of the second cascade convolution block in the fusion module as the sub-fusion feature , the output of the third cascade convolution block is recorded as the sub-fusion feature ; Using content loss function, pair fusion features The distance and sub-fusion features between the shortwave clear image V2 in the training set The distance between the visible light clear image V1 in the training set is minimized and optimized respectively; Step c1, using the structural loss function, the sub-fusion features The distance and sub-fusion features between the shortwave clear image V2 in the training set The distance between the visible light clear image V1 in the training set is minimized and optimized respectively.

7. The domain generalization method for turbulence-degraded image restoration under wide spectral distribution shift according to claim 6, characterized in that: In step 4, the content loss function is a pre-trained VGG19 network; The structural loss function is the multi-scale structure function MS-SSIM.

Images