GAN-based phase coherent image generation method

Abstract

The invention discloses a GAN-based phase coherent image generation method, and relates to the technical field of image processing. The method comprises the following steps: firstly, constructing a PCGNet network comprising a preprocessing module, an image generation module and a phase correction module, designing a generator loss function and setting a discriminator; the method comprises the following steps: acquiring a source image, preprocessing the source image, and inputting the preprocessed source image into a PCGNet network, wherein the preprocessing comprises Gaussian blur and shallow feature extraction; carrying out three times of down-sampling and up-sampling processing on the preprocessed feature map, and carrying out frequency domain processing through a phase correction module to optimize phase coherence; the generator integrates pixel-level, perception, confrontation and phase loss functions to form a total loss function, and a Real-ESRGAN discriminator is adopted to provide pixel-level feedback; and evaluating the image phase difference based on an optical transfer function (OTF). The method can eliminate the influence of the phase information of the imaging target, the evaluation result is only related to the imaging system, and the phase coherence image can be effectively generated and evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a GAN-based phase coherent image generation method. Background Art

[0002] In many scientific research and practical application fields today, such as astronomical observation, medical imaging, remote sensing detection, etc., the acquisition and processing of high-quality images are crucial. Among them, phase coherence plays a key role in the image generation and analysis process. Especially for imaging systems based on technologies such as synthetic aperture, maintaining the phase coherence of the image is one of the core elements to achieve high-resolution and precise imaging.

[0003] Synthetic aperture imaging technology improves image resolution by integrating information obtained from multiple sub-apertures. However, it faces many challenges in practical applications. On the one hand, it is not easy for the signals received by the physical device to maintain coherence. Various factors will lead to the generation of common phase errors, thus affecting the image quality. These factors include both internal factors such as the unavoidable aberrations of the optical system itself and sub-aperture installation errors, as well as external interferences such as atmospheric turbulence, mechanical vibrations, and photosensitive element noise. Although researchers have conducted a lot of research and proposed a variety of common phase error detection and correction methods, there is still a lot of room for research in using advanced image generation technology to generate high-quality images that meet phase coherence conditions.

[0004] Since the generative adversarial network (GAN) was proposed in 2014, it has achieved remarkable results in the field of image generation. Through the mutual game between the generative network and the discriminative network, GAN can learn the distribution of real data and then generate outputs that are very similar to the real data. Based on the GAN framework, many excellent network variants have emerged and are widely used in many fields such as image super-resolution, image denoising, image restoration, and natural image conversion. However, there is currently insufficient research on the specific task of phase coherent image generation. Most of the existing GAN-based image generation methods focus on generating results that are visually similar to real images, but often ignore the phase information of the image, which makes it difficult to generate high-quality images that meet the phase coherence requirements.

[0005] In the field of image evaluation, although there are many mature evaluation indicators, such as the mean opinion score (MOS) in subjective evaluation indicators, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) in objective evaluation indicators. These indicators play an important role in evaluating image quality from different perspectives. For example, PSNR mainly measures the difference in image signal strength, SSIM focuses on evaluating the similarity of image structure, and MOS evaluates image quality based on human subjective perception. However, they mainly focus on the intensity, structure and human perception of the image, and do not fully consider the phase information of the image. Therefore, they cannot be effectively used to evaluate the phase coherence between the generated image and the source image, and it is difficult to accurately judge whether the generated image truly meets the phase coherence requirements.

[0006] In summary, in application scenarios such as synthetic aperture imaging that have strict requirements on phase coherence, there is an urgent need for a method that can effectively generate phase-coherent images, and at the same time, an indicator that can accurately evaluate the phase coherence of the image, so as to fill the gap in the existing technology in this regard and meet the needs of related fields for high-quality imaging. Summary of the invention

[0007] In view of the shortcomings of the prior art, the object of the present invention is to provide a phase coherent image generation method based on GAN.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A phase coherent image generation method based on GAN, comprising the following steps:

[0010] Construct the PCGNet network, including the preprocessing module, image generation module and phase correction module; design the generator loss function and set the discriminator;

[0011] Get the source image I and perform preprocessing operations to generate an image through the PCGNet network;

[0012] The preprocessed feature map X is downsampled and upsampled three times to obtain the output feature map;

[0013] The phase information of the input feature map is corrected and optimized by frequency domain analysis and processing to improve the phase coherence of the final generated image;

[0014] In the generator, multiple loss functions are combined to form the total loss function of the generator; the discriminator in Real-ESRGAN is used to provide pixel-level feedback to the generator;

[0015] Image phase difference evaluation is performed based on the optical transfer function (OTF).

[0016] Preferably, the preprocessing module includes Gaussian blur processing and shallow feature extraction and channel number change;

[0017] The image generation module is constructed using an encoder-decoder structure, including a downsampling module and an upsampling module; the downsampling module includes an adaptive downsampling operation, a normalization process, and an activation function application; the upsampling module includes bilinear interpolation upsampling, feature fusion, skip connection application, and phase information processing;

[0018] The phase correction module consists of three convolutional layers and three correctors, including two-dimensional discrete Fourier transform, phase value adaptive adjustment and inverse Fourier transform reconstruction;

[0019] The generator loss function design includes calculating pixel-level L1 loss, perceptual loss L percep , against the loss L ADVG And the phase loss L phase ;

[0020] The discriminator directly adopts the discriminator in Real-ESRGAN; the discriminator has two inputs, one is the real image, that is, the preprocessed source image, and the other is the image generated by the generator.

[0021] Preferably, a source image I is obtained and preprocessed to generate an image through a PCGNet network, including:

[0022] Apply a Gaussian blur operation to the source image I, which is implemented by the Gaussian blur function G(·) to reduce the noise in the image;

[0023] The source image I is retained as the label information, and the image after Gaussian blur processing is used as the input of the PCGNet network, denoted as X, that is, X = G(I);

[0024] After Gaussian blurring, a 3×3 convolutional layer is used to process the blurred image X.

[0025] The preprocessed image X is input into the PCGNet network, and after a series of processing by the network structure function F(·), the generated image Y is finally obtained. The whole process can be expressed as Y=F(X).

[0026] Preferably, the preprocessed feature map is subjected to three downsampling and upsampling processes to obtain an output feature map, comprising:

[0027] The preprocessed feature map X is input into the downsampling module of the image generation module, and three downsampling operations are performed according to the design of the downsampling module;

[0028] Each downsampling operation includes three steps: adaptive downsampling, normalization, and activation function application, which halves the height and width of the feature map, doubles the number of channels, and gradually extracts feature information at different levels.

[0029] After completing the three downsampling operations, the feature map finally output by the downsampling module is input into the upsampling module for three upsampling operations;

[0030] Each upsampling operation includes four steps: bilinear interpolation upsampling, feature fusion, jump connection application and phase information processing. The size of the downsampled feature map is gradually restored to a relatively large size, while integrating the feature information of different channels, retaining more detail information, and enhancing the ability to retain phase information. Finally, the output feature map is obtained after three downsampling and upsampling processes.

[0031] Preferably, the phase information of the input feature map is corrected and optimized by frequency domain analysis and processing to improve the phase coherence of the final generated image, including:

[0032] Get the feature map H of the upper layer input, which has a specific image size of M×N, where (x, y) represents the coordinates of the image pixel in the spatial domain, and each pixel has a corresponding pixel value h(x, y);

[0033] The input feature map H is subjected to a two-dimensional discrete Fourier transform to calculate the amplitude spectrum and phase spectrum of the feature map, as shown in the following formula:

[0034]

[0035] A H (u,v)=|H(u,v)

[0036] Φ H (u,v)=arg(H(u,v))

[0037] This transformation operation converts the feature map from the spatial domain to the spectral domain. In the spectral domain, (u, v) represents the coordinates of the spatial frequency on the spectrum, and after transformation, the complex frequency value H(u, v) is obtained. e and i are the Euler number and imaginary unit. A H (u,v) is the calculated amplitude value, and Φ H (u,v) is the calculated phase value;

[0038] A learnable parameter ξ with the same dimension as the phase spectrum is introduced to enable adaptive adjustment of each phase value of the feature map. The adjusted phase spectrum and amplitude spectrum are reconstructed through inverse Fourier transform to obtain a spatial domain image, so as to improve the phase coherence of the final generated image. The process can be expressed as:

[0039] Φ' H (u,v)=Φ H (u,v)+ξ(u,v)π

[0040]

[0041] Preferably, multiple loss functions are integrated in the generator to form the total loss function of the generator; the discriminator in Real-ESRGAN is used to provide pixel-level feedback to the generator, including:

[0042] In the process of generating images, various loss function values ​​of the generator are calculated, and these loss function values ​​are combined according to certain weights to obtain the total loss function value L of the generator. G ;

[0043] The perceptual loss L percep Calculation based on pre-trained vgg19 network;

[0044] The phase loss L phase The final phase loss is the sum of the phase difference between the generated image and the real image and the phase difference between the corresponding hierarchical feature maps in the encoding and decoding stages. The calculation can be expressed as:

[0045]

[0046] Among them, φ x (u,v) represents the phase spectrum of the source image, φ y (u,v) represents the phase spectrum of the generated image, and They represent the phase spectra of the feature maps corresponding to the feature levels in the codec;

[0047] The total loss function of the generator is L G It can be expressed as:

[0048] L G =λ1L percep +λ2L ADVG +λ3L1+λ4L phase

[0049] Among them, λ i is the weight of each loss;

[0050] Using the discriminator in Real-ESRGAN, the image generated by the generator and the real image, i.e. the preprocessed source image, are input into the discriminator respectively;

[0051] The discriminator discriminates the input image based on its own network architecture and functional characteristics, and outputs the authenticity value of each pixel; these authenticity values ​​are provided to the generator as pixel-level feedback, allowing the generator to make adjustments and improvements to improve image quality.

[0052] Preferably, performing image phase difference evaluation based on optical transfer function OTF includes:

[0053] Firstly, it is determined that the optical imaging system under study is a multi-aperture imaging system and belongs to an incoherent light imaging system; according to the evaluation method of its imaging performance in the frequency domain, its optical transfer function (OTF) is determined;

[0054] The optical transfer function OTF is composed of a modulation transfer function MTF and a phase transfer function PTF, wherein the modulation transfer function MTF is the modulus part of the optical transfer function OTF, and the phase transfer function PTF is the phase part of the optical transfer function OTF; the expression form of the optical transfer function is OTF(u,v)=MTF(u,v)·e i·PTF(u,v) , where (u,v) are the spatial frequency coordinates in the frequency domain, e and i are the Euler number and imaginary unit respectively;

[0055] The imaging of the optical imaging system in the frequency domain can be expressed as: G(f x ,f y )=F(f x ,f y )H(f x ,f y )+N(f x ,f y ), where H(f x ,f y ) is the optical sensing function of the incoherent light imaging system; F(f x ,f y ) is the spectrum of the real object image; G(f x ,f y ) is the spectrum of the image directly observed by the incoherent light imaging system; N(f x ,f y ) is the noise spectrum;

[0056] The ideal spectrum of the imaging target is represented as F, and the image formed by the ideal image through the imaging device 1 is I1, so the imaging process can be expressed as: I1 = F * OTF1, where OTF1 is the optical transfer function of imaging device 1; similarly, the image generation model is regarded as another imaging device denoted as imaging device 2, and the image generated by the image generation model is denoted as I2. The imaging process can be expressed as: I2 = F * OTF2, where OTF2 is the optical transfer function of the image generation model;

[0057] Assuming that the whole process is noise-free, let the correlation factor OTF is abbreviated as: but:

[0058]

[0059] because F(u,v)·F*(u,v)=|F(u,v) 2

[0060]

[0061]

[0062] e iθ +e -iθ =2cos(θ)

[0063]

[0064] λ=cos(θ d1 -θ d2 )

[0065] The phase correlation factor not only reflects the phase difference between images, but also has nothing to do with the imaging target and is only related to the phase transfer function PTF in the optical transfer function of the imaging system.

[0066] According to the method of directly calculating the difference in image phase values, based on the mathematical modeling of the imaging process of the optical imaging system in the frequency domain, the cosine function is used to eliminate the influence of the period 2π on the difference, and the phase value difference between images I1 and I2 is calculated. The calculation process is as follows:

[0067] λ'=cos[P(F·OTF1)-P(F·OTF2)]=cos[P(F)·P(OTF1-OTF2)]

[0068] Where P represents the calculation of the image phase value, and the use of the cosine function can eliminate the influence of the period 2π on the difference;

[0069] It can be seen from the calculation results that the direct calculation of the phase value difference is also related to the phase information of the image itself, that is, when the ideal spectrum F of the imaging target changes, the evaluation result of this method will change accordingly.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] 1. In the present invention, through the designed PCGNet network architecture, the phase correction module therein can perform detailed analysis and processing on the phase information of the image; in the phase correction module, the amplitude spectrum and phase spectrum of the feature map are calculated by two-dimensional discrete Fourier transform, and the phase value is adaptively adjusted by introducing learnable parameters, and then the spatial domain image is reconstructed by inverse Fourier transform, which effectively corrects the phase information of the image, making the generated image more accurate and coherent in terms of phase, and meeting the requirements of many application scenarios with strict requirements on phase coherence, such as optical synthetic aperture imaging, certain high-precision medical imaging and other fields.

[0072] 2. In the present invention, the image generation module adopts an encoder-decoder structure, which can not only effectively extract and fuse multi-level features during downsampling and upsampling, but also retain more detailed information through jump connections. At the same time, the phase difference of the feature map at the same level is used as one of the losses of the generator, which prompts the generator to fully consider the phase information when generating the image, thereby generating an image with high coherence with the source image, which can effectively capture the global structure and key detail information of the input image, and improve the overall quality of the generated image.

[0073] 3. In this invention, pixel-level L1 loss and perceptual loss L are comprehensively used in the generator. percep , against the loss L AVDG And the phase loss L phase There are many loss functions. The pixel-level L1 loss measures the difference between the generated image and the real image at the pixel level, prompting the generator to generate more accurate pixel values; the perceptual loss is calculated based on the pre-trained vgg19 network, and evaluates the image quality from the perspective of human perception, so that the generated image is more in line with human perception in terms of visual effects; the adversarial loss interacts with the discriminator to prompt the generator to generate high-quality images that can deceive the discriminator; the phase loss focuses on the phase difference between the generated image and the real image, ensuring that the generated image meets the phase coherence standard. By combining these loss functions of different dimensions into the total loss function of the generator according to certain weights, the difference between the generated image and the desired target can be comprehensively and accurately measured, thereby generating better quality images.

[0074] 4. In the present invention, the discriminator in Real-ESRGAN is used, which can output the authenticity value of each pixel, providing extremely detailed pixel-level feedback to the generator. This pixel-level feedback enables the generator to clearly understand at which pixels the generated image differs from the real image, so as to make targeted adjustments and improvements.

[0075] 5. In the present invention, a method for evaluating image phase difference based on optical transfer function OTF is proposed. Compared with existing image evaluation indicators, this method can accurately evaluate the phase coherence between the generated image and the source image. It combines the ideal spectrum of the imaging target with the optical transfer function of different imaging devices (including conventional imaging devices and image generation models as virtual imaging devices) by reasonably mathematically modeling the imaging process, and derives a phase correlation factor that is independent of the imaging target and only related to the phase transfer function PTF in the optical transfer function of the imaging system. In this way, the evaluation method can completely eliminate the influence of the phase information of the imaging target itself, so that the evaluation result is only related to the imaging system, providing a reliable and accurate basis for judging whether the generated image meets the phase coherence condition, and ensuring that the generated image meets the expected standard in terms of phase coherence. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of the process of a phase coherent image generation method based on GAN proposed in the present invention;

[0077] Figure 2 This is a schematic diagram of the PCGNet network structure proposed by the present invention;

[0078] Figure 3 This is a schematic diagram of the structure of the downsampling module proposed in the present invention;

[0079] Figure 4 This is a schematic diagram of the structure of the upsampling module proposed in the present invention;

[0080] Figure 5 This is a schematic diagram of the phase corrector structure proposed by the present invention;

[0081] Figure 6 This is a schematic diagram of the imaging process in the frequency domain of the optical imaging system proposed in the present invention. DETAILED DESCRIPTION

[0082] Reference Figures 1 to 6 .

[0083] The embodiment further illustrates a GAN-based phase coherent image generation method proposed in the present invention.

[0084] Construct the PCGNet network, including the preprocessing module, image generation module and phase correction module; design the generator loss function and set the discriminator;

[0085] Get the source image I and perform preprocessing operations to generate an image through the PCGNet network;

[0086] The preprocessed feature map X is downsampled and upsampled three times to obtain the output feature map;

[0087] The phase information of the input feature map is corrected and optimized by frequency domain analysis and processing to improve the phase coherence of the final generated image;

[0088] In the generator, multiple loss functions are combined to form the total loss function of the generator; the discriminator in Real-ESRGAN is used to provide pixel-level feedback to the generator;

[0089] Image phase difference evaluation is performed based on the optical transfer function (OTF).

[0090] The preprocessing module includes Gaussian blur processing, shallow feature extraction and channel number change.

[0091] The image generation module is constructed using an encoder-decoder structure, including a downsampling module and an upsampling module; the downsampling module includes an adaptive downsampling operation, normalization processing, and activation function application; the upsampling module includes bilinear interpolation upsampling, feature fusion, jump connection application, and phase information processing.

[0092] The phase correction module consists of three convolutional layers and three correctors, including two-dimensional discrete Fourier transform, phase value adaptive adjustment and inverse Fourier transform reconstruction.

[0093] The generator loss function design includes pixel-level L1 loss, perceptual loss L percep , against the loss L ADVG And the phase loss L phase .

[0094] The discriminator directly adopts the discriminator in Real-ESRGAN; the discriminator has two inputs, one is the real image, that is, the source image after preprocessing, and the other is the image generated by the generator; the discriminator can discriminate the input image and output the authenticity value of each pixel. Through this pixel-level feedback, the generator is provided with information about the difference between the generated image and the real image at each pixel point, so that the generator can make targeted adjustments and improvements based on this, thereby continuously improving the quality of the generated image, making the image generated by the generator closer to the characteristics and quality standards of the real image.

[0095] Get the source image I and perform preprocessing operations to generate an image through the PCGNet network, including:

[0096] A Gaussian blur operation is applied to the source image I, which is implemented by a Gaussian blur function G(·) to reduce the noise in the image.

[0097] The source image I is retained as label information, and the image after Gaussian blur processing is used as the input of the PCGNet network, denoted as X, that is, X = G(I).

[0098] After Gaussian blur processing, a 3×3 convolutional layer is used to process the blurred image X; on the one hand, this convolutional layer can extract shallow features of the image and capture local feature information at different positions of the image; on the other hand, it can also change the number of channels of the image and integrate the extracted feature information into different channels, thus laying the foundation for the subsequent processing of other modules in the PCGNet network (such as image generation module, phase correction module, etc.).

[0099] The preprocessed image X is input into the PCGNet network, and after a series of processing by the network structure function F(·), the generated image Y is finally obtained. The whole process can be expressed as Y=F(X).

[0100] The preprocessed feature map is downsampled and upsampled three times to obtain the output feature map, including:

[0101] The preprocessed feature map X is input into the downsampling module of the image generation module, and three downsampling operations are performed according to the design of the downsampling module.

[0102] Each downsampling operation includes three steps: adaptive downsampling, normalization, and activation function application, which halves the height and width of the feature map, doubles the number of channels, and gradually extracts feature information at different levels. Specifically:

[0103] For the feature map processed by the preprocessing module, a convolution layer is used to perform adaptive downsampling. In this convolution process, the parameters of the convolution layer are set so that the height and width of the feature map are halved and the number of channels is doubled after the convolution operation.

[0104] Since the generated image is specific and mainly depends on the preprocessed source image, Instance Norm needs to be used to normalize it in the HW dimension to accelerate model convergence and maintain the independence of each image instance. Instance Norm will independently normalize each image instance (that is, each input feature map). It calculates the mean and variance of each instance in the HW dimension, and then normalizes the feature value.

[0105] The feature map after the above processing will be activated by the Leaky ReLU activation function to obtain the output of the downsampling module; the implementation of the Leaky ReLU activation function is that when the input is greater than zero, the output is equal to the input; when the input is less than zero, the output is a smaller slope multiplied by the input value (such as the slope is set to 0.01). This can avoid the neuron "death" problem that may occur in the traditional ReLU activation function, so that the neuron can have a certain output under different input conditions, thereby obtaining the output feature map of the downsampling module.

[0106] After completing the three downsampling operations, the feature map finally output by the downsampling module is input into the upsampling module to perform three upsampling operations.

[0107] Each upsampling operation includes four steps: bilinear interpolation upsampling, feature fusion, jump connection application and phase information processing. The size of the downsampled feature map is gradually restored to a relatively large degree, while integrating the feature information of different channels, retaining more detail information, and enhancing the ability to retain phase information. Finally, the output feature map after three downsampling and upsampling processes is obtained, which is as follows:

[0108] First, the input feature map of the upsampling module, that is, the output feature map of the downsampling module, is subjected to bilinear interpolation upsampling operation; bilinear interpolation is a method of estimating the value of new pixels based on the values ​​of surrounding pixels. In this way, the size of the downsampled feature map can be restored to a relatively large extent, making it closer to the size scale of the original feature map;

[0109] After upsampling is completed, the features of different channels are fused through the convolution layer; the convolution kernel of the convolution layer slides on the feature maps of different channels to perform convolution operations, integrating the feature information contained in different channels to better generate a complete image;

[0110] Skip connections are set up between the encoder and decoder. These skip connections will directly transfer the features of the encoding stage to the corresponding level of the decoding stage. In specific implementation, by establishing appropriate indexes or pointers, it is ensured that the features extracted at a certain level in the encoding stage can be accurately transferred to the corresponding level in the decoding stage, so as to retain more detailed information and avoid excessive loss of information during multiple processing, thus helping to generate clearer and more detailed images.

[0111] In order to enhance the ability of the entire generation module to retain phase information, the phase information of the feature map is extracted at each layer in the encoding stage and the corresponding decoding stage; the L1 distance is used to measure the difference between the phase information of these feature maps, that is, the phase information of each pair of feature maps at corresponding levels is subtracted and the absolute value is taken, and then the difference of all corresponding levels is summed to obtain a numerical value representing the difference in phase information; this value is used as one of the loss functions of the generator, which prompts the generator to pay more attention to the accurate generation of phase when generating images, so that the generated image is closer to the source image in terms of phase.

[0112] The phase information of the input feature map is corrected and optimized by frequency domain analysis and processing to improve the phase coherence of the final generated image, including:

[0113] Get the feature map H of the upper layer input, which has a specific image size of M×N, where (x, y) represents the coordinates of the image pixel in the spatial domain, and each pixel has a corresponding pixel value h(x, y).

[0114] The input feature map H is subjected to a two-dimensional discrete Fourier transform to calculate the amplitude spectrum and phase spectrum of the feature map, as shown in the following formula:

[0115]

[0116] A H (u,v)=|H(u,v)

[0117] Φ H (u,v)=arg(H(u,v))

[0118] This transformation operation converts the feature map from the spatial domain to the spectral domain. In the spectral domain, (u, v) represents the coordinates of the spatial frequency on the spectrum, and after transformation, the complex frequency value H(u, v) is obtained. e and i are the Euler number and imaginary unit. A H (u,v) is the calculated amplitude value, and Φ H (u,v) is the calculated phase value.

[0119] A learnable parameter ξ with the same dimension as the phase spectrum is introduced to enable adaptive adjustment of each phase value of the feature map. The adjusted phase spectrum and amplitude spectrum are reconstructed through inverse Fourier transform to obtain a spatial domain image, so as to improve the phase coherence of the final generated image. The process can be expressed as:

[0120] Φ' H (u,v)=Φ H (u,v)+ξ(u,v)π

[0121]

[0122] In the generator, multiple loss functions are combined to form the total loss function of the generator; the discriminator in Real-ESRGAN is used to provide pixel-level feedback to the generator, including:

[0123] In the process of generating images, various loss function values ​​of the generator are calculated, and these loss function values ​​are combined according to certain weights to obtain the total loss function value L of the generator. G .

[0124] The perceptual loss L percep Computed based on a pre-trained vgg19 network.

[0125] The phase loss L phaseThe final phase loss is the sum of the phase difference between the generated image and the real image and the phase difference between the corresponding hierarchical feature maps in the encoding and decoding stages. The calculation can be expressed as:

[0126]

[0127] Among them, φ x (u,v) represents the phase spectrum of the source image, φ y (u,v) represents the phase spectrum of the generated image, and They represent the phase spectra of the feature maps corresponding to the feature levels in the codec.

[0128] The total loss function L of the generator can be expressed as:

[0129] L G =λ1L percep +λ2L ADVG +λ3L1+λ4L phase

[0130] Among them, λ i is the weight of each loss;

[0131] The discriminator in Real-ESRGAN is used, and the image generated by the generator and the real image (preprocessed source image or other real reference image) are input into the discriminator respectively.

[0132] The discriminator discriminates the input image based on its own network architecture and functional characteristics, and outputs the authenticity value of each pixel; the discriminator finally outputs the authenticity value of each pixel, and this authenticity value is usually mapped to the interval [0,1]; a value close to 1 indicates that the discriminator has a high probability of judging that the image is a real image, and a value close to 0 indicates that it has a high probability of judging that the image is a generated image; in this way, the discriminator provides pixel-level feedback to the generator, allowing the generator to clearly understand at which pixels the generated image differs from the real image, so that it can make targeted adjustments and improvements to further improve the quality of the generated image.

[0133] Image phase difference evaluation based on optical transfer function (OTF), including:

[0134] Firstly, it is necessary to determine that the optical imaging system under study is a multi-aperture imaging system and an incoherent light imaging system; according to the evaluation method of its imaging performance in the frequency domain, its optical transfer function OTF is determined.

[0135] It is clear that the optical transfer function OTF is composed of the modulation transfer function MTF and the phase transfer function PTF; among them, the modulation transfer function MTF is the modulus part of the optical transfer function OTF, which reflects the contrast transmission ability of the optical system for different spatial frequency components and is related to the details and clarity of the image; the phase transfer function PTF is the phase part of the optical transfer function OTF, which reflects the phase change of the optical system for different spatial frequency components and reflects the distortion of the wavefront during the imaging process. In an ideal state, the optical system should keep the phase consistent for all frequency components; at the same time, it is clear that the expression form of the optical transfer function is OTF(u,v)=MTF(u,v)·e i·PTF(u,v) , where (u,v) are the spatial frequency coordinates in the frequency domain, e and i are the Euler number and imaginary unit, respectively.

[0136] The imaging of the optical imaging system in the frequency domain can be expressed as: G(f x ,f y )=F(f x ,f y )H(f x ,f y )+N(f x ,f y ), where H(f x ,f y ) is the optical sensing function of the incoherent light imaging system; F(f x ,f y ) is the spectrum of the real object image; G(f x ,f y ) is the spectrum of the image directly observed by the incoherent light imaging system; N(f x ,f y ) is the noise spectrum.

[0137] The ideal spectrum of the imaging target is represented as F, and the image formed by the ideal image through the imaging device 1 is I1, so the imaging process can be expressed as: I1 = F * OTF1, where OTF1 is the optical transfer function of imaging device 1; similarly, the image generation model is regarded as another imaging device denoted as imaging device 2, and the image generated by the image generation model is denoted as I2. The imaging process can be expressed as: I2 = F * OTF2, where OTF2 is the optical transfer function of the image generation model.

[0138] Assuming that the whole process is noise-free, let the correlation factor OTF is abbreviated as: but:

[0139]

[0140] because F(u,v)·F*(u,v)=|F(u,v) 2

[0141]

[0142] e iθ +e -i θ=2cos(θ)

[0143]

[0144] λ=cos(θ d1 -θ d2 )

[0145] The phase correlation factor not only reflects the phase difference between images, but also has nothing to do with the imaging target and is only related to the phase transfer function PTF in the optical transfer function of the imaging system.

[0146] According to the method of directly calculating the difference in image phase values, based on the mathematical modeling of the imaging process of the optical imaging system in the frequency domain, the cosine function is used to eliminate the influence of the period 2π on the difference, and the phase value difference between images I1 and I2 is calculated. The calculation process is as follows:

[0147] λ'=cos[P(F·OTF1)-P(F·OTF2)]=cos[P(F)·P(OTF1-OTF2)]

[0148] Wherein, P represents the calculation of the image phase value, and the use of the cosine function can eliminate the influence of the period 2π on the difference.

[0149] It can be seen from the calculation results that the direct calculation of the phase value difference is also related to the phase information of the image itself, that is, when the ideal spectrum F of the imaging target changes, the evaluation result of this method will change accordingly.

[0150] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A GAN-based phase coherent image generation method, characterized in that: The following steps are involved: Construct the PCGNet network, including the preprocessing module, image generation module and phase correction module; Generator loss function design and discriminator setting; Get the source image I and perform preprocessing operations to generate an image through the PCGNet network; The preprocessed feature map X is downsampled and upsampled three times to obtain the output feature map; The phase information of the input feature map is corrected and optimized by frequency domain analysis and processing to improve the phase coherence of the final generated image; In the generator, multiple loss functions are combined to form the total loss function of the generator; the discriminator in Real-ESRGAN is used to provide pixel-level feedback to the generator; Image phase difference evaluation is performed based on the optical transfer function (OTF).

2. According to claim 1, a GAN-based phase coherent image generation method is characterized in that: The preprocessing module includes Gaussian blur processing, shallow feature extraction and channel number change; The image generation module is constructed using an encoder-decoder structure, including a downsampling module and an upsampling module; the downsampling module includes an adaptive downsampling operation, a normalization process, and an activation function application; the upsampling module includes bilinear interpolation upsampling, feature fusion, skip connection application, and phase information processing; The phase correction module consists of three convolutional layers and three correctors, including two-dimensional discrete Fourier transform, phase value adaptive adjustment and inverse Fourier transform reconstruction; The generator loss function design includes calculating pixel-level L1 loss, perceptual loss L percep , against the loss L ADVG And the phase loss L phase ; The discriminator directly adopts the discriminator in Real-ESRGAN; the discriminator has two inputs, one is the real image, that is, the preprocessed source image, and the other is the image generated by the generator.

3. The method for generating a phase coherent image based on GAN according to claim 2, characterized in that: Get the source image I and perform preprocessing operations to generate an image through the PCGNet network, including: Apply a Gaussian blur operation to the source image I, which is implemented by the Gaussian blur function G(·) to reduce the noise in the image; The source image I is retained as the label information, and the image after Gaussian blur processing is used as the input of the PCGNet network, denoted as X, that is, X = G(I); After Gaussian blurring, a 3×3 convolutional layer is used to process the blurred image X. The preprocessed image X is input into the PCGNet network, and after a series of processing by the network structure function F(·), the generated image Y is finally obtained. The whole process can be expressed as Y=F(X).

4. The method for generating a phase coherent image based on GAN according to claim 3, characterized in that: The preprocessed feature map is downsampled and upsampled three times to obtain the output feature map, including: The preprocessed feature map X is input into the downsampling module of the image generation module, and three downsampling operations are performed according to the design of the downsampling module; Each downsampling operation includes three steps: adaptive downsampling, normalization, and activation function application, which halves the height and width of the feature map, doubles the number of channels, and gradually extracts feature information at different levels. After completing the three downsampling operations, the feature map finally output by the downsampling module is input into the upsampling module for three upsampling operations; Each upsampling operation includes four steps: bilinear interpolation upsampling, feature fusion, jump connection application and phase information processing. The size of the downsampled feature map is gradually restored to a relatively large size, while integrating the feature information of different channels, retaining more detail information, and enhancing the ability to retain phase information. Finally, the output feature map is obtained after three downsampling and upsampling processes.

5. The method for generating a phase coherent image based on GAN according to claim 4, characterized in that: The phase information of the input feature map is corrected and optimized by frequency domain analysis and processing to improve the phase coherence of the final generated image, including: Get the feature map H of the upper layer input, which has a specific image size of M×N, where (x, y) represents the coordinates of the image pixel in the spatial domain, and each pixel has a corresponding pixel value h(x, y); The input feature map H is subjected to a two-dimensional discrete Fourier transform to calculate the amplitude spectrum and phase spectrum of the feature map, as shown in the following formula: A H (u,v)=|H(u,v) Φ H (u,v)=arg(H(u,v)) This transformation operation converts the feature map from the spatial domain to the spectral domain. In the spectral domain, (u, v) represents the coordinates of the spatial frequency on the spectrum, and after transformation, the complex frequency value H(u, v) is obtained. e and i are the Euler number and imaginary unit. A H (u,v) is the calculated amplitude value, and Φ H (u,v) is the calculated phase value; A learnable parameter ξ with the same dimension as the phase spectrum is introduced to enable adaptive adjustment of each phase value of the feature map. The adjusted phase spectrum and amplitude spectrum are reconstructed through inverse Fourier transform to obtain a spatial domain image, so as to improve the phase coherence of the final generated image. The process can be expressed as: F' H (u,v)=Φ H (u,v)+ξ(u,v)π 6. The method for generating a phase coherent image based on GAN according to claim 5, characterized in that: In the generator, multiple loss functions are combined to form the total loss function of the generator; the discriminator in Real-ESRGAN is used to provide pixel-level feedback to the generator, including: In the process of generating images, various loss function values ​​of the generator are calculated, and these loss function values ​​are combined according to certain weights to obtain the total loss function value L of the generator. G ; The perceptual loss L percep Calculation based on pre-trained vgg19 network; The phase loss L phase The final phase loss is the sum of the phase difference between the generated image and the real image and the phase difference between the corresponding hierarchical feature maps in the encoding and decoding stages. The calculation can be expressed as: Among them, φ x (u,v) represents the phase spectrum of the source image, φ y (u,v) represents the phase spectrum of the generated image, and They represent the phase spectra of the feature maps corresponding to the feature levels in the codec; The total loss function of the generator is L G It can be expressed as: Among them, λ i is the weight of each loss; Using the discriminator in Real-ESRGAN, the image generated by the generator and the real image, i.e. the preprocessed source image, are input into the discriminator respectively; The discriminator discriminates the input image based on its own network architecture and functional characteristics, and outputs the authenticity value of each pixel; these authenticity values ​​are provided to the generator as pixel-level feedback, allowing the generator to make adjustments and improvements to improve image quality.

7. The method for generating phase coherent images based on GAN according to claim 6, characterized in that: Image phase difference evaluation based on optical transfer function (OTF), including: Firstly, it is determined that the optical imaging system under study is a multi-aperture imaging system and belongs to an incoherent light imaging system; according to the evaluation method of its imaging performance in the frequency domain, its optical transfer function (OTF) is determined; The optical transfer function OTF is composed of a modulation transfer function MTF and a phase transfer function PTF, wherein the modulation transfer function MTF is the modulus part of the optical transfer function OTF, and the phase transfer function PTF is the phase part of the optical transfer function OTF; the expression form of the optical transfer function is OTF(u,v)=MTF(u,v)·e i·PTF(u,v) , where (u,v) are the spatial frequency coordinates in the frequency domain, e and i are the Euler number and imaginary unit respectively; The imaging of the optical imaging system in the frequency domain can be expressed as: G(f x ,f y )=F(f x ,f y )H(f x ,f y )+N(f x ,f y ), where H(f x ,f y ) is the optical sensing function of the incoherent light imaging system; F(f x ,f y ) is the spectrum of the real object image; G(f x ,f y ) is the spectrum of the image directly observed by the incoherent light imaging system; N(f x ,f y ) is the noise spectrum; The ideal spectrum of the imaging target is represented as F, and the image formed by the ideal image through the imaging device 1 is I1, so the imaging process can be expressed as: I1 = F * OTF1, where OTF1 is the optical transfer function of imaging device 1; similarly, the image generation model is regarded as another imaging device denoted as imaging device 2, and the image generated by the image generation model is denoted as I2. The imaging process can be expressed as: I2 = F * OTF2, where OTF2 is the optical transfer function of the image generation model; Assuming that the whole process is noise-free, let the correlation factor OTF is abbreviated as: but: because F(u,v)·F*(u,v)=|F(u,v) 2 and iθ +e -iθ =2cos(θ) λ=cos(θ d1 -θ d2 ) The phase correlation factor not only reflects the phase difference between images, but also has nothing to do with the imaging target and is only related to the phase transfer function PTF in the optical transfer function of the imaging system. According to the method of directly calculating the difference in image phase values, based on the mathematical modeling of the imaging process of the optical imaging system in the frequency domain, the cosine function is used to eliminate the influence of the period 2π on the difference, and the phase value difference between images I1 and I2 is calculated. The calculation process is as follows: λ'=cos[P(F·OTF1)-P(F·OTF2)]=cos[P(F)·P(OTF1-OTF2)] Where P represents the calculation of the image phase value, and the use of the cosine function can eliminate the influence of the period 2π on the difference; It can be seen from the calculation results that the direct calculation of the phase value difference is also related to the phase information of the image itself, that is, when the ideal spectrum F of the imaging target changes, the evaluation result of this method will change accordingly.

Images