Speckle image restoration method and system based on generative adversarial network

Abstract

The invention discloses a speckle image restoration method and system based on a generative adversarial network, and relates to the technical field of image processing, and the method comprises the steps: obtaining an original image and a corresponding speckle image, and carrying out the preprocessing of the original image and the corresponding speckle image, and obtaining a training set; constructing an image restoration model based on the generative adversarial network; training an image restoration model based on the training set and the target loss function to obtain a trained image restoration model; acquiring a to-be-processed speckle image, and inputting the to-be-processed speckle image into the trained image restoration model; inputting a to-be-processed speckle image into the initial convolution module to obtain shallow layer features; inputting the shallow-layer features into a down-sampling module to obtain deep-layer features; inputting the deep features into a residual module to obtain processing features; inputting the processed features into an upper sampling module to obtain reduced features; and inputting the restored features to an output module to obtain a restored image. And the speckle noise of the speckle image is accurately removed, so that the quality of the restored image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a speckle image restoration method and system based on a generative adversarial network. Background Art

[0002] At present, multimode fiber (MMF) has a wider application prospect in the fields of optical communication, medical endoscopy, micro imaging, etc. due to its larger mode field diameter and higher mode capacity. However, during the optical transmission process, complex mode coupling, phase perturbation, and interference effects cause the incident image to form a highly complex speckle pattern at the output end, resulting in serious distortion or even loss of image information, which greatly limits its practical application in the field of high-quality imaging. Traditional multimode fiber imaging methods mainly include physical methods such as wavefront shaping, optical phase conjugation, and transmission matrix modeling. These methods rely on precise optical modeling and hardware calibration, usually requiring high equipment costs, being sensitive to environmental changes, and being difficult to perform real-time imaging or extend to complex scenarios, making it difficult to meet the actual requirements of high-resolution and high-robustness imaging.

[0003] With the development of deep learning, data-driven methods have gradually shown powerful performance in multimode fiber image restoration. Convolutional neural network (CNN) is widely used to learn the non-linear mapping relationship between the input speckle image and the original image, realizing end-to-end image reconstruction, and significantly outperforming traditional methods in terms of reconstruction accuracy and speed. However, most methods only focus on the mapping between single-modal speckle images and original images, ignoring the joint modeling and mining of composite information such as multi-scale, multi-frequency, structure, and texture in the speckle image, which limits the restoration quality of the final image.

[0004] Therefore, how to accurately remove speckle noise from the speckle image and thereby improve the quality of the restored image is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a speckle image restoration method and system based on a generative adversarial network, which accurately removes speckle noise from the speckle image and thereby improves the quality of the restored image.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A speckle image restoration method based on a generative adversarial network, comprising:

[0008] Obtaining the original image and the corresponding speckle image and performing preprocessing to obtain a training set;

[0009] Constructing an image restoration model based on the generative adversarial network;

[0010] Train the image restoration model based on the training set and the target loss function to obtain a trained image restoration model;

[0011] Obtain a speckle image to be processed and input it into the trained image restoration model;

[0012] Based on the input of the speckle image to be processed into the initial convolution module, obtain shallow features;

[0013] Based on the input of the shallow features into the downsampling module, obtain deep features;

[0014] Based on the input of the deep features into the residual module, obtain processed features;

[0015] Based on the input of the processed features into the upsampling module, obtain restored features;

[0016] Based on the input of the restored features into the output module, obtain a restored image.

[0017] Preferably, obtaining a trained image restoration model specifically includes:

[0018] Based on the speckle images in the training set, input them into the generator of the image restoration model according to the steps of generating a restored image to obtain training restored images;

[0019] Based on the splicing of the training restored images and the speckle images in the training set, obtain a first spliced image;

[0020] Based on the splicing of the original images and the speckle images in the training set, obtain a second spliced image;

[0021] Based on the input of the first spliced image and the second spliced image into the discriminator of the image restoration model, respectively obtain a first probability value and a second probability value;

[0022] Judge whether the difference between the first probability value and the second probability value satisfies a set range. If so, obtain the trained image restoration model;

[0023] Otherwise, train the discriminator of the image restoration model based on the target loss function, and repeat the above steps of obtaining probability values based on the trained discriminator until the trained image restoration model is obtained;

[0024] The trained image restoration model includes a trained generator and a trained discriminator;

[0025] The trained generator includes the initial convolution module, the downsampling module, the residual module, the upsampling module and the output module.

[0026] Preferably, the initial convolution module includes: a first convolution layer, a first batch normalization layer, and a first activation layer;

[0027] The speckle image to be processed is sequentially input into the first convolution layer, the first batch normalization layer, and the first activation layer to obtain the shallow features;

[0028] The downsampling module includes: a second convolution layer, a first normalization layer, a second activation layer, a third convolution layer, a second normalization layer, and a third activation layer;

[0029] The shallow features are sequentially input into the second convolution layer, the first normalization layer, the second activation layer, the third convolution layer, the second normalization layer, and the third activation layer to obtain the deep features.

[0030] Preferably, the residual module includes a plurality of residual blocks connected in series in sequence;

[0031] All the residual blocks have the same structure and each includes: a fourth convolution layer, a third normalization layer, a fourth activation layer, a fifth convolution layer, and a fourth normalization layer;

[0032] The first input feature is sequentially input into the fourth convolution layer, the third normalization layer, and the fourth activation layer to obtain a first intermediate feature;

[0033] The first intermediate feature is sequentially input into the fifth convolution layer and the fourth normalization layer to obtain a second intermediate feature;

[0034] The first intermediate feature and the second intermediate feature are fused to obtain a first output feature.

[0035] Preferably, the upsampling module includes: a first transposed convolution layer, a fifth normalization layer, a fifth activation layer, a second transposed convolution layer, a sixth normalization layer, a sixth activation layer, and a sub-pixel convolution layer;

[0036] The processed feature is sequentially input into the first transposed convolution layer, the fifth normalization layer, the fifth activation layer, the second transposed convolution layer, the sixth normalization layer, the sixth activation layer, and the sub-pixel convolution layer to obtain the restored feature;

[0037] The output module includes: a sixth convolution layer and a seventh activation layer;

[0038] The restored feature is sequentially input into the sixth convolution layer and the seventh activation layer to obtain the restored image.

[0039] Preferably, the discriminator includes: a first convolution module, a second convolution module, and a probability output module;

[0040] The processing procedures for inputting the first stitched image and the second stitched image into the discriminator are the same, and both are as follows:

[0041] The stitched image is input into the first convolutional module to obtain a first extracted feature;

[0042] The first extracted feature is input into the second convolutional module to obtain a second extracted feature;

[0043] The second extracted feature is input into the probability output module to obtain a corresponding probability value.

[0044] Preferably, the first convolutional module includes a plurality of sequentially connected convolutional units;

[0045] All the convolutional units have the same structure, and each includes: a seventh convolutional layer, a second batch normalization layer, and an eighth activation layer;

[0046] The second input feature is sequentially input into the seventh convolutional layer, the second batch normalization layer, and the eighth activation layer to obtain a second output feature.

[0047] Preferably, the second convolutional module includes: an eighth convolutional layer, a third batch normalization layer, and a ninth activation layer;

[0048] The first extracted feature is sequentially input into the eighth convolutional layer, the third batch normalization layer, and the ninth activation layer to obtain the second extracted feature;

[0049] The probability output module includes: a global average pooling layer, a spectral normalization layer, and a tenth activation layer;

[0050] The second extracted feature is sequentially input into the global average pooling layer, the spectral normalization layer, and the tenth activation layer to obtain the corresponding probability value.

[0051] Preferably, the target loss function L is specifically:

[0052] L = w1 × L L1 + w2 × L perc + w3 × L adv + w4 × L SSIM ;

[0053] wherein, w1, w2, w3, and w4 respectively represent the weights corresponding to the respective loss functions, and L L1 represents the pixel difference loss function, L perc represents the perceptual loss function, L adv represents the adversarial loss function, and L SSIM represents the structural similarity loss function.

[0054] A speckle image restoration system based on a generative adversarial network, comprising: a data acquisition module, a model construction module, a model training module, a first feature acquisition module, a second feature acquisition module, and a restoration output module;

[0055] The data acquisition module is used to acquire the original image and the corresponding speckle image and perform preprocessing to obtain a training set;

[0056] The model construction module is used to construct an image restoration model based on the generative adversarial network;

[0057] The model training module is used to train the image restoration model based on the training set and the target loss function to obtain a trained image restoration model;

[0058] The first feature acquisition module is used to acquire the speckle image to be processed and input it into the trained image restoration model; based on the input of the speckle image to be processed into the initial convolution module, shallow features are obtained;

[0059] The second feature acquisition module is used to input the shallow features into the downsampling module to obtain deep features; based on the input of the deep features into the residual module, processed features are obtained;

[0060] The restoration output module is used to input the processed features into the upsampling module to obtain restoration features; based on the input of the restoration features into the output module, a restored image is obtained.

[0061] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a speckle image restoration method and system based on a generative adversarial network, adopting a generative adversarial network architecture, in which the generator adopts a structure based on residual blocks (Residual Block) and cross-scale feature fusion, and uses multi-scale convolutional layers for feature extraction and restoration. The discriminator adopts a PatchGAN architecture, optimizes the restoration ability of the generator through adversarial training, and enhances the stability of the network using spectral normalization. It has the following beneficial effects:

[0062] 1. The generator of the present invention introduces a deep fusion of large receptive field convolution, residual blocks and channel attention mechanisms, combined with a cross-scale feature extraction structure, which can accurately model the long-range and short-range dependencies in complex optical speckles, and significantly improve the image detail restoration ability and network generalization ability.

[0063] 2. The upsampling module of the present invention innovatively combines the advantages of transposed convolution and sub-pixel convolution layers, effectively avoiding the checkerboard effect and blurred distortion problems in the traditional upsampling process during the image reconstruction stage, and realizing the clear reconstruction of high-resolution output images.

[0064] 3. The discriminator of the present invention adopts the PatchGAN architecture and introduces the spectral normalization mechanism, which effectively suppresses the gradient explosion and mode collapse during the training of the discriminator, enhances the ability to distinguish local textures, and promotes the stable learning of high-frequency features by the generator.

[0065] 4. The present invention combines the L1 loss, perceptual loss, SSIM loss and adversarial loss, and simultaneously constrains the image structure, texture and perceptual consistency through a multi-objective optimization strategy, realizing a visually more realistic and clear image restoration effect.

[0066] 5. The present invention can be widely applied to high-quality image reconstruction tasks in multimode fiber optic imaging systems, and is suitable for the actual needs of multiple fields such as medical imaging, microscopic imaging, low-light imaging, SAR images, and remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0068] Figure 1 It is a flowchart of a speckle image restoration method based on a generative adversarial network provided by the present invention.

[0069] Figure 2 It is a schematic structural diagram of the generator provided by the present invention.

[0070] Figure 3 It is a schematic structural diagram of the residual block provided by the present invention.

[0071] Figure 4 It is a schematic structural diagram of the upsampling module provided by the present invention.

[0072] Figure 5 It is a schematic structural diagram of the discriminator provided by the present invention.

[0073] Figure 6 It is a schematic structural diagram of a speckle image restoration system based on a generative adversarial network provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0075] Example 1

[0076] As Figure 1 shown, an embodiment of the present invention discloses a speckle image restoration method based on a generative adversarial network, including:

[0077] Obtain the original image and the corresponding speckle image and perform preprocessing to obtain a training set;

[0078] Construct an image restoration model based on the generative adversarial network;

[0079] Train the image restoration model based on the training set and the target loss function to obtain a trained image restoration model;

[0080] Obtain the speckle image to be processed and input it into the trained image restoration model;

[0081] Based on the speckle image to be processed, input it into the initial convolution module to obtain shallow features;

[0082] Based on the shallow features, input them into the downsampling module to obtain deep features;

[0083] Based on the deep features, input them into the residual module to obtain processed features;

[0084] Based on the processed features, input them into the upsampling module to obtain restored features;

[0085] Based on the restored features, input them into the output module to obtain a restored image.

[0086] Example 2

[0087] An embodiment of the present invention discloses a speckle image restoration method based on a generative adversarial network, including:

[0088] Obtain the original image and the corresponding speckle image and perform preprocessing to obtain a training set.

[0089] Preferably, in this embodiment, a multimode fiber image acquisition platform is built to obtain natural scene images, i.e., original images, and the corresponding speckle images under different fiber bending degrees, lengths, etc., and construct a paired data set for training; based on the images in the paired data set, perform normalization preprocessing to ensure that the input images are in the range of [0,1] to obtain a training set.

[0090] Preferably, the multimode fiber image acquisition platform in this embodiment includes: a 532nm laser, a first lens, a first SLM transmissive spatial light modulator, a second SLM transmissive spatial light modulator, a second lens, a beam splitter prism, a third lens, a first objective lens, an optical fiber, a second objective lens, a fourth lens, a first CCD camera, a first PC terminal, a fifth lens, a second CCD camera, and a second PC terminal;

[0091] The light emitted by the 532nm laser passes through the first lens, the first transmissive spatial light modulator (SLM), and the second transmissive SLM in sequence to generate the first transmitted light and transmit it to the second PC terminal;

[0092] The first transmitted light passes through the second lens and the beam splitter prism in sequence to generate the first optical path and the second optical path;

[0093] The first optical path passes through the third lens, the first objective lens, the optical fiber, the second objective lens, and the fourth lens in sequence to generate the third optical path, and the third optical path is transmitted to the first CCD camera and the first PC terminal respectively;

[0094] The second optical path passes through the fifth lens to generate the second transmitted light and transmit it to the second CCD camera.

[0095] Construct an image restoration model based on the generative adversarial network.

[0096] Preferably, the image restoration model is constructed based on the generative adversarial network, including a generator and a discriminator. The generator adopts an architecture based on residual blocks and cross-scale feature fusion to improve the image restoration effect; the discriminator adopts the PatchGAN architecture, which can perform authenticity discrimination on each local region (Patch) of the image, thereby enhancing the model's perception ability of the local structure of the image.

[0097] Train the image restoration model based on the training set and the target loss function to obtain the trained image restoration model.

[0098] Preferably, obtaining the trained image restoration model specifically includes:

[0099] Input the speckle image in the training set into the generator of the image restoration model according to the steps of generating the restored image to obtain the training restored image;

[0100] Based on the training restored image and the speckle image in the training set, splice them to obtain the first spliced image;

[0101] Based on the original image and the speckle image in the training set, splice them to obtain the second spliced image;

[0102] Input the first spliced image and the second spliced image into the discriminator of the image restoration model respectively to obtain the first probability value and the second probability value correspondingly;

[0103] Judge whether the difference between the first probability value and the second probability value satisfies the set range. If so, obtain the trained image restoration model;

[0104] Otherwise, train the discriminator of the image restoration model based on the target loss function, and repeat the above steps of obtaining the probability value based on the trained discriminator until the trained image restoration model is obtained;

[0105] The trained image restoration model includes a trained generator and a trained discriminator;

[0106] The trained generator includes an initial convolution module, a downsampling module, a residual module, an upsampling module, and an output module.

[0107] Preferably, during the training process, the following optimization strategies are adopted:

[0108] Optimizer: Use the Adam optimizer. Among them, the learning rate of the generator is set to 0.0001, and the learning rate of the discriminator is set to 0.00001;

[0109] The target loss function combines the L1 loss, the perceptual loss, the adversarial loss, and the structural similarity loss (SSIM Loss) to improve the quality, clarity, and realism of the generated images;

[0110] Learning rate adjustment: Adopt a dynamic learning rate adjustment strategy, and use the cosine annealing and learning rate decay mechanisms to improve the stability and convergence speed of training.

[0111] Preferably, the target loss function L is specifically:

[0112] L = w1×L L1 +w2×L perc +w3×L adv +w4×L SSIM ;

[0113] Among them, w1, w2, w3, and w4 respectively represent the weights corresponding to the respective loss functions, and L L1 represents the pixel difference loss function, L perc represents the perceptual loss function, L adv represents the adversarial loss function, L SSIM represents the structural similarity loss function.

[0114] Preferably, the pixel difference loss function L L1 is specifically:

[0115] ;

[0116] Among them, represents the gray value of the i-th pixel in the generated image, represents the gray value of the i-th pixel in the real image, and N represents the total number of pixels in the image (N = height × width).

[0117] Preferably, the perceptual loss function L perc Specifically:

[0118] ;

[0119] where C, H, and W respectively represent the number of channels, height, and width of the feature map, represents the feature extraction function of the intermediate layer of the pre-trained VGG network, represents the generated image, represents the real image, represents the Euclidean distance (the square of the L2 norm).

[0120] Preferably, the adversarial loss function L adv Specifically:

[0121] ;

[0122] where, represents the input obtained by concatenating the discriminator's generated image with the conditional input S (such as a speckle pattern), and the output is the predicted probability of "real image", and log( ) represents the logarithmic function.

[0123] Preferably, the structural similarity loss function L SSIM Specifically:

[0124] ;

[0125] ;

[0126] where, and respectively represent the mean brightness of the generated image and the real image, represents the covariance between the generated image and the real image, used to represent the structural consistency, and C1 and C2 both represent stability term constants, generally taking 0.01 2 and 0.03 2 , and respectively represent the variance of the generated image and the variance of the real image.

[0127] Obtain the speckle image to be processed and input it into the trained image restoration model.

[0128] Preferably, in this embodiment, only the trained generator in the trained image restoration model is used to restore the speckle image to be processed. The steps of the trained generator and the generator during training to generate the restored image are the same, only the generator parameters are different.

[0129] Preferably, as Figure 2As shown, the trained generator includes an initial convolution module, a downsampling module, a residual module, an upsampling module, and an output module.

[0130] Based on the input of the speckle image to be processed into the initial convolution module, shallow features are obtained.

[0131] Preferably, the initial convolution module includes: a first convolutional layer, a first batch normalization layer, and a first activation layer;

[0132] The speckle image to be processed is sequentially input into the first convolutional layer, the first batch normalization layer, and the first activation layer to obtain shallow features.

[0133] Preferably, in this embodiment, the convolutional kernel size of the first convolutional layer is 7×7, the stride is 1, and the padding is 3, which is used for feature extraction to expand the receptive field and extract shallow global texture features; the first activation layer uses the ReLU activation function; the features output by the first convolutional layer are sequentially subjected to non-linear transformation through the first batch normalization layer (Batch Normalization) and the first activation layer, which is used to enhance the network expression ability and stabilize the training process.

[0134] Based on the shallow features input into the downsampling module, deep features are obtained.

[0135] Preferably, the downsampling module includes: a second convolutional layer, a first normalization layer, a second activation layer, a third convolutional layer, a second normalization layer, and a third activation layer;

[0136] The shallow features are sequentially input into the second convolutional layer, the first normalization layer, the second activation layer, the third convolutional layer, the second normalization layer, and the third activation layer to obtain deep features.

[0137] Preferably, the downsampling module is used to gradually compress the spatial resolution of the feature map to extract high-level semantic information. In this embodiment, the second convolutional layer and the third convolutional layer have the same structure, the convolutional kernel size is 4×4, the stride is 2, and the padding is 1. The second activation layer and the third activation layer both use the ReLU activation function.

[0138] Preferably, the feature extraction from shallow texture to deep semantics is completed through two convolutions to obtain deep features, providing a semantically rich intermediate representation for the subsequent residual module.

[0139] Based on the deep features input into the residual module, processed features are obtained.

[0140] Preferably, the residual module includes a plurality of sequentially connected residual blocks;

[0141] As Figure 3 shown, all the residual blocks have the same structure and include: a fourth convolutional layer, a third normalization layer, a fourth activation layer, a fifth convolutional layer, and a fourth normalization layer;

[0142] The first input feature is sequentially input into the fourth convolutional layer, the third normalization layer, and the fourth activation layer to obtain a first intermediate feature;

[0143] The first intermediate feature is sequentially input into the fifth convolutional layer and the fourth normalization layer to obtain a second intermediate feature;

[0144] The first intermediate feature and the second intermediate feature are fused to obtain a first output feature.

[0145] Preferably, in this embodiment, the fourth convolutional layer and the fifth convolutional layer have the same structure, the convolutional kernel size is 3×3, and the padding is 1; the fourth activation layer uses the ReLU activation function.

[0146] Preferably, the first intermediate feature and the second intermediate feature are fused by element-wise addition to achieve skip connection, thereby alleviating the problem of gradient disappearance in deep network training and retaining the original feature expression.

[0147] Preferably, in this embodiment, the residual module includes 12 sequentially connected residual blocks.

[0148] Based on the processed feature, it is input into the upsampling module to obtain a restored feature.

[0149] Preferably, as Figure 4 shown, the upsampling module includes: a first transposed convolutional layer, a fifth normalization layer, a fifth activation layer, a second transposed convolutional layer, a sixth normalization layer, a sixth activation layer, and a pixel shuffle layer;

[0150] The processed feature is sequentially input into the first transposed convolutional layer, the fifth normalization layer, the fifth activation layer, the second transposed convolutional layer, the sixth normalization layer, the sixth activation layer, and the pixel shuffle layer to obtain a restored feature.

[0151] Preferably, in this embodiment, the first transposed convolutional layer and the second transposed convolutional layer have the same structure, the convolutional kernel size is 4×4, the stride is 2, and the padding is 1. The fifth activation layer and the sixth activation layer both use the ReLU activation function. The pixel shuffle layer (PixelShuffle) is used for sub-pixel rearrangement operation to improve the reconstruction quality of image edges and texture details.

[0152] Based on the restored feature, it is input into the output module to obtain a restored image.

[0153] The output module includes: a sixth convolutional layer and a seventh activation layer;

[0154] The restored feature is sequentially input into the sixth convolutional layer and the seventh activation layer to obtain a restored image.

[0155] Preferably, in this embodiment, the convolution kernel size of the sixth convolution layer is 7×7, the stride is 1, and the padding is 3, which is used to map the number of channels from 128 to 1 to achieve single-channel output of grayscale images. The seventh activation layer uses the Tanh activation function to compress the pixel values to the interval [-1, 1], so as to be consistent with the dynamic range of the pre-training normalized images and facilitate the collaborative training of perceptual loss and adversarial loss.

[0156] Preferably, to further improve the restoration effect, post-processing is performed on the restored image, such as removing excessive noise, enhancing details, etc., to improve the visual effect.

[0157] Preferably, as Figure 5 shown, the discriminator includes: a first convolution module, a second convolution module, and a probability output module;

[0158] The processing processes of inputting the first spliced image and the second spliced image into the discriminator are the same, both are:

[0159] The spliced image is input into the first convolution module to obtain the first extracted feature;

[0160] The first extracted feature is input into the second convolution module to obtain the second extracted feature;

[0161] The second extracted feature is input into the probability output module to obtain the corresponding probability value.

[0162] Preferably, the first convolution module includes a plurality of sequentially connected convolution units;

[0163] All the convolution units have the same structure, and each includes: a seventh convolution layer, a second batch normalization layer, and an eighth activation layer;

[0164] The second input feature is sequentially input into the seventh convolution layer, the second batch normalization layer, and the eighth activation layer to obtain the second output feature.

[0165] Preferably, in this embodiment, the stride of the seventh convolution layer is 2, the padding is 2, the eighth activation layer uses the ReLU activation function, and the first convolution module includes 4 sequentially connected convolution units.

[0166] Preferably, the second convolution module includes: an eighth convolution layer, a third batch normalization layer, and a ninth activation layer;

[0167] The first extracted feature is sequentially input into the eighth convolution layer, the third batch normalization layer, and the ninth activation layer to obtain the second extracted feature.

[0168] Preferably, in this embodiment, the stride of the eighth convolution layer is 2, the padding is 1, and the ninth activation layer uses the ReLU activation function.

[0169] Preferably, the probability output module includes: a global average pooling layer, a spectral normalization layer, and a tenth activation layer;

[0170] The second extracted feature is sequentially input into the global average pooling layer, the spectral normalization layer, and the tenth activation layer to obtain the corresponding probability value.

[0171] Preferably, the global average pooling layer is used for spatial dimension compression, extracting the global response features of each channel, and outputting a one-dimensional vector with a shape of [Batch, 512].

[0172] Preferably, to improve the stability of the discriminator and prevent the problems of gradient explosion or disappearance during training, a spectral normalization mechanism is applied in the probability output module to constrain the weight tensor, ensuring that the Lipschitz constant of each layer during backpropagation is controlled within a reasonable range, thereby realizing a stable adversarial training process.

[0173] Preferably, the tenth activation layer uses the Sigmoid activation function, which is used to normalize the value to the interval [0, 1], and the output value represents the probability that the current input image pair (original image + speckle image or restored image + speckle image) is a real image.

[0174] Preferably, this structure enables the discriminator to have strong local structure recognition ability while retaining the global judgment ability, thereby improving the training adversariality and restoration quality.

[0175] Preferably, it further includes: evaluating the restoration ability of the trained image restoration model:

[0176] The original image is reconstructed from the test set using the trained model. The powerful learning ability of the deep learning neural network enables the trained model to reconstruct the speckle images of the test set that have not been used. The reconstructed image is compared with the original labeled image, and the SSIM is used as the evaluation index for the comparison. SSIM measures the similarity degree based on the brightness, contrast, and structure of the two images, and the numerical range is from -1 to 1.

[0177] The calculation formula of SSIM is the same as the formula in the structural similarity loss function:

[0178] ;

[0179] PSNR is also one of the commonly used indicators to measure the quality of image reconstruction. It is mainly used to evaluate the difference between the reconstructed image and the reference image (usually a noise-free image). Its calculation is based on the pixel-level error of the image, and it can quantitatively reflect the accuracy of the reconstructed image in restoring the gray value. The effect of the model in solving the endoscopic imaging problem is measured by calculating various evaluation indicators:

[0180] ;

[0181] Among them, MAX I represents the maximum possible value of the image pixels. For an 8-bit grayscale image, MAX = 255, and MSE represents the mean square error;

[0182] ;

[0183] Among them, m and n respectively represent the number of rows and columns (height and width) of the image, I(i, j) represents the grayscale value of the pixel at the i-th row and j-th column in the original image, and K(i, j) represents the grayscale value of the pixel at the i-th row and j-th column in the reconstructed image.

[0184] Example 3

[0185] As Figure 6 shown, a speckle image restoration system based on a generative adversarial network includes: a data acquisition module, a model construction module, a model training module, a first feature acquisition module, a second feature acquisition module, and a restoration output module;

[0186] The data acquisition module is used to acquire the original image and the corresponding speckle image and perform preprocessing to obtain a training set;

[0187] The model construction module is used to construct an image restoration model based on a generative adversarial network;

[0188] The model training module is used to train the image restoration model based on the training set and the target loss function to obtain a trained image restoration model;

[0189] The first feature acquisition module is used to acquire the speckle image to be processed and input it into the trained image restoration model; based on the speckle image to be processed input into the initial convolution module, shallow features are obtained;

[0190] The second feature acquisition module is used to input the shallow features into the downsampling module to obtain deep features; based on the deep features input into the residual module, processed features are obtained;

[0191] The restoration output module is used to input the processed features into the upsampling module to obtain restoration features; based on the restoration features input into the output module, a restored image is obtained.

[0192] Preferably, the functional implementation processes of the functional modules in this embodiment correspond one by one to the above method, and will not be elaborated here one by one.

[0193] Example 4

[0194] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0195] The memory is used to store a computer program;

[0196] When the processor is used to execute the program stored in the memory, it can implement a speckle image restoration method based on a generative adversarial network as described in Embodiment 1 or 2.

[0197] The electronic device may include: a processor, a communications interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus. The processor can call the logical instructions in the memory to execute a speckle image restoration method based on a generative adversarial network as described in Embodiment 1 or 2.

[0198] In addition, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0199] The various embodiments in this specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0200] The foregoing description of the disclosed embodiments enables those skilled in the art to practice or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speckle image restoration method based on a generative adversarial network, characterized in that Including: Obtain the original image and the corresponding speckle image and perform preprocessing to obtain a training set; Construct an image restoration model based on a generative adversarial network; Train the image restoration model based on the training set and a target loss function to obtain a trained image restoration model; Obtain a speckle image to be processed and input it into the trained image restoration model; Based on the speckle image to be processed, input it into an initial convolution module to obtain shallow features; Based on the shallow features, input them into a downsampling module to obtain deep features; Based on the deep features, input them into a residual module to obtain processed features; Based on the processed features, input them into an upsampling module to obtain restored features; Based on the restored features, input them into an output module to obtain a restored image.

2. The speckle image restoration method based on a generative adversarial network according to claim 1, characterized in that Obtaining a trained image restoration model specifically includes: Based on the speckle image in the training set, input it into the generator of the image restoration model and follow the steps of generating a restored image to obtain a training restored image; Based on the training restored image and the speckle image in the training set, splice them to obtain a first spliced image; Based on the original image and the speckle image in the training set, splice them to obtain a second spliced image; Based on the first spliced image and the second spliced image, input them into the discriminator of the image restoration model respectively, and correspondingly obtain a first probability value and a second probability value; Judge whether the difference between the first probability value and the second probability value satisfies a set range. If so, obtain the trained image restoration model; Otherwise, train the discriminator of the image restoration model based on the target loss function, and repeat the above steps of obtaining probability values based on the trained discriminator until the trained image restoration model is obtained; The trained image restoration model includes a trained generator and a trained discriminator; The trained generator includes the initial convolution module, the downsampling module, the residual module, the upsampling module and the output module.

3. A speckle image restoration method based on a generative adversarial network according to claim 1, characterized in that The initial convolution module includes: a first convolutional layer, a first batch normalization layer and a first activation layer; The speckle image to be processed is sequentially input into the first convolutional layer, the first batch normalization layer and the first activation layer to obtain the shallow features; The downsampling module includes: a second convolutional layer, a first normalization layer, a second activation layer, a third convolutional layer, a second normalization layer and a third activation layer; The shallow features are sequentially input into the second convolutional layer, the first normalization layer, the second activation layer, the third convolutional layer, the second normalization layer and the third activation layer to obtain the deep features.

4. A speckle image restoration method based on a generative adversarial network according to claim 1, wherein The residual module includes a plurality of residual blocks connected in series in sequence; All the residual blocks have the same structure and each includes: a fourth convolutional layer, a third normalization layer, a fourth activation layer, a fifth convolutional layer and a fourth normalization layer; The first input feature is sequentially input into the fourth convolutional layer, the third normalization layer and the fourth activation layer to obtain a first intermediate feature; The first intermediate feature is sequentially input into the fifth convolutional layer and the fourth normalization layer to obtain a second intermediate feature; The first intermediate feature and the second intermediate feature are fused to obtain a first output feature.

5. A speckle image restoration method based on a generative adversarial network according to claim 1, characterized in that The upsampling module includes: a first transposed convolution layer, a fifth normalization layer, a fifth activation layer, a second transposed convolution layer, a sixth normalization layer, a sixth activation layer, and a sub-pixel convolution layer; The processed feature is sequentially input into the first transposed convolution layer, the fifth normalization layer, the fifth activation layer, the second transposed convolution layer, the sixth normalization layer, the sixth activation layer, and the sub-pixel convolution layer to obtain the restored feature; The output module includes: a sixth convolution layer and a seventh activation layer; The restored feature is sequentially input into the sixth convolution layer and the seventh activation layer to obtain the restored image.

6. The speckle image restoration method based on a generative adversarial network according to claim 2, wherein, The discriminator includes: a first convolution module, a second convolution module, and a probability output module; The process of inputting the first spliced image and the second spliced image into the discriminator is the same, which is: The spliced image is input into the first convolution module to obtain a first extracted feature; The first extracted feature is input into the second convolution module to obtain a second extracted feature; The second extracted feature is input into the probability output module to obtain a corresponding probability value.

7. A speckle image restoration method based on a generative adversarial network according to claim 6, characterized in that The first convolution module includes a plurality of sequentially connected convolution units; All the convolution units have the same structure, and each includes: a seventh convolution layer, a second batch normalization layer, and an eighth activation layer; The second input feature is sequentially input into the seventh convolution layer, the second batch normalization layer, and the eighth activation layer to obtain a second output feature.

8. A speckle image restoration method based on a generative adversarial network according to claim 6, characterized in that, The second convolution module includes: an eighth convolution layer, a third batch normalization layer, and a ninth activation layer; The first extracted feature is sequentially input into the eighth convolution layer, the third batch normalization layer, and the ninth activation layer to obtain the second extracted feature; The probability output module includes: a global average pooling layer, a spectral normalization layer, and a tenth activation layer; The second extracted feature is sequentially input into the global average pooling layer, the spectral normalization layer, and the tenth activation layer to obtain the corresponding probability value.

9. A speckle image restoration method based on a generative adversarial network according to claim 1, characterized in that The target loss function L is specifically: L = w1×L L1 + w2×L perc + w3×L adv + w4×L SSIM ; Among them, w1, w2, w3, and w4 respectively represent the weights corresponding to the respective loss functions, and L L1 represents the pixel difference loss function, L perc represents the perceptual loss function, L adv represents the adversarial loss function, L SSIM represents the structural similarity loss function.

10. A speckle image restoration system based on a generative adversarial network, which is applied to a speckle image restoration method based on a generative adversarial network according to any one of claims 1-9, and is characterized in that, including: a data acquisition module, a model construction module, a model training module, a first feature acquisition module, a second feature acquisition module, and a restoration output module; The data acquisition module is used to acquire the original image and the corresponding speckle image and perform preprocessing to obtain a training set; The model construction module is used to construct an image restoration model based on the generative adversarial network; The model training module is used to train the image restoration model based on the training set and the target loss function to obtain a trained image restoration model; The first feature acquisition module is used to acquire the to-be-processed speckle image and input it into the trained image restoration model; based on the to-be-processed speckle image input into the initial convolution module to obtain a shallow feature; The second feature acquisition module is used to input the shallow feature into the downsampling module to obtain a deep feature; based on the deep feature input into the residual module to obtain a processed feature; The restoration output module is used to input the processed feature into the upsampling module to obtain a restored feature; Based on the restored feature input into the output module to obtain a restored image.

Images