A dynamic scattering medium scattering imaging recovery method and speckle data acquisition device

Abstract

The present application relates to the field of computational optical imaging and the field of scattering medium imaging, and provides a dynamic scattering medium scattering imaging recovery method and speckle data acquisition device, the main idea is to be able to accurately recover the target under unknown scattering scene with high fidelity, the model has high generalization and robustness.Main scheme includes collecting speckle data generated under multiple different scattering media, different scattering medium positions and dynamically changing scattering media, and three speckles are collected for each target image, so as to better learn the statistical invariant under the dynamic scattering scene;The design of the generator and the discriminator structure is included in the generation of the adversarial network building;According to the different functions of the generator and the discriminator, design a suitable loss function to constrain and optimize the network;Adversarial network training, and using the trained generator to predict and generate the target image corresponding to the speckle generated under the unknown scattering scene, which is used for imaging recovery of the target.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computational optical imaging and the field of scattering medium imaging, and in particular to a dynamic scattering medium scattering imaging recovery method and speckle data acquisition device. BACKGROUND

[0002] Imaging a target through a scattering medium is a classic challenging inverse problem. Overcoming the scattering problem of the light beam can enable imaging of objects hidden deep inside the scattering medium or behind the medium, enable high-altitude observation through clouds and remote sensing mapping, enable biological microscopes to observe deep tissues through shallow biological tissues, and improve the observation capability of traditional optical systems in underwater and foggy conditions, which has important practical applications. In real-world scenarios, the distribution of targets at different positions, the random and dynamic changes of the scattering medium, and the interference of environmental variables on the light path system restrict the imaging application in scattering scenarios.

[0003] It is difficult and costly for traditional optical imaging methods and image processing techniques to image and recover speckle images with random light field distribution. Therefore, methods based on computational imaging are widely used to process target image recovery problems in scattering scenarios. The commonly used scattering imaging methods mainly include wavefront shaping, deconvolution imaging, correlation imaging based on traditional optical theory, and deep learning technology which has developed rapidly in recent years. However, wavefront shaping, deconvolution imaging, and correlation imaging require strict light paths and have large computational loads, and can usually only be applied to the measurement and speckle recovery of static scattering media. Realizing scattering imaging in a real dynamic environment is still a great challenge. Deep learning is a powerful technology based on neural networks, which can capture the internal information of data-rich and has obvious advantages in extracting the relationship between the input and the label based on the structure and features of the image. By learning the internal information of the speckle image, the details and features hidden in the speckle pattern can be extracted, and a similar "transfer matrix" can be constructed from the random light intensity distribution of the target image speckle, thereby realizing speckle image reconstruction. However, the data-driven deep learning technology learns the feature information based on the training data, and the scattering scene is single, so the generalization ability is poor, and the imaging recovery effect of the scattering target in an unknown scattering scene is poor.

[0004] Therefore, the present application proposes a dynamic scattering medium scattering imaging recovery method based on a generative adversarial network, which can realize imaging recovery of a target in an unknown scattering scene and a dynamically changing scattering medium, and has higher generalization and robustness. SUMMARY

[0005] The application aims at the deficiencies of the prior art, and provides a dynamic scattering medium scattering imaging recovery method based on a generative adversarial network, which can generate speckle data in multiple different scattering media, different scattering medium positions and dynamically changing scattering media, train a model, accurately recover a hidden target image in a speckle image, accurately recover a target in an unknown scattering scene with high fidelity, and has high generalization and robustness.

[0006] To achieve the above-mentioned purpose, the application adopts the following technical means:

[0007] A speckle data acquisition device, comprising a laser, a beam expander system, a polarizer, a spatial light modulator, a polarizer, a lens, a scattering medium and a CCD arranged in sequence, and the scattering light field of the scattering medium is collected by the CCD camera, characterized in that the scattering medium is placed on a turntable, the scattering medium is in a rotating state by the turntable, and the scattering medium is diverse, and the turntable can change the position.

[0008] The application also provides a dynamic scattering medium scattering imaging recovery method based on a generative adversarial network, comprising the following steps:

[0009] Step 1, data generation:

[0010] Step 1.1, the CelebA face data set and the Mnist handwritten data set are used as target images, and are loaded into the spatial light modulator;

[0011] Step 1.2, the speckle generated by the light field with target information after passing through the optical path is collected;

[0012] Step 1.3, different scattering media are placed in the optical path, the position of the scattering medium is changed, and the scattering medium is dynamically changed, step 1.2 is repeated, and 3 speckles corresponding to all target images are collected each time, and speckle data in multiple scattering scenes is obtained.

[0013] Step 2, data preprocessing: the collected speckle data is processed into a gray scale image of 0-255, and is normalized to an interval of [-1, 1].

[0014] Step 3, generative adversarial network building:

[0015] Step 3.1, the generative adversarial network is composed of a generator G θ and a discriminator D φ ;

[0016] Step 3.2, the generator G θThe U-net network structure is adopted, in which the upsampling and downsampling operations are implemented using convolution. The generator input consists of three speckle maps x1, x2, and x3 generated from the same target image under dynamic scattering medium conditions. The generator output is the reconstructed image G. θ (x1, x2, x3);

[0017] Step 3.3, the discriminator D φ A convolutional neural network is used, with the generator G as the input. θ Reconstructed image G θ (x1, x2, x3) and the real target image, for the generator G θ Reconstructed image G θ (x1, x2, x3), expected discriminant D φ If a value is determined to be false, the output is a matrix consisting entirely of "0". For a real target image, the desired discriminant D is... φ If the condition is true, output a matrix consisting entirely of "1"s;

[0018] Step 4: Loss Function Design

[0019] The loss function of generative adversarial networks is Loss.

[0020]

[0021] In equation (1), The generator loss is represented by the loss function. Dφ To represent the discriminator loss, based on the functions and output results of the generator and discriminator, the generator loss and discriminator loss are designed as follows:

[0022]

[0023]

[0024] In Equation (2), the first two terms represent the negative Pearson correlation coefficient between the restored image and the real target image, describing the degree of similarity between the two, and the last term represents the mean absolute error between the restored image and the real target image. In Equation (3), the first term represents the mean absolute error between the output of the restored image generated by the generator and the label value after passing through the discriminator, and the second term represents the mean absolute error between the output of the real target image and the label value after passing through the discriminator.

[0025] Step 5: Train the generative adversarial network to reconstruct the target image:

[0026] The speckle images x1, x2, and x3 are input into a generative adversarial network for model training. After multiple rounds of iterative optimization, the speckle image can be restored and reconstructed, and it can restore and reconstruct speckle images in unknown scattering scenarios.

[0027] Beneficial effects:

[0028] The present invention provides a dynamic scattering medium scattering imaging restoration method based on generative adversarial networks, which has the following advantages compared with the prior art:

[0029] 1. By collecting three speckle images of the same target as training data in a dynamic scattering scenario, the statistical invariants of the scattering scenario can be effectively extracted, achieving high-quality speckle recovery in dynamic and complex scattering scenarios.

[0030] 2. Based on multiple types of scattering media and different locations of scattering media, after training, the generative adversarial network model can achieve speckle recovery in scenarios with unknown types of scattering media and unknown locations of scattering media, thereby improving the model's generalization performance and robustness.

[0031] 3. Introducing a negative Pearson correlation coefficient into the loss function enables the restored image to match the real target image at the pixel level, accelerates model convergence, and makes the restored image generated by the model more accurate. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2 This is the optical path structure for acquiring speckle data in this invention;

[0034] Figure 3 This is the structure of the generative adversarial network generator of the present invention;

[0035] Figure 4 This is the structure of the generative adversarial network discriminator of the present invention. Detailed Implementation

[0036] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0037] This invention provides a method for speckle image restoration in dynamic scattering media based on generative adversarial networks (GANs). Leveraging the high-quality image generation capabilities of GANs, the generator learns the statistical invariants of speckle patterns generated in dynamic scattering scenarios and can extract and distinguish speckle features and differences between different targets, achieving speckle image restoration under different scattering media, different scattering media locations, and different dynamic scenarios. The GAN model employs a multi-input mechanism, using three speckle images of the same target as a single input, making it easier for the model to learn the statistical invariants under dynamic speckle scenarios. Furthermore, combining speckle data from multiple scattering scenarios enhances the model's robustness and generalization ability. Specifically, as... Figure 1 As shown, it includes the following steps:

[0038] Step 1: Data Generation

[0039] Step 1.1, Speckle Data Acquisition: The experimental system for collecting speckle scattering imaging and reconstruction under various scattering scenarios mainly includes: a helium atmosphere laser (632.8nm) as a coherent light source, which illuminates the target image on the surface of the SLM (spatial light modulator) after passing through a beam expander and polarizer. The light field carrying the target image information is incident on the scattering medium after passing through an analyzer and a focusing lens. The scattered light field passing through the scattering medium is acquired by a CCD camera.

[0040] The scattering medium in the optical path has three polishing specifications: 220 mesh, 600 mesh, and 1500 mesh. Each polishing specification requires the acquisition of 30,000 speckle images, corresponding to 5,000 MNIST target digital images and 5,000 CelebA face target digital images. The position of the scattering medium is offset near the focal point of the focusing lens, and the scattering medium is rotated to simulate a real dynamic environment.

[0041] Step 1.2, Speckle Data Preprocessing: The collected speckle pixel size is 256×256×3, which is converted into a grayscale image and normalized.

[0042] Step 1.3, Dataset Construction: The processed speckle data and the corresponding target images are used as the training data and real labels for the generative adversarial network, respectively.

[0043] Step 2: Building the adversarial network structure:

[0044] Step 2.1: Generate the adversarial network using generator G. θ and discriminator D φ constitute;

[0045] Step 2.2: The generator is based on the U-net network. Its input consists of three speckle maps x1, x2, and x3 generated from the same target image in a dynamic scattering medium. The generator's output is the reconstructed image G. θ (x1, x2, x3), the reconstructed image G is constrained by a loss function. θ (x1, x2, x3) such that G θ (x1, x2, x3) are reconstructed into the corresponding target image; the generator's network structure is as follows: Figure 3As shown, the network structure mainly consists of an encoding structure and a decoding structure, which mainly involves an input layer, convolutional layers, pooling layers, batch normalization (BN) layers, activation layers, an output layer, and skip connections. The pooling layers utilize convolution to achieve upsampling and downsampling functions. The input speckle maps x1, x2, and x3 are first processed by convolution, BN, and LeakyReLU to extract features, and then the feature data are concatenated along the channel dimension. During the encoding process, the lateral path is composed of convolutional layers, BN layers, and LeakyReLU activation functions, while downsampling is performed by convolutional layers. During the decoding process, the lateral path is based on convolutional layers, BN layers, and LeakyReLU activation functions, with upsampling performed by PixelShuffle and skip connections added. This allows the model to fuse the feature data from the encoding process, thus ensuring that the feature map after skip connections has more layers of features.

[0046] Step 2.3: The discriminator is composed of a convolutional neural network, which performs feature extraction on the high-dimensional input data and obtains low-dimensional discriminative information to guide the training of the generator. The discriminator has two input channels, one of which is the image G reconstructed by the generator. θ (x1, x2, x3), another is obtained by inputting the corresponding real target image; the discriminator output also has two channels, representing the image G respectively. θ The discrimination result between (x1, x2, x3) and the real target image is not a specific number, but a feature discrimination matrix, which allows the discriminator to focus on image G. θ (x1, x2, x3) and details of the real target image; for the real target image, the feature matrix of the discriminator is expected to be classified as "1"; for the image G reconstructed by the generator, the feature matrix is ​​expected to be classified as "1"; θ (x1, x2, x2), the feature matrix of the expected discriminator is judged as "0".

[0047] Step 3: Design of the loss function for the generative adversarial network model:

[0048] In generative adversarial networks, the generator and discriminator have separate structures and functions. After the structures are designed separately, the loss functions also need to be designed separately to better utilize the discriminator to guide the generator to generate clear and accurate images.

[0049] Step 3.1, Composition of Generative Adversarial Network Loss: The loss of a generative adversarial network consists of generator loss and discriminator loss, which can be expressed by formula (1):

[0050]

[0051] Step 3.2: Design the target loss function of the generator according to equation (2). Add the negative Pearson correlation coefficient to the generator loss function, i.e., the first two terms of equation (2); the negative Pearson correlation coefficient can be used to describe the similarity between two images, therefore the image G generated by the generator... θ The more similar (x1, x2, x3) is to the real target image, the smaller the negative Pearson correlation coefficient, the lower the loss, and the more the generator loss can converge; the last term is the mean absolute error, which can better reflect the actual situation of the prediction error.

[0052]

[0053] In equation (2), m and n represent the number of rows and columns of the image, respectively, X represents the predicted image, Y represents the target image, and E represents the average grayscale value of the image.

[0054] Step 3.3: Design the target loss function Loss of the discriminator according to equation (3). Gφ It consists of the mean absolute error, with the first term representing the discriminator's judgment on image G. θ (x1, x2, x3) represents the error between the extracted features and the feature matrix “0”, and the second term represents the error between the real target image and the feature matrix “1”.

[0055]

[0056] In equation (3), m and n represent the number of rows and columns of the image, respectively. This represents the discriminator's prediction matrix for the image generated by the generator. This represents the expectation matrix corresponding to the image generated by the generator; this expectation matrix is ​​a "0" matrix. This represents the discriminator's prediction matrix for the target image. This represents the expectation matrix corresponding to the target image, which is a matrix of "1s".

[0057] In step three, the present invention uses the negative Pearson correlation coefficient as part of the generator loss function, which enables the reconstructed image to match the real target image in terms of pixel and detail features, making the generated restored image more accurate and similar.

[0058] Step 4: Train the generative adversarial network and use the trained generator to perform speckle reconstruction in unknown scattering scenarios:

[0059] Step 4.1: Input the constructed dataset into the generative adversarial network, select ADMA as the optimizer, set the learning rate to 0.0005, set the batch size to 32, and use the loss function designed above as the loss function for the generator and discriminator to constrain and optimize the model;

[0060] The working environment of the method of this invention is: Windows 10, Intel(R) Core(TM) i7-9700kf CPU@3.60Ghz, TensorFlow 2.5.1, and NVIDIA GeForce RTX 2080Ti graphics processing unit.

[0061] Step 4.2: Use the trained generator to predict and generate the target image corresponding to the speckle pattern generated in the unknown scattering scene.

[0062] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.

Claims

1. A dynamic scattering medium scattering imaging restoration method based on generative adversarial networks, characterized in that, include: Step 1: The CelebA face dataset and the MNIST handwritten dataset are used as target images and loaded into a spatial light modulator. The target image is then projected onto a scattering medium after being modulated by the light field. Step 2: Collect speckle data generated by the rotating scattering medium under dynamically changing scattering conditions, and use the collected speckle data and the corresponding target image as training data and real labels for the generative adversarial network, respectively. Step 3: Generate and build the adversarial network structure: Step 3.1: Generate the adversarial network using a generator. and discriminator constitute; Step 3.2: The generator is based on the U-net network, constructing a multi-input single-output network model; its input consists of three speckle maps generated from the same target image under dynamic scattering medium conditions. , and The output is the generator-reconstructed image. ; Step 3.3: The discriminator is composed of a convolutional neural network, and its input has two channels. One channel is input to the image reconstructed by the generator. Another input is the corresponding real target image; the discriminator's output also has two channels, representing the different views of the image. The discrimination result between the real target image and the actual target image is a feature discrimination matrix; For a real target image, the feature matrix of the discriminator is expected to be classified as "1", while for an image reconstructed by the generator... The feature matrix of the expected discriminator is judged as "0"; Step 4: Use the training data and real labels obtained in Step 2 to train the adversarial network to obtain a trained generator, and use the trained generator to perform speckle recovery on speckle generated in unknown scattering scenarios. The steps for designing the loss function for a generative adversarial network (GAN) model include: The loss function of a generative adversarial network consists of the generator loss and the discriminator loss, and can be expressed by formula (1): (1) Design the target loss function of the generator according to equation (2). , (2) In equation (2), , These represent the number of rows and columns of the image, respectively. The image represents the prediction. Indicates the target image. This indicates that the average grayscale value of the image is taken. Design the target loss function of the discriminator according to equation (3). , (3) In equation (3), , These represent the number of rows and columns of the image, respectively. This represents the discriminator's prediction matrix for the image generated by the generator. This represents the expectation matrix corresponding to the image generated by the generator; this expectation matrix is ​​a "0" matrix. This represents the discriminator's prediction matrix for the target image. This represents the expectation matrix corresponding to the target image, which is a matrix of "1".

2. The dynamic scattering medium scattering imaging restoration method based on generative adversarial networks according to claim 1, characterized in that, Step 1 includes the following steps: Step 1.1, Speckle Data Acquisition: Acquire speckle data generated under multiple different scattering media, different scattering media positions, and dynamically changing scattering media conditions; Step 1.2, Speckle Data Preprocessing: The collected speckle pixel size is 256×256×3, which is converted into a grayscale image and normalized. Step 1.3, Dataset Construction: The processed speckle data and the corresponding target images are used as the training data and real labels for the generative adversarial network, respectively.

3. The dynamic scattering medium scattering imaging restoration method based on generative adversarial networks according to claim 2, characterized in that, Step 3: Train the Generative Adversarial Network (GAN) and use the trained generator to perform speckle reconstruction in unknown scattering scenarios: Step 3.1: Input the constructed dataset into the generative adversarial network, select ADMA as the optimizer, set the learning rate to 0.0005, set the batch size to 32, and use the loss function of the adversarial network to constrain and optimize the model to obtain the trained model; Step 3.2: Use the trained generator to predict and generate the target image corresponding to the speckle pattern generated in the unknown scattering scene.

4. A speckle data acquisition device, used in the method described in any one of claims 1-3, comprising a laser, a beam expander, a polarizer, a spatial light modulator, an analyzer, a lens, a scattering medium, and a CCD arranged sequentially, wherein the scattered light field transmitted through the scattering medium is acquired by a CCD camera, characterized in that, The scattering medium is placed on a turntable, which causes the scattering medium to rotate.

Images