A scattering imaging device system based on Transformer and CNN
By introducing Transformer and CNN into a deep learning-based scattering imaging system, combined with multi-head self-attention and multilayer perceptron modules, the problem of insufficient non-local feature extraction capability of convolutional neural networks is solved, and high-quality speckle image restoration effect is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG SCI-TECH UNIV
- Filing Date
- 2023-02-09
- Publication Date
- 2026-05-26
AI Technical Summary
Existing deep learning-based scattering imaging systems mainly rely on convolutional neural networks for speckle image reconstruction, which suffers from insufficient non-local feature extraction capabilities.
A scattering imaging device system based on Transformer and CNN is adopted. By building a speckle map restoration network on the PyTorch framework, combining CNN encoding path and Transformer encoding path, multi-head self-attention and multilayer perceptron module are used to learn global context representation, and the loss function is optimized by negative Pearson correlation coefficient for image restoration.
It achieves high-quality speckle image restoration with good image detail recovery and high accuracy, making it suitable for widespread use.
Smart Images
Figure CN116228572B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computational optical imaging and scattering imaging, and particularly to a scattering imaging device system based on Transformer and CNN. Background Technology
[0002] When light passes through a non-uniform scattering medium, it cannot form a clear image due to the influence of the scattering medium. For example, it is impossible to see the target object clearly in turbid water, rainy days, or foggy days. Therefore, achieving clear imaging through a scattering medium is a problem that needs to be solved. Traditional techniques for restoring scattering images are mainly based on physical methods, including wavefront shaping, optical phase conjugation, and optical transfer matrix techniques. However, these techniques have drawbacks such as high computational cost, poor transferability, and poor reconstructability. To overcome these drawbacks, deep learning methods have been introduced to solve the speckle image restoration problem. The scattering process of light passing through a scattering medium is a deterministic process. The output light field obtained after multiple scatterings still retains the information of the incident light field. Neural networks, relying on their powerful data mining capabilities, can learn the complex relationship between the input and output light fields and reconstruct the speckle pattern.
[0003] Existing deep learning-based scattering imaging systems suffer from the following drawbacks: Previous methods for scattering imaging using deep learning primarily rely on convolutional neural networks (CNNs) for speckle image reconstruction. While CNNs offer excellent performance and outstanding feature extraction capabilities, they also have limitations, such as insufficient ability to extract non-local features. Therefore, we propose a scattering imaging system based on Transformer and CNN. Summary of the Invention
[0004] The main objective of this invention is to provide a scattering imaging device system based on Transformer and CNN, which can effectively solve the problems in the background art.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] A scattering imaging device system based on Transformer and CNN includes a speckle image acquisition device, a deep learning image restoration method, and a speckle image restoration method. The speckle image acquisition device includes a laser (1), lens A (2), lens B (3), a spatial light modulator (4), a target image (5), an objective lens (6), a scattering medium (8), an objective lens (9), and a CMOS camera (10) arranged sequentially along the optical path. The speckle image acquisition device also includes a computer (7). The output terminals of the spatial light modulator (4) and the CMOS camera (10) are connected to the computer (7) via wires and simultaneously controlled by a Matlab program to continuously capture and collect speckle images (11). The speckle image restoration method includes a speckle image restoration network based on the PyTorch framework and based on Transformer and CNN.
[0007] Furthermore, a speckle map restoration network based on Transformer and CNN is built on top of PyTorch, including the following steps:
[0008] Step 1: After the image is input into the network, it enters the CNN encoding path and the Transformer encoding path respectively. The input size is... Where H and W represent the image length and width, C represents the number of image channels, and R represents the real number field;
[0009] Step 2: The CNN encoding path contains 4 sub-modules. Each sub-module contains two 3×3 convolutions, a ReLU activation function, and a 2×2 max pooling layer for downsampling.
[0010] Step 3: In the Transformer encoding path, the input size is... The speckle image is divided into dimensions of N patches, that is ∈ To preserve positional information, these patches are projected into a K-dimensional embedding space, and a positional embedding is added for patch embedding. To preserve the spatial information of each patch, the formula for patch embedding is: ,in The projected embedding of the patch is represented, and then a Transformer module containing multi-head self-attention (MSA) and a multilayer perceptron is used to learn a global context representation. The MSA layer consists of M parallel self-attention heads to scale the embedded patch, and the MLP module learns long-range context dependencies. The formulas for MSA and MLP are expressed as follows: , Norm() represents layer normalization. and Encode the output representations of the MSA and MLP of the i-th layer of the Transformer;
[0011] Step 4: After the two encoding paths, the CNN's encoded information and the information encoded from the Transformer are added together through the addition path and fused into the decoder. The decoder consists of 4 sub-modules. Each sub-module contains a 2×2 deconvolution, two 3×3 convolutions and a ReLU activation function to perform feature upsampling and decode the features into a high-resolution image.
[0012] Step 5: Set the loss function to the negative Pearson correlation coefficient (NPCC), the formula of which is:
[0013]
[0014] In the formula, M(i,j) and N(i,j) represent the gray values of the network-reconstructed image and the original target image at the same spatial location, respectively, and i and j represent the two-dimensional coordinate indices of all effective pixels in the speckle image. and This represents the mean of the image restored by the network and the original target.
[0015] Furthermore, the Transformer and CNN scattering imaging device system includes the following steps:
[0016] Step 1: Acquire speckle image. Use the MNIST handwritten digit image to acquire speckle image as the target image. Upsample to 1024×1024 and display on the spatial light modulator. The SLM output light carrying object information passes through the scattering medium and the speckle image is acquired by the CMOS camera.
[0017] Step 2: Establish the dataset. First, process the collected speckle images, crop the central square region to retain useful speckle information, and resize it to 256×256 as the input image for the network model. Upsample the MNIST handwritten digit images to 256×256 as the label images for the dataset. Divide the obtained images into training set, validation set, and test set in a ratio of 0.8:0.1:0.1.
[0018] Step 3: Use the speckle image as the input image and the target image as the label image to train the network and obtain the weight file of the image restoration network. Then, input the speckle image into the trained weight file for testing and reconstruct the target image.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] 1. This invention can effectively restore speckle images, and the restored images are of high quality, with good recovery of image details. In order to evaluate the network's restoration effect on speckle images, peak signal-to-noise ratio and structural similarity are used to measure the network's restoration effect.
[0021] 2. In the past, the methods of scattering imaging using deep learning mainly relied on convolutional neural networks for speckle image reconstruction. Convolutional neural networks have excellent performance and outstanding feature extraction capabilities. Scattering imaging devices based on Transformer and CNN are easy to operate, highly accurate, practical, and effective, making them suitable for widespread use. Attached Figure Description
[0022] Figure 1 This is an overall flowchart of a scattering imaging device system based on Transformer and CNN according to the present invention.
[0023] Figure 2 This is a schematic diagram of the optical device of a scattering imaging device system based on Transformer and CNN according to the present invention.
[0024] Figure 3 This is a schematic diagram of a speckle image restoration network for a scattering imaging device system based on Transformer and CNN according to the present invention.
[0025] Figure 4 This is a rendering of a scattering imaging device system based on Transformer and CNN according to the present invention.
[0026] In the diagram: 1. Laser; 2. Lens A; 3. Lens B; 4. Spatial light modulator; 5. Target image; 6. Objective lens; 7. Computer; 8. Scattering medium; 9. Objective lens; 10. CMOS camera; 11. Speckle image. Detailed Implementation
[0027] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.
[0028] like Figure 1-4As shown; a scattering imaging device system based on Transformer and CNN includes a speckle image acquisition device, a deep learning image restoration method and a speckle image restoration method. The speckle image acquisition device includes a laser 1, lens A (2), lens B (3), spatial light modulator 4, target image 5, objective lens 6, scattering medium 8, objective lens 9 and CMOS camera 10 arranged sequentially along the optical path. The speckle image acquisition device also includes a computer 7. The output terminals of the spatial light modulator 4 and the CMOS camera 10 are connected to the computer 7 through wires and the speckle image 11 collected by continuous imaging is controlled simultaneously by the Matlab program. The light beam emitted by the laser 1 is collimated by lens A (2) and lens B (3) and then incident on the spatial light modulator 4 to modulate the light field of the target image 5. The modulated light field is focused by the first objective lens 6 and forms speckle through the scattering medium 8. The speckle is imaged onto the CMOS camera 10 by the second objective lens 9, and the computer 7 controls the spatial light modulator 4 and the CMOS camera 10 to complete the image acquisition. That is, coherent laser light generated by laser 1 is used as an illumination source. The emitted beam is collimated by a collimation system composed of lens A (2) and lens B (3) and then uniformly incident on the spatial light modulator 4. The target image 5 is loaded and modulated according to the control signal of computer 7. The modulated light field is focused by the first objective lens 6 and then forms a random speckle pattern through the scattering medium 8. The speckle pattern is imaged onto the CMOS camera 10 by the second objective lens 9. The spatial light modulator 4 and the CMOS camera 10 are controlled by computer 7 to record the speckle intensity distribution and transmit it to computer 7 for storage, thus completing the image acquisition. The speckle image restoration method includes a speckle image restoration network based on PyTorch framework using Transformer and CNN.
[0029] The process of building a speckle map restoration network based on Transformer and CNN on top of PyTorch includes the following steps:
[0030] Step 1: After the image is input into the network, it enters the CNN encoding path and the Transformer encoding path respectively. The input size is... Where H and W represent the image length and width, and C represents the number of image channels;
[0031] Step 2: The CNN encoding path contains 4 sub-modules. Each sub-module contains two 3×3 convolutions, a ReLU activation function, and a 2×2 max pooling layer for downsampling.
[0032] Step 3: In the Transformer encoding path, the input size is... The speckle image is divided into 11 dimensions. N patches, that is ∈ To preserve positional information, these patches are projected into a K-dimensional embedding space, and a positional embedding is added for patch embedding. To preserve the spatial information of each patch, the formula for patch embedding is: ,in The projected embedding of the patch is represented, and then a Transformer module containing multi-head self-attention (MSA) and a multilayer perceptron (MLP) is used to learn a global context representation. The MSA layer consists of M parallel self-attention heads to scale the embedded patch, and the MLP module learns long-range context dependencies. The formulas for MSA and MLP are expressed as follows: , Norm() represents layer normalization. and Encode the output representations of the MSA and MLP of the i-th layer of the Transformer;
[0033] Step 4: After the two encoding paths, the CNN's encoded information and the information encoded from the Transformer are added together through the addition path and fused into the decoder. The decoder consists of 4 sub-modules. Each sub-module contains a 2×2 deconvolution, two 3×3 convolutions and a ReLU activation function to perform feature upsampling and decode the features into a high-resolution image.
[0034] Step 5: Set the loss function to the negative Pearson correlation coefficient (NPCC), the formula of which is:
[0035] 1
[0036] In Equation 1, M(i,j) represents the image restored by the network, and N(i,j) represents the original target. and This represents the mean of the image restored by the network and the original target.
[0037] The Transformer and CNN scattering imaging device system includes the following steps:
[0038] Step 1: Acquire speckle image 11. Use the MNIST handwritten digit image to acquire speckle image 11 as target image 5. Upsample to 1024×1024 and display on spatial light modulator 4. The SLM output light carrying object information passes through scattering medium 8 and is acquired by CMOS camera 10.
[0039] Step 2: Establish the dataset. First, process the collected speckle image 11, crop the central square region to retain useful speckle information, and resize it to 256×256 as the input image for the network model. Upsample the MNIST handwritten digit image to 256×256 as the label image for the dataset. Divide the obtained images into training set, validation set, and test set in a ratio of 0.8:0.1:0.1.
[0040] Step 3: Input speckle image 11 as input image and target image 5 as label image into the network for training to obtain the weight file of the image restoration network. Then input speckle image 11 into the trained weight file for testing to reconstruct the target image.
[0041] It should be noted that this invention is a scattering imaging device system based on Transformer and CNN. During operation, in the Transformer encoding path, the input size is... The speckle image is divided into 11 dimensions. N patches, that is ∈ To preserve positional information, these patches are projected into a K-dimensional embedding space, and a positional embedding is added for patch embedding. To preserve the spatial information of each patch, the formula for patch embedding is: ,in The projected embedding of the patch is represented, and then a Transformer module containing multi-head self-attention (MSA) and a multilayer perceptron (MLP) is used to learn a global context representation. The MSA layer consists of M parallel self-attention heads to scale the embedded patch, and the MLP module learns long-range context dependencies. The formulas for MSA and MLP are expressed as follows: , Norm() represents layer normalization. and The output representations of the MSA and MLP of the i-th layer of the Transformer are encoded. After two encoding paths, the information encoded by the CNN and the information encoded by the Transformer are added together through an additive path and fused into the decoder. The decoder consists of four sub-modules, each containing a 2×2 deconvolution, two 3×3 convolutions, and a ReLU activation function to perform feature upsampling and decode the features into a high-resolution image. In previous methods of scattering imaging using deep learning, convolutional neural networks were mainly used for speckle image reconstruction. Convolutional neural networks have excellent performance and outstanding feature extraction capabilities. The scattering imaging device system based on Transformer and CNN is easy to operate, highly accurate, practical, and effective, making it suitable for widespread use.
[0042] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A scattering imaging device system based on Transformer and CNN, comprising a speckle image acquisition device, a deep learning image restoration method, and a speckle image restoration method, characterized in that: The speckle image acquisition device includes a laser (1), lens A (2), lens B (3), spatial light modulator (4), target image (5), objective lens (6), scattering medium (8), objective lens (9), and CMOS camera (10) arranged sequentially along the optical path. The speckle image acquisition device also includes a computer (7). The output terminals of the spatial light modulator (4) and the CMOS camera (10) are connected to the computer (7) through wires and simultaneously controlled by a Matlab program to continuously capture and collect speckle images (11). The speckle image restoration method includes a speckle image restoration network based on PyTorch framework, which is based on Transformer and CNN. Building a speckle map restoration network based on Transformer and CNN in PyTorch includes the following steps: Step 1: After the image is input into the network, it enters the CNN encoding path and the Transformer encoding path respectively. The input size is... , where H and W represent the length and width of the image, C represents the number of image channels, and R represents the real number field; Step 2: The CNN encoding path contains 4 sub-modules. Each sub-module contains two 3×3 convolutions, a ReLU activation function, and a 2×2 max pooling layer for downsampling. Step 3: In the Transformer encoding path, the input size is... The speckle image (11) is divided into dimensions of N patches, that is ∈ To preserve positional information, these patches are projected into a K-dimensional embedding space, and a positional embedding is added for patch embedding. To preserve the spatial information of each patch, the formula for patch embedding is: ,in The projected embedding of the patch is represented, and then a Transformer module containing multi-head self-attention (MSA) and a multilayer perceptron (MLP) is used to learn a global context representation. The MSA layer consists of M parallel self-attention heads to scale the embedded patch, and the MLP module learns long-range context dependencies. The formulas for MSA and MLP are expressed as follows: , Norm() represents layer normalization. and Encode the output representations of the MSA and MLP of the nth layer of the Transformer; Step 4: After the two encoding paths, the CNN's encoded information and the information encoded from the Transformer are added together through the addition path and fused into the decoder. The decoder consists of 4 sub-modules. Each sub-module contains a 2×2 deconvolution, two 3×3 convolutions and a ReLU activation function to perform feature upsampling and decode the features into a high-resolution image. Step 5: Set the loss function to the negative Pearson correlation coefficient (NPCC), the formula of which is: （1） In equation (1), M(i,j) and N(i,j) represent the gray values of the network-reconstructed image and the original target image at the same spatial location, respectively, and i and j represent the two-dimensional coordinate indices of all effective pixels in the speckle image. and This represents the mean of the image restored by the network and the original target.
2. The speckle image restoration method for a scattering imaging device system based on Transformer and CNN according to claim 1, characterized in that: A scattering imaging device system based on Transformer and CNN includes the following steps: Step 1: Acquire speckle image (11). Use the MNIST handwritten digit image to acquire speckle image (11) as target image (5). Upsample to 1024×1024 and display on spatial light modulator (4). SLM output light carrying object information passes through scattering medium (8) and speckle image (11) is acquired by CMOS camera (10). Step 2: Establish the dataset. First, process the collected speckle image (11), crop the central square area to retain useful speckle information, and change the size to 256×256 as the input image of the network model. Upsample the MNIST handwritten digit image to 256×256 as the label image of the dataset. Divide the obtained images into training set, validation set and test set in a ratio of 0.8:0.1:0.
1. Step 3: Input the speckle image (11) as the input image and the target image (5) as the label image into the network for training to obtain the weight file of the image restoration network. Then input the speckle image (11) into the trained weight file for testing to reconstruct the target image.