Device, method and system for recovering image penetrating through scattering medium

Abstract

The invention belongs to the technical field of optical scattering imaging, and particularly relates to a device, a method and a system for recovering an image penetrating through a scattering medium. The device comprises a laser, an attenuation piece, a beam expander, a spatial light modulator, a dynamic scattering medium simulator, a lens and a CMOS camera which are sequentially arranged along a light path. The method comprises the following steps: acquiring a speckle image formed after a target image passes through a scattering medium by using the device; constructing a neural network comprising an encoder, a decoder and multi-layer jump connection for training so as to learn reverse mapping from the speckle image to the target image; and recovering an unknown speckle image by using the trained network. According to the system level, a graphical user interface integrating data acquisition, model reasoning and quality evaluation is adopted. According to the method, through collaborative design of the light path and the algorithm, the limitation of a traditional method on a static or weak scattering medium is effectively overcome, and high-quality and real-time recovery of the image under the dynamic scattering medium is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of optical scattering imaging, and particularly relates to an image recovery device, method and system through a scattering medium. BACKGROUND

[0002] When optical imaging is performed through a scattering medium (such as biological tissue, fog, ground glass), due to uneven distribution of refractive index inside the medium, the incident light wavefront will be severely scattered randomly, the phase and amplitude information thereof are disturbed, and the target image is degraded into a seemingly irregular speckle pattern. This effect seriously restricts the development of many frontier fields, such as deep tissue imaging in biomedicine, visual system of autonomous driving in bad weather, and the like.

[0003] To overcome the influence of scattering, the prior art has proposed various schemes, but all have significant limitations. The specific explanations are as follows: optical gating technology relies on the time difference between ballistic photons and scattered photons, and is only applicable to scenarios with weak scattering effect, and is invalid in strong scattering medium with large optical thickness. Iterative feedback technology (such as wavefront shaping) can compensate for scattering, but needs to be iteratively optimized for multiple times, has low calculation efficiency, and is easy to fall into a local optimal solution, making it difficult to realize real-time imaging of dynamic medium. The transfer matrix method can accurately characterize the transmission characteristics of the scattering medium, but its measurement process is complex, and is extremely sensitive to small vibrations of the environment and dynamic changes of the medium itself, and the practical range is limited to highly stable laboratory environment. In recent years, speckle inversion methods based on deep learning have shown potential, and can learn the complex mapping relationship between speckle and target image through data-driven way. However, the existing methods mostly focus on the algorithm itself, and often lack a matching optical hardware platform that can stably collect high-quality training data. In addition, these schemes are usually isolated algorithm models, and fail to integrate an easy-to-use software interface, limiting their popularization and application among non-professional users.

[0004] Therefore, the prior art lacks a solution that closely integrates a stable and reliable optical acquisition device, a robust algorithm suitable for dynamic scattering environment, and a user-friendly complete system. SUMMARY

[0005] The present application aims to overcome the above-mentioned problems in the prior art, and provides an image recovery device, method and system through a scattering medium, which realizes high-quality and high-efficiency image recovery in a dynamic scattering medium.

[0006] To achieve the above technical purposes and effects, the present application is implemented by the following technical solutions:

[0007] The application provides an image recovery device for penetrating a scattering medium, which comprises a laser, an attenuating sheet, a beam expander, a spatial light modulator, a dynamic scattering medium simulator, a lens and a CMOS camera; the laser emits laser light which is first subjected to light intensity adjustment by the attenuating sheet and then subjected to beam expansion and collimation by the beam expander; the collimated light beam is incident on the spatial light modulator and loaded with target image information and subjected to amplitude modulation; the modulated light beam is irradiated on the dynamic scattering medium simulator which is placed near the focal plane of the lens; the scattered light field after penetrating the scattering medium is converged by the lens and received by the CMOS camera to form a speckle image.

[0008] Further, in the image recovery device, the dynamic scattering medium simulator is a self-built moving scattering medium which is composed of a vibrator and a ground glass scattering sheet; the ground glass scattering sheet is fixed on the vibrator, and the vibration frequency of the vibrator is adjustable.

[0009] The application further provides an image recovery method for penetrating a scattering medium, which comprises the following steps:

[0010] S1, building an image recovery device, loading a target image to a spatial light modulator, collecting a speckle image formed after degradation by a scattering medium by a CMOS camera, and constructing a training data set containing a target image-speckle image pair;

[0011] S2, constructing a neural network model, training the model by using the training data set, and learning an end-to-end nonlinear mapping relationship from a speckle image to a target image;

[0012] S3, inputting an unknown speckle image to be recovered into the trained neural network model, and outputting a recovered clear image.

[0013] Further, in step S2, the neural network model is a U-Net structure, and the core thereof comprises:

[0014] an encoder path: used for extracting multi-level features from an input speckle image, composed of a plurality of convolution modules in cascade, the feature map size is reduced step by step, and the channel number is increased;

[0015] a decoder path: used for mapping and up-sampling the features to recover to the original image size, composed of a plurality of transpose convolution modules in cascade;

[0016] a skip connection: a plurality of shortcut paths are established between the corresponding levels of the encoder and the decoder, the shallow detail features extracted by the encoder are fused with the deep semantic features of the decoder, so as to improve the structure fidelity of the recovered image.

[0017] Further, in step S2, the neural network model further comprises a Dropout layer, wherein the Dropout layer: introduces a Dropout operation at different levels of the encoder and the decoder, randomly discards part of neurons during training, suppresses overfitting, and improves the generalization ability of the model.

[0018] Further, in step S2, the training process is mathematically expressed as: for a scattering imaging system forward model , wherein x is an original light field, y is an observed speckle, f is a scattering operator; the neural network is trained to approximate the inverse process thereof , that is, directly from y reconstructs x .

[0019] The application also provides an image recovery system through a scattering medium, which is a software platform with a graphical user interface, and integrates the following functional modules:

[0020] An image display area: used for displaying an original image, an input speckle image, and a predicted image output by the neural network, respectively;

[0021] An interactive control module: providing operation buttons of "select image", "load model", "start prediction", and "save result", and controlling the entire recovery process;

[0022] A quality evaluation module: automatically calculating and displaying objective evaluation indexes of the predicted image, including a peak signal-to-noise ratio (PSNR) and a structural similarity (SSIM).

[0023] Further, the graphical user interface is developed based on a PyQt framework, and provides a streamlined and visual operation environment.

[0024] The application has the following beneficial effects:

[0025] 1. At the device level, the application provides an optical acquisition platform with simple and stable light paths. The device can simulate and generate real-time dynamic speckle patterns, providing a high-quality data basis for subsequent image recovery. In particular, through precise control of light intensity (attenuation sheet) and collimation of light beams (beam expander), stable speckle signals can still be acquired under dynamic scattering media, overcoming the dependence of traditional methods on static scattering media and being suitable for dynamic scattering environments.

[0026] 2. At the algorithm level, the encoder-decoder neural network designed in this invention, featuring skip connections and Dropout mechanisms, significantly improves the image restoration quality and model robustness. The skip connection structure effectively alleviates the vanishing gradient problem in deep networks, directly transmitting low-level detail features (such as edges and textures) from the encoder to the decoder, fusing them with high-level semantic features. This results in a restored image that is structurally closer to the original target, with more complete detail preservation, significantly improving the Structural Similarity (SSIM) metric. The introduction of Dropout layers as a regularization technique effectively prevents overfitting to the training data. This makes the trained model more generalizable, achieving high-quality restoration even for new degraded speckle images not present in the training set, thus exhibiting a higher Peak Signal-to-Noise Ratio (PSNR), meaning the restored image has less noise and higher fidelity.

[0027] 3. At the system level, this invention integrates data acquisition, model inference, and quality assessment into a complete solution. Through a developed graphical user interface (e.g., based on PyQt5), the complex image restoration process is simplified into intuitive user operations (e.g., button clicks), greatly lowering the barrier to entry and improving efficiency. The system's built-in image quality evaluation module (PSNR / SSIM) can objectively and quantitatively evaluate the restoration effect, providing users with immediate feedback, which is beneficial for the practical application and promotion of the technology.

[0028] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the image restoration device through a scattering medium according to the present invention;

[0031] Figure 2 This is a schematic diagram of the neural network model training system of the present invention;

[0032] Figure 3 This is a schematic diagram of the speckle image restoration system of the present invention;

[0033] Figure 4 This is a diagram showing the binarized image restoration effect achieved in Example 3;

[0034] Figure 5This is a diagram showing the 8-bit image restoration effect achieved in Example 3. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment provides an image restoration device that transmits light through a scattering medium. The device includes a laser 1, an attenuator 2, a beam expander 3, a spatial light modulator 4, a dynamic scattering medium simulator 5, a lens 6, and a CMOS camera 7. The laser emitted by the laser 1 has its light intensity output adjusted by the attenuator 2; then it is expanded and collimated by the beam expander 3; the expanded and collimated laser beam is irradiated by the spatial light modulator 4 for modulation; the modulated beam is scattered by the dynamic scattering medium simulator 5; and finally, it is converged by the lens 6 and received by the CMOS camera 7.

[0038] In this embodiment, the dynamic scattering medium simulator 5 is a self-built motion scattering medium, which consists of a vibrator and a frosted glass scattering sheet. The frosted glass scattering sheet is fixed on the vibrator, and the vibration frequency of the vibrator is adjustable.

[0039] Example 2

[0040] like Figure 2 As shown, this embodiment provides an image restoration method through a scattering medium, based on the device of Embodiment 1 for model training and image restoration. The specific steps are as follows:

[0041] Step S1: Dataset Preparation

[0042] Prepare multiple types of images as the target image set on the computer, including but not limited to binarized images of handwritten digits and letters (such as the MNIST dataset) and 8-bit grayscale images of natural scenes (such as CIFAR-10 or self-collected photos).

[0043] For each target image x The operation is performed using the apparatus of Example 1: the image x Loaded onto spatial light modulator 4, triggering CMOS camera 7 to acquire corresponding speckle images. y Repeat this process thousands of times to build a scaled-up paired dataset D = {( x i , yi ) | i = 1, 2,...}.

[0044] The dataset D is randomly divided into training and validation sets in a ratio of about 8:2.

[0045] Step S2: Neural network model construction and training

[0046] Under the PyTorch or TensorFlow deep learning framework, an encoder-decoder network with skip connections (similar to U-Net) is constructed.

[0047] The encoder consists of 4 convolutional modules connected in series. Each module contains: a 3x3 convolutional layer (stride = 1, padding = 1), a BatchNorm layer, a ReLU activation function layer, and a 2x2 max-pooling layer with a stride of 2 (for downsampling). The output channel numbers of the 4 modules can be set to 32, 64, 128, and 256, respectively.

[0048] The decoder also consists of 4 up-sampling modules connected in series. Each module contains: a 3x3 transpose convolutional layer (stride = 2, for up-sampling), a feature map from the corresponding level of the encoder (skip connection), and a change in the number of channels opposite to the encoder.

[0049] The skip connection establishes a shortcut connection between each corresponding level of the encoder and the decoder, a total of 4 times.

[0050] The Dropout layer is inserted after the ReLU activation function in each convolutional module. The dropout rate after the first convolutional module is set to 0.05, and the rest are all 0.1.

[0051] In the training setting, the loss function uses the mean square error loss (MSE Loss) or the weighted combination of L1 Loss and structural similarity loss (SSIM Loss).

[0052] In the training setting, the optimizer uses the Adam optimizer with an initial learning rate of 0.001 and a learning rate decay strategy.

[0053] In the training setting, the training period is trained on the training set, and the performance is monitored on the validation set. Usually, 100-200 epochs are trained until the loss function converges.

[0054] After training, the optimal model weight is saved. For a new, unknown speckle image y newOnly need to be input into the neural network loaded with trained weights, forward propagation after the output of the recovered image directly x pred .

[0055] Embodiment three

[0056] As Figure 3 shown, the embodiment provides an image restoration system through a scattering medium, the graphical user interface (GUI) of the image restoration system is implemented as follows:

[0057] Develop GUI using Python language and PyQt5 library. The main window is divided into three main image display areas: "original image", "speckle image" and "predicted image".

[0058] The interface on the right side is set to the interactive button module, including: "select original image" (used to load the ideal target image), "select speckle image" (used to load the speckle image to be restored), "load model" (load the pre-trained.pth or.h5 model file), "start prediction" (execute inference), "save predicted image" (save the result).

[0059] The image quality evaluation module is set below the interface. After prediction is completed, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values between the predicted image and the original image are automatically calculated and displayed on the specified label.

[0060] The above embodiment is applied to restore the image through the dynamic ground glass scattering. The restoration effect is shown in Figure 4 and Figure 5 . Quantitative analysis shows that for binary images, the PSNR value of the restoration result can be more than 35dB, and the SSIM value can be more than 0.95; for complex 8-bit grayscale images, the PSNR value can also be more than 24dB, and the SSIM value can be more than 0.65, verifying the effectiveness of the present application.

[0061] The present application combines special optical hardware, innovative deep learning algorithms and friendly software systems to provide a practical solution for real-time, high-quality image restoration through dynamic scattering media, which has broad application prospects in the fields of automatic driving (such as penetrating rain and fog), medical imaging (such as penetrating biological tissues) and the like.

[0062] The preferred embodiments of the application disclosed above are only to facilitate the elucidation of the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments. Obviously, many modifications and variations can be made in light of the teachings above. The description is chosen and described in order to provide the best illustration of the application and its practical application to those skilled in the art and to enable those skilled in the art to best utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. An image restoration device through a scattering medium, characterized by, The image recovery device comprises a laser (1), an attenuating sheet (2), a beam expander (3), a spatial light modulator (4), a dynamic scattering medium simulator (5), a lens (6), and a COMS camera (7); the laser (1) emits laser light which is first subjected to light intensity adjustment by the attenuating sheet (2) and then subjected to beam expansion and collimation by the beam expander (3); the collimated light beam is incident on the spatial light modulator (4) and loaded with target image information and subjected to amplitude modulation; the modulated light beam is irradiated on the dynamic scattering medium simulator (5) which is placed near the focal plane of the lens (6); the scattered light field after penetrating the scattering medium is converged by the lens (6) and received by the CMOS camera (7), forming a speckle image.

2. The image restoration apparatus according to claim 1, characterized by The dynamic scattering medium simulator (5) is a self-built moving scattering medium which is composed of a vibrator and a ground glass scattering sheet; the ground glass scattering sheet is fixed on the vibrator, and the vibration frequency of the vibrator is adjustable.

3. An image restoration method through a scattering medium, characterized by, The method comprises the following steps: S1, the image recovery device is built as described in claim 1 or 2, the target image is loaded to the spatial light modulator (4), the speckle image formed after the scattering medium is degraded is collected by the CMOS camera (7), and a training data set containing the target image-speckle image pair is constructed; S2, a neural network model is constructed, the model is trained by using the training data set, and an end-to-end nonlinear mapping relationship from the speckle image to the target image is learned; S3, an unknown speckle image to be recovered is input into the trained neural network model, and a clear recovered image is output.

4. The image restoration method according to claim 3, characterized by, In step S2, the neural network model is of a U-Net structure, and the core thereof comprises: an encoder path: used for extracting multi-level features from the input speckle image, composed of multiple convolution modules in cascade, the feature map size is gradually reduced, and the channel number is increased; a decoder path: used for mapping and up-sampling the features to recover to the original image size, composed of multiple transposed convolution modules in cascade; a jump connection: multiple shortcut paths are established between the corresponding levels of the encoder and the decoder, the shallow detail features extracted by the encoder are fused with the deep semantic features of the decoder, so that the structure fidelity of the recovered image is improved.

5. The image restoration method according to claim 4, characterized by, In step S2, the neural network model further comprises a Dropout layer, wherein the Dropout layer: introduces a Dropout operation at different levels of the encoder and the decoder, randomly discards part of the neurons during training, so as to suppress overfitting and improve the generalization ability of the model.

6. The image restoration method according to claim 5, characterized by, In step S2, the training process is mathematically formulated as: for a scattering imaging system forward model where x is the original light field, y is the observed speckle, f is the scattering operator; a neural network is trained to approximate its inverse process i.e. to directly reconstruct y from x。 7. An image restoration system that transmits light through a scattering medium, characterized in that, The image recovery system is a software platform with a graphical user interface, and the following functional modules are integrated: an image display area: used for displaying the original image, the input speckle image, and the predicted image output by the neural network respectively; an interactive control module: providing "select image", "load model", "start prediction", and "save result" operation buttons to control the entire recovery process; a quality evaluation module: automatically calculating and displaying objective evaluation indexes of the predicted image, including the peak signal-to-noise ratio and the structural similarity.

8. The image restoration system of claim 7, wherein The graphical user interface is developed based on the PyQt5 framework, and provides a streamlined and visual operation environment.

Images