A method and system for scatter imaging based on optical diffraction neural networks

Abstract

This invention discloses a scattering imaging method and system based on an optical diffraction neural network. The method includes: transmitting incident light through a target image and inputting it into a preset scattering medium phase plate, outputting a corresponding degraded image; the preset scattering medium phase plate includes a preset static or dynamic scattering medium phase plate; inputting the degraded image into a pre-trained optical diffraction neural network, and obtaining a reconstructed image of the target image using a detector placed at the focal point; the optical diffraction neural network includes an input layer, an output layer, and several diffraction structure layers located in the middle, each diffraction structure layer including a diffraction layer and a Droupt layer, and a lens is placed after the last diffraction structure layer. Compared with the prior art, which can only achieve imaging through static scattering media, this invention also achieves imaging through dynamic scattering media and improves the optical diffraction neural network, resulting in better imaging effects for both complex and simple objects.

Description

Technical Field

[0001] This invention belongs to the field of scattering imaging, specifically relating to a scattering imaging method and system based on an optical diffraction neural network. Background Technology

[0002] Deep learning, as one of the most popular methods in artificial intelligence and machine learning, aims to enable machines to analyze input data such as text and images using learning methods similar to humans, quickly obtaining the inherent patterns in sample data to perform complex and difficult tasks. Currently, deep learning, with its excellent algorithms, has solved many complex problems in image classification, object detection, natural language processing, optical imaging, and autonomous driving, achieving significant progress. With the tremendous success and advancement of deep learning, the combination of optical computing and deep learning has become a research focus in recent years, and imaging through scattering media has been a pressing problem in the field of optics for decades. Among these methods, using optical diffraction neural networks to solve scattering imaging offers faster computation speed and lower energy consumption compared to computer-built neural network frameworks, making it of significant research importance.

[0003] For example, an existing technical solution designs and manufactures an optical diffraction neural network for imaging scattering media, specifically in the terahertz band. This network uses a phase plate to simulate a static scattering medium. The input light field first strikes the phase plate, causing phase information disturbance, and then the image transmitted through the phase plate is reconstructed using multiple diffraction layers. However, the simulation effect of this scheme is not very good; the Pearson coherence coefficient (PCC) used to evaluate the simulation results is only 0.75 (the closer to 1, the better the effect). Therefore, after 3D printing, the actual imaging quality is not very good due to the influence of manufacturing precision and the alignment accuracy between layers.

[0004] Overall, current optical diffraction neural network scattering imaging technology has the following drawbacks:

[0005] 1. Simple target recovery: Existing technologies only image simple binary targets and do not image complex grayscale targets. However, the field of scattering imaging requires the ability to image complex targets.

[0006] 2. Limited and limited applications: Existing technologies only simulate imaging through static scattering media, but scattering media are usually dynamic, such as clouds, water, and fog, all of which have time-varying characteristics. Therefore, they cannot be applied to most scattering media.

[0007] 3. The imaging effect of the recovered target is generally poor: it is based only on the conventional optical diffraction neural network framework, resulting in a generally poor imaging effect. Summary of the Invention

[0008] To address the aforementioned problems in the prior art, this invention provides a scattering imaging method and system based on an optical diffraction neural network. The technical problem to be solved by this invention is achieved through the following technical solution:

[0009] In a first aspect, embodiments of the present invention provide a scattering imaging method based on an optical diffraction neural network, the method comprising:

[0010] After the incident light passes through the target image, it is input into a preset scattering medium phase plate, and the corresponding degraded image is output; wherein, the preset scattering medium phase plate includes a preset static scattering medium phase plate or a preset dynamic scattering medium phase plate.

[0011] The degraded image is input into a pre-trained optical diffraction neural network, and a detector placed at the focal point is used to obtain a restored image of the target image. The optical diffraction neural network includes an input layer, an output layer, and several diffraction structure layers in the middle. Each diffraction structure layer includes a diffraction layer and a Droupt layer, and a lens is placed after the last diffraction structure layer.

[0012] In one embodiment of the present invention, the preset static scattering medium phase plate is used to simulate the static scattering medium by convolving a normal distribution conforming to the standard deviation and mean with a zero-mean Gaussian smoothing kernel; the transmittance of the preset static scattering medium phase plate is defined as follows:

[0013]

[0014] Among them, t D (x,y) is the transmittance of the preset static scattering medium phase plate; j is the imaginary unit; Δn is the refractive index difference between air and the static scattering medium phase plate; λ is the wavelength of the incident light; D(x,y) is the height at the static scattering medium phase plate; D(x,y)=W(x,y)*K(σ); W(x,y) follows a normal distribution with mean μ and standard deviation σ0, expressed as W(x,y)~N(μ,σ0); K(σ) is a zero-mean Gaussian smoothing kernel with standard deviation σ; * indicates 2D convolution operation.

[0015] In one embodiment of the present invention, the preset dynamic scattering medium phase plate is an approximation of the dynamic scattering medium body as an infinitely thin axial plane slice along the optical axis, with each slice simulating a phase plate; wherein, the light field distribution of the i-th dynamic scattering medium phase plate is expressed as:

[0016] O i (w)=diag(p i (w i ))HO i-1 (w);

[0017] Where, diag(·) is a function used to construct a diagonal matrix, w is the difference between the refractive index of the particles in the dynamic scattering medium and the constant background, expressed as w = n - n0, and H is the diffraction factor that affects the distribution of the generated light field through free diffraction propagation, H = F -1 diag(h)F,F -1 F and F are the inverse two-dimensional Fourier transform and the two-dimensional Fourier transform, respectively, and h is the abbreviation for h(p,q). p and q are the frequencies in the x and y directions of diffraction propagation between the phase plates of the dynamic scattering medium, respectively; Δz is the distance between the two phase plates of the dynamic scattering medium; k is the wavenumber of the background medium, k = k0n0, where n0 is a constant background refractive index. λ is the wavelength of the incident light, Δk is the sampling interval in Fourier space, and p i (w i ) is the refraction propagation factor. w i It is the i-th dynamic scattering medium phase plate;

[0018] The degraded image output after passing through all the dynamic scattering medium phase plates is represented as follows:

[0019] I = |O(W) 2 |=Aexp(φ);

[0020] Where O(W) is the optical field distribution after passing through the last dynamic scattering medium phase plate, and A and φ represent the optical field amplitude information and phase information, respectively.

[0021] The transmittance of the phase plate of the dynamic scattering medium is expressed as:

[0022] T D (x,y)=exp(φ).

[0023] In one embodiment of the present invention, the training process of the optical diffraction neural network includes:

[0024] Obtain a training set; wherein the training set includes images of various types, all of which are used as sample images; wherein the various types of images include digital images and face images;

[0025] Construct a parallel model; wherein the parallel model includes at least three branches, each branch including a preset scattering medium phase plate connected in sequence, and an optical diffraction neural network as one channel of the parallel model;

[0026] In any training round, for each branch, the incident light selected for that branch passes through the sample image of that branch and is input into the preset scattering medium phase plate of that branch. The degraded image output is input into the optical diffraction neural network of that branch, received by its input layer, modulated by the diffraction layers in each diffraction structure layer, and output by the output layer to the lens for reception, until the modulated predicted recovery image of that branch is obtained by the detector; wherein, the sample image types of the at least three branches are different.

[0027] Based on the predicted recovered image output by the at least three branches in parallel, the value of the total loss function for this round of training is obtained, and backpropagation is performed to optimize the parameters of each optical diffraction neural network based on the value of the total loss function.

[0028] Repeat the training multiple times until the model convergence condition is met, and obtain the trained optical diffraction neural networks.

[0029] In one embodiment of the present invention, each optical diffraction neural network comprises M diffraction structure layers, wherein modulation of each diffraction layer is achieved at a position z = z m The light field u after the m-th diffraction structure layer at point m m (x,y,z m ) is represented as:

[0030]

[0031] Where S is the transmittance of the preset scattering medium phase plate, and if a preset static scattering medium phase plate is selected, S is the transmittance t of the preset static scattering medium phase plate. D (x,y), if a preset dynamic scattering medium phase plate is selected, S is the transmittance T of the preset dynamic scattering medium phase plate. D (x,y);

[0032] D m (x,y) is the Droupt layer, t m (x,y,z m ) is the transmittance function through the diffraction layer, w(x,y,z) is the transmission kernel that follows the Rayleigh-Sommerfeld diffraction formula, and Δz m It is the distance between the two diffraction layers, φ(x,y,z) m ) represents the phase information in the m-th diffraction layer, where m ranges from [1, M]; r is the propagation distance between two diffraction nodes following the Rayleigh-Sommerfeld diffraction formula; and λ is the wavelength of the incident light.

[0033] In one embodiment of the present invention, the predicted recovery image of any branch is represented as:

[0034] o(x,y)=|um*w(x,y,f)*tl(x,y) 2 ;

[0035] Among them, u m For u m (x,y,z m (abbreviation of ); f is the focal length of the lens; t l (x,y) is the phase modulation factor introduced by the lens itself. k is the wave number of the background medium.

[0036] In one embodiment of the present invention, the total loss function is obtained by weighted summation of the loss functions of the at least three branches and their corresponding weighting coefficients.

[0037] In one embodiment of the present invention, the loss function for each branch is the Pearson correlation coefficient.

[0038] In one embodiment of the present invention, the step of transmitting incident light through a target image and inputting it into a preset scattering medium phase plate to output a corresponding degraded image; and inputting the degraded image into a pre-trained optical diffraction neural network to obtain a restored image of the target image using a detector placed at the focal point, includes:

[0039] For all branches of the trained parallel model, the incident light selected for each branch is passed through the target image of that branch and input into the preset scattering medium phase plate of that branch, outputting the corresponding degraded image. The degraded image is then input into the pre-trained optical diffraction neural network of that branch, and the restored image of the target image of that branch is obtained by using a detector placed at the focal point.

[0040] Secondly, embodiments of the present invention provide a scattering imaging method based on an optical diffraction neural network, the system comprising:

[0041] A scattering medium phase plate processing module is used to receive the light waves after the incident light passes through the target image, input them into a preset scattering medium phase plate, and output the corresponding degraded image; wherein, the preset scattering medium phase plate includes a preset static scattering medium phase plate or a preset dynamic scattering medium phase plate.

[0042] An optical diffraction neural network restoration module is used to receive the degraded image using a pre-trained optical diffraction neural network and obtain a restored image of the target image using a detector placed at the focal point of the optical diffraction neural network. The optical diffraction neural network includes an input layer, an output layer, and several diffraction structure layers in the middle. Each diffraction structure layer includes a diffraction layer and a Droupt layer, and a lens is placed after the last diffraction structure layer.

[0043] The beneficial effects of this invention are:

[0044] The scattering imaging method based on optical diffraction neural networks provided in this invention, compared to existing technologies that can only achieve imaging through static scattering media, enables imaging through dynamic scattering media. Furthermore, it improves both the internal structure and overall framework of the optical diffraction neural network, solving both types of scattering imaging problems simply by replacing the phase plate of the scattering medium. This results in a simple, universal imaging framework with superior imaging performance and wider application in the field of scattering imaging. Unlike data-based deep learning methods, this invention achieves imaging through scattering media by recovering light field information and improves the framework model of the optical diffraction neural network, resulting in better imaging performance for both complex and simple objects. This scattering imaging method framework exhibits high robustness and flexibility. Attached Figure Description

[0045] Figure 1 This is a schematic flowchart of a scattering imaging method based on an optical diffraction neural network provided in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram illustrating the principle of optical diffraction neural network-based scattering imaging in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of one branch in the parallel model of an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram illustrating the training process using three branches as an example in an embodiment of the present invention;

[0049] Figure 5 The figure shows the simulation results of the static scattering imaging method based on optical diffraction neural network in the existing technology;

[0050] Figure 6 This is a simulation result diagram of imaging of a static scattering medium based on an optical diffraction neural network in an embodiment of the present invention;

[0051] Figure 7 This is a simulation result of transmission dynamic scattering medium imaging based on optical diffraction neural network in an embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram of a scattering imaging system based on an optical diffraction neural network, provided as an embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0054] Currently, scattering imaging technology based on optical diffraction neural networks has the following problems:

[0055] (1) At the theoretical level, most of the current optical diffraction neural network models are only studied for digital classification tasks, and there is a lack of more detailed research on imaging task models.

[0056] (2) At the application level, only the problem of imaging through static scattering media has been solved so far. However, scattering media are usually dynamic, such as clouds, water and fog, which have time-varying characteristics.

[0057] (3) At the imaging level, it is currently only used for imaging simple objects, and the imaging effect is generally poor.

[0058] To address the aforementioned problems, embodiments of the present invention provide a scattering imaging method and system based on an optical diffraction neural network.

[0059] In a first aspect, embodiments of the present invention provide a scattering imaging method based on an optical diffraction neural network, such as... Figure 1 As shown, the method may include the following steps:

[0060] S1, after the incident light passes through the target image, it is input into the preset scattering medium phase plate and the corresponding degraded image is output;

[0061] The preset scattering medium phase plate includes a preset static scattering medium phase plate or a preset dynamic scattering medium phase plate.

[0062] S2, the degraded image is input into a pre-trained optical diffraction neural network, and a restored image of the target image is obtained using a detector placed at the focal point; wherein, the optical diffraction neural network includes an input layer, an output layer and several diffraction structure layers located in the middle, each diffraction structure layer includes a diffraction layer and a Droupt layer, and a lens is placed after the last diffraction structure layer.

[0063] This invention improves the imaging effect and increases the complexity of the imaging target by refining the framework of the optical diffraction neural network. Furthermore, it redesigns a reasonable phase plate to simulate imaging through dynamic scattering media. Therefore, compared to existing technologies that only target static scattering media imaging, this invention expands the application scope of optical diffraction neural networks in the field of scattering imaging.

[0064] Specifically, scattering phenomena are generally divided into static scattering imaging and dynamic scattering imaging. Static scattering imaging assumes that the scattered light does not change over time, while dynamic scattering imaging assumes that the intensity of the scattered light fluctuates over time. The fluctuation is related to the Brownian motion of the particles, and the degree of fluctuation is related to the intensity of the Brownian motion. In the field of scattering imaging, dynamic scattering imaging is more difficult than static scattering imaging. Current methods for simulating image transmission through a scattering medium generally rely on real-world photographs. However, real-world photographs, received by a camera detector, only record the intensity information of the scattering medium, not its phase information. Optical diffraction neural networks recover the phase information of the image to achieve light field reconstruction. Therefore, to make the simulation more accurate and realistic, and considering that the application of optical diffraction neural networks is currently limited, with most applications involving imaging through static scattering media, it is necessary to construct appropriate phase plates to simulate the imaging of objects through scattering media.

[0065] The following describes the construction process of the preset scattering medium phase plate in the embodiments of the present invention.

[0066] (1) Preset static scattering medium phase plate

[0067] The preset static scattering medium phase plate is used to simulate the static scattering medium by convolving a normal distribution that conforms to the standard deviation and mean with a zero-mean Gaussian smooth kernel.

[0068] The transmittance of the preset static scattering medium phase plate is defined as follows:

[0069]

[0070] Among them, t D (x,y) is the transmittance of the preset static scattering medium phase plate; j is the imaginary unit; Δn is the refractive index difference between air and the static scattering medium phase plate; λ is the wavelength of the incident light; D(x,y) is the height of the static scattering medium phase plate.

[0071] Wherein, D(x,y) is further defined as:

[0072] D(x,y)=W(x,y)*K(σ);

[0073] W(x,y) follows a normal distribution with mean μ and standard deviation σ0, denoted as W(x,y)~N(μ,σ0); K(σ) is a zero-mean Gaussian smoothing kernel with standard deviation σ; * indicates a 2D convolution operation.

[0074] Using a static scattering medium phase plate, we can simulate the image formation of light waves through a static scattering medium. A key feature of the static scattering medium phase plate is that light wave propagation is calculated only once, independent of time evolution. Light wave information consists of amplitude and phase, representing the intensity and propagation characteristics of the light wave, respectively. These two factors together determine the complete behavior of the light wave. The constructed static scattering medium phase plate is essentially a transmissive phase plate; therefore, light waves passing through it will cause phase disturbance, resulting in an image formed by the light passing through the static scattering medium—a degraded image.

[0075] Static scattering media that can be simulated using this method include frosted glass of different mesh counts, optical fibers, etc.

[0076] (2) Preset dynamic scattering medium phase plate

[0077] Dynamic scattering media exhibit time-varying characteristics and are dynamic three-dimensional objects. These dynamic three-dimensional scattering media can be simulated by recording two-dimensional holograms. The specific approach is as follows:

[0078] The preset dynamic scattering medium phase plate approximates the dynamic scattering medium as an infinitely thin axial plane slice along the optical axis, with each slice simulating a phase plate; therefore, there can be multiple dynamic scattering medium phase plates.

[0079] In actual simulations, an infinitely thin wavelength can be in the range of 1 / 10 to 1 / 100 of the wavelength of light.

[0080] The optical field distribution of the i-th dynamic scattering medium phase plate is related to the optical field distribution of the (i-1)-th dynamic scattering medium phase plates; the optical field distribution of the i-th dynamic scattering medium phase plate is expressed as:

[0081] O i (w)=diag(p i (w i ))HO i-1 (w);

[0082] Here, diag(·) is a function used to construct a diagonal matrix, w is the difference between the particle in the dynamic scattering medium and the constant background refractive index, expressed as w = n - n0; the final light field distribution is composed of the diffraction propagation factor and the refraction propagation factor. After passing through a series of dynamic scattering medium phase plates, it is finally converted into a two-dimensional complex field, and the dynamic scattering medium phase plates propagate through free diffraction.

[0083] H is the diffraction factor that influences the distribution of the generated light field through free diffraction propagation, H = F. -1 diag(h)F,F -1 F and F are the inverse two-dimensional Fourier transform and the two-dimensional Fourier transform, respectively, and h is the abbreviation for h(p,q). p and q are the frequencies in the x and y directions of diffraction propagation between the phase plates of the dynamic scattering medium, respectively; Δz is the distance between the two phase plates of the dynamic scattering medium; k is the wavenumber of the background medium, k = k0n0, where n0 is a constant background refractive index. λ is the wavelength of the incident light, Δk is the sampling interval in Fourier space, and p i (w i ) is the refraction propagation factor. w i It is the i-th dynamic scattering medium phase plate;

[0084] The interferogram generated after multiple scatterings is received as hologram I (i.e., the degraded image). The hologram contains the amplitude and phase information of the light field after the object passes through a dynamic scattering medium over a distance.

[0085] Specifically, the degraded image output after passing through all the dynamic scattering medium phase plates is represented as follows:

[0086] I = |O(W) 2 |=Aexp(φ);

[0087] Where O(W) is the light field distribution after passing through the last dynamic scattering medium phase plate, and A and φ represent the light field amplitude information and phase information; the phase information extracted from the hologram can simulate the dynamic scattering medium phase plate;

[0088] The transmittance of the phase plate of the dynamic scattering medium is expressed as:

[0089] T D (x,y)=exp(φ).

[0090] The phase plate of the dynamic scattering medium can be used to simulate the imaging of light waves passing through the image through the dynamic scattering medium. The scattering characteristics of the dynamic medium change with time and have time-varying properties.

[0091] In summary, the specific principle of this part of the invention can be simply described as follows:

[0092] 1. Constructing a multi-phase plate model: The dynamic scattering medium is approximated as multiple thin phase plates along the optical axis, with the light field distribution of each phase plate depending on the previous phase plate. By understanding the dependency relationship between the light field distributions of the preceding and following phase plates, the propagation changes of light waves at each moment can be continuously simulated.

[0093] 2. Obtaining a two-dimensional hologram (i.e., a degraded image) of light waves passing through a dynamic scattering medium: By recording holograms at each time step, the time-varying influence of the dynamic scattering medium on light waves can be captured. The hologram contains amplitude and phase information of the light field, which can be used to reconstruct the influence of the scattering medium on light waves.

[0094] 3. Phase extraction: To simplify the process, the dynamic characteristics of the scattering medium are ultimately obtained by extracting phase information from the hologram.

[0095] This method can simulate scattering media such as seawater, turbid underwater environments, dynamic smoke, and haze.

[0096] For S1, the incident light passes through the target image and is then input to a preset static scattering medium phase plate t. D (x,y) or a preset dynamic scattering medium phase plate T D (x,y), the output degraded image is used as the input image G of the optical diffraction neural network.

[0097] This invention improves upon conventional optical diffraction neural networks. The core of a conventional optical diffraction neural network is based on angular spectral diffraction theory to construct the light propagation coefficients between diffraction layers, with propagation between layers occurring through free diffraction. Each diffraction layer has a complex transmission coefficient, generating a secondary wave. The overall optical diffraction neural network consists of an input layer, an output layer, and several intermediate diffraction layers. A diffraction layer can be understood as a diffraction plate with a transmission coefficient; the pixel particles are very small, and different pixel particles have different optical path lengths, causing changes in the output wavefront after the light wave passes through the diffraction layer.

[0098] However, conventional optical diffraction neural network frameworks generally produce poor imaging results when imaging complex targets. To address this issue, this invention improves conventional optical diffraction neural networks by refining both the internal structure and the overall framework of the network.

[0099] (1) Improvement of network internal structure

[0100] To address the overfitting issue that current diffraction neural networks suffer from in imaging, this invention adds a Droupt layer after each diffraction layer, forming a diffraction structure layer. The Droupt layer prevents overfitting by randomly discarding a portion of neurons. Furthermore, unlike optical diffraction neural networks used in digital image classification, where parallel light waves reach the detector for partitioning, and different light wave information falls into corresponding partitions, this invention differs from optical diffraction neural networks. In imaging, after the incident light wave passes through the last diffraction layer, it passes through the detector as parallel light to receive intensity information. However, the detector's pixel size is finite, preventing it from receiving all light wave information. Therefore, this invention adds a lens after the last diffraction structure layer to converge parallel light rays, placing the detector at the focal point. This framework structure is more suitable for imaging applications.

[0101] (2) Improvement of the overall network framework

[0102] To address the generally poor performance of current diffraction neural networks in target reconstruction imaging, this invention constructs at least three shared optical diffraction neural networks for parallel learning and cascaded processing. Each network processes different targets independently; optionally, it can simultaneously handle simple digital targets and complex grayscale targets, or utilize a multi-wavelength multiplexing framework. During training, random weights are applied to the final output layer to effectively provide the final result of the entire system, improving the inference capability of the overall system model. After training, each optical diffraction neural network can be used individually or in parallel, increasing flexibility.

[0103] The training process of the optical diffraction neural network is described below. For the principle of scattering imaging based on the optical diffraction neural network in this embodiment of the invention, please refer to [link to relevant documentation]. Figure 2 understand.

[0104] In one optional implementation, the training process of the optical diffraction neural network includes:

[0105] ① Obtain the training set; wherein, the training set includes images of various types, all of which are used as sample images;

[0106] The various types of images include digital images and facial images;

[0107] Compared to existing technologies that primarily target digital images, the training set of this invention contains a wider variety of sample images, making it applicable to complex objects.

[0108] ② Construct a parallel model; wherein the parallel model includes at least three branches, each branch including a preset scattering medium phase plate connected in sequence, and an optical diffraction neural network as one channel of the parallel model;

[0109] Each channel of the optical diffraction neural network includes an input layer, an output layer, and several diffraction structure layers in between. Each diffraction structure layer includes a diffraction layer and a Droupt layer. A lens is placed after the last diffraction structure layer, and a detector is placed at the focal point of the optical diffraction neural network. The detector can also be considered as a detector layer. For the structure of one branch in the parallel model, please refer to [link to relevant documentation]. Figure 3 In this diagram, the Input Plane is the input image, the Scattering Media Plate is a preset scattering medium phase plate, the gray Diffractive Layers are the diffraction layers of the optical diffraction neural network (Droupt layer omitted and not shown), the blue elliptical part is the lens, and the Output Plane is the recovery effect after passing through the optical diffraction network, i.e., the recovered image (detector omitted and not shown).

[0110] After the parallel model is built, the main input parameters of the optical diffraction neural network need to be set, including: the wavelength λ of the incident light, the number of diffraction layers M, the number of training epochs, and the learning rate.

[0111] The wavelength λ of the incident light in each branch can be different.

[0112] ③ In any round of training, for each branch, the incident light selected for that branch passes through the sample image of that branch and is input into the preset scattering medium phase plate of that branch. The degraded image output is input into the optical diffraction neural network of that branch, received by its input layer, modulated by the diffraction layers in each diffraction structure layer, and output by the output layer to the lens for reception, until the modulated predicted recovery image of that branch is obtained by the detector; wherein, the sample image types of the at least three branches are different;

[0113] Steps ③ to ④ constitute one round of training.

[0114] Please refer to step ③ Figure 3 To understand, for each branch, firstly, the incident light selected for that branch passes through the sample image of that branch, and then the light wave is input into the preset scattering medium phase plate of that branch. The preset scattering medium phase plate simulates the light wave passing through the sample image and then through the scattering medium to form an image, outputting a degraded image.

[0115] The degraded image is then fed into the optical diffraction neural network of this branch. The input layer of the optical diffraction neural network receives the image, and the diffraction layers in each diffraction structure layer are modulated to achieve optical field recovery. The image is then output by the output layer to the lens for receiving and converging. At the focal point of the lens, the image is detected and received by the detector to obtain the predicted recovered image modulated by this branch.

[0116] Each optical diffraction neural network contains M diffraction structure layers, i.e., M diffraction layers.

[0117] Through modulation of each of its diffraction layers, at z = z m The light field u after the m-th diffraction structure layer at point m m (x,y,z m ) is represented as:

[0118]

[0119] Where S is the transmittance of the preset scattering medium phase plate, and if a preset static scattering medium phase plate is selected, S is the transmittance t of the preset static scattering medium phase plate. D (x,y), if a preset dynamic scattering medium phase plate is selected, S is the transmittance T of the preset dynamic scattering medium phase plate. D (x,y);

[0120] D m (x,y) is the Droupt layer, t m (x,y,z m ) is the transmittance function through the diffraction layer, w(x,y,z) is the transmission kernel that follows the Rayleigh-Sommerfeld diffraction formula, and Δz m It is the distance between the two diffraction layers, φ(x,y,z) m ) represents the phase information in the m-th diffraction layer, where m ranges from [1, M]; r is the propagation distance between two diffraction nodes following the Rayleigh-Sommerfeld diffraction formula; and λ is the wavelength of the incident light.

[0121] The detector receives the image modulated by all diffraction layers. Specifically, after being modulated by all diffraction layers, it is received by a lens with a focal length of f. The light field finally propagates an axial distance of f to the output plane, and the light field information can be recovered after being modulated by the diffraction layers. The final simulation demonstrates how a focusing lens converges the light to the detector to receive its intensity map. Due to the thickness variation of the focusing lens, the equiphase surface of the incident light wave is bent after passing through it. t l (x,y) represents the phase modulation factor introduced by the lens itself.

[0122] Its strength is calculated as the network output, i.e., the predicted restored image of any branch, expressed as:

[0123] o(x,y)=|um*w(x,y,f)*tl(x,y) 2 ;

[0124] Among them, u m For u m (x,y,z m (abbreviation of ); f is the focal length of the lens; t l (x,y) is the phase modulation factor introduced by the lens itself. k is the wave number of the background medium.

[0125] It should be noted that each branch processes the data in parallel according to the above process, and each can obtain the corresponding predicted and restored image. The network architecture of each branch is the same, the difference lies in the selection of different light waves (different wavelength information for each channel) and different input target information (different sample image types, so that each channel processes different targets).

[0126] ④ Based on the predicted restored image output by the at least three branches in parallel, obtain the value of the total loss function for this round of training, and perform backpropagation to optimize the parameters of each optical diffraction neural network based on the value of the total loss function;

[0127] In this embodiment of the invention, the total loss function is obtained by weighted summation of the loss functions of the at least three branches and their corresponding weighting coefficients.

[0128] In one alternative implementation, the loss function for each branch is the Pearson correlation coefficient. This is more suitable for the field of scattering imaging.

[0129] Please see Figure 4 Taking three branches as an example, the total loss function is defined as the result of three optical diffraction neural networks processing in parallel and multiplying by random weighting coefficients, which can be expressed as:

[0130]

[0131] Where Loss is the total loss function, loss n Let w be the loss function of the optical diffraction neural network in the nth branch. fn loss n The corresponding weighting coefficients are n = 1, 2, 3.

[0132]

[0133] Where G is the input sample image, Let G be the label image corresponding to G, and O be the predicted restored image corresponding to the input sample image. O represents the corresponding label image.

[0134] In one optional implementation, the loss function of each branch can be a complex loss function; the optical diffraction neural network itself achieves the purpose of optical field recovery by modulating the phase of the optical field or by co-modulating the amplitude and phase. The complex loss function can further improve the network efficiency.

[0135] The backpropagation process during training is the optimization objective of constructing the optical diffraction neural network, namely, minimizing the total loss function through the backpropagation algorithm. This involves calculating the gradient of the phase parameters with respect to the diffraction plate. The backpropagation (BP) algorithm can be used to obtain all the gradients used to update the diffraction layer parameters of the optical diffraction neural network. That is, by using appropriate optimization strategies (such as stochastic gradient descent (SGD)), these gradients can be used to iteratively update the optical diffraction neural network to achieve transmission and scattering medium imaging.

[0136] ⑤ Repeat the training multiple times until the model convergence condition is met, and obtain the trained optical diffraction neural networks.

[0137] The model convergence condition can be that the number of iterations reaches the training epoch, where epoch represents the maximum number of epochs.

[0138] The above describes the training process of an optical diffraction neural network. Through the above training, at least three optical diffraction neural networks are obtained after training.

[0139] Therefore, the processes for S1 and S2 can be:

[0140] For all branches of the trained parallel model, the incident light selected for each branch is passed through the target image of that branch and input into the preset scattering medium phase plate of that branch, outputting the corresponding degraded image. The degraded image is then input into the pre-trained optical diffraction neural network of that branch, and the restored image of the target image of that branch is obtained by using a detector placed at the focal point.

[0141] Therefore, the present invention can perform parallel restoration imaging processing on at least three target images simultaneously. Of course, it is also reasonable to use one or two branches for restoration imaging processing.

[0142] The scattering imaging method based on optical diffraction neural networks provided in this invention, compared to existing technologies that can only achieve imaging through static scattering media, enables imaging through dynamic scattering media. Furthermore, it improves both the internal structure and overall framework of the optical diffraction neural network, solving both types of scattering imaging problems simply by replacing the phase plate of the scattering medium. This results in a simple, universal imaging framework with superior imaging performance and wider application in the field of scattering imaging. Unlike data-based deep learning methods, this invention achieves imaging through scattering media by recovering light field information and improves the framework model of the optical diffraction neural network, resulting in better imaging performance for both complex and simple objects. This scattering imaging method framework exhibits high robustness and flexibility.

[0143] To facilitate understanding of the effectiveness of the methods in the embodiments of the present invention, relevant simulation results are given below.

[0144] Please see Figure 5 , Figure 5 The image shows the simulation results of the static scattering imaging method based on optical diffraction neural network in the existing technology. The first row, Target Object, represents the target image. The second row, FSP (Output Plane), represents the degraded image after the incident light passes through the target image, passes through the phase plate of the static scattering medium, and then propagates freely to the detector layer. The third row, Diffractive Network, represents the recovered image after the degraded image passes through the existing optical diffraction neural network and propagates freely to the detector layer.

[0145] Figure 6The image shown is a simulation result of imaging through a static scattering medium based on an optical diffraction neural network in an embodiment of the present invention. It is completed in parallel using three branches. The first row is the target image, the second row is the degraded image after the incident light passes through the target image and then through the phase plate of the static scattering medium of the present invention, and the third row is the restored image after the degraded image passes through the optical diffraction neural network of the present invention.

[0146] Figure 7 The image shown is a simulation result of imaging through a dynamic scattering medium based on an optical diffraction neural network in an embodiment of the present invention. The first row is the target image, the second row is the degraded image after the incident light passes through the target image and then through the phase plate of the dynamic scattering medium of the present invention, and the third row is the restored image after the degraded image passes through the optical diffraction neural network of the present invention.

[0147] As can be seen, the scattering imaging method based on optical diffraction neural network provided in this embodiment of the invention can achieve imaging through static scattering media as well as imaging through dynamic scattering media, and the imaging effect is good.

[0148] Secondly, corresponding to the above method embodiments, this invention also provides a scattering imaging system based on an optical diffraction neural network, such as... Figure 8 As shown, the system includes:

[0149] A scattering medium phase plate processing module is used to receive the light waves after the incident light passes through the target image, input them into a preset scattering medium phase plate, and output the corresponding degraded image; wherein, the preset scattering medium phase plate includes a preset static scattering medium phase plate or a preset dynamic scattering medium phase plate.

[0150] An optical diffraction neural network restoration module is used to receive the degraded image using a pre-trained optical diffraction neural network and obtain a restored image of the target image using a detector placed at the focal point of the optical diffraction neural network. The optical diffraction neural network includes an input layer, an output layer, and several diffraction structure layers in the middle. Each diffraction structure layer includes a diffraction layer and a Droupt layer, and a lens is placed after the last diffraction structure layer.

[0151] For details on the specific processing procedures of each module of this system, please refer to the relevant content in the first section, which will not be elaborated here.

[0152] It should be noted that, in the description of this invention, the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0153] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0155] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0156] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A scattering imaging method based on an optical diffraction neural network, characterized in that, include: After the incident light passes through the target image, it is input into a preset scattering medium phase plate, and the corresponding degraded image is output; wherein, the preset scattering medium phase plate includes a preset static scattering medium phase plate or a preset dynamic scattering medium phase plate. The degraded image is input into a pre-trained optical diffraction neural network, and a detector placed at the focal point is used to obtain a restored image of the target image. The optical diffraction neural network includes an input layer, an output layer, and several diffraction structure layers in the middle. Each diffraction structure layer includes a diffraction layer and a Droupt layer, and a lens is placed after the last diffraction structure layer.

2. The scattering imaging method based on optical diffraction neural network according to claim 1, characterized in that, The preset static scattering medium phase plate is used to simulate the static scattering medium by convolving a normal distribution conforming to the standard deviation and mean with a zero-mean Gaussian smoothing kernel; the transmittance of the preset static scattering medium phase plate is defined as follows: Among them, t D (x,y) is the transmittance of the preset static scattering medium phase plate; j is the imaginary unit; Δn is the refractive index difference between air and the static scattering medium phase plate; λ is the wavelength of the incident light; D(x,y) is the height at the static scattering medium phase plate; D(x,y)=W(x,y)*K(σ); W(x,y) follows a normal distribution with mean μ and standard deviation σ0, expressed as W(x,y)~N(μ,σ0); K(σ) is a zero-mean Gaussian smoothing kernel with standard deviation σ; * indicates 2D convolution operation.

3. The scattering imaging method based on optical diffraction neural network according to claim 1, characterized in that, The preset dynamic scattering medium phase plate is an approximation of the dynamic scattering medium as an infinitely thin axial plane slice along the optical axis, with each slice simulating a phase plate; wherein, the light field distribution of the i-th dynamic scattering medium phase plate is expressed as: O i (w)=diag(p i (w i ))HO i-1 (w); Where, diag(·) is a function used to construct a diagonal matrix, w is the difference between the refractive index of the particles in the dynamic scattering medium and the constant background, expressed as w = n - n0, and H is the diffraction factor that affects the distribution of the generated light field through free diffraction propagation, H = F -1 diag(h)F,F -1 F and F are the inverse two-dimensional Fourier transform and the two-dimensional Fourier transform, respectively, and h is the abbreviation for h(p,q). p and q are the frequencies in the x and y directions of diffraction propagation between the phase plates of the dynamic scattering medium, respectively; Δz is the distance between the two phase plates of the dynamic scattering medium; k is the wavenumber of the background medium, k = k0n0, where n0 is a constant background refractive index. λ is the wavelength of the incident light, Δk is the sampling interval in Fourier space, and p i (w i ) is the refraction propagation factor. w i It is the i-th dynamic scattering medium phase plate; The degraded image output after passing through all the dynamic scattering medium phase plates is represented as follows: I=|O(W) 2 |=Aexp(φ); Where O(W) is the optical field distribution after passing through the last dynamic scattering medium phase plate, and A and φ represent the optical field amplitude information and phase information, respectively. The transmittance of the phase plate of the dynamic scattering medium is expressed as: T D (x,y)=exp(φ)。 4. The scattering imaging method based on optical diffraction neural network according to any one of claims 1-3, characterized in that, The training process of the optical diffraction neural network includes: Obtain a training set; wherein the training set includes images of various types, all of which are used as sample images; wherein the various types of images include digital images and face images; Construct a parallel model; wherein the parallel model includes at least three branches, each branch including a preset scattering medium phase plate connected in sequence, and an optical diffraction neural network as one channel of the parallel model; In any training round, for each branch, the incident light selected for that branch passes through the sample image of that branch and is input into the preset scattering medium phase plate of that branch. The degraded image output is input into the optical diffraction neural network of that branch, received by its input layer, modulated by the diffraction layers in each diffraction structure layer, and output by the output layer to the lens for reception, until the modulated predicted recovery image of that branch is obtained by the detector; wherein, the sample image types of the at least three branches are different. Based on the predicted recovered image output by the at least three branches in parallel, the value of the total loss function for this round of training is obtained, and backpropagation is performed to optimize the parameters of each optical diffraction neural network based on the value of the total loss function. Repeat the training multiple times until the model convergence condition is met, and obtain the trained optical diffraction neural networks.

5. The scattering imaging method based on optical diffraction neural network according to claim 4, characterized in that, Each optical diffraction neural network contains M diffraction structure layers. Through modulation of each diffraction layer, at z = z... m The light field u after the m-th diffraction structure layer at point m m (x,y,z m ) is represented as: Where S is the transmittance of the preset scattering medium phase plate, and if a preset static scattering medium phase plate is selected, S is the transmittance t of the preset static scattering medium phase plate. D (x,y), if a preset dynamic scattering medium phase plate is selected, S is the transmittance T of the preset dynamic scattering medium phase plate. D (x,y); D m (x,y) is the Droupt layer, t m (x,y,z m ) is the transmittance function through the diffraction layer, w(x,y,z) is the transmission kernel that follows the Rayleigh-Sommerfeld diffraction formula, and Δz m It is the distance between the two diffraction layers, φ(x,y,z) m ) represents the phase information in the m-th diffraction layer, where m ranges from [1, M]; r is the propagation distance between two diffraction nodes following the Rayleigh-Sommerfeld diffraction formula; and λ is the wavelength of the incident light.

6. The scattering imaging method based on optical diffraction neural network according to claim 5, characterized in that, The predicted restored image for any branch is represented as: o(x,y)=|um*w(x,y,f)*tl(x,y) 2 ; Among them, u m For u m (x,y,z m (abbreviation of ); f is the focal length of the lens; t l (x,y) is the phase modulation factor introduced by the lens itself. k is the wave number of the background medium.

7. The scattering imaging method based on optical diffraction neural network according to claim 4, characterized in that, The total loss function is obtained by weighted summation of the loss functions of the at least three branches and their corresponding weighting coefficients.

8. The scattering imaging method based on optical diffraction neural network according to claim 7, characterized in that, The loss function for each branch is the Pearson correlation coefficient.

9. The scattering imaging method based on optical diffraction neural network according to claim 4, characterized in that, After the incident light passes through the target image, it is input into a preset scattering medium phase plate, and the corresponding degraded image is output. The degraded image is input into a pre-trained optical diffraction neural network, and a reconstructed image of the target image is obtained using a detector placed at the focal point, including: For all branches of the trained parallel model, the incident light selected for each branch is passed through the target image of that branch and input into the preset scattering medium phase plate of that branch, outputting the corresponding degraded image. The degraded image is then input into the pre-trained optical diffraction neural network of that branch, and the restored image of the target image of that branch is obtained by using a detector placed at the focal point.

10. A scattering imaging system based on an optical diffraction neural network, characterized in that, include: A scattering medium phase plate processing module is used to receive the light waves after the incident light passes through the target image, input them into a preset scattering medium phase plate, and output the corresponding degraded image; wherein, the preset scattering medium phase plate includes a preset static scattering medium phase plate or a preset dynamic scattering medium phase plate. An optical diffraction neural network restoration module is used to receive the degraded image using a pre-trained optical diffraction neural network and obtain a restored image of the target image using a detector placed at the focal point of the optical diffraction neural network. The optical diffraction neural network includes an input layer, an output layer, and several diffraction structure layers in the middle. Each diffraction structure layer includes a diffraction layer and a Droupt layer, and a lens is placed after the last diffraction structure layer.

Images