Scattering imaging method and system based on optical diffraction neural network

Abstract

The invention discloses a scattering imaging method and system based on an optical diffraction neural network, and the method comprises the steps: enabling incident light to penetrate a target image, inputting the incident light into a preset scattering medium phase plate, and outputting a corresponding degraded image; the preset scattering medium phase plate comprises a preset static or dynamic scattering medium phase plate; inputting the degraded image into a pre-trained optical diffraction neural network, and obtaining a recovered image of the target image by using a detector placed at a focus; the optical diffraction neural network comprises an input layer, an output layer and a plurality of diffraction structure layers located in the middle, each diffraction structure layer comprises a diffraction layer and a Droupt layer, and a lens is arranged behind the last diffraction structure layer. Compared with the prior art which can only realize imaging through a static scattering medium, the method also realizes imaging through a dynamic scattering medium, improves the optical diffraction neural network, and enables the imaging effect to be better for complex and simple objects.

Description

Technical Field

[0001] The invention belongs to the field of scattering imaging, and in particular relates to a scattering imaging method and system based on optical diffraction neural network. Background Art

[0002] As one of the most popular methods in artificial intelligence and machine learning, deep learning aims to enable machines to use human-like learning methods to analyze input text, pictures and other data, and obtain the inherent laws of sample data at the fastest speed to perform complex and difficult tasks. At present, deep learning has solved many complex problems in the fields of image classification, target detection, natural language processing, optical imaging, and autonomous driving with its excellent algorithms, and has made significant progress. With the great success of deep learning, the combination of optical computing and deep learning has been the focus of research in recent years, and in recent decades, imaging through scattering media has been an urgent problem to be solved in the field of optics. Among them, the method of using optical diffraction neural network to solve scattering imaging has faster computing speed and lower energy consumption than the neural network framework built by computer, which is of great research significance.

[0003] For example, in the prior art solution, an optical diffraction neural network applied to the field of scattering medium imaging is designed and manufactured, and the neural network is applied to the terahertz band. A phase plate is used to simulate a static scattering medium. The input light field is first incident on the phase plate to cause disturbance of the phase information, and then the image transmitted through the phase plate is restored by continuous all-optical reconstruction through multiple diffraction layers. However, the simulation effect of this scheme is not very good. The Pearson coherence coefficient (PCC) is used to evaluate the simulation results, which is only 0.75 (the closer to 1, the better the effect). Therefore, after 3D printing, the actual imaging quality is not very good due to the influence of manufacturing accuracy and the accuracy of alignment between layers.

[0004] Overall, the current scattering imaging technology of optical diffraction neural network has the following disadvantages:

[0005] 1. The target restoration is simple: the existing technology only images simple binary targets, but not complex grayscale targets. However, in the field of scatter imaging, it is necessary to realize complex target imaging.

[0006] 2. Limited application and relatively simple: Existing technologies only simulate imaging through static scattering media, but scattering media are usually dynamic, such as clouds, water, and fog, which all have time-varying characteristics. Therefore, they cannot be applied to most scattering media.

[0007] 3. The imaging effect of restored target is average: It is only based on the conventional optical diffraction neural network framework, resulting in average imaging effect. Summary of the invention

[0008] In order to solve the above problems existing in the prior art, the present invention provides a scattering imaging method and system based on an optical diffraction neural network. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0009] In a first aspect, an embodiment of the present invention provides a scattering imaging method based on an optical diffraction neural network, the method comprising:

[0010] After the incident light passes through the target image, it is input into a preset scattering medium phase plate, and a corresponding degraded image is output; wherein the preset scattering medium phase plate includes a preset static scattering medium phase plate, or a preset dynamic scattering medium phase plate;

[0011] The degraded image is input into a pre-trained optical diffraction neural network, and a restored image of the target image is obtained using a detector placed at the focus; wherein the optical diffraction neural network includes an input layer, an output layer and a plurality of diffraction structure layers located in the middle, each diffraction structure layer includes a diffraction layer and a droupt layer, and a lens is arranged after the last diffraction structure layer.

[0012] In one embodiment of the present invention, the preset static scattering medium phase plate simulates the static scattering medium by convolving a normal distribution that conforms to the standard deviation and the mean value with a zero-mean Gaussian smoothing kernel; the transmittance of the preset static scattering medium phase plate is defined as follows:

[0013]

[0014] Among them, t D (x, y) is the transmittance of the preset static scattering medium phase plate; j is an imaginary unit; Δn is the refractive index difference between air and the static scattering medium phase plate; λ is the wavelength of the incident light; D(x, y) is the height at the static scattering medium phase plate; D(x, y) = W(x, y) * K(σ); W(x, y) obeys a normal distribution with a mean value μ and a standard deviation σ0, expressed as W(x, y) ~ N(μ, σ0); K(σ) is a zero-mean Gaussian smoothing kernel with a standard deviation of σ; * represents a 2D convolution operation.

[0015] In one embodiment of the present invention, the preset dynamic scattering medium phase plate approximates the dynamic scattering medium body as an infinitely thin axial plane slice along the optical axis, and each slice is simulated as a phase plate; wherein the light field distribution of the i-th dynamic scattering medium phase plate is expressed as:

[0016] O i (w) = diag(p i (w i ))HO i-1 (w);

[0017] Where diag(·) is a function used to construct a diagonal matrix, w is the difference between the particle in the dynamic scattering medium and the constant background refractive index, expressed as w = n-n0, and H is the diffraction factor of the light field distribution generated by the free diffraction propagation, H = F -1 diag(h)F,F -1 and F are the two-dimensional inverse Fourier transform and the two-dimensional Fourier transform respectively, h is the abbreviation of h(p,q), p and q are the frequencies in the diffraction propagation directions x and y between the dynamic scattering medium phase plates, Δz is the distance between the two dynamic scattering medium phase plates, k is the wave number of the background medium, k = k0n0, n0 is the constant background refractive index, λ is the wavelength of the incident light, Δk is the sampling interval in Fourier space, and p i (w i ) is the refraction propagation factor, w i is the i-th dynamic scattering medium phase plate;

[0018] The degraded image output after passing through all dynamic scattering medium phase plates is expressed as:

[0019] I=|O(W) 2 | = Aexp(φ);

[0020] Among them, O(W) is the light field distribution after passing through the last dynamic scattering medium phase plate, A and φ represent the light field amplitude information and phase information;

[0021] The transmittance of the dynamic scattering medium phase plate is expressed as:

[0022] T D (x,y)=exp(φ).

[0023] In one embodiment of the present invention, the training process of the optical diffraction neural network includes:

[0024] Acquire a training set; wherein the training set includes multiple types of images, all of which are used as sample images; wherein the multiple types of images include digital images and face images;

[0025] Constructing a parallel model; wherein the parallel model includes at least three branches, each branch includes a preset scattering medium phase plate connected in sequence, and an optical diffraction neural network as one of the channels of the parallel model;

[0026] In any round of training, for each branch, the incident light selected by the branch is transmitted through the sample image of the branch and then input into the preset scattering medium phase plate of the branch, and the degraded image outputted by the branch is input into the optical diffraction neural network of the branch, received by the input layer, modulated by the diffraction layers in each diffraction structure layer, and outputted by the output layer to the lens for reception, until the predicted restored image after the modulation of the branch is obtained by using the detector; wherein the sample images of the at least three branches are of different types;

[0027] Obtaining a value of a total loss function of this round of training according to the predicted restored images output in parallel by the at least three branches, and performing back propagation optimization on parameters of each optical diffraction neural network according to the value of the total loss function;

[0028] Repeat multiple rounds of training until the model convergence condition is reached, and each trained optical diffraction neural network is obtained.

[0029] In one embodiment of the present invention, each optical diffraction neural network comprises M diffraction structure layers, and through the modulation of each diffraction layer, at the position z=z m The light field u after the mth diffraction structure layer at m (x,y,z m ) is expressed as:

[0030]

[0031] Wherein, S is the transmittance of the preset scattering medium phase plate. If the preset static scattering medium phase plate is selected, S is the transmittance of the preset static scattering medium phase plate. D (x, y), if the preset dynamic scattering medium phase plate is selected, S is the transmittance T of the preset dynamic scattering medium phase plate D (x,y);

[0032] D m (x,y) is the Dropt layer, t m (x,y,z m ) is the transmittance function through the diffraction layer, w(x,y,z) is the transmission kernel following the Rayleigh-Sommerfeld diffraction formula, Δz m is the distance between the two diffraction layers, φ(x,y,z m ) is the phase information in the mth diffraction layer, and the value range of m is [1,M]; r is the propagation distance between two diffraction nodes in accordance with the Rayleigh-Sommerfeld diffraction formula; λ is the wavelength of the incident light.

[0033] In one embodiment of the present invention, the predicted restoration image of any branch is expressed as:

[0034] o(x,y)=|um*w(x,y,f)*tl(x,y) 2 ;

[0035] Among them, u m for u m (x,y,z m ) abbreviation; f is the focal length of the lens; t l (x,y) is the phase modulation factor brought by the lens itself, k is the wave number of the background medium.

[0036] In one embodiment of the present invention, the total loss function is obtained by weighted summation of the loss functions of the at least three branches and the corresponding weight coefficients.

[0037] In one embodiment of the present invention, the loss function of each branch is the Pearson correlation coefficient.

[0038] In one embodiment of the present invention, after the incident light passes through the target image, it is input into a preset scattering medium phase plate to output a corresponding degraded image; the degraded image is input into a pre-trained optical diffraction neural network, and a restored image of the target image is obtained using a detector placed at the focus, including:

[0039] For all branches of the trained parallel model, the incident light selected by each branch is transmitted through the target image of the branch and then input into the preset scattering medium phase plate of the branch, and the corresponding degraded image is output. The degraded image is input into the pre-trained optical diffraction neural network of the branch, and the restored image of the target image of the branch is obtained using a detector placed at the focus.

[0040] In a second aspect, an embodiment of the present invention provides a scattering imaging method based on an optical diffraction neural network, the system comprising:

[0041] A scattering medium phase plate processing module, used to receive the light wave after the incident light passes through the target image, and input it into a preset scattering medium phase plate, and output a corresponding degraded image; wherein the preset scattering medium phase plate includes a preset static scattering medium phase plate, or a preset dynamic scattering medium phase plate;

[0042] An optical diffraction neural network restoration module is used to receive the degraded image using a pre-trained optical diffraction neural network, and obtain a restored image of the target image using a detector placed at the focus of the optical diffraction neural network; wherein the optical diffraction neural network includes an input layer, an output layer and a plurality of diffraction structure layers located in the middle, each diffraction structure layer includes a diffraction layer and a droupt layer, and a lens is arranged after the last diffraction structure layer.

[0043] Beneficial effects of the present invention:

[0044] The scattering imaging method based on the optical diffraction neural network provided in the embodiment of the present invention can realize imaging through dynamic scattering media, compared with the prior art which can only realize imaging through static scattering media, and improves the internal network and the overall network framework of the optical diffraction neural network. It only needs to replace the scattering medium phase plate to solve two types of scattering imaging problems, and realizes a simple structure and universal imaging framework, and has good imaging effect, and has a wider application range in the field of scattering imaging. Different from the data-based deep learning method, the present invention realizes imaging through scattering media by restoring light field information, and improves the framework model of the optical diffraction neural network, which can achieve better imaging effect for both complex objects and simple objects. The scattering imaging method framework has high robustness and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic flow chart of a scattering imaging method based on an optical diffraction neural network provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram for understanding the principle of scattering imaging based on optical diffraction neural network in an embodiment of the present invention;

[0047] Figure 3 It is a structural schematic diagram of a branch in the parallel model of an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a training process using three branches as an example in an embodiment of the present invention;

[0049] Figure 5 This is a simulation result diagram of a static scattering imaging method based on an optical diffraction neural network in the prior art;

[0050] Figure 6 This is a diagram of simulation results of imaging through static scattering media based on an optical diffraction neural network in an embodiment of the present invention;

[0051] Figure 7 This is a diagram of simulation results of imaging through dynamic scattering media based on an optical diffraction neural network in an embodiment of the present invention;

[0052] Figure 8 A schematic structural diagram of a scattering imaging system based on an optical diffraction neural network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0054] At present, the scattering imaging technology based on optical diffraction neural network has the following problems:

[0055] (1) On the theoretical level, most of the current optical diffraction neural network models are only studied for digital classification tasks, and there is a lack of more detailed research on imaging task models.

[0056] (2) At the application level, only the problem of imaging through static scattering media has been solved. However, scattering media are usually dynamic, such as clouds, water, and fog, and have time-varying characteristics.

[0057] (3) At the imaging level, only simple objects are imaged, and the imaging effect is average.

[0058] To solve the above problems, an embodiment of the present invention provides a scattering imaging method and system based on an optical diffraction neural network.

[0059] In a first aspect, an embodiment of the present invention provides a scattering imaging method based on an optical diffraction neural network, such as Figure 1 As shown, the method may include the following steps:

[0060] S1, after the incident light passes through the target image, it is input into the preset scattering medium phase plate and the corresponding degraded image is output;

[0061] Wherein, the preset scattering medium phase plate includes a preset static scattering medium phase plate, or a preset dynamic scattering medium phase plate;

[0062] S2, inputting the degraded image into a pre-trained optical diffraction neural network, and obtaining a restored image of the target image using a detector placed at the focus; wherein the optical diffraction neural network includes an input layer, an output layer and a plurality of diffraction structure layers located in the middle, each diffraction structure layer includes a diffraction layer and a droupt layer, and a lens is arranged after the last diffraction structure layer.

[0063] The embodiment of the present invention improves the framework of the optical diffraction neural network to achieve improved imaging effects and increased complexity of imaging targets. A reasonable phase plate is redesigned to simulate imaging through dynamic scattering media. Therefore, compared with the prior art that only targets static scattering media imaging, the present invention can improve the application scope of the optical diffraction neural network in the field of scattering imaging.

[0064] Specifically, the scattering phenomenon is generally divided into static scattering imaging and dynamic scattering imaging. Static scattering imaging assumes that the scattered light will not change with time, while dynamic scattering imaging assumes that the scattered light intensity will fluctuate with time. The reason for the fluctuation is related to the Brownian motion of the particles, and the degree of fluctuation is related to the intensity of the Brownian motion. In the field of scattering imaging, dynamic scattering imaging is more difficult than static scattering imaging. How to simulate the image through the scattering medium for imaging, the current implementation method is generally real-shot images, but the real-shot images are received through the camera detector, which only records the intensity information of the scattering medium, and cannot record its phase information. The optical diffraction neural network realizes light field recovery by restoring the phase information of the image. Therefore, in order to make the simulation effect more accurate and practical, and in view of the fact that the optical diffraction neural network is not widely used at present, most of the problems are imaging through static scattering media, and it is necessary to construct reasonable phase plates to simulate the imaging of objects through scattering media.

[0065] The following first describes the construction process of the preset scattering medium phase plate in the embodiment of the present invention.

[0066] (1) Preset static scattering medium phase plate

[0067] The preset static scattering medium phase plate simulates the static scattering medium by convolving a normal distribution that conforms to the standard deviation and the average value with a zero-mean Gaussian smoothing kernel;

[0068] The transmittance of the preset static scattering medium phase plate is defined as follows:

[0069]

[0070] Among them, t D (x, y) is the transmittance of the preset static scattering medium phase plate; j is an imaginary unit; Δn is the refractive index difference between air and the static scattering medium phase plate; λ is the wavelength of the incident light; D(x, y) is the height at the static scattering medium phase plate;

[0071] Where D(x,y) is further defined as:

[0072] D(x,y)=W(x,y)*K(σ);

[0073] W(x,y) obeys a normal distribution with mean μ and standard deviation σ0, expressed as W(x,y)~N(μ,σ0); K(σ) is a zero-mean Gaussian smoothing kernel with standard deviation σ; * represents a 2D convolution operation.

[0074] Through the static scattering medium phase plate, it is possible to simulate the imaging of light waves through static scattering media. The characteristic of the static scattering medium phase plate is that the propagation of light waves is calculated once and does not depend on time evolution. Light wave information consists of amplitude and phase, which represent the intensity and propagation characteristics of light waves respectively. The two together determine the complete behavior of light waves. The constructed static scattering medium phase plate is equivalent to a phase plate with transmittance, so the light wave will cause phase disturbance after passing through it, and the image obtained through the static scattering medium is a degraded image.

[0075] Static scattering media that can be simulated by this method include frosted glass of different mesh sizes, optical fibers, etc.

[0076] (2) Preset dynamic scattering medium phase plate

[0077] Dynamic scattering media have time-varying characteristics and are dynamic three-dimensional objects. Dynamic three-dimensional scattering media with time-varying characteristics can be simulated by recording two-dimensional holograms. The specific ideas are as follows:

[0078] The preset dynamic scattering medium phase plate approximates the dynamic scattering medium body as an infinitely thin axial plane slice along the optical axis, and each slice is simulated as a phase plate; therefore, there may be multiple dynamic scattering medium phase plates.

[0079] In actual simulation, the infinitely thin layer may be in the range of 1 / 10 to 1 / 100 of the wavelength of light.

[0080] The light field distribution of the i-th dynamic scattering medium phase plate is related to the light field distribution of the i-1 dynamic scattering medium phase plate; the light field distribution of the i-th dynamic scattering medium phase plate is expressed as:

[0081] O i (w) = diag(p i (w i ))HO i-1 (w);

[0082] Among them, diag(·) is a function used to construct a diagonal matrix, w is the difference between the particle particles in the dynamic scattering medium and the constant background refractive index, expressed as w=n-n0; the final light field distribution is composed of the diffraction propagation factor and the refraction propagation factor. Finally, after passing through a series of dynamic scattering medium phase plates, it is converted into a two-dimensional complex field, and the dynamic scattering medium phase plates propagate through free diffraction;

[0083] H is the diffraction factor of the light field distribution generated by the free diffraction propagation, H = F -1 diag(h)F,F -1 and F are the two-dimensional inverse Fourier transform and the two-dimensional Fourier transform respectively, h is the abbreviation of h(p,q), p and q are the frequencies in the diffraction propagation directions x and y between the dynamic scattering medium phase plates, Δz is the distance between the two dynamic scattering medium phase plates, k is the wave number of the background medium, k = k0n0, n0 is the constant background refractive index, λ is the wavelength of the incident light, Δk is the sampling interval in Fourier space, and p i (w i ) is the refraction propagation factor, w i is the i-th dynamic scattering medium phase plate;

[0084] The interference pattern generated after multiple scattering is finally received as a hologram I (i.e., a degraded image), which contains the amplitude and phase information of the light field after the object passes through a dynamic scattering medium for a certain distance.

[0085] Specifically, the degraded image output after passing through all dynamic scattering medium phase plates is expressed as:

[0086] I=|O(W) 2 | = Aexp(φ);

[0087] Wherein, O(W) is the light field distribution after passing through the last dynamic scattering medium phase plate, A and φ represent the light field amplitude information and phase information; extracting the phase information in the hologram can simulate the dynamic scattering medium phase plate;

[0088] The transmittance of the dynamic scattering medium phase plate is expressed as:

[0089] T D (x,y)=exp(φ).

[0090] The dynamic scattering medium phase plate can be used to simulate the imaging of light waves passing through the dynamic scattering medium. The scattering characteristics of the dynamic medium change with time and are time-varying.

[0091] In summary, the specific principle of this part of the present invention can be simply described as follows:

[0092] 1. Construct multiple phase plate models: Approximate the dynamic scattering medium as multiple thin phase plates along the optical axis. The light field distribution of each phase plate depends on the previous phase plate. Through the dependence of the light field distribution of the previous and next phase plates, the propagation changes of the light wave at each moment can be continuously simulated.

[0093] 2. Obtain a two-dimensional hologram (i.e., a degraded image) of a light wave passing through a dynamic scattering medium: By recording the hologram at each time step, the time-varying effect of the dynamic scattering medium on the light wave can be captured. The hologram contains the amplitude and phase information of the light field, which can be used to reconstruct the effect of the scattering medium on the light wave.

[0094] 3. Phase extraction: In order to simplify the processing, the dynamic characteristics of the scattering medium are finally obtained by extracting phase information from the hologram.

[0095] The scattering media that can be simulated by this method include sea water, turbid underwater environment, dynamic smoke, haze, etc.

[0096] For S1, the incident light passes through the target image and is input into the preset static scattering medium phase plate t D (x, y) or preset dynamic scattering medium phase plate T D (x, y), the output degraded image is used as the input image G of the optical diffraction neural network.

[0097] The present invention improves the conventional optical diffraction neural network. The core of the conventional optical diffraction neural network is to construct the light propagation coefficient between the diffraction layers based on the angular spectrum diffraction theory, and the layers are propagated by free diffraction. Each diffraction layer has a complex transmission coefficient, which produces a secondary wave. The overall optical diffraction neural network is composed of an input layer, an output layer and several intermediate diffraction layers. The diffraction layer can be understood as a diffraction plate with a transmission coefficient. The pixel particles are very small, and different pixel particles have different optical paths, which can change the output light wavefront after the light wave passes through the diffraction layer.

[0098] However, when the conventional optical diffraction neural network framework is faced with complex target imaging, the imaging effect is average. To address this problem, the present invention improves the conventional optical diffraction neural network by improving the internal structure of the network and the overall framework of the network.

[0099] (1) Improvement of internal network structure

[0100] In view of the current phenomenon of overfitting of the model in the field of imaging of diffraction neural networks, the present invention adds a Droutt layer after each diffraction layer. The two constitute a diffraction structure layer. The Droutt layer prevents overfitting by randomly discarding a part of neurons. In addition, the present invention is different from the application of optical diffraction neural networks in the field of digital image classification. Parallel light waves eventually reach the detector for partition processing, and light wave information of different numbers falls on the corresponding partition positions of the detector. In the field of imaging, after the incident light wave passes through the last diffraction layer, the light is received by the detector in the form of parallel light for intensity information. The pixel size of the detector is limited and cannot receive all light wave information. Therefore, the present invention adds a lens after the last diffraction structure layer to converge parallel light rays, and places the detector at the focus. Such a framework structure is more suitable for the field of imaging.

[0101] (2) Improvement of the overall network framework

[0102] In view of the fact that the current diffraction neural network has a general effect in restoring target imaging, the present invention constructs at least three common optical diffraction neural networks for parallel learning and cascade processing. Each network processes objects of different targets separately. Optionally, it can process simple digital targets and complex grayscale targets at the same time, or a multi-wavelength multiplexing framework. During the training process, random weights are finally used in the final output layer to effectively give the final result of the entire system, which can improve the reasoning ability of the entire system model. After the training is completed, each optical diffraction neural network can be used alone or in parallel, which increases flexibility.

[0103] The following is an explanation of the training process of the optical diffraction neural network. Figure 2 understand.

[0104] In an optional implementation manner, the training process of the optical diffraction neural network includes:

[0105] ① Obtaining a training set; wherein the training set includes multiple types of images, all of which are used as sample images;

[0106] Wherein, the multiple types of images include digital images and face images;

[0107] Compared with the prior art which mainly targets digital images, the training set of the embodiments of the present invention has more types of sample images and can be applied to complex objects.

[0108] ② Building a parallel model; wherein the parallel model includes at least three branches, each branch includes a preset scattering medium phase plate connected in sequence, and an optical diffraction neural network as one of the channels of the parallel model;

[0109] The optical diffraction neural network of each channel includes an input layer, an output layer and several diffraction structure layers in the middle. Each diffraction structure layer includes a diffraction layer and a droupt layer. A lens is arranged after the last diffraction structure layer. A detector is placed at the focus of the optical diffraction neural network. The detector can also be regarded as a detector layer. For the structure of a branch in the parallel model, please refer to Figure 3 It is understood that Input Plane is the input image, the color plate Scattering Media Plate is the preset scattering medium phase plate, the gray plate Diffractive Layers is the diffraction layer of the optical diffraction neural network (Droupt layer is omitted and not shown), the blue oval part is the lens, and Output Plane is the restoration effect after passing through the optical diffraction network, that is, the restored image (the detector is omitted and not shown).

[0110] After building the parallel model, you need to set the main input parameters of the optical diffraction neural network, including: the wavelength λ of the incident light, the number of diffraction layers M, the number of training rounds epoch, and the learning rate learning rate.

[0111] The wavelength λ of the incident light of each branch may be different.

[0112] ③ In any round of training, for each branch, the incident light selected by the branch is transmitted through the sample image of the branch and then input into the preset scattering medium phase plate of the branch, and the degraded image outputted by the branch is input into the optical diffraction neural network of the branch, received by the input layer, modulated by the diffraction layers in each diffraction structure layer, and outputted by the output layer to the lens for reception, until the predicted restored image after the modulation of the branch is obtained by using the detector; wherein the sample images of the at least three branches are of different types;

[0113] Steps ③ to ④ constitute a round of training process.

[0114] Step ③ See Figure 3 Understand that for each branch, first, after the incident light selected by the branch passes through the sample image of the branch, the light wave will be input into the preset scattering medium phase plate of the branch, and the preset scattering medium phase plate simulates the light wave passing through the sample image and then passing through the scattering medium to perform imaging, and outputs a degraded image;

[0115] Then the output degraded image is further input into the optical diffraction neural network of the branch, the input layer of the optical diffraction neural network receives it, the diffraction layers in each diffraction structure layer are modulated to realize light field restoration, and the output layer outputs it to the lens for reception and convergence, and the detector is detected and received at the focus of the lens to obtain the predicted restored image after the modulation of the branch;

[0116] Wherein, each optical diffraction neural network includes M diffraction structure layers, that is, M diffraction layers;

[0117] By modulating each of the diffraction layers, at z=z m The light field u after the mth diffraction structure layer at m (x,y,z m ) is expressed as:

[0118]

[0119] Wherein, S is the transmittance of the preset scattering medium phase plate. If the preset static scattering medium phase plate is selected, S is the transmittance of the preset static scattering medium phase plate. D (x, y), if the preset dynamic scattering medium phase plate is selected, S is the transmittance T of the preset dynamic scattering medium phase plate D (x,y);

[0120] D m (x,y) is the Dropt layer, t m (x,y,z m ) is the transmittance function through the diffraction layer, w(x,y,z) is the transmission kernel following the Rayleigh-Sommerfeld diffraction formula, Δz m is the distance between the two diffraction layers, φ(x,y,z m ) is the phase information in the mth diffraction layer, and the value range of m is [1,M]; r is the propagation distance between two diffraction nodes in accordance with the Rayleigh-Sommerfeld diffraction formula; λ is the wavelength of the incident light.

[0121] The detector receives the image after being modulated by all the diffraction layers. Specifically, after being modulated by all the diffraction layers, it is received by a lens with a focal length of f. The light field finally propagates an axial distance of f to the output plane, and the light field information can be restored after being modulated by the diffraction layers. The final simulation is to use a focusing lens to converge the light to the detector to receive its intensity map. Due to the change in thickness of the focusing lens itself, the equiphase surface after the incident light wave passes through is bent. t l (x,y) is the phase modulation factor brought by the lens itself.

[0122] Its intensity is calculated as the output of the network, that is, the predicted restored image of any branch, expressed as:

[0123] o(x,y)=|um*w(x,y,f)*tl(x,y) 2 ;

[0124] Among them, u m for u m (x,y,z m ) abbreviation; f is the focal length of the lens; t l (x,y) is the phase modulation factor brought by the lens itself, k is the wave number of the background medium.

[0125] It should be noted that each branch performs parallel processing according to the above process and can obtain the corresponding predicted restored image. The network architecture of each branch is the same, the difference is that different light waves can be selected (different wavelength information of each channel) and different input target information (different sample image types, so that each channel processes different targets).

[0126] ④ According to the predicted restored images output in parallel by the at least three branches, the value of the total loss function of this round of training is obtained, and the parameters of each optical diffraction neural network are optimized by back propagation according to the value of the total loss function;

[0127] In the embodiment of the present invention, the total loss function is obtained by weighted summing the loss functions of the at least three branches and the corresponding weighting coefficients.

[0128] In an optional implementation, the loss function of each branch is a Pearson correlation coefficient, which is more suitable for the field of scatter imaging.

[0129] See also Figure 4 , taking three branches as an example, the total loss function is defined as three optical diffraction neural networks processed in parallel and multiplied by random weight coefficients, which can be expressed as:

[0130]

[0131] Among them, Loss is the total loss function, loss n is the loss function of the optical diffraction neural network of the nth branch, w fn loss n The corresponding weighting coefficients are n=1, 2, 3.

[0132]

[0133] Among them, G is the input sample image, is the label image corresponding to G, O is the predicted restored image corresponding to the input sample image, O is the corresponding label image.

[0134] In an optional implementation, the loss function of each branch may be a complex loss function; the optical diffraction neural network itself achieves the purpose of light field recovery by modulating the phase of the light field or jointly modulating the amplitude and phase. The complex loss function can further improve the network efficiency.

[0135] The back propagation process in the training process is to build the optimization goal of the optical diffraction neural network, that is, to minimize the value of the total loss function through the back propagation algorithm. All gradients used to update the parameters of the diffraction layer of the optical diffraction neural network can be obtained through the back propagation (BP) algorithm. That is, by adopting appropriate optimization strategies (such as stochastic gradient descent (SGD)), these gradients can be used to iteratively update the optical diffraction neural network to achieve imaging through scattering media.

[0136] ⑤ Repeat multiple rounds of training until the model convergence condition is reached to obtain each trained optical diffraction neural network.

[0137] Among them, the model convergence condition can be that the iteration round reaches the training round number epoch, which represents the maximum round.

[0138] The above is the training process of the optical diffraction neural network. Through the above training, at least three trained optical diffraction neural networks are obtained.

[0139] Then, the process for S1 and S2 can be:

[0140] For all branches of the trained parallel model, the incident light selected by each branch is transmitted through the target image of the branch and then input into the preset scattering medium phase plate of the branch, and the corresponding degraded image is output. The degraded image is input into the pre-trained optical diffraction neural network of the branch, and the restored image of the target image of the branch is obtained using a detector placed at the focus.

[0141] Therefore, the present invention can perform parallel restoration imaging processing on at least three target images at the same time. Of course, one branch or two branches can also be used to perform restoration imaging processing, which is reasonable.

[0142] The scattering imaging method based on the optical diffraction neural network provided in the embodiment of the present invention can realize imaging through dynamic scattering media, compared with the prior art which can only realize imaging through static scattering media, and improves the internal network and the overall network framework of the optical diffraction neural network. It only needs to replace the scattering medium phase plate to solve two types of scattering imaging problems, and realizes a simple structure and universal imaging framework, and has good imaging effect, and has a wider application range in the field of scattering imaging. Different from the data-based deep learning method, the present invention realizes imaging through scattering media by restoring light field information, and improves the framework model of the optical diffraction neural network, which can achieve better imaging effect for both complex objects and simple objects. The scattering imaging method framework has high robustness and flexibility.

[0143] In order to facilitate understanding of the effectiveness of the method of the embodiment of the present invention, relevant simulation results are given below.

[0144] See also Figure 5 , Figure 5 This is a simulation result diagram of the static scattering imaging method based on the optical diffraction neural network in the prior art. The Target Object in the first row represents the target image, the FSP (Output Plane) in the second row represents the degraded image of the incident light passing through the target image and then passing through the static scattering medium phase plate and then propagating to the detector layer by free diffraction, and the Diffractive Network in the third row represents the restored image of the degraded image after passing through the existing optical diffraction neural network and then propagating to the detector layer by free diffraction.

[0145] Figure 6This is a diagram of the simulation results of imaging through a static scattering medium based on an optical diffraction neural network in an embodiment of the present invention. It is completed in parallel using three branches. The first row is the target image, the second row is the degraded image of the incident light after passing through the target image and then passing through the static scattering medium phase plate of the present invention; the third row is the restored image of the degraded image after passing through the optical diffraction neural network of the present invention.

[0146] Figure 7 The first row is the target image, the second row is the degraded image after the incident light passes through the target image and then passes through the dynamic scattering medium phase plate of the present invention; the third row is the restored image after the degraded image passes through the optical diffraction neural network of the present invention;

[0147] It can be seen that the scattering imaging method based on optical diffraction neural network provided by the embodiment of the present invention can realize imaging through static scattering media, and can also realize imaging through dynamic scattering media, and the imaging effect is good.

[0148] In a second aspect, corresponding to the above method embodiment, the embodiment of the present invention further provides a scattering imaging system based on an optical diffraction neural network, such as Figure 8 As shown, the system includes:

[0149] A scattering medium phase plate processing module, used to receive the light wave after the incident light passes through the target image, and input it into a preset scattering medium phase plate, and output a corresponding degraded image; wherein the preset scattering medium phase plate includes a preset static scattering medium phase plate, or a preset dynamic scattering medium phase plate;

[0150] An optical diffraction neural network restoration module is used to receive the degraded image using a pre-trained optical diffraction neural network, and obtain a restored image of the target image using a detector placed at the focus of the optical diffraction neural network; wherein the optical diffraction neural network includes an input layer, an output layer and a plurality of diffraction structure layers located in the middle, each diffraction structure layer includes a diffraction layer and a droupt layer, and a lens is arranged after the last diffraction structure layer.

[0151] For the specific processing process of each module of the system, please refer to the relevant content of the first aspect and will not be repeated here.

[0152] It should be noted that, in the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0153] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0154] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0155] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A scattering imaging method based on optical diffraction neural network, characterized in that: include: After the incident light passes through the target image, it is input into a preset scattering medium phase plate, and a corresponding degraded image is output; wherein the preset scattering medium phase plate includes a preset static scattering medium phase plate, or a preset dynamic scattering medium phase plate; The degraded image is input into a pre-trained optical diffraction neural network, and a restored image of the target image is obtained using a detector placed at the focus; wherein the optical diffraction neural network includes an input layer, an output layer and a plurality of diffraction structure layers located in the middle, each diffraction structure layer includes a diffraction layer and a droupt layer, and a lens is arranged after the last diffraction structure layer.

2. The scattering imaging method based on optical diffraction neural network according to claim 1, characterized in that: The preset static scattering medium phase plate simulates the static scattering medium by convolving a normal distribution that conforms to the standard deviation and the average value with a zero-mean Gaussian smoothing kernel; the transmittance of the preset static scattering medium phase plate is defined as follows: Among them, t D (x, y) is the transmittance of the preset static scattering medium phase plate; j is an imaginary unit; Δn is the refractive index difference between air and the static scattering medium phase plate; λ is the wavelength of the incident light; D(x, y) is the height at the static scattering medium phase plate; D(x, y) = W(x, y) * K(σ); W(x, y) obeys a normal distribution with a mean value μ and a standard deviation σ0, expressed as W(x, y) ~ N(μ, σ0); K(σ) is a zero-mean Gaussian smoothing kernel with a standard deviation of σ; * represents a 2D convolution operation.

3. The scattering imaging method based on optical diffraction neural network according to claim 1, characterized in that: The preset dynamic scattering medium phase plate approximates the dynamic scattering medium body as an infinitely thin axial plane slice along the optical axis, and each slice is simulated as a phase plate; wherein the light field distribution of the i-th dynamic scattering medium phase plate is expressed as: O i (w)=diag(p i (w i ))HO i-1 (w); Where diag(·) is a function used to construct a diagonal matrix, w is the difference between the particle in the dynamic scattering medium and the constant background refractive index, expressed as w = n-n0, and H is the diffraction factor of the light field distribution generated by the free diffraction propagation, H = F -1 diag(h)F,F -1 and F are the two-dimensional inverse Fourier transform and the two-dimensional Fourier transform respectively, h is the abbreviation of h(p,q), p and q are the frequencies in the diffraction propagation directions x and y between the dynamic scattering medium phase plates, Δz is the distance between the two dynamic scattering medium phase plates, k is the wave number of the background medium, k = k0n0, n0 is the constant background refractive index, λ is the wavelength of the incident light, Δk is the sampling interval in Fourier space, and p i (w i ) is the refraction propagation factor, w i is the i-th dynamic scattering medium phase plate; The degraded image output after passing through all dynamic scattering medium phase plates is expressed as: I=|O(W) 2 |=Aexp(φ); Among them, O(W) is the light field distribution after passing through the last dynamic scattering medium phase plate, A and φ represent the light field amplitude information and phase information; The transmittance of the dynamic scattering medium phase plate is expressed as: T D (x,y)=exp(φ)。 4. The scattering imaging method based on optical diffraction neural network according to any one of claims 1 to 3, characterized in that: The training process of the optical diffraction neural network includes: Acquire a training set; wherein the training set includes multiple types of images, all of which are used as sample images; wherein the multiple types of images include digital images and face images; Constructing a parallel model; wherein the parallel model includes at least three branches, each branch includes a preset scattering medium phase plate connected in sequence, and an optical diffraction neural network as one of the channels of the parallel model; In any round of training, for each branch, the incident light selected by the branch is transmitted through the sample image of the branch and then input into the preset scattering medium phase plate of the branch, and the degraded image outputted by the branch is input into the optical diffraction neural network of the branch, received by the input layer, modulated by the diffraction layers in each diffraction structure layer, and outputted by the output layer to the lens for reception, until the predicted restored image after the modulation of the branch is obtained by using the detector; wherein the sample images of the at least three branches are of different types; Obtaining a value of a total loss function of this round of training according to the predicted restored images output in parallel by the at least three branches, and performing back propagation optimization on parameters of each optical diffraction neural network according to the value of the total loss function; Repeat multiple rounds of training until the model convergence condition is reached, and each trained optical diffraction neural network is obtained.

5. The scattering imaging method based on optical diffraction neural network according to claim 4, characterized in that: Each optical diffraction neural network contains M diffraction structure layers, through the modulation of each diffraction layer, at z = z m The light field u after the mth diffraction structure layer at m (x,y,z m ) is expressed as: Wherein, S is the transmittance of the preset scattering medium phase plate. If the preset static scattering medium phase plate is selected, S is the transmittance of the preset static scattering medium phase plate. D (x, y), if the preset dynamic scattering medium phase plate is selected, S is the transmittance T of the preset dynamic scattering medium phase plate D (x,y); D m (x,y) is the Dropt layer, t m (x,y,z m ) is the transmittance function through the diffraction layer, w(x,y,z) is the transmission kernel following the Rayleigh-Sommerfeld diffraction formula, Δz m is the distance between the two diffraction layers, φ(x,y,z m ) is the phase information in the mth diffraction layer, and the value range of m is [1,M]; r is the propagation distance between two diffraction nodes in accordance with the Rayleigh-Sommerfeld diffraction formula; λ is the wavelength of the incident light.

6. The scattering imaging method based on optical diffraction neural network according to claim 5, characterized in that: The predicted recovery image of any branch is expressed as: o(x,y)=|um*w(x,y,f)*tl(x,y) 2 ; Among them, u m for u m (x,y,z m ) abbreviation; f is the focal length of the lens; t l (x,y) is the phase modulation factor brought by the lens itself, k is the wave number of the background medium.

7. The scattering imaging method based on optical diffraction neural network according to claim 4, characterized in that: The total loss function is obtained by weighted summing the loss functions of the at least three branches and the corresponding weighting coefficients.

8. The scattering imaging method based on optical diffraction neural network according to claim 7, characterized in that: The loss function of each branch is the Pearson correlation coefficient.

9. The scattering imaging method based on optical diffraction neural network according to claim 4, characterized in that: After the incident light passes through the target image, it is input into a preset scattering medium phase plate, and a corresponding degraded image is output; The degraded image is input into a pre-trained optical diffraction neural network, and a restored image of the target image is obtained using a detector placed at the focus, including: For all branches of the trained parallel model, the incident light selected by each branch is transmitted through the target image of the branch and then input into the preset scattering medium phase plate of the branch, and the corresponding degraded image is output. The degraded image is input into the pre-trained optical diffraction neural network of the branch, and the restored image of the target image of the branch is obtained using a detector placed at the focus.

10. A scattering imaging system based on optical diffraction neural network, characterized in that: include: A scattering medium phase plate processing module, used to receive the light wave after the incident light passes through the target image, and input it into a preset scattering medium phase plate, and output a corresponding degraded image; wherein the preset scattering medium phase plate includes a preset static scattering medium phase plate, or a preset dynamic scattering medium phase plate; An optical diffraction neural network restoration module is used to receive the degraded image using a pre-trained optical diffraction neural network, and obtain a restored image of the target image using a detector placed at the focus of the optical diffraction neural network; wherein the optical diffraction neural network includes an input layer, an output layer and a plurality of diffraction structure layers located in the middle, each diffraction structure layer includes a diffraction layer and a droupt layer, and a lens is arranged after the last diffraction structure layer.

Images