Single diffraction intensity image reconstruction method and device based on self-supervised deep learning, equipment and storage medium

Abstract

The invention relates to a single diffraction intensity image reconstruction method and device based on self-supervised learning, equipment and a storage medium. The method comprises the following steps: acquiring a single diffraction measurement diagram, extracting complementary measurement values from the single diffraction measurement diagram by adopting complementary mask downsampling, and constructing a data set for self-supervised training; reconstructing different estimations of the original scene image by using a dual-channel deep neural network based on the complementary measurement values in the training set; constructing a loss function for self-supervised training, minimizing the loss function by using an optimizer, and performing self-supervised training on the deep neural network by using the loss function; and estimating a single diffraction intensity image scene by using the trained deep neural network based on the complementary measurement value in the test set. According to the invention, phases and amplitudes of different types of images can be reconstructed.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a single diffraction intensity image reconstruction method, device, equipment and storage medium based on self-supervised learning. Background Art

[0002] In recent years, the development of deep learning has changed the fields of computational imaging, microscopy, and holographic imaging. Deep learning can not only optimize the image reconstruction algorithm by learning a large amount of data, thereby improving the quality and resolution of holographic images, making the imaging results clearer and more accurate; it can also optimize the reconstruction algorithm to achieve faster image reconstruction and processing, thereby achieving real-time imaging and real-time processing requirements. The introduction of deep learning technology into holographic imaging technology can enhance the performance and function of the imaging system, improve image quality and accuracy, and expand the application and development prospects of holographic imaging technology in various fields.

[0003] In 2018, Rivenson et al. proposed a new framework for holographic image reconstruction based on deep learning in "Phase recovery and holographic image reconstruction using deep learning in neural networks" - Holographic Imaging using Deep learning for Extended Focus (HIDEF). Compared with the holographic phase recovery method of the same period, it can quickly eliminate the artifacts related to double images and self-interference using a holographic intensity map, which can not only improve the quality of image reconstruction, but also speed up the image reconstruction. However, the performance of this deep learning model is highly dependent on the quantity and quality of training data. It may perform well on a specific data set, but may not perform well on holographic images of other types or conditions, indicating that the generalization ability of the model is limited. It may be difficult to migrate from one type of holographic image to another type of holographic image, requiring additional tuning and retraining. This model is sensitive to noise and distortion in the input data, and the stability and robustness of the model may be insufficient when facing different types of interference and noise.

[0004] In 2022, Chen et al. proposed an end-to-end deep neural network called Fourier Image Network (FIN) in "Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization". Compared with the existing deep learning models based on multi-height phase retrieval (MH-PR), FIN showed unprecedented generalization performance and much faster inference speed. However, the above deep learning-based methods all rely on supervised learning models to supervise the optimization process, requiring large-scale, high-quality and diverse training datasets, which in turn require a lot of labor, time and cost to acquire, align and preprocess training, which may lead to inference errors and limited generalization to new types of objects that have never been seen during training.

[0005] Compared with supervised learning, self-supervised learning has many advantages: (1) it does not require a large amount of labeled data, which reduces data requirements and labor costs; (2) it does not need to obtain the true value of the image; (3) it can learn useful representations from the data itself, improving the adaptability and generalization of the algorithm; (4) it can effectively learn when the amount of data is insufficient, and has good scalability. Because of these advantages of self-supervised deep learning, it solves the shortcomings of traditional hologram reconstruction, such as low resolution, slow data processing speed, weak processing ability for complex scenes, and high imaging cost. Therefore, it is useful and necessary to introduce self-supervised deep learning in hologram reconstruction.

[0006] In 2023, Huang Lei et al. proposed a self-supervised learning method for hologram reconstruction based on physical consistency. This method proposed a self-supervised learning model called GedankenNet, which eliminates the need for labeled or experimental training data. In the absence of prior knowledge of relevant samples, physical consistency loss and artificial random images are used to train the self-supervised learning model, which is effective and has superior generalization in hologram reconstruction tasks. However, the problems are: first, the input holograms must be two or more, which limits the input distance; second, the situation when the training image is damaged is not considered; finally, the construction of the loss function still has room for optimization.

[0007] Therefore, it is necessary to implement a single diffraction intensity image reconstruction method, device, equipment and storage medium based on self-supervised learning. Summary of the invention

[0008] In order to address the deficiencies in the prior art, the object of the present invention is to provide a single diffraction intensity image reconstruction method, device, equipment and storage medium based on self-supervised learning.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect of the present invention, a single diffraction intensity image reconstruction method based on self-supervised learning is disclosed, the method comprising the following steps:

[0011] S1. Obtain a single diffraction measurement image, extract complementary measurement values ​​from the single diffraction measurement image by using complementary mask down sampling, and construct a data set for self-supervised training; the data set includes a training set and a test set.

[0012] S2. Based on the complementary measurements in the training set, a dual-channel deep neural network is used to reconstruct different estimates of the original scene image.

[0013] S3. Construct a loss function for self-supervised training, use an optimizer to minimize the loss function, and use the loss function to perform self-supervised training on the deep neural network.

[0014] S4. Based on the complementary measurements in the test set, the trained deep neural network is used to estimate the single diffraction intensity image scene.

[0015] Preferably, according to the present invention, step S1, obtaining a single diffraction measurement image, extracting complementary measurement values ​​from the single diffraction measurement image by using complementary mask down sampling, and constructing a data set for self-supervised training, comprises:

[0016] S11. Use formula (1) to obtain the complex-valued function of the light field O on the z=0 plane:

[0017] U obj (x,y;z=0)=A(x,y)exp(jφ(x,y)) (1)

[0018] In formula (1), U obj (x, y; z = 0) represents the light field distribution at z = 0, which is a complex function that describes the amplitude and phase information of the light wave on the plane z = 0; A(x, y) is the amplitude distribution, which represents the amplitude or intensity of the light wave on the (x, y) plane, and is a real-valued function that is usually used to describe the brightness distribution of the light wave; exp(jφ(x, y)) is a complex exponential function that represents the phase information of the light wave on the (x, y) plane, where j is an imaginary unit and φ(x, y) is the phase distribution, which is a real-valued function that describes the phase change of the light wave;

[0019] S12, based on the complex-valued function of the light field O, using formula (2) to obtain the incident coherent light wave U of the object with a wavelength of λinc (x,y;z s ) after irradiation at a distance z from the object s The image on the plane U(x,y;z=z s ):

[0020]

[0021] In formula (2), U(x, y; z = z s ) means z=z s The light field distribution at the distance z s The light field after -1 is the inverse Fourier transform operator, which converts the function in the frequency domain back to the spatial domain; P(v,λ,z s ) is the propagation function, i.e., the propagation phase factor, which is a function in the frequency domain and describes the phase change of the light wave during propagation. In this case, v represents the frequency variable, λ represents the wavelength, and z s Indicates the propagation distance; is the Fourier transform operator, which transforms the function in the spatial domain into the frequency domain; U inc (x, y; 0) is the distribution of the incident light field at z = 0, indicating the amplitude and phase information of the light wave on the initial plane z = 0; U obj (x, y; 0) is the distribution of the object light field at z = 0, indicating the amplitude and phase information of the light wave on the object plane z = 0;

[0022] in,

[0023]

[0024] In formula (3), P(v,λ,z) represents the propagation function, which describes the phase change of the light wave during propagation; is the first half of the propagation phase factor, where z is the propagation distance, indicating the distance the light wave propagates, and λ is the wavelength of the light wave. represents the second half of the propagation phase factor, v x represents the contribution of the frequency component in the x direction, v y represents the contribution of the frequency component in the y direction;

[0025] S13, using the detector to obtain the diffraction image of the object imaging, and using formula (4) to obtain the intensity of the diffraction image, to obtain a single diffraction measurement image:

[0026] I(x,y;z=0)=|U(x,y;z=0)| 2 ≡P λ,z U0:I0 (4)

[0027] In formula (4), I(x, y; z = 0) represents the light intensity distribution at z = 0. The light intensity is the square of the light field amplitude, which represents the energy density of the light wave on the plane z = 0. |U(x, y; z = 0)| 2 is the square of the modulus of the light field U(x,y;z=0); U(x,y;z=0) is the light field on the z=0 plane, |U| represents the amplitude of the light field, and the square represents the light intensity; P λ,z represents a proportional factor related to the wavelength λ and the propagation distance z, which represents a constant or function used to normalize the intensity of the light field; U0 is the amplitude of the initial light field or a reference amplitude, which represents the light field before modulation or propagation; I0 is the hologram captured by the detector D;

[0028] S14, using a pair of complementary masks to sample a single diffraction measurement pattern, that is, sampling the hologram I0, to obtain complementary measurement values;

[0029] S15. Based on the complementary measurement values, construct a data set for self-supervised training, wherein the data set includes a training set and a test set.

[0030] Preferably according to the present invention, the pair of complementary masks comprises a sampling mask SM1 and a sampling mask SM2;

[0031] The holographic image acquired by the detector is sampled using two sampling branches, and one branch is downsampled through the sampling mask SM1 to obtain the measured value. The other branch downsamples through the sampling mask SM2 which is complementary to the sampling mask SM1 to obtain the measured value

[0032] A pair of complementary masks satisfies the following conditions: The intersection of SM1 and SM2 is And the union of SM1 and SM2 is the complete set.

[0033] Preferably, according to the present invention, step S2, based on complementary measurements in the training set, using a dual-channel deep neural network to reconstruct different estimates of the original scene image, comprises:

[0034] S21, constructing a deep neural network based on the SSDL-CS framework; the deep neural network includes two parallel SSDL-CS;

[0035] S22, input the two complementary measurements in the training set into two parallel SSDL-CS, and reconstruct the hologram using formula (5) to obtain different estimates of the original scene image:

[0036]

[0037] In formula (5), is the optimal solution or estimated light field, the light field obtained through the optimization process, which represents the best light field distribution under given conditions; It means optimizing the light field O to minimize the objective function; is the loss function, measuring the processed light field The difference between the hologram I0 captured by the detector D; R(O) is a regularization term, which is used to introduce constraints or prior knowledge on the light field O to avoid overfitting or maintain specific characteristics; the regularization term is used to maintain the smoothness, sparsity or other desired characteristics of the light field;

[0038] Among them, thin samples are under coherent illumination. Simplified to

[0039]

[0040] In formula (6), is the free space transformation matrix, For the light field, is the random detection noise, and f(·) is the sampling function of the photosensor array that records the light field intensity.

[0041] Preferably, in step S21, a deep neural network based on the SSDL-CS framework is constructed, including:

[0042] S211, determine the architecture of the SSDL-CS framework;

[0043] The head and tail of the SSDL-CS framework contain two 1×1 convolutional layers respectively, and the middle is composed of several SPAF groups and a large-scale residual connection; each SPAF group contains two recursive SPAF modules, the two SPAF groups share the same parameters, there is a short jump connection between each SPAF group, thereby forming a medium-scale residual connection, and there is a small-scale residual connection between the input and output of each SPAF module;

[0044] S212, using a two-dimensional discrete Fourier transform to transform the tensor of the SPAF group into the frequency domain, and using formula (7) to perform a linear transformation on the data transformed into the frequency domain, using a window of size k / 2 to truncate the high-frequency signal:

[0045]

[0046] In formula (7), Represented by the weight matrix W i,j,u,v The frequency domain data F of different categories or channels are i.u,v Weighted summation is performed to obtain the processed frequency domain data F′ j.u,vThis is a typical linear transformation operation, which uses a window of size k / 2 to cut off high-frequency signals and is widely used in image processing, signal processing and other fields. It represents the truncated frequency domain after the SFAP module input is subjected to the two-dimensional discrete Fourier transform. represents the trainable weight, c is the number of channels, and the window size is k / 2;

[0047] S213, using a two-dimensional discrete Fourier inverse transform to obtain processed data in the spatial domain, and using the parameterized rectified linear unit activation function shown in formula (8):

[0048]

[0049] In formula (8), x is the input value of the activation function, and a is a learnable parameter;

[0050] S214. Optimize the linear transformation using formula (9):

[0051]

[0052] In formula (9), is the truncated frequency component, are trainable weights.

[0053] Step S211 defines the architecture of the SSDL-CS framework and provides a foundation for data flow and computational processes. Step S212 processes the SPAF group in the frequency domain to prepare for the next step of spatial domain processing. Step S213 converts the frequency domain data back to the spatial domain and applies an activation function to achieve nonlinear transformation. Step S214 optimizes the linear transformation and improves the weights in the processing process to further improve system performance. These steps together constitute the complete processing flow of the SSDL-CS framework, from network architecture design to frequency domain processing, to spatial domain activation and optimization, each step provides support for the final light field reconstruction or signal processing.

[0054] Preferably, according to the present invention, step S3, constructing a loss function for self-supervised training, minimizing the loss function using an optimizer, and performing self-supervised training on the deep neural network using the loss function, comprises:

[0055] S31. Construct a loss function for self-supervised training and minimize the loss function using an optimizer;

[0056] The loss function L(·) is composed of Loss1, Loss2 and Loss Different-loss It consists of three parts, as shown in formula (11):

[0057] L(·)=Loss1+Loss2+Loss Different-loss(11)

[0058] In formula (11), Loss1 is the predicted value The loss between the hologram I0 and the hologram I0 is the weighted sum of the mean square error loss MSE, the Fourier domain mean absolute error loss FDMAE and the total variation loss:

[0059]

[0060] In formula (12), Loss1 is the predicted value and the loss between hologram I0, is the predicted value The mean absolute error loss in the Fourier domain between the hologram I0 and the hologram I1 is For the predicted value and the mean square error loss between the hologram I0, α and β are the loss weight values ​​respectively;

[0061]

[0062] In formula (13), N is the total number of pixels, Represents the predicted value Perform Fourier transform, represents the Fourier transform of the hologram I0;

[0063]

[0064] In formula (14), I0 is the hologram, is the measured value The predicted value of, N is the total number of pixels;

[0065] In formula (11), Loss2 is the predicted value and the loss between hologram I0, expressed as:

[0066]

[0067] In formula (15), Loss2 is the predicted value and the loss between hologram I0, L FDMAE ( I0) is the predicted value The Fourier domain mean absolute error loss between the hologram I0 and the hologram I1 is L MSE ( I0) is the predicted value and the mean square error loss between the hologram I0, α and β are the loss weight values;

[0068]

[0069] LossDifferent-loss is the loss between the predicted value and the predicted value, expressed as:

[0070]

[0071] In formula (18), Loss Different-loss is the predicted value and predicted values The loss between FDMAE is the predicted value and predicted values The Fourier domain mean absolute error loss between MSE For the predicted value and predicted values The mean square error loss between α' and β' is the loss weight value, and the calculation method is as follows:

[0072]

[0073] In formula (19) and formula (20), N is the total number of pixels, Represents the predicted value Perform Fourier transform, Represents the predicted value Perform Fourier transform;

[0074] S32. Use the loss function to train the deep neural network through a self-supervised deep learning method.

[0075] In a second aspect of the present invention, a single diffraction intensity image reconstruction device based on self-supervised learning is disclosed. The device comprises a measurement value extraction module, an original scene image reconstruction module, a deep neural network training module and a single diffraction intensity image scene estimation module.

[0076] The measurement value extraction module is used to obtain a single diffraction measurement image, extract complementary measurement values ​​from the single diffraction measurement image by complementary mask down sampling, and construct a data set for self-supervised training; the data set includes a training set and a test set.

[0077] The original scene image reconstruction module is used to reconstruct different estimates of the original scene image based on complementary measurements in the training set using a dual-channel deep neural network.

[0078] The deep neural network training module is used to construct a loss function for self-supervised training, minimize the loss function using an optimizer, and perform self-supervised training on the deep neural network using the loss function.

[0079] The single diffraction intensity image scene estimation module is used to estimate the single diffraction intensity image scene based on the complementary measurement values ​​in the test set using the trained deep neural network.

[0080] In the third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the above-mentioned single diffraction intensity image reconstruction method based on self-supervised learning.

[0081] In a fourth aspect of the present invention, a machine-readable storage medium is disclosed, which stores executable instructions, and when the instructions are executed, the machine performs the above-mentioned single diffraction intensity image reconstruction method based on self-supervised learning.

[0082] Compared with the prior art, the advantages of the present invention are:

[0083] In view of the above problems, the present invention proposes a complementary sampling self-supervised deep learning reconstruction framework based on a single diffraction intensity image. The framework first starts from a single diffraction measurement image and uses a complementary down-sampling mask to extract a pair of complementary measurement values ​​for constructing a data set for self-supervised training. The single diffraction intensity image is used as the input of the deep neural network, which can eliminate the distance limitation; secondly, starting from the complementary training data set, a dual-channel deep neural network is used to reconstruct different estimates of the original scene image from the complementary under-sampled measurement values; then, by constructing a loss function for self-supervised training, it mainly includes two parts, one is the error between the estimated scene and the original scene, and the other is the error between two different estimated scenes; by introducing a new differential loss between the predicted values, the solution space can be effectively constrained and the reconstruction performance can be improved; finally, when training the neural network, a complementary sampling mask method is proposed, using two masks without intersection to divide the measurement values ​​into two non-intersecting data sets as input, and the loss function between the training input and the predicted value is used to optimize the reconstruction quality of the network. The parameters of the deep neural network are trained by minimizing the loss function through the optimizer, and the scene is estimated from the measurement values ​​through the trained DNN network. After self-supervised training, the complementary sampling self-supervised deep learning reconstruction framework based on single diffraction intensity images can be successfully extended to most experimental holograms to reconstruct the phase and amplitude of different types of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a method flow chart of a single diffraction intensity image reconstruction method based on self-supervised learning in the present invention;

[0085] Figure 2 It is a schematic diagram of constructing a complementary self-supervised training data set by complementary downsampling masks in the present invention;

[0086] Figure 3 is a schematic diagram of a single diffraction intensity image reconstruction method based on self-supervised learning in the present invention;

[0087] Figure 4 It is a schematic diagram of the network architecture of the deep neural network in the present invention. DETAILED DESCRIPTION

[0088] The present disclosure is further described below with reference to the accompanying drawings and embodiments:

[0089] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0090] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0091] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0092] Example 1

[0093] This embodiment provides a single diffraction intensity image reconstruction method based on self-supervised learning, such as Figure 1 As shown, the method comprises the following steps:

[0094] S1. Starting from a single diffraction measurement image, a pair of complementary measurement values ​​are extracted using a complementary downsampling mask to construct a dataset for self-supervised training.

[0095] Usually, the light field O on the z = 0 plane can be represented by a complex-valued function:

[0096] U obj (x,y;z=0)=A(x,y)exp(jφ(x,y)) (1)

[0097] Where A and φ are the amplitude and phase of the light field. Consider the incident coherent light wavefront as U inc (x,y;z s ), with a wavelength of λ, at a distance of z from the object s On the plane, we can use U(x,y; z=z s ) to indicate:

[0098] U(x,y;z=z s )=F -1 {P(v,λ,z s )·F(U inc (x,y;0)U obj (x,y;0))} (2)

[0099] Among them, F -1 and F are the inverse Fourier transform operator and the Fourier transform operator. P(.) is the angular spectrum transfer function, which depends on the spatial frequency component v = (ν x ,ν y ), wavelength λ and propagation distance z. P(v,λ,z) can be expressed as:

[0100]

[0101] Now assume that the detector plane only includes the diffraction pattern of the object, and consider that U in equation (2) inc (x,y;0)U obj (x, y; 0) = U0, then the intensity of the measured image can be:

[0102] I(x,y;z=0)=|U(x,y;z=0)| 2 ≡P λ,z U0:I0 (4)

[0103] Among them, P λ,z is the propagation operator, and I0 is the hologram captured by detector D.

[0104] Figure 2 Schematic diagram of the principle of constructing a complementary self-supervised training data set through complementary downsampling masks in the present invention. Figure 2 In the example, O is a two-dimensional light field, z s is the distance from the light field to the detector, and I0 is the hologram captured by the detector D after the light field O propagates through the angular spectrum. Mask SM1 and mask SM2 are a pair of complementary masks, and the measured value is I0 is sampled through mask SM1, the measured value It is obtained by sampling I0 through mask SM2.

[0105] S2. Starting from the complementary training dataset, a two-channel deep neural network is used to reconstruct different estimates of the original scene image from the complementary undersampled measurements.

[0106] In general, the hologram reconstruction task can be formulated as an inverse problem:

[0107]

[0108] In formula (5), Represents a vectorized hologram, the dimension of the hologram is N×N, For the light field. is the forward imaging model, L(·) is the loss function, and R(·) is the regularization term. When a thin sample is under coherent illumination, Can be simplified to

[0109]

[0110] In formula (6), is the free space transformation matrix, is the random detection noise and f(·) is the sampling function of the (photoelectric) sensor array that records the light field intensity.

[0111] In order to optimize the above problems, the present invention proposes a SSDL-CS framework, constructs a training data set with two complementary measurements, inputs them into two parallel SSDL-CS, and designs a special loss function that fits the physical constraints for training deep convolutional neural networks through self-supervised deep learning methods. The network used for reconstruction is SSDL-CS, which contains two 1×1 convolutional layers at the head and tail, respectively, and a series of spatial Fourier transform (SPAF) groups and a large-scale residual connection in the middle. The network architecture is as follows: Figure 3 As shown. Figure 3 In the example, O is a two-dimensional light field, z s is the distance from the light field to the detector, and I0 is the hologram captured by the detector D after the light field O propagates through the angular spectrum. Mask SM1 and mask SM2 are a pair of complementary masks, and the measured value is I0 is sampled through mask SM1, the measured value It is obtained by sampling I0 through mask SM2. For the measured value The output obtained through the network is Output The predicted value obtained by angular spectrum propagation, For the measured value The output obtained through the network is Output The predicted value obtained by angular spectrum propagation. ASP (Angular Spectrum Propagation) stands for "Angular Spectrum Propagation". This is a numerical method used to simulate the propagation of light waves in space. ASM (Angular Spectrum Method) stands for "Angular Spectrum Method". Angular Spectrum Propagation is a numerical calculation method used to simulate the propagation of light waves in space. Loss1 is the predicted value and the loss between hologram I0, Loss2 is the predicted value and the loss between hologram I0,Loss Different-loss is the loss between the predicted value and the predicted value.

[0112] Each SPAF group contains two recursive SPAF modules, and the two SPAF modules share the same parameters, which can improve the network capacity without significantly expanding the network size. There is a short jump connection between each SPAF group, thus forming a medium-scale residual connection, and there is a small-scale residual connection between the input and output of each SPAF module. Figure 3 Schematic diagram of the SPAF module. After transforming the tensor to the frequency domain using the two-dimensional discrete Fourier transform, it is linearly transformed and then the high-frequency signal is truncated with a window of size k / 2, that is,

[0113]

[0114] in, It is represented as the truncated frequency domain after the SFAP module input is subjected to the two-dimensional discrete Fourier transform. represents the trainable weights, c is the number of channels, and the window size is k / 2. After this linear transformation, the processed data is obtained in the spatial domain using a two-dimensional discrete Fourier inverse transform, and then the parametric rectified linear unit (PReLU) activation function is used.

[0115]

[0116] Among them, a is a learnable parameter.

[0117] PreLU (Parametric Rectified Linear Unit) is an activation function commonly used in deep learning. It is an improvement on ReLU (Rectified Linear Unit), and its main purpose is to solve the "dead ReLU" problem that may occur during the training process of ReLU, that is, some neurons may never be activated during the training process (the output is always 0). PreLU is mainly used in deep neural networks, especially when training deep networks, which can significantly improve the training effect and model performance. It is widely used in different types of neural networks such as convolutional neural networks (CNN) and recurrent neural networks (RNN). When the input x is greater than or equal to 0, PreLU outputs x itself. When the input is negative, the output is ax, where a is a learnable parameter, usually a positive number. This allows PreLU to have a non-zero slope when the input x is negative.

[0118] The present invention adopts the PreLU activation function, which has the following characteristics:

[0119] (1) Flexibility: By introducing the learning parameter a, the PreLU activation function allows different slopes on the negative half-axis, thereby increasing the nonlinear expression ability of the model.

[0120] (2) Avoiding “dead neurons”: Unlike the ReLU function, PreLU allows gradients in negative regions, which helps alleviate the “dead neuron” problem common in the ReLU function, that is, when the input is negative, the output of ReLU is zero, resulting in the inability to update neurons.

[0121] (3) Adaptability: The parameter a is learnable, so the network can adaptively adjust the slope of the negative region during training. PreLU is a common activation function, especially in deep convolutional neural networks (CNNs), which helps improve model performance and training efficiency.

[0122] In order to adapt the SPAF module to high-resolution image processing in deeper networks, we shrink the matrix W, which significantly reduces the model size. The optimized linear transformation is defined as:

[0123]

[0124] in, is the truncated frequency component. are trainable weights.

[0125] Specifically, the two SPAF modules in each SPAF group have shared hyperparameters, but a pyramid-like setting is applied so that the k value in the deeper SPAF module decreases. This pyramid-like structure maps the high-frequency information of the holographic diffraction pattern to the low-frequency region of the first few layers and passes the low-frequency information to the subsequent layers with a smaller window size, thereby better utilizing the multi-scale features while greatly reducing the model size and avoiding potential overfitting and generalization problems.

[0126] In order to obtain two complementary measurements in self-supervised deep learning, two complementary masks are used to downsample the hologram, one of which is sampled by mask SM1 to obtain the measurement value The other is sampled by mask SM2 which is complementary to SM1. The two complementary measurements are fed into two parallel networks SSDL-CS (complementary single layer) for self-supervised deep learning training. During the network optimization process, the predicted values ​​are used. and the loss between hologram I0 and Loss1; 2) predicted value and the loss between the hologram I0 and the predicted value, Loss2; 3) the loss between the predicted value and the predicted value, Loss Different-lossThe reconstruction loss is calculated, and the final reconstruction stage can achieve real-time reconstruction of the hologram through SSDL-CS.

[0127] Figure 4 The overall process of the complementary sampling self-supervised deep learning reconstruction framework for single diffraction intensity images proposed in this invention is given. Figure 4 In the above description, conv stands for Convolution, which represents the convolution operation in mathematics. SPAF group stands for Spatial Fourier Transform, which represents spatial Fourier transform. Residual connection stands for residual connection. Short skip connection stands for short jump connection. FT stands for Fourier transform, and IFT stands for inverse Fourier transform. The network architecture is existing, but the present invention proposes a new architecture SSDL-CS. The full name of the SSDL-CS framework is Self-Supervised Deep Learning Reconstruction Framework Using Single Diffraction Intensity Image and Complementary Sampling, which represents the self-supervised deep learning reconstruction framework of single diffraction intensity image and complementary sampling.

[0128] like Figure 4 As shown, the entire architecture works as follows:

[0129] (1) Data input: The input data first passes through two 1×1 convolutional layers at the head to adjust the number of channels and perform preliminary processing.

[0130] (2) SPAF group processing: After several SPAF groups, each SPAF group contains two recursive SPAF modules. There are medium-scale residual connections between SPAF groups and small-scale residual connections within each module.

[0131] (3) Frequency domain processing: In step S212, the data is converted to the frequency domain using a two-dimensional discrete Fourier transform, and a linear transformation and high-frequency signal truncation are performed. In step S213, the data is converted back to the spatial domain using a two-dimensional inverse discrete Fourier transform, and a PreLU activation function is applied for nonlinear processing.

[0132] (4) Optimization and output: The linear transformation is optimized in step S214 and the weights are further adjusted. Finally, the data passes through the two 1×1 convolutional layers at the end and the processing results are output.

[0133] The SSDL-CS framework proposed in this paper combines the processing methods in the spatial domain and the frequency domain, extracting and processing features through the recursive SPAF module and residual connection. The two-dimensional discrete Fourier transform and PreLU activation function are used for processing and nonlinear transformation in the frequency domain and the spatial domain, and the performance is further improved through optimization steps. The entire architecture is designed to efficiently extract and process image or signal features and realize the combination of compressed sensing and deep learning.

[0134] S3. By constructing a loss function for self-supervised training, it mainly includes two parts: one is the error between the estimated scene and the original scene, and the other is the difference error (Difference Loss) between two different estimated scenes.

[0135] The general supervised deep learning method has only one reconstruction network. After training on a large number of training data sets, a deep convolutional neural network is obtained. After the training process is completed, it can be directly used to reconstruct the original image from the measured values. However, the shortcomings of supervised learning are as follows: 1) A large amount of manually annotated data is required to train the model; 2) The generalization ability of the model depends on the quality and quantity of the annotated data; 3) There is still room for improvement in the construction of the deep neural network architecture and the optimization of the training mechanism.

[0136] The loss function in classic supervised learning is defined as:

[0137]

[0138] However, this paper intends to use a self-supervised deep learning method to construct a training data set through the measurements obtained by sampling under complementary masks. The loss function used for self-supervised deep learning is defined by Loss1, Loss2 and Loss Different-loss It consists of three parts, expressed as:

[0139] L(·)=Loss1+Loss2+Loss Different-loss (11)

[0140] (1) Loss1 is the predicted value The loss between the hologram I0 and the hologram I0 is the mean square error (MSE) loss and the Fourier domain mean absolute error (FDMAE) loss:

[0141] Loss1=αL FDMAE +βL MSE (12)

[0142] Among them, α, β, and γ are loss weight values. They are calculated as follows:

[0143]

[0144] where I0 is the hologram, is the measured value The predicted value, N is the total number of pixels, Stands for two-dimensional discrete Fourier transform.

[0145] (2) Loss2 is the predicted value and the loss between hologram I0, expressed as:

[0146] Loss2=αL FDMAE +βL MSE (15)

[0147] Among them, α, β, γ and δ are loss weight values ​​respectively. They are calculated as follows:

[0148]

[0149] (3) Loss Different-loss The loss between the predicted value and the predicted value can be expressed as:

[0150] Loss Different-loss =α′L′ FDMAE +β′L′ MSE (19)

[0151] Among them, α′, β′ are loss weight values ​​respectively. They are calculated as follows:

[0152]

[0153] S4. The parameters of the deep neural network are trained by minimizing the loss function through the optimizer, and finally the scene is estimated from the measured values ​​through the trained CNN network.

[0154] Example 2

[0155] This embodiment provides a single diffraction intensity image reconstruction device based on self-supervised learning, which includes a measurement value extraction module, an original scene image reconstruction module, a deep neural network training module and a single diffraction intensity image scene estimation module.

[0156] The measurement value extraction module is used to obtain a single diffraction measurement image, extract complementary measurement values ​​from the single diffraction measurement image by complementary mask down sampling, and construct a data set for self-supervised training; the data set includes a training set and a test set.

[0157] The original scene image reconstruction module is used to reconstruct different estimates of the original scene image based on complementary measurements in the training set using a dual-channel deep neural network.

[0158] The deep neural network training module is used to construct a loss function for self-supervised training, minimize the loss function using an optimizer, perform self-supervised training on the deep neural network using the loss function, and train the parameters of the deep neural network.

[0159] The single diffraction intensity image scene estimation module is used to estimate the single diffraction intensity image scene based on the complementary measurement values ​​in the test set using the trained deep neural network.

[0160] Example 3

[0161] This embodiment also provides an electronic device, comprising: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the single diffraction intensity image reconstruction method based on self-supervised learning as described above.

[0162] In this embodiment, the electronic device may include, but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, and the like.

[0163] Example 4

[0164] This embodiment also provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the single diffraction intensity image reconstruction method based on self-supervised learning as described above.

[0165] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0166] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0167] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0168] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] The above-described embodiments are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for reconstructing a single diffraction intensity image scene based on self-supervised learning, characterized in that: The method comprises the following steps: S1, obtaining a single diffraction measurement image, extracting complementary measurement values ​​from the single diffraction measurement image by complementary mask downsampling, and constructing a data set for self-supervised training; the data set includes a training set and a test set; S2, based on the complementary measurements in the training set, using a two-channel deep neural network to reconstruct different estimates of the original scene image; S3. Construct a loss function for self-supervised training, minimize the loss function using an optimizer, and use the loss function to perform self-supervised training on the deep neural network; S4. Based on the complementary measurements in the test set, the trained deep neural network is used to estimate the single diffraction intensity image scene.

2. The method for reconstructing a single diffraction intensity image scene based on self-supervised learning according to claim 1, characterized in that: The step S1, obtaining a single diffraction measurement image, extracting complementary measurement values ​​from the single diffraction measurement image by using complementary mask down sampling, and constructing a data set for self-supervised training, includes: S11. Use formula (1) to obtain the complex-valued function of the light field O on the z=0 plane: U obj (x,y;z=0)=A(x,y)exp(jφ(x,y)) (1) In formula (1), U obj (x, y; z = 0) represents the light field distribution at z = 0, which is a complex function that describes the amplitude and phase information of the light wave on the plane z = 0; A(x, y) is the amplitude distribution, which represents the amplitude or intensity of the light wave on the (x, y) plane, and is a real-valued function used to describe the brightness distribution of the light wave; exp(jφ(x, y)) is a complex exponential function that represents the phase information of the light wave on the (x, y) plane, where j is an imaginary unit and φ(x, y) is the phase distribution, which is a real-valued function that describes the phase change of the light wave; S12, based on the complex-valued function of the light field O, using formula (2) to obtain the incident coherent light wave U of the object with a wavelength of λ inc (x, y; z s ) after irradiation at a distance z from the object s The image U(x, y; z = z s ): In formula (2), U(x, y; z = z s ) means z=z s The light field distribution at the distance z s The light field after is the inverse Fourier transform operator, which converts the function in the frequency domain back to the spatial domain; P(v, λ, z s ) is the propagation function, i.e., the propagation phase factor, which is a function in the frequency domain and describes the phase change of the light wave during propagation. In this case, v represents the frequency variable, λ represents the wavelength, and z s Indicates the propagation distance; is the Fourier transform operator, which transforms the function in the spatial domain into the frequency domain; U inc (x, y; 0) is the distribution of the incident light field at z = 0, indicating the amplitude and phase information of the light wave on the initial plane z = 0; U obj (x, y; 0) is the distribution of the object light field at z = 0, indicating the amplitude and phase information of the light wave on the object plane z = 0; in, In formula (3), P(v, λ, z) represents the propagation function, which describes the phase change of the light wave during propagation; is the first half of the propagation phase factor, where z is the propagation distance, indicating the distance the light wave propagates, and λ is the wavelength of the light wave. represents the second half of the propagation phase factor, v x represents the contribution of the frequency component in the x direction, v y represents the contribution of the frequency component in the y direction; S13, using the detector to obtain the diffraction image of the object imaging, and using formula (4) to obtain the intensity of the diffraction image, to obtain a single diffraction measurement image: I(x,y;z=0)=|U(x,y;z=0)| 2 ≡P λ，z U0:I0 (4) In equation (4), I(x, y; z = 0) represents the light intensity distribution at z = 0. The light intensity is the square of the light field amplitude and represents the energy density of the light wave on the plane z = 0. |U(x, y; z = 0)| 2 is the square of the modulus of the light field U(x, y; z = 0); U(x, y; z = 0) is the light field on the z = 0 plane, |U| represents the amplitude of the light field, and the square represents the light intensity; P λ，z represents a proportional factor related to the wavelength λ and the propagation distance z, which represents a constant or function used to normalize the intensity of the light field; U0 is the amplitude of the initial light field or a reference amplitude, which represents the light field before modulation or propagation; I0 is the hologram captured by the detector D; S14, using a pair of complementary masks to sample a single diffraction measurement pattern, that is, sampling the hologram I0, to obtain complementary measurement values; S15. Based on the complementary measurement values, construct a data set for self-supervised training, wherein the data set includes a training set and a test set.

3. The method for reconstructing a single diffraction intensity image scene based on self-supervised learning according to claim 2, characterized in that: The pair of complementary masks includes a sampling mask SM1 and a sampling mask SM2; The holographic image acquired by the detector is sampled using two sampling branches, and one branch is downsampled through the sampling mask SM1 to obtain the measured value. The other branch downsamples through the sampling mask SM2 which is complementary to the sampling mask SM1 to obtain the measured value A pair of complementary masks satisfies the following conditions: The intersection of SM1 and SM2 is And the union of SM1 and SM2 is the complete set.

4. The method for reconstructing a single diffraction intensity image scene based on self-supervised learning according to claim 3, characterized in that: The step S2, based on the complementary measurements in the training set, uses a dual-channel deep neural network to reconstruct different estimates of the original scene image, including: S21, constructing a deep neural network based on the SSDL-CS framework; the deep neural network includes two parallel SSDL-CS; S22, input the two complementary measurements in the training set into two parallel SSDL-CS, and reconstruct the hologram using formula (5) to obtain different estimates of the original scene image: In formula (5), is the optimal solution or estimated light field, the light field obtained through the optimization process, which represents the best light field distribution under given conditions; It means optimizing the light field O to minimize the objective function; is the loss function, measuring the processed light field The difference between the hologram I0 captured by the detector D; R(O) is a regularization term, which is used to introduce constraints or prior knowledge on the light field O to avoid overfitting or maintain specific characteristics; the regularization term is used to maintain the smoothness, sparsity or other desired characteristics of the light field; Among them, thin samples are under coherent illumination. Simplified to In formula (6), is the free space transformation matrix, For the light field, is the random detection noise, and f(·) is the sampling function of the photosensor array that records the light field intensity.

5. The method for reconstructing a single diffraction intensity image scene based on self-supervised learning according to claim 4, characterized in that: In step S21, a deep neural network based on the SSDL-CS framework is constructed, including: S211, determine the architecture of the SSDL-CS framework; The head and tail of the SSDL-CS framework contain two 1×1 convolutional layers respectively, and the middle is composed of several SPAF groups and a large-scale residual connection; each SPAF group contains two recursive SPAF modules, the two SPAF groups share the same parameters, there is a short jump connection between each SPAF group, thereby forming a medium-scale residual connection, and there is a small-scale residual connection between the input and output of each SPAF module; S212, using a two-dimensional discrete Fourier transform to transform the tensor of the SPAF group into the frequency domain, and using formula (7) to perform a linear transformation on the data transformed into the frequency domain, using a window of size k / 2 to truncate the high-frequency signal: In formula (7), Represented by the weight matrix W i,j,u,v The frequency domain data F of different categories or channels are i.u,v Weighted summation is performed to obtain the processed frequency domain data F′ j.u,v ; It represents the truncated frequency domain after the SFAP module input is subjected to the two-dimensional discrete Fourier transform. represents the trainable weight, c is the number of channels, and the window size is k / 2; S213, using a two-dimensional discrete Fourier inverse transform to obtain processed data in the spatial domain, and using the parameterized rectified linear unit activation function shown in formula (8): In formula (8), x is the input value of the activation function, and a is a learnable parameter; S214. Optimize the linear transformation using formula (9): In formula (9), is the truncated frequency component, are trainable weights.

6. The method for reconstructing a single diffraction intensity image scene based on self-supervised learning according to claim 5, characterized in that: The step S3, constructing a loss function for self-supervised training, minimizing the loss function using an optimizer, and performing self-supervised training on the deep neural network using the loss function, includes: S31. Construct a loss function for self-supervised training and minimize the loss function using an optimizer; The loss function L(·) is composed of Loss1, Loss2 and Loss Different-loss It consists of three parts, as shown in formula (11): L(·)=Loss1+Loss2+Loss Different-loss (11) In formula (11), Loss1 is the predicted value The loss between the hologram I0 and the hologram I0 is the weighted sum of the mean square error loss MSE, the Fourier domain mean absolute error loss FDMAE and the total variation loss: In formula (12), Loss1 is the predicted value and the loss between hologram I0, is the predicted value The mean absolute error loss in the Fourier domain between the hologram I0 and the hologram I1 is For the predicted value and the mean square error loss between the hologram I0, α and β are the loss weight values ​​respectively; In formula (13), N is the total number of pixels, Represents the predicted value Perform Fourier transform, represents the Fourier transform of the hologram I0; In formula (14), I0 is the hologram, is the measured value The predicted value of, N is the total number of pixels; In formula (11), Loss2 is the predicted value and the loss between hologram I0, expressed as: In formula (15), Loss2 is the predicted value and the loss between hologram I0, is the predicted value The Fourier domain mean absolute error loss between the hologram I0 and the hologram I1 is For the predicted value and the mean square error loss between the hologram I0, α and β are the loss weight values; Loss Different-loss is the loss between the predicted value and the predicted value, expressed as: In formula (18), Loss Different-loss is the predicted value and predicted values The loss between FDMAE is the predicted value and predicted values The Fourier domain mean absolute error loss between For the predicted value and predicted values The mean square error loss between α' and β' is the loss weight value, and the calculation method is as follows: In formula (19) and formula (20), N is the total number of pixels, Represents the predicted value Perform Fourier transform, Represents the predicted value Perform Fourier transform; S32. Use the loss function to train the deep neural network through a self-supervised deep learning method.

7. A single diffraction intensity image reconstruction device based on self-supervised learning, characterized in that: The device includes a measurement value extraction module, an original scene image reconstruction module, a deep neural network training module and a single diffraction intensity image scene estimation module; The measurement value extraction module is used to obtain a single diffraction measurement image, extract complementary measurement values ​​from the single diffraction measurement image by using complementary mask down sampling, and construct a data set for self-supervised training; the data set includes a training set and a test set; The original scene image reconstruction module is used to reconstruct different estimates of the original scene image based on the complementary measurements in the training set using a dual-channel deep neural network; The deep neural network training module is used to construct a loss function for self-supervised training, minimize the loss function using an optimizer, perform self-supervised training on the deep neural network using the loss function, and train the parameters of the deep neural network; The single diffraction intensity image scene estimation module is used to estimate the single diffraction intensity image scene based on the complementary measurement values ​​in the test set using the trained deep neural network.

8. An electronic device, characterized in that: include: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to perform the single diffraction intensity image reconstruction method based on self-supervised learning as described in any one of claims 1 to 7.

9. A machine-readable storage medium, characterized in that: The machine stores executable instructions, which, when executed, enable the machine to perform the single diffraction intensity image reconstruction method based on self-supervised learning as claimed in any one of claims 1 to 7.

Images