Particle field reconstruction model training method, particle field reconstruction method, equipment and medium
By integrating the physical constraints of light field microscopy imaging into the deep learning model and optimizing network parameters, the problems of insufficient generalization ability and limited accuracy of three-dimensional particle field reconstruction are solved, and higher-quality particle field reconstruction is achieved.
Patent Information
- Application Number
- CN202510459868.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-14
- Publication Date
- 2025-07-25
AI Technical Summary
The existing three-dimensional particle field reconstruction method based on deep learning has insufficient generalization capabilities and limited reconstruction accuracy, so it cannot adapt to the reduction in reconstruction quality when the microscope system's numerical aperture, particle concentration or imaging noise deviates from the training set distribution.
By integrating the physical constraints of the light field microscopy imaging model into the deep learning process, convolutional neural networks are used for pre-training and fine-tuning, combined with light field imaging calculations, network parameters are optimized to improve reconstruction quality.
Effectively eliminate artifacts, inhibit particle reconstruction tensile effect, improve the quality of three-dimensional particle field reconstruction, and improve network generalization performance and interpretability.
Smart Images
Figure CN120374852A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of the integration of light field microscopy particle image velocimetry and deep learning, and particularly relates to a deep learning reconstruction model of a light field microscopic particle field incorporating physical constraints, which is used to improve the generalization ability and accuracy of three-dimensional particle field reconstruction. Background Technique
[0002] The light field microscopy particle image velocimetry (light field Micro-PIV) technology is the core means for three-dimensional microscale flow field measurement. It collects four-dimensional light field images containing position and angle information through a light field imaging system, reconstructs the three-dimensional spatial distribution of tracer particles, and calculates the velocity field based on the cross-correlation algorithm to achieve high-precision measurement of the three-dimensional velocity field of the microscale flow field. In this technical framework, the accuracy and efficiency of particle three-dimensional reconstruction directly affect the reliability of the final velocity field measurement.
[0003] Currently, the three-dimensional particle field reconstruction method based on a deep learning model has become a cutting-edge technology in this field. Compared with the traditional deconvolution reconstruction algorithm, the deep learning method using a convolutional neural network (CNN) has improved in both the axial resolution and reconstruction speed of particle reconstruction, and can achieve fast and accurate reconstruction of the three-dimensional particle field. However, such a pure data-driven deep learning model has an inherent defect of insufficient generalization ability. The network highly depends on the completeness of the training data and the matching degree of the optical parameters of the test scenario. The model can only learn the apparent features of discrete training samples and cannot extrapolate to unseen parameter intervals. When the numerical aperture of the microscopic system, particle concentration, or imaging noise deviates from the training set distribution, the reconstruction quality significantly deteriorates.
[0004] Integrating physical information into the deep learning network framework provides a new idea for light field microscopic three-dimensional particle field reconstruction. By establishing the coupling relationship between data-driven and imaging physical mechanisms, it can not only fully exploit the feature learning advantages of deep learning, but also make the reconstruction results follow the physical laws of light field microscopy, and is expected to improve the defects of insufficient generalization ability and poor interpretability existing in the conventional light field microscopic deep learning particle field reconstruction. Summary of the Invention
[0005] The present invention proposes a training method, a particle field reconstruction method, a device, and a medium for a light field microscopic three-dimensional particle field reconstruction model incorporating physical information. By integrating the physical constraints of the light field microscopy imaging model into the deep learning process, it solves the problems of insufficient generalization ability and limited reconstruction accuracy existing in the conventional light field microscopic deep learning particle field reconstruction, and improves the particle field reconstruction quality.
[0006] To solve the above technical problems, the technical solution adopted by the present invention is:
[0007] The present invention first provides a training method for a light field microscopic three-dimensional particle field reconstruction model incorporating physical information, including:
[0008] Network model pre-training: The network model is pre-trained using a training data set, and the training data set of the network model is a data sample of particle light intensity distribution - light field image; the input of the network model is the light field image, and the output of the network model is the predicted value of the particle light intensity distribution.
[0009] Network model fine-tuning: Fine-tune the parameters of the network model obtained by pre-training; the fine-tuning method is: perform light field imaging calculation on the predicted value of the particle light intensity distribution output by the network model to obtain the calculated light field image; use the loss function to calculate the deviation between the original light field image and the calculated light field image, and set the number of training times to make the loss function tend to a stable value, and the model converges to obtain the fine-tuned neural network model of the particle light intensity distribution.
[0010] The present invention also provides a method for reconstructing a three-dimensional particle field by light field microscopy, including:
[0011] Obtain a light field image;
[0012] Input the obtained light field image into the three-dimensional particle field reconstruction model, and output the predicted value of the particle light intensity distribution;
[0013] Wherein, the three-dimensional particle field reconstruction model is trained according to the model training method provided above.
[0014] The present invention also provides an electronic device, including:
[0015] A processor; a memory for storing instructions executable by the processor;
[0016] Wherein, the processor is configured to: implement the model training method provided above, or implement the method for reconstructing a three-dimensional particle field by light field microscopy provided above.
[0017] The present invention also provides a computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal can execute the model training method provided above, or implement the method for reconstructing a three-dimensional particle field by light field microscopy provided above.
[0018] Compared with the existing pure data-driven deep learning reconstruction model, the present invention has the following advantages:
[0019] (1) The PIDLR model proposed by the present invention can eliminate the artifact problem that appears in the particle field reconstruction of traditional deep learning models, suppress the stretching effect of particle reconstruction, and effectively improve the quality of three-dimensional particle field reconstruction.
[0020] (2) Incorporate the physical process into the deep learning network framework. By introducing the light field imaging process as a physical constraint, the result can conform to the imaging mechanism model, which can improve the reconstruction performance of the network for unseen samples and enhance the generalization performance and interpretability of the network. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 Light field microscopy three-dimensional particle field reconstruction network structure incorporating physical information;
[0022] Figure 2 Schematic diagram of the pre-trained neural network structure;
[0023] Figure 3 Schematic diagram of the light field imaging convolution calculation process. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0024] Embodiment 1
[0025] This embodiment provides a method for training a light field microscopy three-dimensional particle field reconstruction model incorporating physical information. The overall architecture of the reconstruction model is as Figure 1 shown, including two processes: model pre-training and model fine-tuning. Model pre-training provides a pre-trained neural network structure and initial values for model fine-tuning. During the model fine-tuning process, the pre-trained neural network parameters are fine-tuned in combination with the light field imaging process to make the predicted value of the particle light intensity distribution output closer to the true value.
[0026] The model training method includes:
[0027] Pre-training of the network model: Use the training data set to pre-train the network model. The training data set of the network model is the data sample of particle light intensity distribution - light field image. The input of the network model is the light field image, and the output of the network model is the predicted value of the particle light intensity distribution.
[0028] Fine-tuning of the network model: Fine-tune the parameters of the pre-trained network model. The fine-tuning method is: Perform light field imaging calculation on the predicted value of the particle light intensity distribution output by the network model to obtain the calculated light field image. Use the loss function to calculate the deviation between the original light field image and the calculated light field image, and set the number of training times to make the loss function tend to a stable value, and the model converges to obtain the fine-tuned neural network model of the particle light intensity distribution.
[0029] During the pre-training process of the reconstruction model of the present invention, a convolutional neural network is used as the core architecture. The input of the network is the light field image, and the output is the predicted value of the particle light intensity distribution. The specific process is as Figure 2As shown. The core process covers three links: dataset construction, light field image preprocessing, and network architecture design. First, based on the microchannel parameters, the spatial dimensions of the three-dimensional measurement volume are defined, and the central coordinates of the particles are randomly generated to construct the theoretical three-dimensional particle light intensity distribution. Then, it is convolved with the point spread function (PSF) of the light field microscopy system to generate the light field image. Finally, the "particle light intensity distribution - light field image" data samples are formed as the training dataset. Next, the light field images in the samples are used as inputs, and the pixels with the same angle relative to the center of each microlens in each light field image are extracted and rearranged to generate sub-aperture images. Since the U-Net network model has a simple structure and excellent performance, a convolutional neural network with the U-Net model architecture is constructed based on the actual requirements of three-dimensional distribution reconstruction of light field microscopic particles, and the dataset is used for training. The network upsampling layer performs sub-pixel filling on the sub-aperture images extracted from the light field images to achieve data dimension expansion. The encoder part contains multiple convolutional layers and pooling layers. The convolutional layers extract features from the input data, and the pooling layers reduce the data dimension. The decoder part contains multiple upsampling layers, convolutional layers, and skip connection layers. The upsampling layers are used to restore the image size, and the skip connection layers are used to fuse features to enhance the detail restoration ability. The network establishes the mapping relationship between the light field image and the three-dimensional distribution through supervised learning, and finally outputs the predicted value of the three-dimensional particle light intensity distribution. During the model pre-training process, a loss function is needed to quantify the difference between the predicted value and the true value. When the loss function no longer decreases significantly or tends to be stable in multiple consecutive training epochs, it is considered that the model has converged, and at this time, the neural network of the initial particle light intensity distribution is obtained. The mean square error loss function (Loss1 mse ) is used in the pre-training process:
[0030]
[0031] where, is the true value of the particle light intensity distribution of the i-th voxel in the dataset, is the predicted value of the particle light intensity distribution obtained by reconstructing the light field image of the i-th voxel through the neural network, and p is the number of voxels.
[0032] During the model fine-tuning process, the physical constraints of the light field imaging process are incorporated into the pre-trained neural network of the light intensity distribution. The light field imaging calculation is performed on the predicted value of the particle light intensity distribution output by the network to obtain the calculated light field image. The loss function is used to calculate the deviation between the original light field image and the calculated light field image, and the number of training times is set to make the loss function tend to a stable value, and the model converges, then the fine-tuned neural network of the particle light intensity distribution can be obtained. The mean square error loss function (Loss2 mse ) is also used in the model fine-tuning process:
[0033]
[0034] wherein, b j is the intensity value of the j-th pixel point in the original light field image, is the intensity value of the j-th pixel point in the calculated light field image, and q is the number of pixels in the light field image.
[0035] The light field imaging convolution calculation process is as Figure 3 shown. The predicted value of the particle light intensity distribution output by the network model is divided into K layers according to the depth (it is generally appropriate to take 1μm or 2μm), where K is generally in the range of 50 - 100. Then each layer is divided into A×B equally sized block regions, each block region contains M×N voxels, and the center of each block region is conjugate to the center of the corresponding microlens (A and B are the numbers of microlenses in the x and y directions respectively, and M×N is the number of pixels in the x and y directions covered by a single microlens), and the light intensity value in each voxel is represented by f. In the k-th layer of the predicted particle light intensity distribution after block division, the light intensity values of the voxels with the same coordinates in each block region are extracted and recombined, so that M×N two-dimensional voxel light intensity value matrices can be obtained, and the values F mnk are denoted as:
[0036]
[0037] where A and B are the numbers of block regions divided in the x and y directions respectively, m and n are the relative position coordinates of the voxels in each block region, m = 1, 2, 3,..., M, and n = 1, 2, 3,..., N.
[0038] The voxel light intensity value matrix F mnk of the k-th layer of the predicted particle field is convolved with its corresponding point spread function H mnk . Such convolution calculations are carried out in M×N groups to obtain M×N calculation results, and all the results are accumulated to obtain the light field image corresponding to this layer, which is expressed as:
[0039]
[0040] where "*" represents the convolution process, H mnk represents the point spread function corresponding to the light intensity value recombination matrix F mnk , and G k represents the light field image after the light field imaging calculation of the k-th layer of the predicted particle light intensity distribution.
[0041] The light field images G k of each depth layer are accumulated to obtain the total light field image G obtained by the light field imaging process of the predicted particle light intensity distribution, which is expressed as:
[0042]
[0043] The strategy of block loading can significantly optimize the video memory occupancy problem, effectively reduce the computational burden and storage overhead, enabling the light field imaging process model to be embedded in the deep learning network as the physical information layer.
[0044] During the pre-training and model fine-tuning processes, considering both the convergence speed and stability, the Adam optimizer is selected. By setting the learning rate and the number of model training iterations to make the model converge, the network training and fine-tuning can be completed, thus forming a complete deep learning reconstruction model of light field microscopy three-dimensional particle field that integrates physical information.
[0045] The following is an embodiment that details the working principle and implementation method of the model.
[0046] According to the microchannel parameters, the relevant parameters of the measurement volume matching the microlens array are selected. The size of the measurement volume is 330μm×220μm×100μm, the particle concentration in the measurement volume is 0.8ppm (particle per microlen, ppm), and the positions of the particle centers are randomly distributed. The number of microlens arrays used is A×B = 24×16, the number of pixels covered by each microlens is M×N = 25×25, and the optical parameters of the imaging system are shown in Table 1.
[0047] Table 1 Optical Parameters of Light Field Microscope
[0048]
[0049] First, use the dataset to pre-train the three-dimensional particle field light intensity distribution reconstruction model. Convolve the three-dimensional particle light intensity distribution of the measurement volume with the PSF to obtain the particle light field image. A total of 500 groups of "particle light intensity distribution - light field image" samples are generated as the training dataset. Extract the pixels with the same angle relative to the center of each microlens in each light field image and rearrange them to generate the sub-aperture image. In this embodiment, each microlens covers 25×25 pixels, and the microlens array is arranged in 24×16. Therefore, a total of 625 sub-aperture images with a size of 24×16 can be extracted from 1 light field image in the dataset. At the input layer of the U-Net network, the size of the sub-aperture image is 24×16×625. After the upsampling layer, the image size is expanded to 1536×1024×64. The encoder extracts features from the image and reduces the dimension to obtain 512 feature maps of 96×64. The decoder improves the accuracy of the size recovery process, and finally outputs the predicted value of the three-dimensional particle light intensity distribution with a size of 600×400×51. Through the loss function Loss1 mse Quantify the difference between the predicted value and the true value of the corresponding particle light intensity distribution in the dataset, use the ReLU activation function to enhance the non-linear fitting ability of the network, select the Adam optimizer for training optimization, and train the network for 300 Epoch to make Loss1mse Minimize, with the initial learning rate set to 10 -5 to balance the convergence speed and stability. After the pre-training is completed, a neural network structure and initial values that can reconstruct the three-dimensional particle light intensity distribution are obtained.
[0050] Next, the light field imaging process is introduced as a physical constraint to fine-tune the model parameters of the pre-trained light intensity distribution neural network. When performing the light field imaging convolution calculation of the reconstruction result by block loading, first divide the predicted value of the particle light intensity distribution into 51 layers at a depth of 2μm. Each layer is further divided into 24×16 periodically distributed block regions, and each block contains 25×25 voxels. Therefore, the voxel size is determined to be 0.55μm×0.55μm×2μm. Next, extract and reorganize the light intensity values of the voxels with the same coordinates within each block region, so that a voxel light intensity value matrix of 25×25×51 with a size of 24×16 can be obtained. Then, load the corresponding PSF (with a size of 201×201) for convolution respectively, and finally linearly superimpose all the results to obtain the calculated light field image. Use the loss function Loss2 mse Calculate the deviation between the original light field image and the calculated light field image, and select the Adam optimizer for training optimization, with the learning rate set to 10 -6 to balance the convergence speed and stability. The number of model training epochs is set to 20 to minimize Loss2 mse and fine-tune the parameters of the pre-trained network. After the training is completed, a complete deep learning reconstruction model of the light field microscopy three-dimensional particle field incorporating physical information can be obtained.
[0051] Through the trained deep learning reconstruction model of the light field microscopy three-dimensional particle field, the prediction of the three-dimensional particle light intensity distribution of the input light field image can be realized, that is, the three-dimensional particle field reconstruction.
[0052] Example 2
[0053] This example provides a method for reconstructing a light field microscopy three-dimensional particle field, including:
[0054] Obtain a light field image;
[0055] Input the obtained light field image into the three-dimensional particle field reconstruction model trained in Example 1, and output the predicted value of the particle light intensity distribution.
[0056] Example 3
[0057] This example provides an electronic device, including:
[0058] A processor; a memory for storing processor-executable instructions;
[0059] Wherein, the processor is configured to implement the model training method provided in Embodiment 1 or implement the light field microscopy three-dimensional particle field reconstruction method provided in Embodiment 2.
[0060] Embodiment 4
[0061] This embodiment provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can execute the model training method provided in Embodiment 1 or implement the light field microscopy three-dimensional particle field reconstruction method provided in Embodiment 2.
Claims
1. A training method for a three-dimensional particle field reconstruction model of a light field microscope integrating physical information, characterized in that, Including: Pre-training of the network model: The convolutional neural network model is pre-trained using a training dataset. The training dataset of the network model is a data sample of particle light intensity distribution - light field image. The input of the network model is the light field image, and the output of the network model is the predicted value of the particle light intensity distribution. Fine-tuning of the network model: Fine-tuning the parameters of the network model obtained through pre-training. The fine-tuning method is as follows: Perform light field imaging calculation on the predicted value of the particle light intensity distribution output by the pre-trained network model to obtain the calculated light field image. Use the loss function to calculate the deviation between the original light field image and the calculated light field image, and set the number of training times to make the loss function tend to a stable value, and the model converges to obtain the fine-tuned neural network model of the particle light intensity distribution.
2. The training method of the optical field microscopy three-dimensional particle field reconstruction model incorporating physical information according to claim 1, characterized in that The network model is a convolutional neural network, which includes upsampling, an encoder, and a decoder. Upsampling is used to perform sub-pixel filling on the sub-aperture image extracted from the light field image to achieve data dimension expansion.
3. The training method of the optical field microscopy three-dimensional particle field reconstruction model incorporating physical information according to claim 2, characterized in that The encoder part contains multiple convolutional layers and pooling layers. The convolutional layers extract features from the input data, and the pooling layers reduce the data dimension.
4. The training method of the optical field microscopy three-dimensional particle field reconstruction model integrating physical information according to claim 2, characterized in that, The decoder part contains multiple upsampling layers, convolutional layers, and skip connection layers. The upsampling layers are used to restore the image size, and the skip connection layers are used to fuse features to enhance the detail restoration ability.
5. The training method of the optical field microscopy three-dimensional particle field reconstruction model integrating physical information according to claim 1, characterized in that, The loss function in the network model pre-training step is: Among them, Loss1 mse is the loss function in the pre-training step of the network model, is the true value of the particle light intensity distribution of the i-th voxel in the dataset, is the predicted value of the particle light intensity distribution reconstructed by the neural network from the light field image of the i-th voxel, and p is the number of voxels.
6. The training method of the optical field microscopy three-dimensional particle field reconstruction model integrating physical information according to claim 1, characterized in that, The mean square error loss function is also used in the model fine-tuning process: Among them, Loss2 mse is the loss function in the fine-tuning step of the network model, and b j is the intensity value of the j-th pixel point in the original light field image, is for calculating the intensity value of the j-th pixel point in the light field image, and q is the number of pixels in the light field image.
7. The training method of the optical field microscopy three-dimensional particle field reconstruction model incorporating physical information according to claim 1, characterized in that, Performing light field imaging calculation on the predicted value of the particle light intensity distribution output by the network model to obtain the calculated light field image, including: The predicted value of the particle light intensity distribution output by the network model is divided into K layers according to the depth H, where K takes values in the range of 50 - 100; then each layer is divided into A×B equal-sized block regions, each block region contains M×N voxels, and the center of each block region is conjugate to the center of the corresponding microlens. A and B are the numbers of microlenses in the x and y directions respectively, and M×N is the number of pixels covered by a single microlens. In the k-th layer of the predicted value of the particle light intensity distribution after block division, the light intensity values of voxels with the same coordinates in each block region are extracted and recombined to obtain an M×N two-dimensional voxel light intensity value matrix, and the values of each matrix are denoted as F mnk : where f m,n represents the light intensity value of the voxel within the relative coordinate positions m and n of each block region, where m = 1, 2, 3, ..., M and n = 1, 2, 3, ..., N; The voxel light intensity value matrix F of the k-th layer of the predicted particle field mnk is convolved with the corresponding point spread function H mnk to obtain M×N calculation results, and all the results are accumulated to obtain the light field image corresponding to this layer, expressed as: Among them, G k represents the light field image after performing light field imaging calculation on the predicted value of the k-th layer of the particle light intensity distribution; "*" represents the convolution process; H mnk represents the light intensity value recombination matrix F mnk corresponding point spread function; Accumulate the light field images G of each depth layer k to obtain the total light field image G obtained through the light field imaging process of the predicted value of the particle light intensity distribution, expressed as: Where G is the calculated light field image.
8. A method for reconstructing a three-dimensional particle field by light field microscopy, characterized in that, Including: Obtaining the light field image; Inputting the obtained light field image into the three-dimensional particle field reconstruction model to output the predicted value of the particle light intensity distribution; Wherein, the three-dimensional particle field reconstruction model is trained according to the model training method described in any one of claims 1 to 7.
9. An electronic device, characterized in that, Including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to: implement the model training method described in any one of claims 1 to 7, or implement the light field microscopy three-dimensional particle field reconstruction method described in claim 8.
10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal can execute the model training method described in any one of claims 1 to 7, or implement the light field microscopy three-dimensional particle field reconstruction method described in claim 8.