A method for detecting the position of microsphere particles based on back focal plane images of confocal optical paths

Abstract

This invention discloses a method for detecting the position of microsphere particles based on the back focal plane image of a confocal optical path. This method combines an interferometric position detection device with an artificial neural network. Based on the camera's acquisition of the interference pattern of the back focal plane of the confocal optical path, a trained artificial neural network is used to output the position data of the microsphere particles. Unlike methods that detect the position of microsphere particles based on a four-quadrant detector in the back focal plane, this method has no nonlinearity and a wide position detection range. Furthermore, unlike microphotographic position detection methods, this method can simultaneously and more accurately detect the position of microsphere particles in three dimensions.

Description

Technical Field

[0001] The invention relates to a microsphere particle position detection method based on a back focal plane image of a confocal optical path, and belongs to the technical field of photoelectric measurement. Background Art

[0002] Single-particle position measurement is a key tool for studying the dynamic motion of biomolecules and nanoparticles. By measuring the three-dimensional position of micro- and nanoscale microspheres, the trajectory of labeled biomolecules and nanoparticles can be tracked, thereby advancing applications and research in biomolecular science and the mechanisms of light-matter interactions.

[0003] Among the existing single-particle position measurement methods, one type is the microscopic position detection method. In this type of method, the position of a single spherical particle can be tracked by taking a microscopic image of the particle with a digital camera with nanometer- and millisecond-level resolution, with a position measurement accuracy of several nanometers. By fitting the image to a two-dimensional Gaussian function for position measurement, the measurement accuracy can reach several nanometers (journal article "Single-image axial localization precision analysis for individual fluorophores," Opt. Express, 2012, 20(3): 3057-3065). By measuring objects such as polystyrene microspheres with a diameter of 1.5μm in a theoretical algorithm matching method, a position measurement accuracy of the order of 10nm can be achieved (journal article "Characterizing and tracking single colloidal particles with video holographic microscopy," Optics Express, 2012, 20(2): 3057~3065). This method can generally only measure in the transverse plane and cannot measure the longitudinal position or requires side observation to increase the longitudinal measurement.

[0004] Another method, based on the interferometric light field at the back focal plane of a confocal optical path, uses quadrant detectors to measure the position of individual particles in three dimensions, enabling even more precise sub-nanometer measurements (see "Three-dimensional tracking of small spheres in focused laser beams: influence of the detection angular aperture," Optics Letters, 2003, 28: 411). However, this quadrant-based method suffers from nonlinearities when measuring particle position, significantly limiting its range and measurement scope. This limitation has driven researchers to explore new methods for measuring single-particle position. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a method for detecting the position of microsphere particles based on the back focal plane image of the confocal optical path, so as to detect the position of microsphere particles in a larger range, including the nonlinear region, and achieve high-precision measurement of the three-dimensional position of a single microsphere particle.

[0006] The present invention adopts the following technical solutions to solve the technical problems:

[0007] The present invention provides a method for detecting the position of microsphere particles based on a back focal plane image of a confocal optical path, which comprises the following steps:

[0008] Step 1: construct a confocal optical path and collect interference patterns;

[0009] After the incident laser beam is focused by the objective lens, it irradiates the microsphere particles near the focal point. The part of the incident laser beam scattered by the microsphere particles and the part not scattered by the microsphere particles are collected by the condenser and interfere near the back focal plane of the condenser. The interference pattern is collected by the camera on the back focal plane of the condenser or a conjugate plane of the back focal plane.

[0010] Step 2: obtaining a data set of interference patterns and their corresponding microsphere particle positions, and preprocessing the data set;

[0011] Step 2.1, based on the radius of the microsphere particles, the refractive index of the microsphere particles, the vacuum wavelength of the incident laser beam, and the parameters of the objective lens, perform electromagnetic field scattering simulation calculations on the confocal optical path to obtain the interference patterns corresponding to the back focal plane of the condenser when the microsphere particles are at different positions near the focus;

[0012] Step 2.2: Using the position data of the microsphere particles as labels for the corresponding interference patterns, and preprocessing the interference patterns collected at different positions of the microsphere particles, thereby forming a data set consisting of the preprocessed interference patterns and their labels;

[0013] Step 3: Build a neural network , and use the dataset to Perform training to obtain a trained microsphere particle position detection model;

[0014] Step 4: Utilize the trained microsphere particle position detection model to detect any interference pattern acquired in the confocal optical path to obtain the position of the microsphere particle near the focus.

[0015] The microsphere particle position detection method based on the back focal plane image of the confocal optical path described in the present invention is also characterized by using formula (1) to characterize the light intensity value I of the interference pattern collected by the camera:

[0016] The light intensity value I of the interference pattern on the rear focal plane of the condenser is shown in formula (1):

[0017] (1)

[0018] In formula (1), is the dielectric constant of vacuum, is the propagation speed of electromagnetic waves in vacuum, is the coordinate value on the back focal plane, is the aperture radius of the condenser, E is the total field at the back focal plane, and:

[0019] (2)

[0020] In formula (2), E s is the scattered field at the back focal plane, is the incident field at the rear focal plane.

[0021] The neural network Includes: feature extraction module, intermediate module, output module;

[0022] The feature extraction module consists of P convolution blocks, where each convolution block includes: a two-dimensional convolution layer, a batch normalization layer, Activation function layer, maximum pooling layer;

[0023] Input any i-th interference pattern in the data set into the feature extraction module and process it through P convolution blocks in sequence to obtain the feature map feResult i ;

[0024] The middle module consists of Q fully connected blocks, where each fully connected block includes: a fully connected layer, Activation function layer and random dropout layer;

[0025] feResult i Input into the middle module and transform through Q fully connected blocks in turn to obtain the feature vector midResult i ;

[0026] The output module consists of a fully connected layer and a regression layer, and first i After dimensionality conversion, the regression layer outputs the predicted position label y of the i-th interference pattern i ;

[0027] Based on the position label and the predicted position, the loss function L is constructed and the back propagation algorithm is used to calculate the effect of the loss function L on the neural network. The parameters are updated to obtain the trained microsphere particle position detection model.

[0028] The loss function L is constructed using formula (3):

[0029] (3)

[0030] In formula (3), N is the number of interference patterns in the data set, t i is the position label of the i-th interference pattern.

[0031] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the microsphere particle position detection method, and the processor is configured to execute the program stored in the memory.

[0032] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the microsphere particle position detection method when the computer program is executed by a processor.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. This invention proposes a method for detecting the position of microsphere particles based on back-focal plane images of confocal optical paths. This method can detect the position of microsphere particles in liquid or gaseous environments. Compared to position detection methods based on microscopic photography, this method captures interference images at the back-focal plane using a camera, thereby obtaining more information about the microsphere particles and enabling more accurate detection of their three-dimensional position. Compared to methods that detect the position of microsphere particles using quadrant detectors based on the back-focal plane interference light field, this method lacks nonlinearity and offers a wider range of position detection.

[0035] 2. This invention combines the interference pattern captured by the camera at the back focal plane with an artificial neural network. The neural network used is an AlexNet-like regression network. Compared with conventional convolutional neural networks, this neural network adopts a deeper network structure, uses local response normalization, and uses larger convolution kernels to increase the model's expressive power. The fully connected layer design of this neural network also better integrates the features extracted by the convolution layer, thereby improving the detection accuracy of microsphere particle positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart in this embodiment;

[0037] Figure 2 is the system optical path diagram in this embodiment;

[0038] Figure 3 is a network structure diagram in this embodiment;

[0039] Figure 4 2 is a comparison chart of the test results in this embodiment;

[0040] Figure 5 2 is a diagram of the detection results in this embodiment.

[0041] Numbers in the figure: 1. Camera, 2. Condenser, 3. Microsphere particles, 4. Sample cell, 5. Objective lens, 6. Laser. DETAILED DESCRIPTION

[0042] In this embodiment, a method for detecting the position of microsphere particles based on the back focal plane image of the confocal optical path is shown in the flowchart of the implementation. Figure 1 As shown, the following steps are included:

[0043] Step 1: construct a confocal optical path and collect interference patterns;

[0044] Build the confocal optical path as follows Figure 2 As shown, after the incident fundamental mode laser beam is focused by the objective lens, it is irradiated on the microsphere particles located near the focus. The part of the incident laser beam scattered by the microsphere particles and the part not scattered by the microsphere particles are collected by the condenser, and interfere near the back focal plane of the condenser. The interference pattern is collected by the camera on the back focal plane of the condenser or the conjugate plane of the back focal plane. In the specific example here, glass microspheres are used, the refractive index of the microsphere particles is 1.55, the radius of the microsphere particles is 1.0μm, the vacuum wavelength of the light beam is 1.064μm, the environment in which the microsphere particles are located is water, and the medium refractive index of water is 1.33.

[0045] Step 2: obtaining a data set of interference patterns and their corresponding microsphere particle positions, and preprocessing the data set;

[0046] In step 2.1, based on the radius of the microsphere particle, the refractive index of the microsphere particle, the vacuum wavelength of the incident laser beam, and the parameters of the objective lens, 4000 position data of the microsphere particle uniformly distributed between -2 μm and 2 μm on the x-axis are generated. The electromagnetic field scattering simulation calculation of the particles at these 4000 positions is performed on the above-mentioned confocal optical path, and the interference pattern corresponding to the back focal plane of the condenser when the microsphere particle is at different positions near the focus is collected.

[0047] The light intensity value I of the interference pattern on the rear focal plane of the condenser is shown in formula (1):

[0048] (1)

[0049] In formula (1), is the dielectric constant of vacuum, is the propagation speed of electromagnetic waves in vacuum, is the coordinate value on the back focal plane, is the aperture radius of the condenser, E is the total field at the back focal plane, and:

[0050] (2)

[0051] In formula (2), E s is the scattered field at the back focal plane, is the incident field at the rear focal plane.

[0052] In step 2.2, the position data of the microsphere particles is used as the label for the corresponding interference pattern. The interference patterns collected at different positions of the microsphere particles are preprocessed and the interference pattern size is uniformly adjusted to 227 × 227 × 3. The position data and interference patterns form a unified dataset with a training set, validation set, and test set ratio of 7:2:1.

[0053] Step 3: Build a neural network , and train with the training set to obtain a trained microsphere particle position detection model;

[0054] Step 3.1, the network constructed here is as follows Figure 3 As shown, it is the Alexnet convolutional neural network Based on The last three layers are replaced by fully connected layers and regression layers; neural network Includes: feature extraction module, intermediate module, output module; and inputs interference pattern into neural network Process it in and get the predicted position label;

[0055] In this embodiment, the feature extraction module is composed of 5 convolution blocks. Any i-th interference pattern in the data set is input into the feature extraction module to obtain the feature processing result feResulti ; In the convolution block, it includes a two-dimensional convolution layer, a batch normalization layer, Activation function layer, maximum pooling layer; the middle module consists of 2 fully connected blocks, and the input feature processing result feResult i , convert it into a feature vector midResult i , in the fully connected block, including the fully connected layer, Activation function layer and random dropout layer, introduce ReLU activation function and random dropout to reduce overfitting; the output module consists of a fully connected layer and a regression layer, and the input feature vector midResult i , convert it into the dimension of regression output, and the regression layer obtains the predicted position label y of the i-th interference pattern i .

[0056] Step 3.2, initialize the neural network The network parameters in ; Set the maximum number of training rounds to 300, the initial learning rate to 0.01, the learning rate decay factor to 0.1, and the learning rate decay period to 50 rounds; input the training set data into the neural network Training, through the neural network Output the microsphere particle position prediction result;

[0057] Step 3.3, use the mean square error loss function MSELOSS to calculate the loss L between the predicted position data and the true position data as shown in formula (3), use the back propagation algorithm to solve the model gradient according to the loss, and use the stochastic gradient descent method to adjust the network parameters Perform iterative updates; in each round of iterative training, the validation set is used for validation, and the Adam optimizer is used to update the network parameters ; Until the maximum number of iterations reaches 300, the training stops, the model is tested using the test set data, the model is further optimized based on the test results, and the optimized network parameters are saved , and then the trained neural network save.

[0058] (3)

[0059] In formula (3), N is the number of interference patterns in the data set, t i is the position label of the i-th interference pattern, y i is the predicted position of the i-th interference pattern.

[0060] Step 4: Using the trained microsphere particle position detection model to detect any interference pattern obtained in the confocal light path, the position of the microsphere particle near the focus is obtained;

[0061] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0062] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0063] Importing a trained neural network , establish a mapping relationship between interference pattern and position data; import the interference pattern of the focal plane of the confocal optical path when the microsphere particles are at different positions, and in this embodiment, import the interference pattern of the simulated microsphere particles at positions between 0 and 1.2 μm on the x-axis, and pre-process the interference pattern; through the neural network The prediction of the position data corresponding to each interference pattern is realized, and the results are as follows Figure 4 The figure also compares the results with those of the quadrant detection method. The quadrant detection method predicts the position by analyzing the difference in the integrated light intensity between different quadrants of the back focal plane, thereby establishing a corresponding relationship between the predicted position of the microsphere particle and the light intensity difference. By performing a one-dimensional linear fit within the linear range, the particle position can be inferred from the signal output of the quadrant detector.

[0064] In this embodiment, more simulated microsphere particles at random positions between 0 and 1.5 μm on the x-axis are introduced into the focal plane interference pattern of the confocal optical path, and the interference pattern is pre-processed and the neural network is used to calculate the interference pattern. Realize the prediction of the position data corresponding to each interference pattern, and the prediction results are as follows Figure 5 shown.

[0065] In the implementation examples of the present invention, the contents of the program are implemented using MATLAB R2023a®, using the Adam optimizer, and all calculations are run on a computer with a 12th generation Intel(R) Core(TM) i9-12900K, 64GB RAM and an NVIDIA GeForce RTX A4000 graphics processing unit (GPU).

[0066] The present invention compares different methods for tracking the position information of a single microsphere particle, including the quadrant detection method and the method proposed by the present invention. The simulation results are as follows: Figure 4 As shown. Figure 4 The results show that both methods have high accuracy in tracking the position information of a single microsphere particle within the linear range, but the method of the present invention can also detect the position of a single microsphere particle with high accuracy outside the linear range.

[0067] The above-described embodiments are merely examples of preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting the position of microsphere particles based on the back focal plane image of the confocal optical path, characterized in that: The following steps are involved: Step 1: construct a confocal optical path and collect interference patterns; After the incident laser beam is focused by the objective lens, it irradiates the microsphere particles near the focal point. The part of the incident laser beam scattered by the microsphere particles and the part not scattered by the microsphere particles are collected by the condenser and interfere near the back focal plane of the condenser. The interference pattern is collected by the camera on the back focal plane of the condenser or a conjugate plane of the back focal plane. Step 2: obtaining a data set of interference patterns and their corresponding microsphere particle positions, and preprocessing the data set; Step 2.1, based on the radius of the microsphere particles, the refractive index of the microsphere particles, the vacuum wavelength of the incident laser beam, and the parameters of the objective lens, perform electromagnetic field scattering simulation calculations on the confocal optical path to obtain the interference patterns corresponding to the back focal plane of the condenser when the microsphere particles are at different positions near the focus; Step 2.2: Using the position data of the microsphere particles as labels for the corresponding interference patterns, and preprocessing the interference patterns collected at different positions of the microsphere particles, thereby forming a data set consisting of the preprocessed interference patterns and their labels; Step 3: Build a neural network , and use the dataset to Perform training to obtain a trained microsphere particle position detection model; Step 4: Utilize the trained microsphere particle position detection model to detect any interference pattern acquired in the confocal optical path to obtain the position of the microsphere particle near the focus.

2. The microsphere particle position detection method based on the back focal plane image of the confocal optical path according to claim 1 is characterized in that: Formula (1) is used to represent the light intensity value I of the interference pattern collected by the camera: The light intensity value I of the interference pattern on the rear focal plane of the condenser is shown in formula (1): (1) In formula (1), is the dielectric constant of vacuum, is the propagation speed of electromagnetic waves in vacuum, is the coordinate value on the back focal plane, is the aperture radius of the condenser, E is the total field at the back focal plane, and: (2) In formula (2), E s is the scattered field at the back focal plane, is the incident field at the rear focal plane.

3. The microsphere particle position detection method based on the back focal plane image of the confocal optical path according to claim 1 is characterized in that: The neural network Includes: feature extraction module, intermediate module, output module; The feature extraction module consists of P convolution blocks, where each convolution block includes: a two-dimensional convolution layer, a batch normalization layer, Activation function layer, maximum pooling layer; Input any i-th interference pattern in the data set into the feature extraction module and process it through P convolution blocks in sequence to obtain the feature map feResult i ; The middle module consists of Q fully connected blocks, where each fully connected block includes: a fully connected layer, Activation function layer and random dropout layer; feResult i Input into the middle module and transform through Q fully connected blocks in turn to obtain the feature vector midResult i ; The output module consists of a fully connected layer and a regression layer, and first i After dimensionality conversion, the regression layer outputs the predicted position label y of the i-th interference pattern i ; Based on the position label and the predicted position, the loss function L is constructed and the back propagation algorithm is used to calculate the loss function L. The parameters are updated to obtain the trained microsphere particle position detection model.

4. The method for detecting the position of microsphere particles based on the back focal plane image of the confocal optical path according to claim 3, characterized in that: The loss function L is constructed using formula (3): (3) In formula (3), N is the number of interference patterns in the data set, t i is the position label of the i-th interference pattern.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the microsphere particle position detection method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the microsphere particle position detection method according to any one of claims 1 to 4 are executed.

Images