Method for positioning by using light field microscope

Abstract

The invention discloses a method for performing three-dimensional positioning by using a light field microscope, which belongs to the field of light field microscope imaging, and comprises the following steps of: S1, performing light field imaging of an object by using the light field microscope, calibrating an observed original light field image and decoding the original light field image into a biplane parameterized four-dimensional format, therefore, a real polar plane image, namely a real EPI, is formed; s2, wave light forward direction models of spherical volumes located at different depths are used for simulating a set of light fields, a depth perception dictionary is constructed, and each atom is a synthetic polar plane image related to the specific depth, namely a synthetic EPI; s3, implementing three-dimensional positioning by executing a convolutional sparse coding model based on a depth perception dictionary and a dry friction inertial near-end gradient algorithm (IPGDF) on the observed polar plane image; according to the invention, the positioning precision and the positioning speed of three-dimensional positioning under the light field microscope are improved.

Description

Technical Field

[0001] The invention relates to a three-dimensional positioning method, in particular to a positioning method using a light field microscope, and belongs to the field of light field microscope imaging. Background Art

[0002] The mechanisms of perception, cognition, and complex behaviors that emerge from the activity of neuronal networks is a fundamental question in neuroscience, and progress depends on the development of techniques to track the activity of hundreds to thousands of neurons simultaneously. Neuronal localization is fundamental to studying information about neuronal activity, and optical microscopy imaging techniques are widely used to record neuronal activity with the help of advanced fluorescent indicators. Despite the advantages of high spatial resolution and potential scalability, large-scale imaging using optical microscopy is rather time-consuming.

[0003] By utilizing light field imaging, light field microscopy provides an alternative for 3D imaging of neural activity at fast frame rates. Unlike traditional optical microscopy, which only records lateral information as two-dimensional projections of light, light field microscopy is actually a three-dimensional optical imaging technology that can simultaneously collect the position and angle information of the incident light reaching the sensor by combining microlenses, the lens array is on the original imaging plane, and by moving the imaging sensor to the back focal plane of the microlenses. Through this structure, light field microscopy is able to obtain four-dimensional data containing the spatial position and propagation direction of light in a single snapshot. This non-scanning imaging mechanism helps to increase the imaging speed and achieve high frame rate recording of neural population activity, which has great potential in observing the structure and dynamics of the entire brain volume.

[0004] However, light field microscope imaging is easily affected by noise, which can lead to deviations in positioning accuracy. In addition, the calculation process for achieving three-dimensional positioning is complex, which affects the positioning speed. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a method for positioning using a light field microscope, which improves the positioning accuracy and positioning speed of three-dimensional positioning under a light field microscope.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for positioning using a light field microscope comprises the following steps:

[0008] Step S1: using a light field microscope to perform light field imaging of the object, calibrating and decoding the observed original light field image into a dual-plane parameterized four-dimensional format, thereby forming a true epipolar plane image, i.e., a true EPI;

[0009] Step S2: simulate a set of light fields using a wave-light forward model of a spherical volume at different depths and construct a depth-aware dictionary, where each atom is a synthetic epipolar image associated with a specific depth, i.e., a synthetic EPI;

[0010] Step S3: 3D positioning is achieved by executing a depth-aware dictionary-based convolutional sparse coding model and a dry friction inertial proximal gradient algorithm (IPGDF) on the observed polar plane image.

[0011] A further improvement of the technical solution of the present invention is that the specific operation of step S1 is as follows:

[0012] The experimental body is modeled as a sphere with a diameter of 5 microns, and the light field is observed using a light field microscope lens array. Each lens is regarded as an ideal small hole (i, j, k, l), which means that the lens at the position (i, j) observes the image of the pixel at the position (k, l), where (i, j) represents the lens coordinates and (j, k) represents the coordinates of the pixel behind each lens;

[0013] Subaperture image I obtained from the original light field image 4D (i, j, k, l), including the lens plane and pixel plane parameters, is called the dual-plane parameterized four-dimensional format, and the real EPI is obtained.

[0014] A further improvement of the technical solution of the present invention is that the specific operation of step S2 is as follows:

[0015] A sphere at different depths is simulated with a wave light forward model to obtain a sub-aperture image (i, j, k, l). The lens position is fixed at (i, :), and the pixel position is fixed at (k, :), resulting in a two-dimensional slice image (i, :, k:) with angle and spatial direction. The depth-related properties of EPI are used: the deeper the point source, the greater the slope of the straight line. Different depths are fixed for capturing images, generating a series of synthetic EPI images related to each depth, and forming a depth-aware EPI dictionary.

[0016] The further improvement of the technical solution of the present invention is that: step S3 converts the light field three-dimensional positioning problem into a convolutional sparse coding solution problem, takes the observed real EPI image as input Y, takes the synthetic EPI of different depths as a dictionary, and obtains the depth of the maximum sparse coefficient obtained by solving the convolutional sparse coding as the estimated depth to achieve positioning; the specific operations are as follows:

[0017] S3.1: Given the observed true EPI image as input Y, solve the convolutional sparse coding model:

[0018]

[0019] Solving in the real domain and introducing the slice model Simplified convolutional sparse coding model:

[0020]

[0021] Where min x (f(x)+g(x)) means to find the value that minimizes (f(x)+g(x)) as much as possible. i ∈R θ×n (θ≤Θ,n≤N) is the i-th atom in the EPI dictionary, Θ and N represent the angular and spatial dimensions, M represents the number of atoms in the dictionary, and λ is the regularization parameter. A larger λ indicates a greater sparsity in the coefficients. i ∈R Θ×N Represents the dictionary atom d i The relevant sparse coding coefficients, ||·||2 is the l2 norm, which is used to measure the sum of squares of errors, and the l1 norm ||·||1 is a regularization term used to promote the sparsity of the solution; P i represents the operation of extracting the signal at the i-th position from the input EPI image Y, P i T is the transpose operation; D L is composed of EPI atoms The local dictionary composed of i ∈R M represents the sparse coding coefficient, S i =D L α i is the i-th slice of the signal;

[0022] S3.2: Introducing the dry friction inertial proximal gradient algorithm (IPGDF), the optimization problem formula of the slice-based convolutional sparse coding model is divided into a differentiable function f(α i ) and the non-differentiable function g(α i )Two parts:

[0023]

[0024] For the function f(α i ) to find the gradient and obtain the gradient operator formula:

[0025]

[0026] Where ▽ represents the gradient operator symbol, is the sparse coding coefficient of the i-th slice for the k-th iteration;

[0027] The proximal mapping algorithm is used Update:

[0028]

[0029] in is the updated sparse coding coefficient, h is the time step, and the proximal function prox is the proximal operator that helps to achieve the sparse solution. Its parameters is the sparse coding constant of the (k-1)th iteration; γ is the constant of the dry friction function φ = γ||·||1, g k is g(α i )'s helper function;

[0030] Assume that the univariate threshold function make a j for The jth component of j The relationship between the size of T and v is a The solution of (x) is obtained, and the proximal gradient expression is obtained as follows:

[0031]

[0032] The further improvement of the technical solution of the present invention is that in step S3.2, the Nesterov accelerated dry friction inertial proximal gradient algorithm (Fast-IPGDF) is introduced into the proximal mapping formula. The specific operation is: use ▽f(z k ) replaces the proximal mapping formula The Nesterov acceleration boost point is Get the new sparse coefficient update formula:

[0033]

[0034] in

[0035] A further improvement of the technical solution of the present invention is that within a depth range of -60 to 60 μm, the positioning accuracy is 5 μm, and the positioning is accurate and has high positioning precision.

[0036] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

[0037] The present invention introduces a dry friction inertial proximal gradient algorithm (IPGDF) to update the gradient, avoid falling into the local minimum, have better and faster convergence results, and improve the speed and accuracy of three-dimensional positioning based on extreme plane image matching depth; and the coefficient sparsity is high, with good anti-noise performance, which can realize simultaneous positioning of multiple point sources and effective positioning in a noisy environment.

[0038] The convolutional sparse coding model of the present invention is updated by using a local slice representation method, thereby reducing the complexity of calculation.

[0039] The present invention introduces a Nesterov accelerated dry friction inertial proximal gradient algorithm (Fast-IPGDF) to update the coefficients of a convolutional sparse coding model, thereby achieving accelerated convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the light field microscope three-dimensional positioning method of the present invention;

[0041] Figure 2 This is the optical path diagram of the light field microscope based on microlens;

[0042] Figure 3 is the original two-dimensional light field image at a certain depth;

[0043] Figure 4 This is an example of a sub-aperture image;

[0044] Figure 5 is an atom at a certain depth in the depth-aware dictionary;

[0045] Figure 6 This is the depth positioning effect diagram within the depth range of -60 to 60 μm;

[0046] Figure 7 is the sparse coding coefficient map corresponding to each atom obtained by convolutional sparse coding;

[0047] Figure 8 This is a graph showing the relationship between the model residual value and the number of iterations;

[0048] Fig. 9 This is the result diagram of dual point source positioning;

[0049] Fig.10 This is the flow chart of the three-dimensional positioning algorithm. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below with reference to the accompanying drawings and embodiments:

[0051] like Figure 1 As shown, a method for positioning using a light field microscope comprises the following steps:

[0052] Step S1: using a light field microscope to perform light field imaging of the object, calibrating and decoding the observed original light field image into a dual-plane parameterized four-dimensional format, thereby forming a true epipolar plane image, i.e., a true EPI;

[0053] Step S2: simulate a set of light fields using a wave-light forward model of a spherical volume at different depths and construct a depth-aware dictionary, where each atom is a synthetic epipolar image associated with a specific depth, i.e., a synthetic EPI;

[0054] Step S3: 3D positioning is achieved by executing a depth-aware dictionary-based convolutional sparse coding model and a dry friction inertial proximal gradient algorithm (IPGDF) on the observed polar plane image.

[0055] The specific operations of step S1 are:

[0056] The observed raw light field image is calibrated and then decoded into a two-plane parameterized four-dimensional format to form a sub-aperture image. The experimental body is modeled as a 5-micron diameter sphere, and the light field is observed using a light field microscope lens array. The device is as follows: Figure 2 As shown. Each lens is regarded as an ideal pinhole. (i,j,k,l) ​​represents the image of the pixel at position (k,l) observed by the lens at position (i,j). Where (i,j) represents the lens coordinates, and (j,k) represents the coordinates of the pixel behind each lens; if i and j are fixed, that is, captured from one of the lenses, an image I(i,j,:,:) at different positions under a lens will be obtained, that is, the sub-aperture image, the sub-aperture image I 4D The array of (i,j,k,l) ​​contains the lens plane and pixel plane parameters, so it is also called the dual-plane parameterized four-dimensional format, such as Figure 4 The white line indicates the edge of the lens. Figure 3 A particular view is obtained by fixing (i, j) in the original light field image in , where the viewing angle varies along rows (from left to right) and columns.

[0057] The specific operations of step S2 are:

[0058] In the sub-aperture image (i,j,k,l), the lens position is fixed at (i,:) and the pixel position is fixed at (k,:); this will give a two-dimensional slice image (i,:,k:) with angle and spatial direction, called a polar plane image (EPI). By using the EPI depth-related property: the deeper the point source, the greater the slope of the line, the different depths are fixed for capturing imaging. In this way, a series of EPIs related to each depth are generated, and a depth-aware EPI dictionary is formed, as shown in Figure 4.

[0059] The specific operations of step S3 are:

[0060] S3.1: The convolutional sparse coding model reveals the intrinsic structure and pattern of the data by representing the input data as a sparse linear combination of a set of basis functions. The observed EPI is used as the input Y, and the EPIs at different depths are used as the dictionary; the depth at which the maximum sparse coefficient is obtained by solving the convolutional sparse coding is the depth of the input EPI. In this way, the three-dimensional positioning model can be converted into a convolutional dictionary sparse coding model; given the observed EPI as input, solve the convolutional sparse coding model:

[0061]

[0062] Solving in the real domain and introducing the slice model Simplified convolutional sparse coding model:

[0063]

[0064] Where min x (f(x)+g(x)) means to find the value that minimizes (f(x)+g(x)) as much as possible. i ∈R θ×n (θ≤Θ,n≤N) is the i-th atom in the EPI dictionary, Θ and N represent the angular and spatial dimensions, M represents the number of atoms in the dictionary, and λ is the regularization parameter. A larger λ indicates a greater sparsity in the coefficients. i ∈R Θ×N Represents the dictionary atom d i The relevant sparse coding coefficients, ||·||2 is the l2 norm, which is used to measure the sum of squares of errors, and the l1 norm ||·||1 is a regularization term used to promote the sparsity of the solution; P i represents the operation of extracting the signal at the i-th position from the input EPI image Y, P i T is the transpose operation; D L is composed of EPI atoms The local dictionary composed of i ∈R M represents the sparse coding coefficient, S i =D L α i is the i-th slice of the signal;

[0065] S3.2: Introduce the dry friction inertial proximal gradient algorithm (IPGDF) to improve the convergence of the algorithm. According to the IPGDF convergence theorem, solve {min(f+g)} and divide the objective function into the differentiable function f(α i ) and the non-differentiable function g(α i )Two parts:

[0066]

[0067] For the function f(α i ) to find the gradient, we can get the gradient operator formula, where ▽ represents the gradient operator symbol, is the sparse coding coefficient of the kth iteration:

[0068]

[0069] During the experiment, it was assumed Satisfying the proximal mapping gradient algorithm, considering the dry friction function and the non-differentiable function g(α i ), using the proximal mapping algorithm Updates.

[0070]

[0071] in is the updated sparse coding coefficient, h is the time step, and the proximal function prox is the proximal operator that helps to achieve the sparse solution. Its parameters is the sparse coding constant of the (k-1)th iteration; γ is the constant of the dry friction function φ = γ||·||1, g k is g(α i )'s helper function;

[0072] The calculation formula given by the present invention is used to calculate the near-end mapping Assume that the univariate threshold function make a j yes The jth component, T a (x) can be discussed separately j The solution is obtained by the relationship between the size of and v; and then the proximal gradient expression is obtained as:

[0073] This method is used to perform three-dimensional positioning simulation. Within the depth range of -60-60μm, the positioning accuracy is 5μm. This method is used for positioning. The positioning results are shown in the figure below. Figure 6 As shown in the figure, the positioning using this method is accurate and has high positioning accuracy.

[0074] Figure 7 The figure shows the relationship between the sparse coefficients obtained through convolutional sparse coding and the dictionary model. The three-dimensional positioning coefficients of this method are relatively sparse and the positioning is accurate. Moreover, the sparsity is high and the anti-noise performance is strong under certain noise influence.

[0075] Figure 8 The figure shows the absolute value of the difference between the original data and the positioning data when the method is positioned, also known as the curve of the residual changing with the number of iterations. It can be seen from the figure that the method reaches the convergence condition when the number of iterations is 30. The fewer the number of iterations, the faster the convergence and the more efficient the operation.

[0076] Fig. 9 The figure shows the positioning results of this method when the double point source depths are set to 15μm and -5μm respectively. The blue represents the actual position of the ball, and the red represents the estimated value obtained by this method. It can be seen from the figure that this method still has accurate positioning capabilities in the case of double point sources.

[0077] Bring the Nesterov accelerated IPGDF algorithm into the proximal mapping formula and use ▽f(z k ) replaces the proximal mapping formula ▽f(α i k ),here The sparse coefficient update formula using the Fast-IPGDF algorithm can be obtained:

[0078]

[0079] in

[0080] like Fig.10 As shown, it is a flow chart of the inertial proximal gradient algorithm based on Nesterov acceleration and dry friction. The specific steps are as follows:

[0081] (1) Input observed EPI Y, local convolution dictionary D L

[0082] (2) Setting parameters: γ>0,

[0083] (3) Calculation

[0084] (4) Calculate the proximal mapping

[0085] (5) Update {α i} k+1

[0086] (6) Determine whether k is less than or equal to K-1. If so, loop (3), (4), (5); otherwise, output the estimated value {α i} k .

[0087] The inertial proximal method with Nesterov accelerated dry friction (Fast-IPGDF) updates {α i} k+1 , which can reduce oscillation and accelerate convergence.

[0088] The present invention provides a method for three-dimensional positioning using a light field microscope to effectively detect the three-dimensional position of neuronal cells from a single light field image, with higher positioning accuracy and good noise immunity. The present invention can achieve simultaneous positioning of multiple point sources and effective positioning in noisy environments. Compared with existing research, the present invention has significantly improved positioning accuracy and positioning speed.

Claims

1. A method for positioning using a light field microscope, characterized in that: The following steps are involved: Step S1: using a light field microscope to perform light field imaging of the object, calibrating and decoding the observed original light field image into a dual-plane parameterized four-dimensional format, thereby forming a true epipolar plane image, i.e., a true EPI; Step S2: simulate a set of light fields using a wave-light forward model of a spherical volume at different depths and construct a depth-aware dictionary, where each atom is a synthetic epipolar image associated with a specific depth, i.e., a synthetic EPI; Step S3: 3D positioning is achieved by executing a depth-aware dictionary-based convolutional sparse coding model and a dry friction inertial proximal gradient algorithm (IPGDF) on the observed polar plane image.

2. The method for positioning using a light field microscope according to claim 1, characterized in that: The specific operations of step S1 are as follows: The experimental body is modeled as a sphere with a diameter of 5 microns. The light field is observed using a light field microscope lens array. Each lens is regarded as an ideal small hole (i, j, k, l). The image of the pixel at the position (k, l) observed by the lens at the position (i, j) is represented, where (i, j) represents the lens coordinates and (j, k) represents the coordinates of the pixel behind each lens. The sub-aperture image I obtained from the original light field image is 4D (i, j, k, l), including the lens plane and pixel plane parameters, is called the dual-plane parameterized four-dimensional format, and the real EPI is obtained.

3. The method for positioning using a light field microscope according to claim 1, characterized in that: The specific operations of step S2 are as follows: A sphere at different depths is simulated with a wave light forward model to obtain a sub-aperture image (i, j, k, l). The lens position is fixed at (i, :), and the pixel position is fixed at (k, :), resulting in a two-dimensional slice image (i, :, k:) with angle and spatial direction. The depth-related properties of EPI are used: the deeper the point source, the greater the slope of the straight line. Different depths are fixed for capturing images, generating a series of synthetic EPI images related to each depth, and forming a depth-aware EPI dictionary.

4. The method for positioning using a light field microscope according to claim 1, characterized in that: Step S3 converts the light field 3D positioning problem into a convolutional sparse coding problem, takes the observed real EPI image as input Y, and uses the synthetic EPI at different depths as a dictionary. The depth at which the maximum sparse coefficient is obtained by solving the convolutional sparse coding is the estimated depth, and positioning is achieved. The specific operations are as follows: S3.1: Given the observed true EPI image as input Y, solve the convolutional sparse coding model: Solving in the real domain and introducing the slice model The simplified convolutional sparse coding model is: Where min x (f(x)+g(x)) means to find the value that minimizes (f(x)+g(x)) as much as possible. i ∈R θ×n (θ≤Θ,n≤N) is the i-th atom in the EPI dictionary, Θ and N represent the angular and spatial dimensions, M represents the number of atoms in the dictionary, and λ is the regularization parameter. A larger λ indicates a greater sparsity in the coefficients. z i ∈R Θ×N Represents the dictionary atom d i The relevant sparse coding coefficients, ||·||2 is the l2 norm, which is used to measure the sum of squares of errors, and the l1 norm ||·||1 is a regularization term used to promote the sparsity of the solution; P i represents the operation of extracting the signal at the i-th position from the input EPI image Y, P i T is the transpose operation; D L is composed of EPI atoms The local dictionary composed of i ∈R M represents the sparse coding coefficient, S i =D L α i is the i-th slice of the signal; S3.2: Introducing the dry friction inertial proximal gradient algorithm (IPGDF), the optimization problem formula of the slice-based convolutional sparse coding model is divided into a differentiable function f(α i ) and the non-differentiable function g(α i )Two parts: For the function f(α i ) to find the gradient and obtain the gradient operator formula: in represents the gradient operator symbol, is the sparse coding coefficient of the i-th slice for the k-th iteration; The proximal mapping algorithm is used Update: in is the updated sparse coding coefficient, h is the time step, and the proximal function prox is the proximal operator that helps to achieve the sparse solution. Its parameters is the sparse coding constant of the (k-1)th iteration; γ is the constant of the dry friction function φ = γ||·||1, g k is g(α i )'s helper function; Assume that the univariate threshold function make a j for The jth component of j The relationship between the size of T and v is a The solution of (x) is obtained, and the proximal gradient expression is obtained as follows:

5. The method for positioning using a light field microscope according to claim 4, characterized in that: In step S3.2, the Nesterov accelerated dry friction inertial proximal gradient algorithm (Fast-IPGDF) is introduced into the proximal mapping formula. The specific operation is: Replace the proximal mapping formula The Nesterov acceleration boost point is Get the new sparse coefficient update formula: in 6. The method for positioning using a light field microscope according to claim 4, characterized in that: Within the depth range of -60 to 60 μm, the positioning accuracy is 5 μm, with accurate positioning and high positioning precision.

Images