A parallax-based automatic focusing method for light field microscopy system

Abstract

The present invention discloses a parallax-based automatic focusing and locking method for a light field microscope system, wherein the light field microscope system is used to shoot at a position of a sample defocus distance to obtain four-dimensional phase space data, and the parallax of each perspective of the four-dimensional phase space data relative to the central perspective is calculated using cross-correlation, and a two-dimensional defocus coefficient matrix of the light field microscope system is obtained based on the parallax and defocus distance calculation; the light field microscope system is used to shoot the sample at the current shooting distance to obtain the current two-dimensional parallax matrix of the sample; the two-dimensional parallax matrix S and the two-dimensional defocus coefficient matrix K are linearly fitted to obtain the current defocus amount of the sample; the current shooting distance is adjusted in real time according to the defocus amount until focusing and / or locking is completed. By analyzing the parallax characteristics of the light field image, the present invention can realize the rapid automatic focusing of the light field microscope system and the locking function during the long-term shooting process.

Description

Technical Field

[0001] The invention relates to the technical field of multi-color light field microscopic imaging, and in particular to an automatic focusing and locking method of a light field microscopic system based on parallax. Background Art

[0002] Light field microscopy is a new type of microscopic imaging technology. By recording the light field information of an object, it can obtain three-dimensional images without mechanical scanning. It has broad application prospects in biomedicine and other fields. However, the image clarity of light field microscopy depends largely on the relative position between the microscope objective and the sample, that is, the working distance. If the working distance is not appropriate, the image will be defocused and blurred, making it difficult to meet the needs of subsequent analysis. Although the image can be focused at the beginning of the shooting, the living sample will inevitably deviate from the focal plane during long-term shooting, resulting in blurred images.

[0003] Although automatic focusing technologies are currently available, most of them are based on image clarity analysis and require multiple imaging and comparison at different focal planes. They are slow and may cause phototoxic damage to the sample, and cannot achieve the focus lock function for long-term photography.

[0004] Therefore, how to provide a parallax-based light field microscope system automatic focusing and locking method for realizing fast automatic focusing of the light field microscope system is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] In view of the above research status and existing problems, the present invention provides a parallax-based light field microscope system automatic focusing and locking method, which can realize fast automatic focusing of the light field microscope system and the focus locking function during long-term shooting by analyzing the parallax characteristics of the light field image.

[0006] The present invention provides a parallax-based light field microscopy system automatic focusing and locking method, wherein the light field microscopy system includes a microlens array for acquiring multi-viewing angle information of an image; the method includes the following steps:

[0007] Defocus coefficient matrix calibration steps:

[0008] Using the light field microscope system to shoot at a position of the sample defocus distance z to obtain first two-dimensional original light field data with four-dimensional information;

[0009] The first two-dimensional raw light field data is rearranged into first four-dimensional phase space data, the parallax of each viewing angle of the first four-dimensional phase space data relative to the central viewing angle is calculated using cross-correlation, and the two-dimensional defocus coefficient matrix K of the light field microscope system is calculated based on the parallax and the defocus distance z;

[0010] Sample focus locking steps:

[0011] Using the light field microscope system to shoot a sample at a current shooting distance, obtaining second two-dimensional original light field data having four-dimensional information;

[0012] The second two-dimensional original light field data is rearranged into second four-dimensional phase space data, and the disparity of each perspective of the second four-dimensional phase space data relative to the central perspective is calculated by cross-correlation to obtain a current two-dimensional disparity matrix S of the sample;

[0013] Performing linear fitting on the two-dimensional disparity matrix S and the two-dimensional defocus coefficient matrix K to obtain the current defocus value of the sample;

[0014] The current shooting distance is adjusted in real time according to the defocus amount until focusing and / or locking is completed.

[0015] Preferably, in the defocus coefficient matrix calibration step, the step of rearranging the first two-dimensional original light field data into the first four-dimensional phase space data comprises:

[0016] The first two-dimensional original light field data is rearranged to obtain a shape v i *h*w multi-view rearranged sub-images, where v i is the i-th view in the multi-view rearranged image, h and w are the pixel height and pixel width of each sub-image respectively.

[0017] Preferably, the step of calculating the parallax of each viewing angle of the first four-dimensional phase space data relative to the central viewing angle by using cross-correlation comprises:

[0018] ;

[0019] Among them, u and v represent the longitudinal and lateral angular coordinates of a single microlens respectively, the value ranges of m and n are [0,w],[0,h] respectively, h and w are the pixel height and pixel width of each sub-image respectively, W1 is the first four-dimensional phase space data; u0 and v0 represent the longitudinal and lateral angular coordinates of the central viewing angle respectively.

[0020] Preferably, the step of calculating and obtaining the two-dimensional defocus coefficient matrix K of the light field microscopy system based on the parallax and the defocus distance z comprises:

[0021] The parallax is divided by the defocus distance z to calculate and obtain a two-dimensional defocus coefficient matrix K of the light field microscope system.

[0022] Preferably, in the sample focus locking step, the step of linearly fitting the two-dimensional disparity matrix S and the two-dimensional defocus coefficient matrix K includes: using the least squares method to linearly fit the sample two-dimensional disparity matrix S and the system's two-dimensional defocus coefficient matrix K to obtain the current defocus amount of the sample.

[0023] Preferably, the step of adjusting the current shooting distance in real time according to the defocus amount until focusing is completed includes:

[0024] Determine whether the defocus amount is equal to 0. If so, the sample is in focus and the focusing is completed; if not, adjust the current shooting distance according to the defocus amount, and repeat the sample focus locking step until the defocus amount is equal to 0, then the sample focusing is completed.

[0025] Preferably, the step of adjusting the current shooting distance in real time according to the defocus amount until the focus is locked comprises:

[0026] During the long-term shooting process of the sample, the trigger condition of the focusing step is set, and the focusing step is automatically executed in real time to complete the focus lock under the long-term; the long-term shooting process of the sample refers to the shooting process in which the shooting time is greater than the preset value.

[0027] Preferably, the triggering condition for setting the focusing step includes setting a timed triggering focusing step.

[0028] Compared with the prior art, it has the following beneficial effects:

[0029] The present invention realizes automated processing, and the focusing speed is faster and more efficient;

[0030] The present invention utilizes the parallax characteristics of light field data acquired by a light field microscope system to achieve fast automatic focusing and focus locking for long-term living body photography. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0032] Figure 1 This is a flow chart of a parallax-based light field microscope system automatic focusing and locking method provided in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The light field microscope system includes a microlens array. The light field microscope uses the microlens array to record light field information and can obtain sub-images of multiple perspectives at the same time. The position of the object point on the focal plane remains consistent in the sub-images of different perspectives, while the object point on the defocused plane will produce parallax. Therefore, by analyzing the parallax characteristics of the light field image, fast automatic focusing can be achieved.

[0035] The embodiment of the present invention discloses a parallax-based light field microscope system automatic focusing and locking method, which uses the parallax characteristics of light field data obtained by the light field microscope system to realize automatic focusing and locking functions. The execution steps are as follows:

[0036] Defocus coefficient matrix calibration steps:

[0037] Using a light field microscope system to shoot at a sample defocus distance z to obtain first two-dimensional raw light field data with four-dimensional information;

[0038] The first two-dimensional original light field data is rearranged into the first four-dimensional phase space data, the parallax of each viewing angle of the first four-dimensional phase space data relative to the central viewing angle is calculated using cross-correlation, and the two-dimensional defocus coefficient matrix K of the light field microscope system is calculated based on the parallax and the defocus distance z;

[0039] Sample focus locking steps:

[0040] Using the light field microscope system to photograph the sample at the current shooting distance, obtaining the second two-dimensional original light field data with four-dimensional information;

[0041] The second two-dimensional original light field data is rearranged into second four-dimensional phase space data W2, and the disparity of each perspective of the second four-dimensional phase space data relative to the central perspective is calculated using cross-correlation to obtain the current two-dimensional disparity matrix S of the sample;

[0042] Perform linear fitting on the two-dimensional disparity matrix S and the two-dimensional defocus coefficient matrix K to obtain the current defocus value of the sample;

[0043] The current shooting distance is adjusted in real time according to the defocus amount until the focus is achieved and / or locked.

[0044] It should be noted that the two-dimensional defocus coefficient matrix K is used to characterize the parallax of each viewing angle of the four-dimensional phase space data caused by the sample being defocused by one unit distance.

[0045] In the embodiment, the method of rearranging to obtain the second four-dimensional phase space data is the same as the method of rearranging to obtain the first thinking phase space data, and the cross-correlation method used to calculate the parallax of each perspective of the second four-dimensional phase space data relative to the central perspective is the same as the cross-correlation method used to calculate the parallax of each perspective of the first four-dimensional phase space data relative to the central perspective.

[0046] In one embodiment, the step of rearranging the first two-dimensional raw light field data into first four-dimensional phase space data comprises:

[0047] The first two-dimensional original light field data is rearranged to obtain a shape v i *h*w multi-view rearranged sub-images, where v i is the i-th view in the multi-view rearranged image, h and w are the pixel height and pixel width of each sub-image respectively.

[0048] In a specific implementation, the longitudinal and lateral dimensions of the two-dimensional light field image are u*h and v*w, respectively, where h and w represent the longitudinal and lateral spatial coordinates in the microlens array light field imaging system, and u and v represent the longitudinal and lateral angular coordinates of a single microlens, respectively.

[0049] The angle coordinate is fixed, and the pixel values ​​of the angle coordinate under different spatial coordinates are obtained to form the spatial coordinate sub-image of the angle coordinate; the spatial coordinate sub-images of all angle coordinates are arranged in sequence to form the four-dimensional phase space light field data W1 of the two-dimensional light field image.

[0050] In this embodiment, the rearrangement method includes pre-processing steps such as cropping and alignment.

[0051] In one embodiment, the step of calculating the disparity S0 of each viewing angle (u, v) of the first four-dimensional phase space data relative to the central viewing angle (u0, v0) by using cross-correlation includes:

[0052] ;

[0053] Among them, u and v represent the longitudinal and lateral angular coordinates of a single microlens respectively, the value ranges of m and n are [0,w],[0,h] respectively, h and w are the pixel height and pixel width of each sub-image respectively, W1 is the first four-dimensional phase space data; u0 and v0 represent the longitudinal and lateral angular coordinates of the central viewing angle respectively.

[0054] In one embodiment, the step of calculating and obtaining a two-dimensional defocus coefficient matrix K of a light field microscopy system based on parallax and defocus distance z comprises:

[0055] Divide the parallax by the defocus distance z to calculate the two-dimensional defocus coefficient matrix K of the light field microscope system:

[0056] .

[0057] In one embodiment, the step of linearly fitting the two-dimensional disparity matrix S and the two-dimensional defocus coefficient matrix K includes: using the least squares method to linearly fit the sample two-dimensional disparity matrix S and the system two-dimensional defocus coefficient matrix K to obtain the current defocus value of the sample:

[0058] .

[0059] In one embodiment, the step of adjusting the current shooting distance in real time according to the defocus amount until focusing is completed includes:

[0060] Determine whether the defocus amount is equal to 0. If so, the sample is in focus and the focusing is completed; if not, adjust the current shooting distance according to the defocus amount, and repeat the sample focus locking step until the defocus amount is equal to 0, and the sample focusing is completed.

[0061] In one embodiment, the step of adjusting the current shooting distance in real time according to the defocus amount until the focus is locked includes:

[0062] During the long-time shooting of the sample, the trigger conditions of the focus step are set, and the focus step is automatically executed in real time to complete the focus lock under the long time; the long time refers to the shooting time being greater than the preset value.

[0063] In one embodiment, setting the triggering condition of the focusing step includes setting a timed triggering focusing step.

[0064] In one embodiment, when adjusting the second two-dimensional original light field data of the sample photographed by the light field microscope system, the process of photographing the current shooting distance from the sample includes: controlling the sample stage to move along the vertical Z axis and in a direction close to or away from the light field microscope system shooting component.

[0065] The above embodiment of the present invention realizes fast automatic focusing of the sample by analyzing the parallax characteristics of the four-dimensional phase space of the defocused plane light field data. In the application scenario of long-term photography of living bodies, the sample is kept in continuous focus and accurate, and the image is clear, realizing the function of long-term focus locking.

[0066] The above is a detailed introduction to the parallax-based light field microscopy system automatic focus locking method provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0067] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A parallax-based automatic focusing method for a light field microscopy system, wherein the light field microscopy system comprises a microlens array for acquiring multi-viewing angle information of an image; characterized in that: The steps include: Defocus coefficient matrix calibration steps: Using the light field microscope system to shoot at a position of the sample defocus distance z to obtain first two-dimensional original light field data with four-dimensional information, and rearrange it into first four-dimensional phase space light field data; The cross-correlation is used to calculate the parallax of each perspective of the first four-dimensional phase space light field data relative to the central perspective, including: The step of calculating the disparity of each perspective of the first four-dimensional phase space light field data relative to the central perspective by using cross-correlation includes: R u,v (m,n)=∑ x ∑ y W1(x,y,u,v)W1(x+m,y+n,u0,v0); S0(u,v)=argmax R u，v (m,n); Among them, u and v represent the longitudinal and lateral angular coordinates of a single microlens respectively, the value ranges of m and n are [0,w], [0,h] respectively, h and w are the pixel height and pixel width of each sub-image respectively, W1 is the first four-dimensional phase space light field data; u0 and v0 represent the longitudinal and lateral angular coordinates of the central viewing angle respectively; The two-dimensional defocus coefficient matrix K of the light field microscope system is calculated based on the parallax and the defocus distance z, comprising: Dividing the parallax by the defocus distance z, a two-dimensional defocus coefficient matrix K of the light field microscope system is calculated; Sample focus locking steps: Using the light field microscope system to shoot a sample at a current shooting distance, obtaining second two-dimensional original light field data having four-dimensional information; The second two-dimensional original light field data is rearranged into second four-dimensional phase space data, and the disparity of each perspective of the second four-dimensional phase space data relative to the central perspective is calculated by cross-correlation to obtain a current two-dimensional disparity matrix S of the sample; Performing linear fitting on the two-dimensional disparity matrix S and the two-dimensional defocus coefficient matrix K to obtain the current defocus value of the sample; Adjusting the current shooting distance in real time according to the defocus amount until focusing and / or locking the focus is completed, including: Determine whether the defocus amount is equal to 0. If so, the sample is in focus and the focusing is completed; if not, adjust the current shooting distance according to the defocus amount, and repeat the sample focus locking step until the defocus amount is equal to 0, then the sample focusing is completed.

2. The method for automatically focusing and locking a light field microscope system based on parallax according to claim 1, characterized in that: In the defocus coefficient matrix calibration step, the step of rearranging the first two-dimensional original light field data into first four-dimensional phase space light field data comprises: The first two-dimensional original light field data is rearranged to obtain a shape v i *h*w multi-view rearranged sub-images, where v i is the i-th view in the multi-view rearranged image, h and w are the pixel height and pixel width of each sub-image respectively.

3. The method for automatically focusing and locking a light field microscope system based on parallax according to claim 1, characterized in that: In the sample focus locking step, the step of linearly fitting the two-dimensional disparity matrix S and the two-dimensional defocus coefficient matrix K includes: using the least squares method to linearly fit the sample two-dimensional disparity matrix S and the system's two-dimensional defocus coefficient matrix K to obtain the current defocus amount of the sample.

4. The method for automatically focusing and locking a light field microscope system based on parallax according to claim 1 or 3, characterized in that: The step of adjusting the current shooting distance in real time according to the defocus amount until the focus is locked comprises: During the long-term shooting process of the sample, the trigger condition of the focusing step is set, and the focusing step is automatically executed in real time to complete the focus lock under the long-term; the long-term shooting process of the sample refers to the shooting process in which the shooting time is greater than the preset value.

5. The method for automatically focusing and locking a light field microscope system based on parallax according to claim 4, characterized in that: The triggering condition of setting the focusing step includes setting a timing-triggered focusing step.

Images