3D Auto-Focusing Method, Device and Readable Storage Medium Based on Multimodal Data Fusion and Kalman Filtering

Abstract

The present invention provides a three-dimensional autofocus method, device and readable storage medium based on multi-modal data fusion and Kalman filtering, which realizes high-precision real-time focusing through multiple steps: using a phase and image sensor to collect sample three-dimensional phase information and image data of different focal planes, calculating the phase depth and sharpness; constructing a multi-plane Kalman filtering cluster to model the focus states of each focal plane, fusing the influence of neighborhood coupling to predict the focus; combining the image sharpness and phase gradient to update and correct the focus state, and determining the optimal three-dimensional focus position with the help of a collaborative optimization decision maker; dynamically adjusting the noise matrix and coupling coefficient to control the objective lens to complete focusing. This method effectively solves the problems of focus drift and jump through multi-modal fusion and multi-plane collaboration, significantly improves the focusing accuracy and stability of the imaging system in complex samples and dynamic environments, and is applicable to devices such as biomedical fully automatic microscopes and digital pathology slide scanners.

Description

Technical Field

[0001] The present invention relates to the field of optical microscopy imaging technology, specifically to the autofocus technology in a microscopic system, and particularly to a three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering, which is applicable to scenarios such as biomedical microscopy imaging and digital pathology slide scanning that require high precision and long-term stable focusing. Background Art

[0002] In recent years, microscopy imaging technology has been widely applied in the biomedical field (such as fully automated microscopes, digital pathology slide scanners). A fully automated microscope needs to perform long-term automated observation of biological samples (lasting for several hours to dozens of hours), and a digital pathology slide scanner needs to quickly scan pathology slides to generate whole-slide digital images. Both rely on a stable and reliable autofocus technology to compensate for focus drift caused by factors such as environmental temperature changes and mechanical vibrations to ensure imaging quality.

[0003] Traditional autofocus methods determine the defocus amount by scanning different imaging planes along the optical axis of the objective lens and using an image quality evaluation function (such as a sharpness function). However, such methods have significant drawbacks:

[0004] 1. Single-modal dependence: Relying only on single-modal data such as sharpness, it is vulnerable to noise interference and has insufficient focusing accuracy in low-contrast samples (such as transparent biological tissues);

[0005] 2. Three-dimensional scene failure: For samples with complex three-dimensional structures, due to the relatively deep range of the clear focal plane, the optimal focus position cannot be accurately identified, resulting in focusing failure;

[0006] 3. Independent multi-plane modeling: Each focal plane independently predicts the focus position, ignoring the interaction between planes, which is prone to cause focus jumps or instability;

[0007] 4. Lack of real-time performance: It is necessary to scan each plane and calculate the evaluation function one by one, which takes a long time and cannot meet the real-time focusing requirements in a dynamic environment.

[0008] With the development of microscopy imaging towards high speed, high precision, and automation, there is an urgent need for an autofocus technology that can fuse multi-modal data, process the multi-plane coupling effect, and achieve real-time dynamic optimization. Summary of the Invention

[0009] Embodiments of the present invention provide a three-dimensional autofocus method, device, and readable storage medium based on multi-modal data fusion and Kalman filtering, aiming at the problems existing in the current technology that traditional microscopic autofocus methods have poor real-time performance, insufficient multi-plane coordination, and weak anti-interference ability, and cannot meet the requirements of long-term stable observation of complex three-dimensional samples.

[0010] The core technology of the present invention mainly constructs a multi - plane Kalman filter cluster to fuse three - dimensional phase information and image sharpness data, dynamically predict and collaboratively optimize the focal positions of multiple focal planes, achieve high - precision, real - time, and stable three - dimensional autofocus, and solve the problems of focus drift and jump in complex samples and dynamic environments.

[0011] In a first aspect, the present invention provides a three - dimensional autofocus method based on multi - modal data fusion and Kalman filtering, and the method includes the following steps:

[0012] Collect the three - dimensional phase information of the sample and the image data of different focal planes in real - time;

[0013] Perform normalization processing on the three - dimensional phase information and the image data to obtain the phase depth information and sharpness metrics of each focal plane;

[0014] Model the focal states of each focal plane based on a multi - plane Kalman filter cluster, where the focal states include the axial position of the focal plane, focal velocity, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes;

[0015] Predict the current focal states of each focal plane by fusing the historical focal states and the coupling effects of adjacent focal planes through a multi - plane Kalman filter cluster;

[0016] Update and correct the predicted focal states using the image sharpness metric and the phase gradient observation value;

[0017] Collaboratively optimize the focal positions of multiple focal planes through a collaborative optimization decision - maker to determine the final three - dimensional focal position;

[0018] Control the adjustment of the objective lens according to the three - dimensional focal position to achieve real - time three - dimensional autofocus.

[0019] Further, the three - dimensional phase information is collected through a microlens array to obtain the phase gradients of the sample in the x - axis and y - axis directions, and the sharpness of the image data is obtained by calculating the sum of the squares of the image gradients.

[0020] Further, the state modeling of the multi - plane Kalman filter cluster includes: defining a state vector containing the axial position, focal velocity, acceleration, phase gradient, and neighborhood coupling coefficient for each focal plane, and constructing an initial error covariance matrix, a process noise matrix, and a measurement noise matrix.

[0021] Further, when predicting the focal state, describe the time change of the focal state through a state transition matrix, and quantify the interaction between adjacent focal planes through the neighborhood coupling coefficient.

[0022] Further, when updating and correcting the focal state, balance the weights of the predicted value and the measured value through the Kalman gain, and the measured values include image sharpness and phase gradient.

[0023] Further, the collaborative optimization decision - maker performs conflict detection and collaborative optimization on the focal positions of multiple focal planes based on a multi - objective optimization algorithm to eliminate the mutual interference between the focal planes.

[0024] Further, it also includes dynamically adjusting the process noise matrix and the coupling coefficient between adjacent focal planes. The process noise matrix is dynamically adjusted according to the image sharpness gradient, and the coupling coefficient is updated in real - time according to the sharpness change between the focal planes.

[0025] In a second aspect, the present invention provides a three - dimensional autofocus device based on multi - modal data fusion and Kalman filtering, including:

[0026] A phase sensor for collecting three - dimensional phase information of a sample;

[0027] An image sensor for collecting image data of different focal planes and calculating a sharpness metric;

[0028] A multi - plane Kalman filtering cluster for modeling, predicting, and updating and correcting the focal states of each focal plane. The focal states include axial position, focal velocity, acceleration, phase gradient observation values, and the coupling coefficient between adjacent focal planes;

[0029] A collaborative optimization decision - maker for collaboratively optimizing the focal positions of multiple focal planes to determine the final three - dimensional focal position;

[0030] A focal length adjustment module for controlling the objective lens adjustment according to the three - dimensional focal position to achieve real - time three - dimensional autofocus.

[0031] In a third aspect, the present invention provides an electronic device including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the above - mentioned three - dimensional autofocus method based on multi - modal data fusion and Kalman filtering.

[0032] In a fourth aspect, the present invention provides a readable storage medium in which a computer program is stored. The computer program includes program codes for controlling a process to execute the process, and the process includes the above - mentioned three - dimensional autofocus method based on multi - modal data fusion and Kalman filtering.

[0033] The main contributions and innovations of the present invention are as follows:

[0034] 1. Multi - modal data fusion improves the focusing accuracy and robustness. By fusing three - dimensional phase gradient (reflecting the sample spatial structure) and image sharpness (traditional single - modal), it solves the problem of focusing blur caused by the noise sensitivity of a single sharpness evaluation function for low - contrast samples (such as transparent biological tissues). Especially, the focusing success rate is improved in samples with weak texture and transparency.

[0035] 2. Multi - plane collaborative modeling to suppress focus jumping and drifting: By quantifying the interaction between focal planes through neighborhood coupling coefficients, a multi - plane Kalman filter cluster is constructed to replace the traditional independent modeling method, eliminating focus jumping caused by independent prediction of each focal plane. And through collaborative optimization of the decision - maker, global optimal focusing is achieved to adapt to the three - dimensional complex structure of the sample.

[0036] 3. Dynamic adaptive mechanism to enhance environmental adaptability: The process noise matrix is dynamically adjusted according to the clarity gradient, and the coupling coefficient is updated in real - time with the change of clarity between focal planes, improving the focusing stability of the system in dynamic environments such as temperature changes and mechanical vibrations, and maintaining high - precision focusing for a long time without manual intervention.

[0037] 4. Optimization of real - time performance and efficiency: Based on the state prediction and observation update mechanism of Kalman filter, the time - consuming process of traditional plane - by - plane scanning is avoided, and the single - frame focusing time is significantly shortened, meeting the real - time focusing requirements of high - speed dynamic samples (such as real - time observation of living cells).

[0038] 5. Expansion of application scenarios: It is applicable to high - precision imaging devices such as biomedical fully - automatic microscopes and digital pathology slide scanners. Especially in scenarios such as three - dimensional imaging of thick samples and long - term tracking of living cells, it breaks through the dependence of traditional methods on a single focal plane or simple samples, promoting the development of microscopic imaging technology towards automation and intelligence.

[0039] Details of one or more embodiments of the present invention are set forth in the following drawings and description to make other features, objects, and advantages of the present invention more concise and understandable. Brief Description of the Drawings

[0040] The drawings described herein are used to provide a further understanding of the present invention and form a part of the present invention. The illustrative embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0041] Figure 1 is a flowchart of a three - dimensional autofocus method based on multi - modal data fusion and Kalman filter according to an embodiment of the present invention;

[0042] Figure 2 is a schematic hardware structure diagram of an electronic device according to an embodiment of the present invention. Detailed Embodiments

[0043] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0044] It should be noted that: In other embodiments, the steps of the corresponding methods are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0045] Traditional microscopic autofocus methods rely on single-modal data, independently process multiple focal planes, and lack real-time collaborative optimization, resulting in insufficient focusing accuracy, being vulnerable to noise interference, and being unable to effectively address the focus drift and jump problems in complex three-dimensional samples and dynamic environments.

[0046] Based on this, the present invention solves the problems existing in the prior art by fusing three-dimensional phase information and image sharpness data based on a multi-plane Kalman filter cluster.

[0047] Embodiment 1

[0048] The present invention aims to propose a three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering. Through a multi-plane Kalman filter cluster and phase / sharpness multi-modal data fusion technology, it realizes the collaborative optimization focusing of multiple focal planes of three-dimensional samples, and solves the deficiencies of traditional methods in terms of real-time performance, stability, and accuracy.

[0049] Specifically, an embodiment of the present invention provides a three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering. Specifically, referring to Figure 1 , the method includes the following steps:

[0050] S1. Real-time collect the three-dimensional phase information of the sample and the image data of different focal planes;

[0051] In this embodiment, a phase sensor is used to obtain the original light field image I(x, y, z, t) through a microlens array. This sensor can provide the phase gradients Ф x (x, y, z) and Ф y (x, y, z) of the sample along the x-axis and y-axis at different focal planes, and these phase information are calculated through phase gradients:

[0052]

[0053]

[0054] To represent the spatial structure change of the sample, it is used to estimate the preliminary information of the focus position. Here, Δ is the pixel offset, which is usually set to 1 pixel.

[0055] At the same time, the image sensor is used to collect image data of each focal plane in parallel to obtain image information of different depth planes.

[0056] The improved Tenengrad function is used to calculate the sum of squared gradients for each image plane, and the sharpness metric S(z i ), reflecting the clarity of the sample at different focal planes:

[0057]

[0058] Among them, σ=λ / 4NA is the weight attenuation coefficient, λ is the wavelength of light, NA is the numerical aperture of the objective lens, and x c ,y c is the center coordinate of the region, and I is the original light field image.

[0059] While traditional Tenengrad functions typically use first-order derivatives (such as the Sobel operator, which calculates the sum of squared gradient amplitudes), this method employs the sum of squared second-order derivatives, which is more sensitive to image edge sharpness and can more accurately distinguish edge blur caused by slight defocus. Traditional methods typically sum pixels across the entire image with equal weights. This method introduces Gaussian weighting, focusing on the image's central region (the core region of the sample in microscopic imaging is typically located at the center of the field of view). This reduces the impact of edge noise (such as reflections from the edge of the slide and stray light) on clarity evaluation and improves robustness in complex environments. The weight decay coefficient σ is directly linked to the optical system parameters (λ, NA), ensuring that the clarity metric conforms to the actual physical properties of imaging (such as the influence of the diffraction limit), avoiding empirical parameter adjustments and enabling automatic adaptation to the microscope objective.

[0060] S2, normalizing the three-dimensional phase information and image data to obtain phase depth information and clarity measurement of each focal plane;

[0061] In this embodiment, phase-depth mapping: Calculate the phase depth d of each focal plane i :

[0062]

[0063] where d iCharacterize the phase depth of the focal plane i, which characterizes the three-dimensional structure information of the sample at this focal plane (directly related to the focal position); λ is the wavelength of the incident light (unit: μm), which is determined by the light source of the microscopic system.

[0064] Clarity normalization: Normalize the clarity S(z i ) to S norm,i :

[0065]

[0066] where min(S) and max(S) are the minimum and maximum values of the clarity of all current focal planes respectively, ensuring that the normalization is based on the local data range (rather than a global fixed threshold) to adapt to the dynamic changes of different samples and environments. In this way, clarity normalization solves the problem of inconsistent scales in multi-modal data fusion through a simple and effective linear transformation, enabling the clarity information to be seamlessly combined with phase data, Kalman filter state variables, etc. This step seems basic, but it is a key link for the present invention to achieve high-precision focusing - it ensures that subsequent complex algorithms such as Kalman filtering and collaborative optimization can correctly process data from different sources, and finally achieve stable and reliable autofocus in a dynamic environment and for complex samples.

[0067] S3. Model the focal state of each focal plane based on the multi-plane Kalman filter cluster, where the focal state includes the axial position of the focal plane, focal velocity, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes;

[0068] In this embodiment, the specific steps of the modeling are as follows:

[0069] 1. Initialize the Kalman filter model

[0070] Set the initial state of the Kalman filter that defines the state vector for each focal plane to:

[0071]

[0072] where, is the axial position of the focal plane; is the initial focal velocity; is the initial acceleration; is the phase gradient observation value in the x direction (calculated by the difference between adjacent pixels of the light field image, that is, representing the phase difference of the focal plane i in the x direction), reflecting the lateral defocus offset, unit: rad / pixel; is the phase gradient observation value in the y direction (representing the phase difference of the focal plane i in the y direction), reflecting the longitudinal defocus offset, unit: rad / pixel; is the neighborhood coupling coefficient, representing the influence of a single adjacent focal plane, dimensionless.

[0073] Set the initial error covariance matrix For estimating the accuracy of the prediction process:

[0074]

[0075] Where , and are the initial estimation errors of the focus position and focus velocity respectively, , are the initial estimation errors of the phase sensor in the x and y directions respectively, is the coupling coefficient error between multiple planes, representing the interaction between each focal plane.

[0076] Set the process noise matrix of the Kalman filter parameters:

[0077]

[0078] For describing the uncertainty in the model prediction process, where , and represent the process noise of the focus position and velocity respectively, , are the process errors of the phase data in the x and y directions respectively, is the process error of the multi-plane coupling coefficient. The noise matrix will be dynamically adjusted according to the magnitude of the clarity gradient to ensure that the system can cope with rapid changes.

[0079] Set the measurement noise matrix , for describing the noise when measuring the image clarity. The clarity noise is negatively correlated with the score, and the tolerance is higher at low signal-to-noise ratios. and are the noises of the phase data respectively.

[0080] 2. Multi-plane Kalman filter cluster prediction model

[0081] Estimate the current focus state based on the historical focus position and velocity:

[0082]

[0083] Where, is the predicted state vector of the i-th focal plane at the current moment (including position, velocity, acceleration, phase gradient, coupling coefficient, etc.); is the historical state vector of the i-th focal plane at the previous moment; A i is the state transition matrix, describing the change of the focus from the previous moment to the current moment; B i u iIt is a control input item, representing the influence of an external control signal on the state (e.g., the control signal of an electric focusing mechanism), B i is the control matrix, u i is the external control signal (such as the displacement command of an electric focusing mechanism); γ is the global elastic coefficient, C ij is the coupling coefficient between focal planes i and j, representing the interaction between focal planes; , are the axial positions (focal coordinates) of focal planes i and j at the previous moment.

[0084] This formula is the core mathematical carrier of the "multi-plane collaborative focusing" technology of the present invention. By introducing a neighborhood coupling mechanism, it fundamentally improves the accuracy and stability of the autofocus system in three-dimensional complex samples and dynamic environments, and is a revolutionary improvement over traditional single-modal and independent modeling methods.

[0085] 3. Update the multi-plane Kalman filter cluster model

[0086] Kalman gain is used to balance the weights between the predicted value and the measured value:

[0087] , where: is the predicted error covariance matrix; is the measurement matrix, used to associate the state variables with the measurement data; is the measurement noise matrix; T represents the transpose of the matrix, that is, the operation of interchanging rows and columns of the measurement matrix is performed.

[0088] Observation matrix design:

[0089]

[0090] 80% of the weight is given to sharpness, and 20% is given to phase (10% for each of the two phase components), balancing accuracy and real-time performance.

[0091] 4. Collaborative optimization decision

[0092] Conflict detection and modeling: First, for the spacing constraint, => trigger optimization, and then optimize the objective function:

[0093]

[0094] ADMM fast solution:

[0095] Augmented Lagrangian function:

[0096]

[0097] where p i , p j are the axial positions (focal coordinates) of focal planes i and j; is the minimum focal plane spacing resolvable by the optical system (derived from the Rayleigh criterion, NA is the numerical aperture of the objective lens, and λ is the optical wavelength), and d min is essentially the axial resolution limit of the optical system. When the focal plane spacing is greater than this value, the mutual influence can be ignored and no coordination is required; when it is less than this value, the coupling effect must be modeled; z i is the focal position of focal plane i after optimization (decision variable); is the position of the i-th focal plane predicted by the Kalman filter; ρ is the augmented Lagrangian penalty coefficient, which controls the consistency strength between z and u and can be set to 1; z is the vector set of the target positions of all focal planes.

[0098] ADMM (Alternating Direction Method of Multipliers) iterative update (converges in 3 times):

[0099]

[0100] where z is the optimization variable (the focal plane focus position vector, with dimension N, and N is the number of focal planes); u is the auxiliary variable (strongly related to z and used to separate the constraint terms, usually set u = z); v is the Lagrange multiplier (reflecting the tightness of the constraint conditions, with the same dimension as z); A, b are the coefficient matrix and the constant term (from the linear part of the objective function, and the specific form depends on the problem modeling); ρ>0 is the penalty parameter (controlling the cost of constraint violation, usually increasing with iterations); is the projection operator (projecting the result into the constraint set C); I is the identity matrix (ensuring the feasibility of matrix inversion).

[0101] This system of equations transforms the multi-focal plane cooperative optimization problem into a sub-problem that can be efficiently solved through a cycle of unconstrained optimization → constraint projection → multiplier update, ensuring global optimal focusing under optical constraints.

[0102] S4. Predict the current focal states of each focal plane by fusing the historical focal states and the coupling effects of adjacent focal planes through multi-plane Kalman filtering clustering;

[0103] S5. Update and correct the predicted focal states using the image sharpness metric and the phase gradient observation values;

[0104] In this embodiment, the specific steps are as follows:

[0105] 1. Phase-Kalman depth fusion

[0106] Fuse the phase data with the prediction results of the Kalman filter to correct the predicted position of the focus. Further adjust the focus position by weighting the phase gradient information (i.e., the sensitivity of the phase change). The formula for phase correction is:

[0107]

[0108] where is the focus position predicted by the Kalman filter, is the focus position at the previous moment. k φ is the phase sensitivity coefficient, which represents the sensitivity of the focus change to the phase gradient change and is usually obtained through system calibration. and are the actually observed phase gradient and the predicted phase gradient, respectively.

[0109] At the same time, use reverse prediction guidance. To reduce the computational burden and improve the real-time performance of the system, the area near the focus can be deeply analyzed, and an ROI is defined. Without affecting the accuracy, the computational amount can be greatly reduced and the system response speed can be improved:

[0110]

[0111] ROI optimization can improve the computational efficiency by introducing an adaptive area and reduce the ineffective phase calculation. and are the predicted focus positions, σ p is the standard deviation of the position estimate, which reflects the accuracy of the focus prediction.

[0112] 2. Dynamic Parameter Adjustment

[0113] The process noise matrix is used to describe the errors that cannot be fully measured in the system (such as the small changes caused by the sample movement). Since the motion characteristics of the sample may be different under different environmental conditions, the noise level of the system will also change. Here, the tanh function is used to smoothly transition the process noise and dynamically adjust the threshold:

[0114]

[0115] where is the sharpness gradient on the current focal plane, which reflects the change degree of the current focus. θ is the threshold for noise adjustment, and δ is the smoothing coefficient, which is used to control the adaptive response speed of the noise.

[0116] The elastic coefficient γk controls the elastic coupling effect between the focal planes and is used to describe the mutual influence between adjacent focal planes. In a dynamic system, when the focus changes significantly (for example, when the focal plane moves relatively fast), the system may encounter the problem of overtracking, so it is necessary to reduce the elastic coefficient to make the system more stable.

[0117]

[0118] Among them, v max is the maximum speed that the system can track, Δt is the time step, is the change in the focus position, indicating the magnitude of the current focus offset.

[0119] In addition, during the process of multi-focal plane focusing, there is a certain coupling effect between the focal planes. When the focal length of one focal plane changes, it will affect the image quality of other focal planes. Especially in a microscope system, the mutual influence (or called coupling effect) between the focal planes is significant. For example, adjusting the position of focal plane 𝑖 may cause a change in the imaging quality of focal plane 𝑗. Therefore, it is necessary to adjust the coupling coefficient C between each focal plane in real time according to the observed sharpness data and phase gradient ij During each round of optimization, the system needs to dynamically adjust these coefficients to make the relationship between multiple focal planes optimal and avoid focus jumps or instability.

[0120] The formula for updating the coupling coefficient is:

[0121]

[0122] is the coupling coefficient between focal planes i and j in the current iteration. η is the adaptive learning rate, which controls the step size of coefficient update. is the gradient of the sharpness function of focal plane i with respect to the position of focal plane j, reflecting the influence of focus change on the sharpness of other focal planes. γ is a parameter that controls the attenuation of the coefficient and is used to suppress over-adjustment. The update mechanism of the coupling coefficient enables the system to gradually adjust the mutual relationship between each focal plane according to the changes in the sharpness function and phase gradient in each iteration.

[0123] S6. Co-optimize the focus positions of multiple focal planes through the co-optimization decision-making device to determine the final three-dimensional focus position;

[0124] In this embodiment, the focus adjustment signal received by the final controller is specifically an output generated by the co-optimization decision-making device and the Kalman filter system, which is used to accurately control the position adjustment of the microscope objective lens to achieve the best focus. This signal is based on various information such as the mutual influence between multiple focal planes, focus position prediction, sharpness evaluation, and phase data correction:

[0125]

[0126] where p final is the finally determined focal plane i focal point position; is the focal point position predicted by the Kalman filter cluster (based on historical state and neighborhood coupling influence); is the focal point position correction amount (fusing multi-source information such as clarity evaluation and phase data correction).

[0127] The final focal point position p final is the comprehensive result of the mutual influence between focal planes (synergistic prediction driven by coupling coefficient), Kalman filter dynamic prediction, clarity and phase data fusion correction, and optical constraint forced optimization. This mechanism breaks through the limitation of traditional single focal plane independent processing. Through the spatio-temporal fusion of multi-source information and physical constraint guidance, it realizes high-precision and stable autofocus under complex three-dimensional samples, which is an important progress of microscopic imaging technology from "single-modal trial and error" to "multi-dimensional intelligent optimization".

[0128] S7. Adjust the objective lens according to the three-dimensional focal point position to achieve real-time three-dimensional focusing.

[0129] Embodiment 2

[0130] Based on the same concept, the present invention also proposes a three-dimensional autofocus device based on multi-modal data fusion and Kalman filter, including:

[0131] A phase sensor for collecting three-dimensional phase information of the sample;

[0132] In this embodiment, the phase sensor collects the light field phase data of multiple planes (focal planes) through a microlens array. This phase data is Ф x (x, y, z) and Ф y (x, y, z) respectively represent the phase changes of the sample in the x-axis and y-axis directions.

[0133] An image sensor for collecting image data of different focal planes and calculating the clarity metric;

[0134] In this embodiment, the clarity metric is calculated using the Tenengrad algorithm, that is, the clarity function S(z i ) is obtained through the sum of the squares of the image gradients, which represents the imaging clarity of each focal plane.

[0135] A multi-plane Kalman filter cluster for modeling, predicting, and updating and correcting the focal point state of each focal plane. The focal point state includes the axial position, focal point velocity, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes;

[0136] In this embodiment, multi-plane Kalman filtering is performed based on the fused multi-modal data to process the interaction between each focal plane. The Kalman filtering cluster will be used to predict the focal point position and eliminate the errors caused by the interference between each focal plane.

[0137] A collaborative optimization decision maker is used to collaboratively optimize the focal point positions of multiple focal planes to determine the final three-dimensional focal point position;

[0138] In this embodiment, the collaborative optimization decision maker uses a multi-objective optimization algorithm (such as Pareto optimization), and according to the output result of the Kalman filtering, optimizes the focusing decision between multiple focal planes to finally determine the accurate three-dimensional focal point position.

[0139] A focal length adjustment module is used to control the objective lens adjustment according to the three-dimensional focal point position to achieve real-time three-dimensional focusing.

[0140] In this embodiment, the focal length adjustment module controls the precise adjustment of the objective lens according to the optimized focal point position to achieve real-time three-dimensional focusing.

[0141] Embodiment III

[0142] This embodiment also provides an electronic device, refer to Figure 2 , including a memory 404 and a processor 402. A computer program is stored in the memory 404, and the processor 402 is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0143] Specifically, the above-mentioned processor 402 may include a central processing unit (CPU), or a specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), or may be configured as one or more integrated circuits for implementing the embodiments of the present invention.

[0144] Among them, the memory 404 may include a mass storage 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 404 may include removable or non-removable (or fixed) media. Where appropriate, the memory 404 may be internal or external to the data processing device. In a particular embodiment, the memory 404 is non-volatile memory. In a particular embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. Where appropriate, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended date out dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0145] The memory 404 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402.

[0146] By reading and executing the computer program instructions stored in the memory 404, the processor 402 implements any one of the three-dimensional autofocus methods based on multi-modal data fusion and Kalman filtering in the above embodiments.

[0147] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. Among them, the transmission device 406 is connected to the above processor 402, and the input / output device 408 is connected to the above processor 402.

[0148] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above network may include wired or wireless networks provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0149] The input / output device 408 is used to input or output information.

[0150] Embodiment 4

[0151] This embodiment also provides a readable storage medium. The readable storage medium stores a computer program, and the computer program includes program codes for controlling a process to execute the process. The process includes the three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to Embodiment 1.

[0152] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated here.

[0153] In general, various embodiments can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the present invention can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, a microprocessor, or other computing devices, but the present invention is not limited thereto. Although various aspects of the present invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as a non-limiting example, the blocks, devices, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or a controller or other computing devices, or some combination thereof.

[0154] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. A computer software or program (also referred to as a program product), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product can include one or more computer-executable components that are configured to perform the embodiments when the program runs. The one or more computer-executable components can be at least one software code or a part thereof. Additionally, in this regard, it should be noted that any box in the logical flow, as Figure 1 described herein, can represent a program step, or interconnected logical circuits, boxes, and functions, or a combination of program steps and logical circuits, boxes, and functions. The software can be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs. The physical media are non-transitory media.

[0155] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as falling within the scope described in this specification.

[0156] The above embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.

Claims

1. A three-dimensional autofocus method based on multimodal data fusion and Kalman filtering, characterized in that It includes the following steps: Collect the three-dimensional phase information of the sample and the image data of different focal planes in real time; Perform normalization processing on the three-dimensional phase information and image data to obtain the phase depth information and clarity metric of each focal plane; Model the focal state of each focal plane based on the multi-plane Kalman filter cluster, where the focal state includes the axial position of the focal plane, focal velocity, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes; Predict the focal state of each current focal plane by fusing the historical focal state and the coupling influence of adjacent focal planes through the multi-plane Kalman filter cluster; Update and correct the predicted focal state using the image clarity metric and phase gradient observation value; Perform collaborative optimization on the focal positions of multiple focal planes through a collaborative optimization decision-making device to determine the final three-dimensional focal position; Control the adjustment of the objective lens according to the three-dimensional focal position to achieve real-time three-dimensional focusing.

2. The three-dimensional autofocus method based on multimodal data fusion and Kalman filtering according to claim 1, wherein The three-dimensional phase information is collected through a microlens array to obtain the phase gradients of the sample in the x-axis and y-axis directions, and the clarity of the image data is obtained by calculating the sum of the squares of the image gradients.

3. The three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to claim 1, characterized in that, The state modeling of the multi-plane Kalman filter cluster includes: defining a state vector containing the axial position, focal velocity, acceleration, phase gradient, and neighborhood coupling coefficient for each focal plane, and constructing an initial error covariance matrix, process noise matrix, and measurement noise matrix.

4. The three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to claim 1, characterized in that When predicting the focal state, the time change of the focal state is described by a state transition matrix, and the interaction between adjacent focal planes is quantified by the neighborhood coupling coefficient.

5. The three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to claim 1, wherein When updating and correcting the focal state, the weights of the predicted value and the measured value are balanced by the Kalman gain, and the measured values include image clarity and phase gradient.

6. The three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to claim 1, wherein The collaborative optimization decision-making device is based on a multi-objective optimization algorithm to perform conflict detection and collaborative optimization on the focal positions of multiple focal planes to eliminate the mutual interference between focal planes.

7. The three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to any one of claims 1-6, characterized in that, It also includes dynamically adjusting the process noise matrix and the coupling coefficient between adjacent focal planes. The process noise matrix is dynamically adjusted according to the image clarity gradient, and the coupling coefficient is updated in real time according to the clarity change between focal planes.

8. A three-dimensional autofocus device based on multimodal data fusion and Kalman filtering, characterized in that, It includes: A phase sensor for collecting the three-dimensional phase information of the sample; An image sensor for collecting the image data of different focal planes and calculating the clarity metric; A multi-plane Kalman filter cluster for modeling, predicting, and updating and correcting the focal state of each focal plane, where the focal state includes the axial position, focal velocity, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes; A collaborative optimization decision-making device for performing collaborative optimization on the focal positions of multiple focal planes to determine the final three-dimensional focal position; A focal length adjustment module for controlling the adjustment of the objective lens according to the three-dimensional focal position to achieve real-time three-dimensional focusing.

9. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, and the computer program includes program codes for controlling a process to execute the process, and the process includes the three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to any one of claims 1 to 7.

Images