Three-dimensional automatic focusing method and device based on multi-modal data fusion and Kalman filtering and readable storage medium thereof

Abstract

The invention provides a three-dimensional automatic focusing method and device based on multi-modal data fusion and Kalman filtering and a readable storage medium, and high-precision real-time focusing is realized through multiple steps: collecting sample three-dimensional phase information and different focal plane image data by using a phase and image sensor, and calculating phase depth and definition; constructing a multi-plane Kalman filtering cluster, modeling the focus state of each focal plane, and predicting the focus by fusing the neighborhood coupling influence; the focus state is updated and corrected in combination with the image definition and the phase gradient, and the optimal three-dimensional focus position is determined by means of a collaborative optimization decision maker; dynamically adjusting the noise matrix and the coupling coefficient, and controlling the objective lens to complete focusing. According to the method, through multi-modal fusion and multi-plane cooperation, the problems of focus drifting and jumping are effectively solved, the focusing precision and stability of an imaging system in a complex sample and a dynamic environment are remarkably improved, and the method is suitable for equipment such as a biomedical full-automatic microscope and a digital pathological section scanner.

Description

Technical Field

[0001] The present invention relates to the field of optical microscopy imaging technology, specifically to the autofocus technology in a microscope system, and particularly to a three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering, which is applicable to scenarios such as biomedical microscopy imaging and digital pathology slide scanning that require high-precision and long-term stable focusing. Background Art

[0002] In recent years, microscopy imaging technology has been widely applied in the biomedical field (such as fully automated microscopes, digital pathology slide scanners). A fully automated microscope needs to automatically observe biological samples for a long time (lasting from several hours to dozens of hours), and a digital pathology slide scanner needs to quickly scan pathology slides to generate whole-slide digital images. Both rely on a stable and reliable autofocus technology to compensate for focus drift caused by factors such as environmental temperature changes and mechanical vibrations to ensure imaging quality.

[0003] Traditional autofocus methods determine the defocus amount by scanning different imaging planes along the optical axis of the objective lens and using an image quality evaluation function (such as a sharpness function). However, such methods have significant defects: 1. Single-modal dependence: Only relying on single-modal data such as sharpness, it is easily interfered by noise and has insufficient focusing accuracy in samples with low contrast (such as transparent biological tissues); 2. Three-dimensional scene failure: For samples with complex three-dimensional structures, due to the relatively deep range of the clear focal plane, the optimal focal position cannot be accurately identified, resulting in focusing failure; 3. Independent modeling of multiple planes: Each focal plane independently predicts the focal position, ignoring the interaction between planes, which easily causes focus jumps or instability; 4. Lack of real-time performance: It is necessary to scan each plane and calculate the evaluation function one by one, which takes a long time and cannot meet the real-time focusing requirements in a dynamic environment.

[0004] With the development of microscopy imaging towards high speed, high precision, and automation, there is an urgent need for an autofocus technology that can fuse multi-modal data, process the coupling effect of multiple planes, and achieve real-time dynamic optimization. Summary of the Invention

[0005] Embodiments of the present invention provide a three-dimensional autofocus method, device, and readable storage medium based on multi-modal data fusion and Kalman filtering, aiming at the problems existing in the current technology that traditional microscopic autofocus methods have poor real-time performance, insufficient multi-plane collaboration, and weak anti-interference ability, and cannot meet the requirements of long-term stable observation of complex three-dimensional samples.

[0006] The core technology of the present invention mainly constructs a multi-plane Kalman filter cluster to fuse three-dimensional phase information and image sharpness data, dynamically predicts and collaboratively optimizes the focus positions of multiple focal planes, realizes high-precision, real-time, and stable three-dimensional autofocus, and solves the problems of focus drift and jump under complex samples and dynamic environments.

[0007] In a first aspect, the present invention provides a three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering, and the method includes the following steps: Collect the three-dimensional phase information of the sample and the image data of different focal planes in real time; Perform normalization processing on the three-dimensional phase information and the image data to obtain the phase depth information and sharpness metrics of each focal plane; Model the focus states of each focal plane based on a multi-plane Kalman filter cluster, where the focus states include the axial position of the focal plane, focus speed, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes; Predict the focus states of the current focal planes by fusing the historical focus states and the coupling effects of adjacent focal planes through a multi-plane Kalman filter cluster; Update and correct the predicted focus states using the image sharpness metric and the phase gradient observation value; Collaboratively optimize the focus positions of multiple focal planes through a collaborative optimization decision-making device to determine the final three-dimensional focus position; Control the adjustment of the objective lens according to the three-dimensional focus position to achieve real-time three-dimensional autofocus.

[0008] Further, the three-dimensional phase information is collected through a microlens array to obtain the phase gradients of the sample in the x-axis and y-axis directions, and the sharpness of the image data is obtained by calculating the sum of the squares of the image gradients.

[0009] Further, the state modeling of the multi-plane Kalman filter cluster includes: defining a state vector containing the axial position, focus speed, acceleration, phase gradient, and neighborhood coupling coefficient for each focal plane, and constructing an initial error covariance matrix, a process noise matrix, and a measurement noise matrix.

[0010] Further, when predicting the focus states, describe the time change of the focus states through a state transition matrix, and quantify the interaction between adjacent focal planes through the neighborhood coupling coefficient.

[0011] Further, when updating and correcting the focus states, balance the weights of the predicted value and the measured value through the Kalman gain, and the measured values include the image sharpness and the phase gradient.

[0012] Further, the collaborative optimization decision-making device is based on a multi-objective optimization algorithm to perform conflict detection and collaborative optimization on the focus positions of multiple focal planes to eliminate the mutual interference between the focal planes.

[0013] Furthermore, it further includes dynamically adjusting the process noise matrix and the coupling coefficient between adjacent focal planes. The process noise matrix is dynamically adjusted according to the image sharpness gradient, and the coupling coefficient is updated in real time according to the sharpness change between the focal planes.

[0014] In a second aspect, the present invention provides a three-dimensional autofocus device based on multi-modal data fusion and Kalman filtering, including: A phase sensor for collecting three-dimensional phase information of a sample; An image sensor for collecting image data of different focal planes and calculating a sharpness metric; A multi-plane Kalman filtering cluster for modeling, predicting, and updating and correcting the focus state of each focal plane. The focus state includes the axial position, focus speed, acceleration, phase gradient observation value, and the coupling coefficient between adjacent focal planes; A cooperative optimization decision maker for cooperatively optimizing the focus positions of multiple focal planes to determine the final three-dimensional focus position; A focal length adjustment module for controlling the objective lens adjustment according to the three-dimensional focus position to achieve real-time three-dimensional autofocus.

[0015] In a third aspect, the present invention provides an electronic device including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the above-mentioned three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering.

[0016] In a fourth aspect, the present invention provides a readable storage medium in which a computer program is stored. The computer program includes program codes for controlling a process to execute the process, and the process includes the above-mentioned three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering.

[0017] The main contributions and innovations of the present invention are as follows: 1. Multi-modal data fusion improves the focus accuracy and robustness. By fusing the three-dimensional phase gradient (reflecting the sample spatial structure) and the image sharpness (traditional single modality), it solves the problem of focus blur caused by the noise sensitivity of a single sharpness evaluation function for low-contrast samples (such as transparent biological tissues), especially improving the focus success rate in weakly textured and transparent samples. 2. Multi-plane collaborative modeling suppresses focus jumping and drifting. By quantifying the interaction between focal planes through the neighborhood coupling coefficient, a multi-plane Kalman filtering cluster is constructed to replace the traditional independent modeling method, eliminating the focus jumping caused by independent prediction of each focal plane, and achieving global optimal focus through a cooperative optimization decision maker to adapt to the three-dimensional complex structure of the sample. 3. Dynamic Adaptive Mechanism to Enhance Environmental Adaptability The process noise matrix of the environment is adjusted dynamically according to the sharpness gradient, and the coupling coefficient is updated in real time with the change of sharpness between focal planes, improving the focusing stability of the system in dynamic environments such as temperature changes and mechanical vibrations, and maintaining high-precision focusing for a long time without manual intervention. 4. Real-time Performance and Efficiency Optimization Based on the state prediction and observation update mechanism of Kalman filtering, the time-consuming process of traditional plane-by-plane scanning is avoided, and the single-frame focusing time is significantly shortened, meeting the real-time focusing requirements of high-speed dynamic samples (such as real-time observation of living cells). 5. Application Scenario Expansion Applicable to high-precision imaging devices such as biomedical fully automatic microscopes and digital pathology slide scanners. Especially in scenarios such as three-dimensional imaging of thick samples and long-term tracking of living cells, it breaks through the dependence of traditional methods on a single focal plane or simple samples, promoting the development of microscopy imaging technology towards automation and intelligence.

[0018] Details of one or more embodiments of the present invention are set forth in the following drawings and description to make other features, objects, and advantages of the present invention more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and form a part of the present invention. The illustrative embodiments and descriptions thereof are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flowchart of a three-dimensional autofocus method based on multimodal data fusion and Kalman filtering according to an embodiment of the present invention; Figure 2 is a schematic hardware structure diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of one or more embodiments detailed in the appended claims of this specification.

[0021] It should be noted that: In other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0022] Traditional microscopic autofocus methods rely on single-modal data, independently process multiple focal planes, and lack real-time collaborative optimization, resulting in insufficient focusing accuracy, being vulnerable to noise interference, and being unable to effectively handle focus drift and jump problems in complex three-dimensional samples and dynamic environments.

[0023] Based on this, the present invention solves the problems existing in the prior art by fusing three-dimensional phase information and image sharpness data based on a multi-plane Kalman filter cluster.

[0024] Embodiment 1 The present invention aims to propose a three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering. Through a multi-plane Kalman filter cluster and phase / sharpness multi-modal data fusion technology, it realizes the collaborative optimization focusing of multiple focal planes of three-dimensional samples, and solves the deficiencies of traditional methods in terms of real-time performance, stability, and accuracy.

[0025] Specifically, the embodiment of the present invention provides a three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering. Specifically, referring to Figure 1 , the method includes the following steps: S1. Real-time collect the three-dimensional phase information of the sample and the image data of different focal planes; In this embodiment, a phase sensor is used to obtain the original light field image I(x, y, z, t) through a microlens array. This sensor can provide the phase gradients Ф x (x, y, z) and Ф y (x, y, z) of the sample along the x-axis and y-axis at different focal planes. These phase information are calculated through the phase gradient:

[0026]

[0027] to represent the spatial structure change of the sample and be used to estimate the preliminary information of the focus position. Here, Δ is the pixel offset, usually set to 1 pixel.

[0028] At the same time, an image sensor is used to parallel collect the image data of each focal plane to obtain the image information of different depth planes.

[0029] For each image plane, the sum of squared gradients is calculated using an improved Tenengrad function to obtain the sharpness measure S(z i ) in the depth direction for each focal plane, reflecting the sharpness of the sample at different focal planes:

[0030] where σ = λ / 4NA is the weight decay coefficient, λ is the wavelength of light, NA is the numerical aperture of the objective lens, x c , y c is the center coordinate of the region, and I is the original light field image.

[0031] The traditional Tenengrad function usually uses the first derivative (such as calculating the sum of squared gradient magnitudes using the Sobel operator), while the present invention uses the sum of squared second derivatives, which is more sensitive to the sharpness of image edges and can more precisely distinguish the edge blurring caused by slight defocus. The traditional method usually sums all image pixels with equal weights, while the present invention introduces Gaussian weighting, focusing on the central region of the image (the core region of the sample in microscopic imaging is usually located at the center of the field of view), reducing the interference of edge noise (such as reflected light and stray light at the edge of the glass slide) on the sharpness evaluation, and improving the robustness in complex environments. The weight decay coefficient σ is directly related to the optical system parameters (λ, NA), making the sharpness measure conform to the actual imaging physical characteristics (such as the influence of the diffraction limit), avoiding the adjustment of empirical parameters, and achieving automatic adaptation to the microscopic objective lens.

[0032] S2. Standardize the three-dimensional phase information and image data to obtain the phase-depth information and sharpness measure for each focal plane; In this embodiment, phase-depth mapping: Calculate the phase depth d i :

[0033] where d i represents the phase depth of focal plane i, representing the three-dimensional structural information of the sample at this focal plane (directly related to the focal position); λ is the wavelength of the incident light (unit: μm), determined by the light source of the microscopic system.

[0034] Sharpness normalization: Normalize the sharpness S(z i ) to S norm,i :

[0035] Among them, min(S) and max(S) are respectively the minimum and maximum values of the clarity of all current focal planes, ensuring that the normalization is based on the local data range (rather than a global fixed threshold) and adapting to the dynamic changes of different samples and environments. In this way, the clarity normalization solves the problem of inconsistent scales in multi-modal data fusion through a simple and effective linear transformation, enabling the clarity information to be seamlessly combined with phase data, Kalman filter state variables, etc. This step seems basic, but it is a key link for the present invention to achieve high-precision focusing - it ensures that subsequent complex algorithms such as Kalman filtering and collaborative optimization can correctly process data from different sources, and finally achieve stable and reliable autofocus in dynamic environments and complex samples.

[0036] S3. Model the focus states of each focal plane based on a multi-plane Kalman filter cluster, where the focus states include the axial position of the focal plane, focus velocity, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes; In this embodiment, the specific steps of modeling are as follows: 1. Initialize the Kalman filter model Set the initial state of the Kalman filter that defines the state vector for each focal plane as:

[0037] Among them, is the axial position of the focal plane; is the initial focus velocity; is the initial acceleration; is the phase gradient observation value in the x direction (calculated by the difference between adjacent pixels of the light field image, that is, representing the phase difference of the focal plane i in the x direction), reflecting the lateral defocus offset, unit: rad / pixel; is the phase gradient observation value in the y direction (representing the phase difference of the focal plane i in the y direction), reflecting the longitudinal defocus offset, unit: rad / pixel; is the neighborhood coupling coefficient, representing the influence of a single adjacent focal plane, dimensionless.

[0038] Set the initial error covariance matrix for estimating the accuracy of the prediction process:

[0039] Among them , and are respectively the initial estimation errors of the focus position and focus velocity, , are respectively the initial estimation errors of the phase sensor in the x and y directions, is the coupling coefficient error between multi-planes, representing the interaction between each focal plane.

[0040] Set the process noise matrix of the Kalman filter parameters:

[0041] Used to describe the uncertainty in the model prediction process, where , and respectively represent the process noise of the focus position and velocity, , respectively represent the process errors of the phase data in the x and y directions, is the process error of the multi-plane coupling coefficient. The noise matrix will be dynamically adjusted according to the magnitude of the clarity gradient to ensure that the system can cope with rapid changes.

[0042] Set the measurement noise matrix , used to describe the noise when measuring the clarity of the image. The clarity noise is negatively correlated with the score, and the tolerance is higher at low signal-to-noise ratios. and respectively represent the noise of the phase data.

[0043] 2. Multi-plane Kalman filter cluster prediction model Estimate the current focus state based on the historical focus position and velocity:

[0044] where, is the predicted state vector of the focal plane i at the current moment (including position, velocity, acceleration, phase gradient, coupling coefficient, etc.); is the historical state vector of the focal plane i at the previous moment; A i is the state transition matrix, which describes the change of the focus from the previous moment to the current moment; B i u i is the control input term, which represents the influence of the external control signal on the state (for example, the control signal of the electric focusing mechanism), B i is the control matrix, u i is the external control signal (such as the displacement command of the electric focusing mechanism); γ is the global elastic coefficient, C ij is the coupling coefficient between the focal planes i and j, which represents the interaction between the focal planes; , are the axial positions (focus coordinates) of the focal planes i and j at the previous moment.

[0045] This formula is the core mathematical carrier of the "multi-plane collaborative focusing" technology of the present invention. By introducing a neighborhood coupling mechanism, it fundamentally improves the accuracy and stability of the autofocus system in three-dimensional complex samples and dynamic environments, representing a revolutionary improvement over traditional single-modal and independent modeling methods.

[0046] 3. Update the multi-plane Kalman filter cluster model Kalman gain Used to balance the weights between the predicted value and the measured value: , where: is the predicted error covariance matrix; is the measurement matrix, used to associate the state variables with the measurement data; is the measurement noise matrix; T represents the transpose of the matrix, that is, the operation of interchanging the rows and columns of the measurement matrix is performed.

[0047] Observation matrix design:

[0048] 80% of the weight is given to sharpness, and 20% is given to phase (10% is evenly distributed to each of the two phase components), balancing accuracy and real-time performance.

[0049] 4. Collaborative optimization decision Conflict detection and modeling: First, impose a spacing constraint, => Trigger optimization, and then optimize the objective function:

[0050] ADMM fast solution: Augmented Lagrangian function:

[0051] Among them, p i , p j are the axial positions (focal coordinates) of the i-th and j-th focal planes; is the minimum focal plane spacing resolvable by the optical system (derived from the Rayleigh criterion, NA is the numerical aperture of the objective lens, and λ is the light wavelength), d min essentially represents the axial resolution limit of the optical system. When the focal plane spacing is greater than this value, the mutual influence can be ignored and no collaboration is required; when it is less than this value, the coupling effect must be modeled; z i is the focal position of the i-th focal plane after optimization (decision variable); is the position of the i-th focal plane predicted by the Kalman filter; ρ is the augmented Lagrangian penalty coefficient, which controls the consistency strength between z and u and can be set to 1; z is the vector set of the target positions of all focal planes.

[0052] ADMM (Alternating Direction Method of Multipliers) iterative update (converges in 3 times):

[0053] Among them, z is the optimization variable (the focal plane focus position vector, with dimension N, where N is the number of focal planes); u is the auxiliary variable (strongly correlated with z, used to separate the constraint terms, usually set u = z); v is the Lagrange multiplier (reflecting the tightness of the constraint conditions, with the same dimension as z); A and b are the coefficient matrix and the constant term (from the linear part of the objective function, and the specific form depends on the problem modeling); ρ>0 is the penalty parameter (controlling the cost of constraint violation, usually increasing with iteration); (∙) is the projection operator (projecting the result into the constraint set C); I is the identity matrix (ensuring the feasibility of matrix inversion).

[0054] This system of equations transforms the multi-focal plane collaborative optimization problem into a sub-problem that can be efficiently solved through a cycle of unconstrained optimization → constraint projection → multiplier update, ensuring global optimal focusing under optical constraints.

[0055] S4. Predict the focus states of current focal planes by fusing historical focus states and the coupling effects of adjacent focal planes through multi-plane Kalman filter clustering; S5. Update and correct the predicted focus states using image sharpness metrics and phase gradient observations; In this embodiment, the specific steps are as follows: 1. Phase-Kalman depth fusion Fuse the phase data with the prediction results of the Kalman filter to correct the predicted position of the focus. Further adjust the focus position by weighting the phase gradient information (i.e., the sensitivity of phase change). The formula for phase correction is:

[0056] Among them, is the focus position predicted by the Kalman filter, is the focus position at the previous moment. k φ is the phase sensitivity coefficient, indicating the sensitivity of focus change to phase gradient change, usually obtained through system calibration. and are the actually observed phase gradient and the predicted phase gradient respectively.

[0057] At the same time, use reverse prediction guidance. To reduce the computational burden and improve the real-time performance of the system, the area near the focus can be deeply analyzed, define a ROI, and without affecting the accuracy, greatly reduce the computational amount and improve the system response speed:

[0058] ROI optimization can improve the computational efficiency and reduce the ineffective phase calculations by introducing an adaptive region. and is the predicted focus position, and σp is the standard deviation of the position estimate, which reflects the accuracy of the focus prediction.

[0059] 2. Dynamic parameter adjustment The process noise matrix is used to describe the errors that cannot be fully measured in the system (such as the minute changes caused by sample movement). Since the motion characteristics of the sample may vary under different environmental conditions, the noise level of the system will also change. Here, the tanh function is used to smoothly transition the process noise and dynamically adjust the threshold:

[0060] where is the sharpness gradient on the current focal plane, which reflects the degree of change of the current focus. θ is the threshold for noise adjustment, and δ is the smoothing coefficient, which is used to control the adaptive response speed of the noise.

[0061] The elastic coefficient γk controls the elastic coupling effect between the focal planes and is used to describe the mutual influence between adjacent focal planes. In a dynamic system, when the focus changes significantly (for example, when the focal plane moves relatively fast), the system may encounter the problem of over-tracking, so it is necessary to reduce the elastic coefficient to make the system more stable.

[0062]

[0063] where v max is the maximum speed that the system can track, Δt is the time step, is the change in the focus position, which represents the magnitude of the current focus offset.

[0064] In addition, during the multi-focal plane focusing process, there is a certain coupling effect between the focal planes. When the focal length of one focal plane changes, it will affect the image quality of other focal planes. Especially in a microscope system, the mutual influence (or called coupling effect) between the focal planes is significant. For example, adjusting the position of focal plane 𝑖 may cause a change in the imaging quality of focal plane 𝑗. Therefore, it is necessary to adjust the coupling coefficient C between each focal plane in real time according to the observed sharpness data and phase gradient ij During each round of optimization process, the system needs to dynamically adjust these coefficients to make the relationship between multiple focal planes optimal and avoid focus jumps or instability.

[0065] The formula for updating the coupling coefficient is:

[0066] is the coupling coefficient between focal planes i and j in the current iteration. η is the adaptive learning rate, which controls the step size of coefficient update. is the gradient of the clarity function of focal plane i with respect to the position of focal plane j, reflecting the influence of focus change on the clarity of other focal planes. γ is a parameter that controls the decay of the coefficient and is used to suppress over-adjustment. The update mechanism of the coupling coefficient enables the system to gradually adjust the mutual relationship between each focal plane according to the changes in the clarity function and phase gradient in each iteration.

[0067] S6. Co-optimize the focus positions of multiple focal planes through a co-optimization decision maker to determine the final three-dimensional focus position; In this embodiment, the focus adjustment signal received by the final controller is specifically an output generated by a co-optimization decision maker and a Kalman filter system, which is used to precisely control the position adjustment of the microscope objective to achieve the best focus. This signal is based on various information such as the mutual influence between multiple focal planes, focus position prediction, clarity evaluation, and phase data correction:

[0068] where p final is the finally determined focus position of focal plane i; is the focus position predicted by the Kalman filter cluster (based on historical states and neighborhood coupling effects); is the focus position correction amount (fusing multi-source information such as clarity evaluation and phase data correction).

[0069] The final focus position p final is the comprehensive result of the mutual influence between focal planes (cooperative prediction driven by the coupling coefficient), Kalman filter dynamic prediction, fusion and correction of clarity and phase data, and optical constraint forced optimization. This mechanism breaks through the limitation of traditional single-focal-plane independent processing and realizes high-precision and stable autofocus under complex three-dimensional samples through the spatio-temporal fusion of multi-source information and physical constraint guidance, which is an important progress in microscopic imaging technology from "single-modal trial and error" to "multi-dimensional intelligent optimization".

[0070] S7. Control the adjustment of the objective lens according to the three-dimensional focus position to achieve real-time three-dimensional focusing.

[0071] Embodiment 2 Based on the same concept, the present invention also proposes a three-dimensional autofocus device based on multi-modal data fusion and Kalman filter, including: A phase sensor for collecting three-dimensional phase information of the sample; In this embodiment, the phase sensor collects the light field phase data containing multiple planes (focal planes) through a microlens array. This phase data is Ф x(x, y, z) and Ф y (x, y, z) respectively represent the phase changes of the sample in the x-axis and y-axis directions.

[0072] An image sensor, configured to collect image data of different focal planes and calculate a sharpness metric; In this embodiment, the sharpness metric is calculated using the Tenengrad algorithm, that is, the sharpness function S(z i ) is obtained through the sum of squares of image gradients, which represents the imaging sharpness of each focal plane.

[0073] A multi-plane Kalman filter cluster, configured to model, predict, and update and correct the focus states of each focal plane, where the focus states include axial position, focus velocity, acceleration, phase gradient observation values, and coupling coefficients between adjacent focal planes; In this embodiment, multi-plane Kalman filtering is performed based on the fused multi-modal data to handle the interactions between each focal plane. The Kalman filter cluster will be used to predict the focus position and eliminate errors caused by interference between each focal plane.

[0074] A collaborative optimization decision maker, configured to collaboratively optimize the focus positions of multiple focal planes to determine the final three-dimensional focus position; In this embodiment, the collaborative optimization decision maker uses a multi-objective optimization algorithm (such as Pareto optimization), and according to the output result of the Kalman filter, optimizes the focusing decision between multiple focal planes to finally determine the accurate three-dimensional focus position.

[0075] A focal length adjustment module, configured to control the adjustment of the objective lens according to the three-dimensional focus position to achieve real-time three-dimensional focusing.

[0076] In this embodiment, the focal length adjustment module controls the precise adjustment of the objective lens according to the optimized focus position to achieve real-time three-dimensional focusing.

[0077] Embodiment III This embodiment also provides an electronic device, refer to Figure 2 , including a memory 404 and a processor 402, where a computer program is stored in the memory 404, and the processor 402 is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0078] Specifically, the above-mentioned processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC for short), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0079] Among them, the memory 404 may include a mass storage 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 404 may include removable or non-removable (or fixed) media. In a suitable case, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). In a suitable case, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory (FLASH), or a combination of two or more of these. In a suitable case, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0080] The memory 404 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402.

[0081] By reading and executing the computer program instructions stored in the memory 404, the processor 402 implements any one of the three-dimensional autofocus methods based on multi-modal data fusion and Kalman filtering in the above embodiments.

[0082] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. Among them, the transmission device 406 is connected to the above processor 402, and the input / output device 408 is connected to the above processor 402.

[0083] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above network may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0084] The input / output device 408 is used to input or output information.

[0085] Embodiment 4 This embodiment also provides a readable storage medium. The readable storage medium stores a computer program, and the computer program includes program code for controlling a process to execute the process. The process includes the three-dimensional autofocus method based on multi-modal data fusion and Kalman filtering according to Embodiment 1.

[0086] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be repeated here.

[0087] Generally, various embodiments can be implemented in hardware or special circuits, software, logic, or any combination thereof. Some aspects of the present invention can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, a microprocessor, or other computing devices, but the present invention is not limited thereto. Although various aspects of the present invention can be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as a non-limiting example, the blocks, devices, systems, technologies, or methods described herein can be implemented in hardware, software, firmware, special circuits or logic, general hardware or a controller, or other computing devices, or some combination thereof.

[0088] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. A computer software or program (also referred to as a program product), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product can include one or more computer-executable components configured to perform the embodiments when the program runs. One or more computer-executable components can be at least one software code or a part thereof. Additionally, at this point, it should be noted that any box in the logical flow, as Figure 1 shown, can represent a program step, or interconnected logical circuits, boxes, and functions, or a combination of program steps and logical circuits, boxes, and functions. The software can be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs. The physical media is a non-transitory medium.

[0089] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0090] The above embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.

Claims

1. A three-dimensional autofocus method based on multimodal data fusion and Kalman filtering, characterized in that: The following steps are involved: Collect the three-dimensional phase information of the sample and image data of different focal planes in real time; Standardizing the three-dimensional phase information and image data to obtain phase depth information and clarity measurement of each focal plane; Modeling the focus state of each focal plane based on a multi-plane Kalman filter cluster, wherein the focus state includes the focal plane axial position, focal velocity, acceleration, phase gradient observation value, and coupling coefficient between adjacent focal planes; The multi-plane Kalman filter cluster fuses the historical focus state with the coupling influence of the adjacent focal planes to predict the current focus state of each focal plane; The predicted focus state is updated and corrected using image clarity metrics and phase gradient observations; The focal positions of multiple focal planes are collaboratively optimized through a collaborative optimization decision maker to determine the final three-dimensional focal position; The objective lens adjustment is controlled according to the three-dimensional focal position to achieve real-time three-dimensional focusing.

2. The three-dimensional auto-focusing method based on multimodal data fusion and Kalman filtering as claimed in claim 1, characterized in that: The three-dimensional phase information is collected through a microlens array to obtain the phase gradient of the sample in the x-axis and y-axis directions. The clarity of the image data is obtained by calculating the square sum of the image gradient.

3. The three-dimensional auto-focusing method based on multimodal data fusion and Kalman filtering as claimed in claim 1, characterized in that: The state modeling of the multi-plane Kalman filter cluster includes: defining a state vector including axial position, focal velocity, acceleration, phase gradient and neighborhood coupling coefficient for each focal plane, and constructing an initial error covariance matrix, a process noise matrix and a measurement noise matrix.

4. The three-dimensional auto-focusing method based on multimodal data fusion and Kalman filtering as claimed in claim 1, characterized in that: When predicting the focus state, the temporal variation of the focus state is described by the state transfer matrix, and the interaction between adjacent focal planes is quantified by the neighborhood coupling coefficient.

5. The three-dimensional auto-focusing method based on multimodal data fusion and Kalman filtering as claimed in claim 1, characterized in that: When updating the corrected focus state, the Kalman gain is used to balance the weights of the predicted values ​​with the measured values, which include image sharpness and phase gradient.

6. The three-dimensional auto-focusing method based on multimodal data fusion and Kalman filtering as claimed in claim 1, characterized in that: The collaborative optimization decision maker performs conflict detection and collaborative optimization on the focal positions of multiple focal planes based on a multi-objective optimization algorithm to eliminate mutual interference between focal planes.

7. The three-dimensional auto-focusing method based on multimodal data fusion and Kalman filtering according to any one of claims 1 to 6, characterized in that: It also includes dynamically adjusting a process noise matrix and a coupling coefficient between adjacent focal planes, wherein the process noise matrix is ​​dynamically adjusted according to an image clarity gradient, and the coupling coefficient is updated in real time according to a clarity change between focal planes.

8. A three-dimensional autofocus device based on multimodal data fusion and Kalman filtering, characterized in that: include: A phase sensor, used to collect three-dimensional phase information of the sample; An image sensor for collecting image data at different focal planes and calculating sharpness metrics; A multi-plane Kalman filter cluster for modeling, predicting, and updating the focus state of each focal plane, including axial position, focus velocity, acceleration, phase gradient observations, and coupling coefficients between adjacent focal planes; A collaborative optimization decision maker, used to collaboratively optimize the focal positions of multiple focal planes to determine the final three-dimensional focal position; The focal length adjustment module is used to control the adjustment of the objective lens according to the three-dimensional focal position to achieve real-time three-dimensional focusing.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the three-dimensional autofocus method based on multimodal data fusion and Kalman filtering according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes a three-dimensional autofocus method based on multimodal data fusion and Kalman filtering according to any one of claims 1 to 7.

Images