A positioning method for three-dimensional real-time tracking of micro-scale targets in a bright-field microscopic environment
By using YOLO and residual network models to process the two-dimensional position and defocus height information of microscale targets in a bright-field microscopic environment, the problem of three-dimensional real-time tracking of microscale targets in complex backgrounds was solved, and stable and continuous three-dimensional positioning was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHWEST UNIV
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-26
AI Technical Summary
In complex bright-field microscopic imaging environments, the imaging differences between microscale targets and background pseudostructures are not significant. Traditional methods are prone to false detection and false negative detection, resulting in unstable three-dimensional positioning. Furthermore, the high-speed movement of microscopic particles makes them prone to defocusing or running out of the field of view, causing the tracking process to be interrupted.
A target detection model based on the YOLO architecture and a defocus regression model with a residual network structure are used to process the two-dimensional position and defocus height information of micro-scale targets, respectively, and three-dimensional real-time tracking is achieved through temporal correlation of continuous image frames.
It can stably output the two-dimensional position of the target in complex backgrounds, reduce the risk of false detection and drift, realize the three-dimensional real-time stable tracking of micro-scale targets, and improve the reliability and continuity of positioning.
Smart Images
Figure CN122089780A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition and reconstruction technology, specifically a method for real-time three-dimensional tracking and localization of microscale targets in a bright-field microscopic environment. Background Technology
[0002] In the fields of biophysics, microbiology, and microfluidics, tracking and locating the motion of microscale targets such as bacteria and microparticles in three-dimensional space is an important technique for studying their dynamic characteristics and interactions with complex environments. In practical applications, the Lagrange tracking method is typically used. This involves adjusting the imaging field of view in real time during the imaging process to keep the target focused within the field of view and near the focal plane, thereby obtaining a long-term continuous record of the trajectory of a single target.
[0003] However, in complex bright-field microscopy environments, the low contrast inherent in bright-field microscopy, coupled with various interference factors such as stray light, background textures, and floating particles, often results in insignificant imaging differences between microscale targets and background pseudostructures. Traditional methods based on threshold segmentation, template matching, or feature extraction are prone to false detections and false negatives under complex background conditions. This is especially true when the target is moving rapidly or when image quality fluctuates, leading to unstable two-dimensional planar position recognition and directly impacting the reliability of subsequent three-dimensional localization.
[0004] Furthermore, during Lagrange tracking, some microscopic particles move at high speeds in three-dimensional space, making them prone to defocusing or rapidly deviating from the imaging field of view. Once the target leaves the focal plane or runs out of the field of view, it will be difficult to accurately identify the target in subsequent images, thus forcing the tracking process to be interrupted. Summary of the Invention
[0005] (a) Technical problems to be solved This invention provides a method for real-time three-dimensional tracking and localization of microscale targets in a bright-field microscopic environment, achieving stable three-dimensional tracking of microscopic targets without the need for complex optical modifications.
[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a method for real-time three-dimensional tracking and localization of microscale targets in a bright-field microscopic environment, comprising the following steps: In a bright-field microscopic imaging optical system, a sequence of microscopic image frames is continuously acquired within the target spatial region; Each frame of the microscopic image is input into a target detection model based on the YOLO architecture. The target detection model is used to locate and output the two-dimensional position information of the microscale target in the frame of the microscopic image by distinguishing between background pseudostructures and microscale targets in a bright field background. Based on the two-dimensional position information of the micro-scale target, a local image region centered on the micro-scale target is cropped from the corresponding frame microscopic image as a fine ROI; The refined ROI is input into the defocus regression model based on the residual network structure. The defocus regression model extracts the grayscale and blur features of the microscale target under different defocus states and regresses and outputs the defocus height and corresponding defocus direction information of the microscale target relative to the current focal plane. The two-dimensional position information of the micro-scale target in the image plane is fused with the defocus axis information of the corresponding frame to obtain the three-dimensional spatial position of the micro-scale target at the current moment, and the three-dimensional position is reconstructed under single-frame conditions. Based on the two-dimensional position information output in each frame, the two-dimensional moving platform is driven to perform planar position compensation so that the micro-scale target is kept in the center of the field of view. At the same time, based on the defocus height and defocus direction information output in the corresponding frame, the focus compensation device is driven to adjust the focal plane position. In a continuous sequence of microscopic image frames, a temporal correlation mechanism is introduced. The three-dimensional position of the microscale target acquired in the previous frame is used as a priori constraint to generate a candidate ROI for the next frame. The candidate ROI is then input into the target detection model for identification and localization. By performing temporal consistency correlation on the reconstructed three-dimensional positions in the continuous frames, a continuous three-dimensional position sequence of the microscale target in the target space region is formed, and the real-time three-dimensional motion trajectory of the microscale target is output.
[0007] In some feasible embodiments, during the training process of the target detection model and the defocus regression model, microscopic images of target samples at different axial positions are collected, and the two-dimensional position information, defocus height, and defocus direction of each target sample are labeled. At the same time, brightness unevenness, illumination fluctuations, and particulate impurities in the bright field background are distinguished and labeled to construct a unified training dataset. The target detection model and the defocus regression model are then trained under supervision so that they can learn the target discrimination and axial imaging feature change rules, respectively.
[0008] In some feasible embodiments, in the bright-field microscopy imaging optical system, a camera and a microscope objective are configured to simultaneously perform continuous imaging of the target spatial region to acquire a sequence of microscopic image frames; each frame of microscopic image maintains a consistent resolution and optical magnification.
[0009] In some feasible embodiments, each frame of the microscopic image is input into a target detection model based on the YOLO architecture. The target detection model is used to perform multi-scale convolutional feature extraction on a single frame of microscopic image under background interference conditions such as uneven brightness, illumination fluctuations, and particulate impurities, in order to distinguish microscale targets from background pseudostructures. Subsequently, regression prediction is performed on the detected microscale targets, and their two-dimensional position information in the image coordinate system and the corresponding detection confidence are output. The two-dimensional location information is used to crop a local image region centered on a micro-scale target in the current frame, forming a fine ROI containing only a single micro-scale target, and the fine ROI is input into a defocus regression model based on a residual network structure.
[0010] In some feasible embodiments, the defocus regression model utilizes a residual network to perform deep feature extraction on the image corresponding to the fine ROI region of the input image; wherein, the residual blocks in the residual network capture the grayscale distribution changes, edge blurring degree and morphological structure differences of the micro-scale target under different defocus states layer by layer, thereby establishing a mapping relationship between defocus imaging features and axial position deviation. Based on the extracted defocus imaging features, the defocus regression model regresses and outputs the defocus height and corresponding defocus direction information of the microscale target relative to the current focal plane; wherein, the defocus height and direction information are used to fuse with the two-dimensional position information to realize the position reconstruction of the microscale target in the corresponding frame in three-dimensional space, and as a feedback control quantity to drive the focus adjustment device to dynamically adjust the focal plane.
[0011] In some feasible embodiments, the two-dimensional position information of the micro-scale target identified in each frame is used to drive the microscope in the two-dimensional moving platform to perform planar position compensation, so that the micro-scale target is always kept in the center of the microscope's field of view; at the same time, based on the defocus height and defocus direction information of the corresponding frame, the focusing mechanism is driven to dynamically adjust the focal plane of the microscope, so that the micro-scale target is focused near the focal plane in real time.
[0012] In some feasible embodiments, during the 3D position reconstruction of microscale targets, for each frame of a microscopic image, the 2D position information output by the target detection and the defocus height information output by the defocus regression model are combined to calculate the coordinates of the microscale target in 3D space, forming single-frame 3D position data; for a scene with multiple microscale targets in the microscopic image frame sequence, the 3D coordinates of each microscale target are independently reconstructed to achieve 3D localization of multiple targets within the microscopic image frame sequence; specifically, the following steps are performed: Target association is performed on a continuous sequence of microscopic image frames. The three-dimensional coordinates of the microscale target located in the previous frame are used as a priori reference to determine the region where the microscale target may appear in the next frame. Using the two-dimensional location information of the micro-scale target in the previous frame as the center, candidate ROIs are cropped out in the next frame image; The candidate ROI is input into the target detection model for identification and localization, and the two-dimensional position information of the target in the next frame is output. The reconstructed 3D positions of each frame are correlated in chronological order to form a continuous 3D motion trajectory of the microscale target.
[0013] In some feasible embodiments, during the acquisition of training data for the target detection model and the defocus regression model, multiple imaging acquisitions are performed on the microscale target under the microscopic imaging system. Specifically, at each known axial position, a corresponding microscopic image frame is acquired to form an axially continuous image stack. Each set of the image stacks corresponds to the imaging results of the same microscale target under different defocus states.
[0014] In some feasible embodiments, for each set of image stacks, the following annotation operation is performed on each frame of the micrograph: In each frame of the microscopic image, the target location region corresponding to the microscale target in the image plane is labeled as the supervision signal of the target detection model; for multi-microscale target scenes, independent labels are assigned to each target. By combining the control parameters of the axial displacement mechanism of the two-dimensional driving platform, the defocus height value of the axial offset relative to the focal plane is marked for each frame of microscopic image, and the defocus direction information is used as the continuous numerical supervision target of the defocus regression model.
[0015] (III) Beneficial Effects: Compared with the prior art, this invention has the following beneficial effects: This invention inputs continuous microscopic images into a target detection model and explicitly introduces distinguishing annotations for interference types such as uneven brightness, illumination fluctuations, and particulate impurities during the training phase. This enables the model to distinguish between real targets and background pseudo-structures, thereby stably outputting the two-dimensional position of the target even under complex bright-field imaging conditions, reducing the risk of false detection and drift.
[0016] The defocus regression model based on fine ROI does not rely on multi-view or multi-focal plane scanning, transforming axial position estimation from indirect judgment to quantitative prediction, providing continuous and usable input for real-time focus compensation control.
[0017] By separating 2D target detection and defocus regression into two functionally distinct models and using ROI clipping as the connection interface, the training complexity caused by a single model simultaneously undertaking spatial localization and axial regression is avoided. The 2D position information of each frame is fused with the corresponding defocus axial information, and the position information of the previous frame is introduced as a prior using a temporal association mechanism to achieve continuous association of micro-scale targets in the temporal dimension, thereby stably reconstructing the 3D position of the target and outputting a smooth real-time motion trajectory. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating a method for real-time three-dimensional tracking and localization of microscale targets in a bright-field microscopic environment, as provided in an embodiment of the present invention. Figure 2This is a schematic diagram of the sequence of microscopic image frames acquired in a localization method for real-time three-dimensional tracking of microscale targets in a bright-field microscopic environment provided in an embodiment of the present invention. Figure 3 This is a flowchart illustrating the ROI in a method for real-time three-dimensional tracking and localization of microscale targets in a bright-field microscopic environment, as provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the target detection model and the defocus regression model in a localization method for real-time three-dimensional tracking of microscale targets in a bright-field microscopic environment provided in an embodiment of the present invention. Figure 5 In a localization method for real-time three-dimensional tracking of microscale targets in a bright-field microscopic environment provided by an embodiment of the present invention, a regression scatter plot of the actual defocus position and the model-predicted defocus position is shown. Figure 6 The regression error distribution diagram of the actual defocus position and the model-predicted defocus position in a localization method for real-time three-dimensional tracking of microscale targets in a bright-field microscopic environment provided in an embodiment of the present invention. Figure 7 This is a schematic diagram illustrating the relationship between prediction error and actual defocus position in a localization method for real-time three-dimensional tracking of microscale targets in a bright-field microscopic environment, as provided in an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0020] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.
[0021] Combination Figures 1 to 7 This paper presents a method for real-time three-dimensional tracking and localization of microscale targets in a bright-field microscopic environment. Under bright-field microscopic imaging conditions, this method first continuously acquires time-series microscopic images containing microscale targets. (See reference below.) Figure 1 Specifically, S1 involves continuously acquiring a sequence of microscopic image frames within the target spatial region in a bright-field microscopic imaging optical system.
[0022] It is important to note that this stage does not involve any additional modifications to the imaging system. Instead, it utilizes a conventional CCD / CMOS camera to acquire continuous frame data. The core objective is to preserve the continuity of the target's motion over time, providing a foundation for subsequent temporal correlation and trajectory reconstruction.
[0023] Specifically, firstly, full-field bright-field grayscale images of bacteria are acquired using a microscope and camera. The acquired full-field images are then initially cropped to target candidate ROIs with a pixel size of 640 x 640. (See reference below.) Figure 2 .
[0024] The direct input of a single-frame microscopic image into a YOLO-based object detection model is based on two considerations: On the one hand, targets in bright-field micrographs are usually small-scale structures with low contrast and irregular shapes, and the background is often accompanied by uneven brightness, granular impurities and noise textures, which are difficult to distinguish stably by traditional threshold-based or morphological methods. On the other hand, single-stage detection networks like YOLO have advantages in multi-scale feature extraction and real-time inference, and can directly regress the two-dimensional position of the target in complex backgrounds.
[0025] The acquired full-view image is initially cropped to a target candidate ROI with a pixel size of 640 x 640. This cropping process can be performed based on the first frame image with the target pixel coordinates specified by the user as the center, or the first frame can be used for automatic target recognition of the entire field of view. The process of cropping the target candidate ROI is then performed continuously after the second frame.
[0026] The candidate ROI is input into the YOLOv8-Nano-based object detection model described above. This model is trained under supervision and learns to accurately extract the semantic features of bacteria in noisy backgrounds and low-contrast situations. This enables it to identify and locate microscale targets in complex backgrounds in two dimensions. The output bounding box provides the two-dimensional centroid coordinates of the bacteria in the image coordinate system. , In this embodiment, the size of the candidate ROI is set to 640×640; however, the present invention is not limited to this, and the specific size of the ROI can be adjusted according to the actual application scenario.
[0027] By introducing samples with complex backgrounds and various types of interference during the training phase, the object detection model learns the differences between the target and the background in statistical features and spatial structure, rather than relying on fixed rules. This allows the model to stably output the two-dimensional position of the target in the image coordinate system during the inference phase, even if lighting conditions change or the background structure becomes complex, providing reliable input for subsequent steps.
[0028] After obtaining the target's two-dimensional position information, instead of directly estimating the axial position on the entire image, a local image region is cropped from the original microscopic image using these two-dimensional coordinates as the center, forming a refined ROI. Specifically, S2 involves inputting each frame of the microscopic image into a target detection model based on the YOLO architecture. This model, against a bright-field background, distinguishes between background pseudo-structures and microscale targets, locating and outputting the two-dimensional position information of the microscale target in that frame of the microscopic image. S3 involves cropping a local image region centered on the microscale target from the corresponding frame of the microscopic image based on the microscale target's two-dimensional position information, serving as the refined ROI. These two steps are for spatial constraints and task decoupling, and are referenced here. Figure 3 .
[0029] Specifically, based on the two-dimensional centroid coordinates output by the object detection model, a local image region of 224 x 224 pixels centered on the object is further cropped out, called a fine ROI, to further reduce the computational overhead per frame.
[0030] This refined ROI preserves all the details of the bacterial point spread function, including the structure and grayscale features of the diffraction spots. The refined ROI is then input into the second-stage defocus regression model based on the ResNet-18 residual network structure. Specifically, in S4, the refined ROI is input into the defocus regression model based on the residual network structure. The defocus regression model extracts the grayscale and blur features of the micro-scale target under different defocus states and regresses the defocus height and corresponding defocus direction information of the micro-scale target relative to the current focal plane.
[0031] The algorithm outputs the defocus height of the bacteria relative to the focal plane and the corresponding defocus direction information through a single forward calculation. Among them, the defocus height value It can be a positive or negative value to indicate whether the bacteria are located above or below the focal plane.
[0032] In the closed-loop control logic, based on the two-dimensional centroid coordinates of the bacteria ( , The defocus height and direction information obtained from regression. The required planar position compensation and axial focus compensation are calculated, and a displacement command is generated and sent to the displacement stage. The displacement stage is driven to move in two dimensions (XY) to perform planar position compensation, and the target is moved back to the center of the field of view. At the same time, the displacement stage is driven to move axially to adjust the position of the focal plane of the microscopic imaging system and the relative position of the bacteria, so as to refocus the defocused target back to the focal plane and achieve automatic focusing.
[0033] Building upon this, the following steps are performed: S5: The two-dimensional position information of the micro-scale target within the image plane is fused with the defocus axis information of the corresponding frame to obtain the three-dimensional spatial position of the micro-scale target at the current moment, thus reconstructing the three-dimensional position under single-frame conditions. S6: Based on the two-dimensional position information output in each frame, the two-dimensional motion platform is driven to perform planar position compensation, keeping the micro-scale target centered in the field of view. Simultaneously, based on the defocus height and defocus direction information output in the corresponding frame, the focus adjustment device is driven to adjust the focal plane position.
[0034] Specifically, based on the two-dimensional centroid coordinates ( , (and defocus height and direction information) The three-dimensional spatial location of the bacteria is reconstructed, and the nearest neighbor matching algorithm is used to correlate the continuous time series frame data to output the real-time three-dimensional motion trajectory of the bacteria.
[0035] Furthermore, during continuous tracking, the two-dimensional centroid coordinates of the bacteria acquired in the previous frame are used ( , Using this as prior information, a new target candidate ROI is generated centered on this location, and this candidate ROI is used as the input of the target detection unit in the next frame, thus forming a closed-loop tracking process to achieve continuous, real-time, and stable tracking of bacteria.
[0036] In principle, the defocus information of micro-scale targets is mainly reflected in the gray-scale distribution near the target, the degree of edge blurring, and local texture changes. These features do not exist in the background far from the target area. By cropping a fine ROI, the spatial range of the input information can be significantly compressed, reducing the interference of irrelevant background on the model.
[0037] From an execution logic perspective, this step explicitly separates the questions of whether something is a target from how much it is out of focus. The target detection model is only responsible for finding the target location in complex backgrounds, while the subsequent defocus regression model only works within a small area where the target is known to exist. This design reduces the learning difficulty of the model and the overall computational load, making the system more suitable for real-time operation.
[0038] For the respective architectures of the object detection model and the defocus regression model, please refer to [reference here]. Figure 4 In the dual-model collaborative training phase, by collecting stacks of microscopic images at different axial positions and uniformly labeling the two-dimensional position, defocus height, defocus direction, and interference type, the target detection model and the defocus regression model can learn under the same imaging conditions and data distribution.
[0039] The purpose of this collaborative training strategy is that the object detection model learns to stably locate objects in complex backgrounds, while the defocus regression model learns the axial feature variation patterns under the ROI conditions corresponding to the actual detection output, thereby avoiding performance degradation caused by inconsistent input distribution during the inference stage.
[0040] More specifically, in order to construct datasets for model training, this embodiment constructs object detection image datasets and defocus regression image datasets, respectively.
[0041] Using a bright-field microscopy system, bacteria in a sample cell were imaged under different imaging conditions. A displacement stage was driven to perform axial scanning, acquiring z-stack images of bacteria arranged in a fixed 1µm step size within a 10µm depth range on both sides of the focal plane. Each frame corresponds to a known axial displacement value, used for subsequent defocus height and orientation calibration. Imaging conditions included varying illumination intensities, uneven brightness, and backgrounds containing particle noise to enhance the training data's coverage of real-world application scenarios.
[0042] In addition, the constructed dataset should also include more images of bacteria moving in complex backgrounds to increase the proportion of this part in the training set of the object detection model and improve the robustness of the object detection model in complex scenes.
[0043] During the dataset annotation phase, 708 target detection images were manually annotated, including the location and region of bacteria in the images. The target detection image dataset was divided into training and validation sets in an 8:2 ratio. The training set was used to train the target detection model, while the validation set was used to observe changes in loss and evaluate model performance. Simultaneously, 1006 defocus regression images were annotated with the corresponding axial defocus height and defocus direction information. They were also divided into training and validation sets in an 8:2 ratio and applied to the training and validation of the defocus regression model.
[0044] Secondly, the construction and training of the two-stage deep learning model are discussed. This embodiment achieves functional decoupling between the object detection task and the defocus regression task by constructing a two-stage deep learning model.
[0045] The first-stage target detection model is used to identify bacterial targets under complex bright-field microscopic backgrounds and output their two-dimensional position information in the image coordinate system. The second-stage defocus regression model is used to regress the defocus height and corresponding defocus direction of a bacterium relative to the focal plane from a local image region containing a single bacterium.
[0046] In this embodiment, the target detection model is constructed using a YOLOv8-Nano-based network structure. Its input is a single-frame bright-field microscopic image, and its output is the location region of the bacterial target in the image and the corresponding two-dimensional location information.
[0047] The defocus regression model is an axial prediction model built on the ResNet-18 residual network structure. This model uses 128x128 local bacterial image regions (fine ROIs) as input. The defocus regression model consists of a feature extraction backbone and a regression head output. The feature extraction backbone extracts salient features related to the defocus state of the bacteria from the local image. This part performs deep feature extraction by stacking convolutional layers, pooling layers, and consecutive residual blocks. The residual structure effectively preserves the extremely weak defocus texture edge gradients in bright-field environments through identity mapping, avoiding signal loss in deep networks.
[0048] It is worth noting that the traditional Softmax classification layer is abandoned at the end of the network. Instead, a regression head design is adopted, which establishes a continuous mapping relationship between image defocus features and defocus positions through a fully connected layer, thereby achieving accurate regression from image features to defocus height and direction information.
[0049] In the object detection model training phase, this embodiment uses an 11th Gen Intel(R) Core(TM) i5-1155G7 CPU as the training platform. The labeled object detection training set is input into the object detection model for training. The training parameters are set to a batch size of 8 and a training epoch of 50. The model adopts the default training strategy of the YOLOv8 framework, with an initial learning rate of 0.01. A warmup mechanism and a learning rate decay strategy are combined to improve the stability and overall convergence performance in the early stages of training. Default data augmentation strategies are used during training, including Mosaic and MixUp combined augmentation, stochastic geometric transformations, and HSV-based illumination perturbations to improve the model's robustness to scale and illumination changes. Simultaneously, Mosaic augmentation is automatically disabled in the last 10 epochs of training to improve object localization accuracy.
[0050] The training process employs a joint loss function consisting of bounding box regression loss and classification loss, and optimizes the model parameters through backpropagation and gradient descent algorithms. After 50 epochs of training, the box_loss decreased from 1.3 to 0.84, and the cls_loss decreased from 2.1 to 0.37. The model with the minimum loss was saved as the optimal object detection model. This model achieved an accuracy of 96% and a recall of 98% on the validation set, demonstrating that it can stably distinguish bacterial targets from non-target regions under complex backgrounds and low contrast conditions, including uneven brightness, varying illumination intensity, and interference from particulate impurities.
[0051] Secondly, during the defocus regression model training phase, local image regions with labeled axial defocus height and direction are input into the defocus regression model. Initial training parameters are set to a batch size of 16, 50 training epochs, and the AdamW optimizer with a fixed learning rate of 1×10⁻. 4 The adaptive gradient mechanism dynamically adjusts the parameter update step size to ensure stable convergence of the model and prevent overfitting.
[0052] During the training phase, data augmentation strategies involving rotation and lighting perturbations are introduced to make the regression network robust to changes in planar orientation and illumination intensity. SmoothL1Loss is used as the loss function for training the regression network. This loss function exhibits quadratic loss when the prediction error is small, thus ensuring regression accuracy; when the prediction error is large, it degenerates into linear loss, reducing the impact of outliers on model training. Model parameters are optimized using backpropagation and gradient descent algorithms.
[0053] After 50 epochs of training, the train_loss decreased from 0.0972 to 0.0038, and the Val_loss decreased from 0.0505 to 0.0029. The model with the minimum loss was saved as the optimal regression model. For the model's performance on the validation set, please refer to [link / reference]. Figure 5 The actual out-of-focus position shown Compared with the model's predicted defocus position The regression scatter plot shows the orange line representing the ideal regression relationship. It can be seen that the scatter points of the predicted and actual values generally follow a certain trend. An ideal diagonal distribution indicates that the model's predictions are in good agreement with the actual values.
[0054] Continue to refer to this diagram. Figure 6 The regression error distribution shown has a mean absolute error (MAE) of 0.75 μm, which is smaller than the typical scale of a single bacterium. Figure 7The relationship between prediction error and actual defocus position is shown. It can be seen that the error fluctuation is small throughout the entire defocus range, indicating that the regression model based on residual structure can still achieve stable and high-precision image-to-defocus position regression under small-scale training dataset conditions.
[0055] In summary, the refined ROI is then input into a defocus regression model based on a residual network structure to estimate the defocus height and direction of the target relative to the focal plane. The key to this step is transforming information traditionally dependent on physical scanning or optical structures into a data-driven regression problem.
[0056] In some embodiments of the present invention, it is considered that under bright-field imaging conditions, microscale targets exhibit subtle but learnable imaging changes under different defocus states, such as edge blurring, brightness distribution symmetry, and texture diffusion characteristics. Residual networks, through multi-layer feature extraction and residual connection mechanisms, can deepen the network while avoiding feature degradation, thereby preserving these subtle but discriminative changes.
[0057] This step outputs both defocus height and defocus direction, rather than just a scalar value, because axial adjustment is a directional control issue. Clearly defining the defocus direction directly guides the focus correction device to adjust in the correct direction, reducing unnecessary search processes and thus improving overall response speed and stability.
[0058] The axial information and the target's two-dimensional position information output by the defocus regression model are not only used for result display, but are directly used as control inputs to drive the motion compensation device. This stage embodies the closed-loop coupling of perception and execution.
[0059] For the execution of motion compensation and autofocus in S6, the two-dimensional position information is used to drive the two-dimensional moving platform so that the target is always adjusted to the center of the field of view, thereby avoiding the target from affecting the subsequent estimation accuracy due to distortion or resolution reduction at the edge of the field of view; the defocus height and direction information are used to drive the focus compensation mechanism to adjust the focal plane or the target axial position in real time so that the target returns to a clear imaging state.
[0060] It is important to understand that the effect of this design lies in the overall execution system, which no longer passively records the out-of-focus state of the target, but continuously corrects the imaging conditions during the tracking process, providing a more stable and consistent input image for subsequent frames, thereby forming positive feedback and improving the reliability of the entire tracking process.
[0061] For 3D reconstruction in a continuous sequence of microscopic images, S7 is performed as follows: In the continuous sequence of microscopic image frames, a temporal correlation mechanism is introduced, using the 3D position of the microscale target acquired in the previous frame as a prior constraint to generate candidate ROIs for the next frame, and inputting the candidate ROIs into the target detection model for identification and localization; by performing temporal consistency correlation on the reconstructed 3D positions in the continuous frames, a continuous 3D position sequence of the microscale target in the target space region is formed, and the real-time 3D motion trajectory of the microscale target is output.
[0062] In each frame, the target's position in three-dimensional space is reconstructed by fusing the two-dimensional planar position with the corresponding axial defocus information. Since this information is output by the model at the same point in time, it is strictly aligned temporally, facilitating correlation between consecutive frames.
[0063] During continuous tracking, the target position of the previous frame is introduced as a priori to generate candidate ROIs for the next frame. This approach leverages the assumption of the continuity of microscale target motion, limiting the detection range to a reasonable area, which reduces the probability of missed detections and minimizes interference from irrelevant areas on the detection model.
[0064] Through this time-prior-based cyclic input mechanism, the system achieves stable three-dimensional trajectory output, rather than independent positioning results between frames, thus better reflecting the continuous characteristics of real microscopic motion processes.
[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Similarly, any equivalent structural changes made based on the description and drawings of the present invention should also be included within the scope of protection of the present invention.
Claims
1. A positioning method for three-dimensional real-time tracking of microscale objects in a bright-field microscopic environment, characterized in that, The method comprises the following steps: In a bright-field microscopic imaging optical system, a sequence of microscopic image frames in a target spatial region is continuously acquired; Each frame of the microscopic image is input into a target detection model based on a YOLO architecture, which is used to locate the two-dimensional position information of the micro-scale target in the frame of the microscopic image by distinguishing the background pseudo-structure from the micro-scale target under the bright-field background; According to the two-dimensional position information of the micro-scale target, a local image region centered on the micro-scale target is cropped from the corresponding frame of the microscopic image as a fine ROI; The fine ROI is input into an out-of-focus regression model based on a residual network structure, which extracts the gray scale and blur features of the micro-scale target under different out-of-focus states and outputs the out-of-focus height and corresponding out-of-focus direction information of the micro-scale target relative to the current focal plane; The two-dimensional position information of the micro-scale target in the image plane and the out-of-focus axial information of the corresponding frame are fused to obtain the three-dimensional spatial position of the micro-scale target at the current time, and the three-dimensional position reconstruction under the condition of a single frame is performed; Based on the output two-dimensional position information of each frame, a two-dimensional moving platform is driven for plane position compensation, so that the micro-scale target remains in the center of the field of view, and based on the out-of-focus height and out-of-focus direction information output by the corresponding frame, a focus adjusting device adjusts the focal plane position; In the sequence of continuous microscopic image frames, a time sequence correlation mechanism is introduced, the three-dimensional position of the micro-scale target obtained in the previous frame is taken as a prior constraint to generate a candidate ROI in the next frame, and the candidate ROI is input into the target detection model for identification and positioning; the three-dimensional positions reconstructed in the continuous frames are associated in time sequence consistency to form a continuous three-dimensional position sequence of the micro-scale target in the target spatial region, and the three-dimensional real-time motion trajectory of the micro-scale target is output.
2. The positioning method of claim 1, wherein, In the training process of the target detection model and the out-of-focus regression model, by acquiring microscopic images of target samples at different axial positions, the two-dimensional position information, out-of-focus height and out-of-focus direction of each target sample are labeled, and the brightness unevenness, light fluctuation and particle impurities in the bright-field background are distinguished and labeled, a unified training data set is constructed, and the target detection model and the out-of-focus regression model are supervised and trained to learn the target discrimination and axial imaging feature variation law respectively.
3. The positioning method of claim 1, wherein, Under the bright-field microscopic imaging optical system, a camera and a microscope objective are configured to continuously image the target spatial region to acquire a sequence of microscopic image frames; each frame of the microscopic image maintains consistent resolution and optical magnification.
4. The positioning method of claim 1, wherein, Each frame of the microscopic image is input into a target detection model based on a YOLO architecture, which is used to perform multi-scale convolution feature extraction on a single frame of the microscopic image under the background interference conditions of brightness unevenness, light fluctuation and particle impurities, so as to distinguish the micro-scale target from the background pseudo-structure; Then, the detected micro-scale target is predicted, and its two-dimensional position information in the image coordinate system and the corresponding detection confidence are output. The two-dimensional position information is used for cropping a local image region centered on the micro-scale target in the current frame to form a fine ROI containing only a single micro-scale target, and inputting the fine ROI into an out-of-focus regression model based on a residual network structure.
5. The positioning method of claim 4, wherein, The out-of-focus regression model uses a residual network to perform deep feature extraction on an image of a region corresponding to the fine ROI input; wherein the residual blocks in the residual network capture the gray scale distribution changes, edge blur degrees and morphological structure differences of the micro-scale target under different out-of-focus states layer by layer, thereby establishing a mapping relationship between the out-of-focus imaging features and the axial position deviation. Based on the extracted out-of-focus imaging features, the out-of-focus regression model outputs the out-of-focus height and the corresponding out-of-focus direction information of the micro-scale target relative to the current focal plane; wherein the out-of-focus height and direction information are used for fusion with the two-dimensional position information to realize position reconstruction of the micro-scale target in three-dimensional space in the corresponding frame, and serve as feedback control quantities to drive the focal plane dynamic adjustment of the focal adjustment device.
6. The positioning method of claim 2, wherein, The two-dimensional position information of the micro-scale target identified in each frame is used to drive the microscope in the two-dimensional moving platform to perform plane position compensation, so that the micro-scale target always remains in the center of the field of view of the microscope; at the same time, based on the out-of-focus height and out-of-focus direction information of the corresponding frame, the focal adjustment mechanism is driven to dynamically adjust the focal plane of the microscope, so that the micro-scale target is focused near the focal plane in real time.
7. The positioning method of claim 6, wherein, In the three-dimensional position reconstruction process of the micro-scale target, for each microscopic image, the two-dimensional position information output by target detection and the out-of-focus height information output by the out-of-focus regression model are combined to calculate the coordinates of the micro-scale target in three-dimensional space, forming single-frame three-dimensional position data; for a scene of multiple micro-scale targets in a sequence of microscopic image frames, the three-dimensional coordinates of each micro-scale target are independently reconstructed to realize three-dimensional positioning of multiple targets in the sequence of microscopic image frames; specifically including the following steps: Target association is performed on a sequence of consecutive microscopic image frames, and the three-dimensional coordinates of the micro-scale target positioned in the previous frame are used as a prior reference to determine the region where the micro-scale target is likely to appear in the next frame; A candidate ROI is cropped in the next frame image centered on the two-dimensional position information of the micro-scale target in the previous frame; The candidate ROI is input into the target detection model for identification and positioning, and the two-dimensional position information of the target in the next frame is output; The three-dimensional positions reconstructed in each frame are associated in time sequence to form a continuous three-dimensional motion trajectory of the micro-scale target.
8. The positioning method of claim 1, wherein, In the data collection process of the target detection model and the out-of-focus regression model, the micro-scale target is imaged and collected multiple times under a microscopic imaging system, specifically, at each known axial position, a corresponding microscopic image frame is collected to form an axially continuous image stack; wherein each set of image stacks corresponds to the imaging results of the same micro-scale target under different out-of-focus states.
9. The positioning method of claim 8, wherein, For each set of image stacks, the following labeling operations are performed on each microscopic image: In each frame of microscopic image, the target position region corresponding to the micro-scale target in the image plane is labeled as the supervision signal of the target detection model; for a multi-micro-scale target scene, each target is assigned an independent label; In combination with the control parameters of the axial displacement mechanism of the two-dimensional driving platform, the defocus height value of the axial offset of the relative focusing plane for each frame of microscopic image is labeled, and the defocus direction information is labeled as the continuous numerical supervision target of the defocus regression model.