Autofocus methods, apparatuses, devices, and storage media for microscopy imaging
By segmenting and labeling the image sequences of the microsphere microscopy system and training a deep learning model, the problem of distinguishing defocused images in the microsphere microscopy system was solved, achieving fast and accurate autofocus and reducing system costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2023-11-07
- Publication Date
- 2026-05-19
AI Technical Summary
Existing autofocus methods in microsphere microscopy systems cannot effectively distinguish between positive and negative defocus in defocused images. Furthermore, existing methods typically require additional sensors or hardware modifications, increasing system costs, or are insufficient in terms of real-time performance and accuracy.
By acquiring single-view microscopic image sequences, uniformly dividing and labeling them, a defocus distance prediction model is constructed. A deep learning model is used to distinguish the positive and negative defocus properties of the defocused images. Feature extraction and mapping sub-networks are used for training to achieve fast and accurate autofocus.
It enables rapid and accurate differentiation of positive and negative defocus in out-of-focus images, improving the efficiency and accuracy of autofocus while avoiding additional hardware costs.
Smart Images
Figure CN117369108B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of microscopic imaging technology, and in particular to an autofocus method, apparatus, device and storage medium for microscopic imaging. Background Technology
[0002] Microsphere super-resolution microscopy is a novel optical super-resolution imaging technique with significant advantages such as no need for fluorescent labeling, wide-field imaging, and simple optical path, making it valuable for applications in semiconductor wafer inspection, nanomaterial characterization, and bioimaging. However, due to the extremely shallow depth of field in microsphere microscopy systems, achieving rapid autofocus is a crucial foundation for successful microsphere microscopy.
[0003] Autofocus refers to the process of receiving light reflected from a sample using a photoelectric sensor, processing it internally within the imaging system, controlling the motorized focusing device, and ultimately obtaining a clear image on the image detector. Microscope autofocus can be categorized into two types: sensor-based and vision-based methods. Sensor-based methods employ auxiliary modules in the optical imaging system, allowing for real-time and efficient measurement of defocus distance. However, they require additional sensors or modifications to existing hardware, significantly increasing system costs. Vision-based methods can be further subdivided into three categories: (a) virtual refocusing methods, (b) multi-image focusing methods, and (c) single-image focusing methods. Virtual refocusing methods describe autofocus as an image deblurring problem. They output a synthesized focused image by performing blind or non-blind deconvolution on the defocused image. This method cannot guarantee the quality of the synthesized image and is prone to overfitting during sample training, exhibiting poor scalability. Multi-image focusing methods involve acquiring images at multiple focal points to form an image stack, then calculating the sharpness value of each image in the stack using morphological operations. The focal point corresponding to the image with the highest sharpness value is the optimal focusing position. While this method is easy to implement, it is very time-consuming in practical applications. Single-image focusing methods generally use deep learning models to establish a mapping between image sharpness and defocus distance. The position of the optimal focal plane is estimated using a single acquired microscopic image. This method can autofocus in real time. However, in some cases, such as defocused images far from the optimal focal plane, they can appear very similar, especially positive and negative defocused images at the same distance from the optimal focal plane, which have only subtle differences. This method often fails to correctly distinguish between positive and negative defocus in defocused images. Summary of the Invention
[0004] Therefore, it is necessary to provide an autofocus method, apparatus, device, and storage medium for microscopic imaging that can correctly distinguish the positive and negative defocus properties of a defocused image, in order to address the aforementioned technical problems.
[0005] An autofocusing method for microscopic imaging, the method comprising:
[0006] Acquire single-field microscopic image sequences of several samples; wherein, each sample corresponds to a single-field microscopic image sequence, each single-field microscopic image sequence includes several microscopic images acquired under the field of focus, each field of focus corresponds to a stack of microscopic images, and each stack of microscopic images includes several microscopic images at different defocus positions.
[0007] Each microscopic image in the current microscopic image stack is uniformly divided into several microscopic image sub-blocks. The image sub-blocks corresponding to the block positions are combined to obtain multiple local microscopic image stacks. The number of local microscopic image stacks corresponding to a microscopic image stack is equal to the number of microscopic image blocks.
[0008] Each microscopic image sub-block in the current local microscopic image stack is labeled. Specifically, the microscopic image sub-block with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other microscopic image sub-blocks in the current local microscopic image stack, labels are made according to their physical distance to the microscopic image sub-block with the label of zero.
[0009] A pre-built defocus distance prediction model is trained using a dataset of local microscopic images to obtain a trained defocus distance prediction model.
[0010] The defocus distance value of the microscopic image under test is obtained based on the trained defocus distance prediction model, and automatic focusing is performed based on the defocus distance value.
[0011] An autofocusing device for microscopic imaging, the device comprising:
[0012] The microscopic image sequence acquisition module is used to acquire single-field microscopic image sequences of several samples; wherein, each sample corresponds to a single-field microscopic image sequence, each single-field microscopic image sequence includes several microscopic images acquired under the field of focus, each field of focus corresponds to a microscopic image stack, and each microscopic image stack includes several microscopic images at different defocus positions.
[0013] The local microscopic image stack acquisition module is used to uniformly divide each microscopic image in the current microscopic image stack into several microscopic image sub-blocks, and combine the image sub-blocks corresponding to the block positions to obtain multiple local microscopic image stacks; wherein, the number of local microscopic image stacks corresponding to a microscopic image stack is equal to the number of microscopic image blocks.
[0014] The labeling module is used to label each microscopic image sub-block in the current local microscopic image stack. Specifically, the microscopic image sub-block with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other microscopic image sub-blocks in the current local microscopic image stack, the labels are assigned based on their physical distance to the microscopic image sub-block with the label of zero.
[0015] The defocus distance prediction model training module is used to train a pre-built defocus distance prediction model using a dataset of local microscopic images to obtain a trained defocus distance prediction model.
[0016] The autofocus module is used to obtain the defocus distance value of the microscopic image under test based on the trained defocus distance prediction model, and then perform automatic focusing based on the defocus distance value.
[0017] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:
[0018] Acquire single-field microscopic image sequences of several samples; wherein, each sample corresponds to a single-field microscopic image sequence, each single-field microscopic image sequence includes several microscopic images acquired under the field of focus, each field of focus corresponds to a stack of microscopic images, and each stack of microscopic images includes several microscopic images at different defocus positions.
[0019] Each microscopic image in the current microscopic image stack is uniformly divided into several microscopic image sub-blocks. The image sub-blocks corresponding to the block positions are combined to obtain multiple local microscopic image stacks. The number of local microscopic image stacks corresponding to a microscopic image stack is equal to the number of microscopic image blocks.
[0020] Each microscopic image sub-block in the current local microscopic image stack is labeled. Specifically, the microscopic image sub-block with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other microscopic image sub-blocks in the current local microscopic image stack, labels are made according to their physical distance to the microscopic image sub-block with the label of zero.
[0021] A pre-built defocus distance prediction model is trained using a dataset of local microscopic images to obtain a trained defocus distance prediction model.
[0022] The defocus distance value of the microscopic image under test is obtained based on the trained defocus distance prediction model, and automatic focusing is performed based on the defocus distance value.
[0023] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0024] Acquire single-field microscopic image sequences of several samples; wherein, each sample corresponds to a single-field microscopic image sequence, each single-field microscopic image sequence includes several microscopic images acquired under the field of focus, each field of focus corresponds to a stack of microscopic images, and each stack of microscopic images includes several microscopic images at different defocus positions.
[0025] Each microscopic image in the current microscopic image stack is uniformly divided into several microscopic image sub-blocks. The image sub-blocks corresponding to the block positions are combined to obtain multiple local microscopic image stacks. The number of local microscopic image stacks corresponding to a microscopic image stack is equal to the number of microscopic image blocks.
[0026] Each microscopic image sub-block in the current local microscopic image stack is labeled. Specifically, the microscopic image sub-block with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other microscopic image sub-blocks in the current local microscopic image stack, labels are made according to their physical distance to the microscopic image sub-block with the label of zero.
[0027] A pre-built defocus distance prediction model is trained using a dataset of local microscopic images to obtain a trained defocus distance prediction model.
[0028] The defocus distance value of the microscopic image under test is obtained based on the trained defocus distance prediction model, and automatic focusing is performed based on the defocus distance value.
[0029] The aforementioned autofocus method, apparatus, computer equipment, and storage medium for microscopic imaging first acquire a sequence of single-field-of-view microscopic images of several samples. Each sample corresponds to a stack of microscopic images under multiple focused fields of view, and each stack includes microscopic images at several different defocus positions. Next, the microscopic images in each stack are divided into non-overlapping, uniform blocks, and the image sub-blocks corresponding to the block positions are combined to obtain multiple local microscopic image stacks. Then, a reference image sub-block is selected from the image sub-blocks in each local microscopic image stack according to sharpness, and other image sub-blocks are labeled according to distance, resulting in a dataset of local microscopic image stacks. Finally, a prediction model is trained using the dataset. After training, it can be used to predict the defocus distance value of the microscopic image under test, and autofocus is performed based on the defocus distance value. This method can accurately and quickly distinguish the positive and negative defocus properties of defocused images, achieving autofocus. Attached Figure Description
[0030] Figure 1 This is a flowchart illustrating an autofocusing method used for microscopic imaging.
[0031] Figure 2 This is a block diagram of an autofocus device used for microscopic imaging.
[0032] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0034] In one embodiment, such as Figure 1 As shown, an autofocusing method for microscopic imaging is provided, comprising the following steps:
[0035] Step 102: Collect single-field microscopic image sequences of several samples.
[0036] Each sample corresponds to a single-field microscopic image sequence, and each single-field microscopic image sequence includes several microscopic images acquired under the field of focus. Each field of focus corresponds to a stack of microscopic images, and each stack of microscopic images includes several microscopic images at different defocus positions.
[0037] Step 104: Divide each microscopic image in the current microscopic image stack into several microscopic image sub-blocks, and combine the image sub-blocks corresponding to the division positions to obtain multiple local microscopic image stacks.
[0038] In this context, the number of local microscopic image stacks corresponding to one microscopic image stack is equal to the number of microscopic image blocks. For example, if the size of an image sub-block is 224×224, and each microscopic image is divided into 90 image sub-blocks, with the remaining parts of the image being discarded, then this microscopic image stack corresponds to 90 local microscopic image stacks. 。
[0039] For a stack of several microscopic images acquired from several samples, each image in the stack is divided into non-overlapping blocks. Each image can be divided into several non-overlapping image sub-blocks. The image sub-blocks corresponding to the block positions of each image in the stack are combined to form a local microscopic image stack. The block processing of the original microscopic images takes into account the possibility that the sample may have unevenness in a single field of view, so the focus position of different regions in a single field of view will be different.
[0040] Step 106: Label each microscopic image sub-block in the current local microscopic image stack.
[0041] Specifically, the sub-block of the microscopic image with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other sub-blocks of the microscopic image in the current local microscopic image stack, labels are assigned based on their physical distance to the sub-block labeled as zero.
[0042] For each image sub-block in the local microscopic image stack, the sharpness value of each sub-block is calculated using an image sharpness evaluation function. The maximum sharpness value is obtained, and the image sub-block corresponding to the maximum sharpness value is labeled as 0μm. Other image sub-blocks in the local microscopic image stack are labeled according to their physical distance from the image sub-block labeled 0μm. The image sharpness evaluation function can be the Tenengrad evaluation function, the Laplacian gradient function, the Brenner function, or other functions; no specific limitation is made here.
[0043] The location corresponding to the most sharp sub-block in each local microscopic image stack is not necessarily the same location.
[0044] Step 108: Train the pre-built defocus distance prediction model using the dataset of local microscopic image stacks to obtain the trained defocus distance prediction model.
[0045] After the above steps, several stacks of locally labeled microscopic images can be obtained, which are the datasets used for model training and testing. express, Z k Let represent the stack of the k-th local microscopic images with true labels in the dataset. This represents the i-th image sub-block in the k-th local microscopic image stack. Let represent the true label corresponding to the i-th image sub-block in the k-th local microscopic image stack, and n represent the total size of the dataset. The dataset... By randomly dividing the dataset into two parts according to a certain ratio, we can obtain the training dataset and the test dataset. For example, the dataset can be divided into the training dataset and the test dataset in a 3:1 ratio.
[0046] Step 110: Obtain the defocus distance value of the microscopic image under test based on the trained defocus distance prediction model, and perform automatic focusing based on the defocus distance value.
[0047] The aforementioned autofocus method for microscopic imaging first acquires a sequence of single-field-of-view microscopic images of several samples. Each sample corresponds to a stack of microscopic images under multiple focused fields of view, and each stack includes microscopic images at several different defocus positions. Next, the microscopic images in each stack are divided into non-overlapping, uniform blocks, and the image sub-blocks corresponding to the block positions are combined to obtain multiple local microscopic image stacks. Then, a reference image sub-block is selected based on sharpness for each local microscopic image stack, and other image sub-blocks are labeled according to distance, resulting in a dataset of local microscopic image stacks. Finally, a prediction model is trained using this dataset. After training, the model can predict the defocus distance value of the microscopic image under test, and autofocus is performed based on the defocus distance value. This method can accurately and quickly distinguish between positive and negative defocus in defocused images, achieving autofocus.
[0048] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0049] In one embodiment, the method for acquiring microscopic images at different defocus positions is as follows:
[0050] Within each field of view to be focused, the microscope objective is moved according to a preset step size to obtain microscopic images at different defocus positions.
[0051] For example, a microsphere super-resolution microscopy system can be used to acquire a sequence of single-field microscopic images of several samples. Specifically, for each sample, N fields of focus are randomly selected. Within each field of focus, the microscope objective is moved with a fixed step size, acquiring M microscopic images at different defocus positions, forming a stack of microscopic images. Each sample can obtain N stacks of microscopic images. At this point, the physical distance from other sub-blocks in the current local microscopic image stack to the sub-block with a label of zero is exactly an integer multiple of the moving step size. This is because the sub-blocks in a local microscopic image stack are all located within the same field of view. Within the same field of view, several microscopic images are acquired according to the moving step size. By uniformly dividing the microscopic images into blocks, the physical distance between corresponding sub-blocks is exactly an integer multiple of the moving step size.
[0052] Images were acquired using a microsphere super-resolution microscopy system, with a single field-of-view image size of 2448×2048. The samples were biological cell or tissue samples. The range of microscope objective movement in each field of view was [-3μm, 3μm], with a step size of 50nm each time, thus allowing 120 microscopic images to be acquired in each field of view.
[0053] The microsphere super-resolution microscopy system can be model OptoNano 200, equipped with the ON LENS-G2 super-resolution objective system, which can achieve 137 nanometer resolution imaging in ambient air.
[0054] In one embodiment, the method further includes:
[0055] Local microscopic image stacks with a maximum resolution value not exceeding a preset threshold are removed.
[0056] Considering that sparsely textured image sub-blocks have a significant impact on autofocus algorithms, when annotating image sub-blocks in a local microscopic image stack, local microscopic image stacks with a maximum sharpness value not exceeding a threshold T are removed.
[0057] In one embodiment, the defocus distance prediction model includes:
[0058] The system consists of a feature extraction subnetwork and a feature mapping subnetwork. The feature extraction subnetwork is used to extract sharpness features, while the feature mapping subnetwork is used to map the sharpness features to specific out-of-focus distance values.
[0059] Training the pre-built defocus distance prediction model includes: training the feature extraction sub-network, and after training, fixing the parameters of the feature extraction sub-network and training the feature mapping sub-network.
[0060] The purpose of this invention is to predict the defocus distance from a single input defocused image using a constructed deep learning model. The constructed deep learning model includes two sub-networks: a feature extraction sub-network FeaNet(.) and a feature mapping sub-network PreNet(.). The feature extraction sub-network FeaNet(.) is used to extract sharpness features from the input defocused image, and the feature mapping sub-network PreNet(.) is used to map the sharpness features to specific defocus distance values.
[0061] The feature extraction sub-network FeaNet(.) can use ResNet34 as the backbone network model. The original ResNet34 network consists of one convolutional layer, four residual blocks, and one global average pooling layer (GAP), which can generate a 512-dimensional image feature representation. Using the original version of ResNet34, FeaNet(.) is constructed. Given a 224×224×3 image as input, FeaNet(.) outputs a sharpness feature vector of size 7×7×512.
[0062] The feature mapping subnetwork PreNet(.) can include the last residual block (RB) of the ResNet network, a 3×3 convolutional layer, a global average pooling layer (GAP), a 1×1 convolutional layer, and a fully connected layer (FC). The number of input and output channels of the residual block (RB), the 3×3 convolutional layer, and the GAP layer are consistent with the original version of the ResNet network. Given a 7×7×512 sharpness feature vector as input, PreNet(.) predicts the output out-of-focus distance value.
[0063] In one embodiment, training the feature extraction subnetwork includes:
[0064] A dataset of local microscopic images is obtained, and the dataset is divided into batches to obtain multiple data subsets;
[0065] Data augmentation is performed on the microscopic image sub-blocks in the current data subset to obtain a new data subset containing the microscopic image sub-blocks and the augmented image sub-blocks; wherein, the microscopic image sub-block and its corresponding augmented image sub-block are a positive sample pair, and the microscopic image sub-block and all other image sub-blocks are several negative sample pairs;
[0066] The new data subset is input into the feature extraction subnetwork, and the trained feature extraction subnetwork is obtained by using the preset feature extraction subnetwork loss function.
[0067] The sharpness features output by the feature extraction subnetwork FeaNet(.) determine the defocus distance value predicted by the subsequent feature mapping subnetwork PreNet(.). Furthermore, since the image feature changes are not significant between defocused images far from the focal plane, and positive and negative defocused images at the same defocus distance from the focal plane are extremely similar, most existing deep learning-based autofocus algorithms cannot effectively distinguish between positive and negative defocus in images.
[0068] Therefore, this invention employs a contrastive learning approach to train the feature extraction sub-network FeaNet(.), comprising the following three steps:
[0069] (1) Data Augmentation
[0070] During model training, the data input to the model is in batches. Therefore, we need to perform data augmentation on all images in the batch to construct a new batch containing the original images and the augmented images. Given images I in the batch... i Let the corresponding enhanced image be Will I i and A positive sample pair is called I. i All other images in the new batch are called negative sample pairs.
[0071] Data augmentation operations can include rotation, flipping, and chroma transformation.
[0072] (2) Feature extraction
[0073] The new batch is fed into the FeaNet(.) network to extract the sharpness features of each image. The goal of contrastive learning is to enable the FeaNet(.) network to extract as similar features as possible from positive samples, while extracting completely different features from negative samples. Specifically, for I... i and A positive sample pair is formed and fed into the feature extraction subnetwork FeaNet(.) to extract the sharpness feature vector F. i and To calculate F i and The similarity between them can be measured using cosine similarity, that is, by calculating the feature vector F. i and The cosine of the angle between two vectors is a key similarity metric. The closer two vectors are in space, the smaller the angle between them, and the larger the cosine value, indicating greater similarity. In contrastive learning, maximizing the vector similarity between positive samples is crucial.
[0074] (3) Minimization of the loss function
[0075] During model training, it is necessary to determine the model's loss function. The model's parameters are determined by minimizing the loss function, with the aim of maximizing the similarity between two feature vectors by minimizing the contrastive loss function.
[0076] In one embodiment, the preset feature extraction sub-network loss function is as follows:
[0077]
[0078] Where L1 represents the loss function of the feature extraction sub-network, B represents the number of microscopic image patches in the current data subset, i.e., the batch size, τ represents the temperature coefficient, exp(.) represents the exponential function, and F...m Represents the microscopic image sub-block I in the new data subset m The sharpness features output by the feature extraction subnetwork, 1 [m≠i] The value is either 0 or 1. It takes the value 0 when m = i and the value 1 when m ≠ i. sim(.) represents the similarity. When cosine similarity is selected, the calculation formula is... F i Represents a microscopic image sub-block I in the data subset. i The sharpness features of the output of the feature extraction subnetwork. Represents a microscopic image sub-block I in the data subset. i Corresponding enhanced image sub-blocks The sharpness features output by the feature extraction subnetwork.
[0079] As shown in the above formula, the numerator only calculates the similarity of positive sample pairs, while the denominator calculates the similarity of all positive and negative sample pairs. The greater the similarity between positive sample pairs and the smaller the similarity between negative sample pairs, the smaller the value of the loss function. Thus, minimizing this loss function is equivalent to maximizing the probability of similarity between positive sample pairs.
[0080] In one embodiment, training the feature mapping subnetwork includes:
[0081] The trained feature mapping subnetwork is obtained through a predefined feature mapping subnetwork loss function; the specific feature mapping subnetwork loss function is as follows:
[0082]
[0083] Among them, L pre_dist Represents the loss function of the feature mapping subnetwork. This represents the i-th microscopic image sub-block in the k-th local microscopic image stack. denoted as the out-of-focus distance value predicted by the feature extraction subnetwork, and RMSE(.) represents the root mean square error between the predicted out-of-focus distance value and the true label.
[0084] A smaller RMSE value indicates higher prediction accuracy of the model. The RMSE(.) loss function squares the error, making larger errors have a greater impact on the loss function, thus focusing more on the model's prediction accuracy.
[0085] In summary, the deep learning model constructed in this invention includes two sub-networks: FeaNet(.) and PreNet(.). During the training phase, the FeaNet(.) network is trained first, and then the PreNet(.) network is trained while keeping the parameters of the FeaNet(.) network fixed. During model training, a Dell PowerDevice T640 server was selected, which contains four NVIDIA RTX 3090 graphics cards and two second-generation processors. Scalable processors, with up to 2 cores per processor. Model training parameters are set as follows: batch size 256, epochs 600, Adam as the optimizer, learning rate set to 0.0001, and gradual warmup optimization. Gradual warmup starts training with a small learning rate to prevent model instability, then increases the learning rate slightly at each step until it reaches the initially set value. Finally, training continues using the initial learning rate. During the training process after warmup, the learning rate decays, which helps accelerate model convergence and stabilize parameter changes.
[0086] After model training, the trained model was tested using a test dataset. During dataset construction, the acquired single-view microscopic images were divided into blocks. Therefore, during testing, the median predicted defocus distance value of each image sub-block corresponding to the single-view microscopic image was used as the final defocus distance value of the single-view microscopic image. After testing, the mean error and variance between the predicted values and the true labels for all images in the test dataset were calculated, which were 0.09 μm and 0.16 μm, respectively.
[0087] In one embodiment, such as Figure 2 As shown, an autofocusing device for microscopic imaging is provided, comprising: a microscopic image sequence acquisition module, a local microscopic image stack acquisition module, a label annotation module, a defocus distance prediction model training and autofocusing module, wherein:
[0088] The microscopic image sequence acquisition module is used to acquire single-field microscopic image sequences of several samples.
[0089] Each sample corresponds to a single-field microscopic image sequence, and each single-field microscopic image sequence includes several microscopic images acquired under the field of focus. Each field of focus corresponds to a stack of microscopic images, and each stack of microscopic images includes several microscopic images at different defocus positions.
[0090] The local microscopic image stack acquisition module is used to uniformly divide each microscopic image in the current microscopic image stack into several microscopic image sub-blocks, and combine the image sub-blocks corresponding to the block positions to obtain multiple local microscopic image stacks.
[0091] The number of local microscopic image stacks corresponding to a microscopic image stack is equal to the number of microscopic image blocks.
[0092] The labeling module is used to label each microscopic image sub-block in the current local microscopic image stack.
[0093] Specifically, the sub-block of the microscopic image with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other sub-blocks of the microscopic image in the current local microscopic image stack, labels are assigned based on their physical distance to the sub-block labeled as zero.
[0094] The defocus distance prediction model training module is used to train a pre-built defocus distance prediction model using a dataset of local microscopic images to obtain a trained defocus distance prediction model.
[0095] The autofocus module is used to obtain the defocus distance value of the microscopic image under test based on the trained defocus distance prediction model, and then perform automatic focusing based on the defocus distance value.
[0096] Specific limitations regarding the autofocus device for microscopic imaging can be found in the limitations of the autofocus method for microscopic imaging described above, and will not be repeated here. The various modules in the aforementioned autofocus device for microscopic imaging can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.
[0097] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as microscopic images. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an autofocus method for microscopic imaging.
[0098] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0099] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.
[0100] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.
[0101] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0103] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. An autofocusing method for microscopic imaging, characterized in that, The method includes: Acquire single-field microscopic image sequences of several samples; wherein, each sample corresponds to a single-field microscopic image sequence, each single-field microscopic image sequence includes several microscopic images acquired under the field of focus, each field of focus corresponds to a stack of microscopic images, and each stack of microscopic images includes several microscopic images at different defocus positions. Each microscopic image in the current microscopic image stack is uniformly divided into several microscopic image sub-blocks. The image sub-blocks corresponding to the block positions are combined to obtain multiple local microscopic image stacks. The number of local microscopic image stacks corresponding to a microscopic image stack is equal to the number of microscopic image blocks. Each microscopic image sub-block in the current local microscopic image stack is labeled. Specifically, the microscopic image sub-block with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other microscopic image sub-blocks in the current local microscopic image stack, labels are made according to their physical distance to the microscopic image sub-block with the label of zero. A pre-built defocus distance prediction model is trained using a dataset of local microscopic images to obtain a trained defocus distance prediction model. The defocus distance value of the microscopic image under test is obtained based on the trained defocus distance prediction model, and automatic focusing is performed based on the defocus distance value; The defocus distance prediction model includes: The system consists of a feature extraction subnetwork and a feature mapping subnetwork. The feature extraction subnetwork is used to extract sharpness features, and the feature mapping subnetwork is used to map the sharpness features to specific out-of-focus distance values. Training a pre-built defocus distance prediction model includes: training the feature extraction sub-network; after training, fixing the parameters of the feature extraction sub-network and training the feature mapping sub-network. Training the feature extraction subnetwork includes: A dataset of local microscopic images is obtained, and the dataset is divided into batches to obtain multiple data subsets; Data augmentation is performed on the microscopic image sub-blocks in the current data subset to obtain a new data subset containing the microscopic image sub-blocks and the augmented image sub-blocks; wherein, the microscopic image sub-block and its corresponding augmented image sub-block are a positive sample pair, and the microscopic image sub-block and all other image sub-blocks are several negative sample pairs; The new data subset is input into the feature extraction subnetwork, and the trained feature extraction subnetwork is obtained by using the preset feature extraction subnetwork loss function. The specific loss function of the preset feature extraction sub-network is as follows: in, Let B represent the loss function of the feature extraction sub-network, and let B represent the number of microscopic image patches in the current data subset. Indicates the temperature coefficient. Represents an exponential function. Represents a microscopic image sub-block in the new data subset. The sharpness features output by the feature extraction subnetwork The value can be 0 or 1, when When the value is 0, The value is 1 at time. Indicates similarity. Represents a microscopic image sub-block within a subset of data. The sharpness features of the output of the feature extraction subnetwork. Represents a microscopic image sub-block within a subset of data. Corresponding enhanced image sub-blocks Clarity features of the output after the feature extraction subnetwork; Training the feature mapping subnetwork includes: The trained feature mapping subnetwork is obtained through a predefined feature mapping subnetwork loss function; the specific feature mapping subnetwork loss function is as follows: in, Represents the loss function of the feature mapping subnetwork. Indicates the first The first in the stack of local microscopic images i A sub-block of microscopic images, This represents the defocus distance value predicted by the feature extraction subnetwork. This represents the root mean square error between the predicted defocus distance value and the actual label.
2. The method according to claim 1, characterized in that, The methods for acquiring microscopic images at different defocus positions are as follows: Within each field of view to be focused, the microscope objective is moved according to a preset step size to obtain microscopic images at different defocus positions.
3. The method according to claim 2, characterized in that, The method further includes: Remove local microscopic images whose maximum resolution value does not exceed a preset threshold.
4. An autofocusing device for microscopic imaging, characterized in that, The apparatus comprising the method according to any one of claims 1 to 3, wherein the apparatus includes: The microscopic image sequence acquisition module is used to acquire single-field microscopic image sequences of several samples; wherein, each sample corresponds to a single-field microscopic image sequence, each single-field microscopic image sequence includes several microscopic images acquired under the field of focus, each field of focus corresponds to a microscopic image stack, and each microscopic image stack includes several microscopic images at different defocus positions. The local microscopic image stack acquisition module is used to uniformly divide each microscopic image in the current microscopic image stack into several microscopic image sub-blocks, and combine the image sub-blocks corresponding to the block positions to obtain multiple local microscopic image stacks; wherein, the number of local microscopic image stacks corresponding to a microscopic image stack is equal to the number of microscopic image blocks. The labeling module is used to label each microscopic image sub-block in the current local microscopic image stack. Specifically, the microscopic image sub-block with the highest sharpness value in the current local microscopic image stack is labeled as zero. For other microscopic image sub-blocks in the current local microscopic image stack, the labels are assigned based on their physical distance to the microscopic image sub-block with the label of zero. The defocus distance prediction model training module is used to train a pre-built defocus distance prediction model using a dataset of local microscopic images to obtain a trained defocus distance prediction model. The autofocus module is used to obtain the defocus distance value of the microscopic image under test based on the trained defocus distance prediction model, and then perform automatic focusing based on the defocus distance value.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.