A multi-feature fusion-based fluorescent cell multi-layer microscopic imaging method and system

Abstract

The application provides a fluorescence cell multilayer microscopic imaging method and system based on multi-feature fusion, and relates to the technical field of microscopic imaging. The method comprises the following steps: acquiring a multilayer Z-axis image sequence of a fluorescence cell sample at an X-Y scanning point; dividing an optimal focal plane image into blocks, and identifying the definition state through a convolutional neural network classification model; searching for a replacement block in a neighboring Z layer according to the classification result for a fuzzy block; and outputting a full field of view image after local registration and feathering fusion. The application solves the problems of unstable automatic focusing and insufficient single focal plane coverage under weak fluorescence conditions by fusing multi-feature definition evaluation and deep learning block classification, and realizes full field of view clear imaging of a fluorescence sample with uneven thickness.

Description

Technical Field

[0001] This invention relates to the field of microscopic imaging technology, and in particular to a method and system for multi-layer microscopic imaging of fluorescent cells based on multi-feature fusion. Background Technology

[0002] Fluorescence microscopy is widely used in cell analysis, pathological diagnosis, and biomedical research due to its advantages such as high target specificity and high contrast. However, the fluorescence signal is usually only 1 / 100 to 1 / 1000 of that in bright-field imaging, and is accompanied by interference such as photon noise and shot noise, resulting in low image signal-to-noise ratio and difficulty in evaluating image sharpness.

[0003] Current microscopic imaging systems primarily rely on manual focusing or autofocus methods based on a single gradient operator, such as the Tenengrad gradient operator or the gray-scale variance evaluation function. These methods perform well in bright-field environments with ample signal and high contrast, but in weak fluorescence and low-contrast scenes, their sharpness score curves often exhibit peak drift and misjudgment of local extrema, making it difficult to stably locate the optimal focal plane. Furthermore, existing equipment often employs a single focal plane imaging strategy, while fluorescent cell samples typically possess three-dimensional structural features such as uneven thickness and local undulations. A single focal plane cannot cover the entire sample area for clear imaging, leading to information loss in locally out-of-focus areas.

[0004] Existing AI-based multi-layer fusion methods (such as CN118628369B) employ a holistic fusion paradigm, inputting images from all layers into a neural network for feature extraction and fusion. This approach fails to distinguish between sharp and blurred areas, easily introducing fusion artifacts into originally sharp regions. Existing depth map-based correction methods (such as CN119313573A) rely on pixel-level sharpness curves for focal plane determination, resulting in severe peak drift in weak fluorescence and low-contrast scenes. Furthermore, they do not utilize deep learning for semantic-level quality assessment of local image patches, making it difficult to handle critically blurred states. Therefore, a local restoration solution is urgently needed that can accurately identify locally out-of-focus areas while avoiding over-processing of sharp areas. Summary of the Invention

[0005] To address the technical problems existing in the background art, the present invention provides a method and system for fluorescence cell multilayer microscopy imaging based on multi-feature fusion.

[0006] To achieve the above objectives, a first aspect of the present invention provides a method for multi-feature fusion-based fluorescence cell multilayer microscopy imaging, comprising: A multi-layer Z-axis image sequence of fluorescent cell samples at XY scan points is acquired. The multi-layer Z-axis image sequence includes the optimal focal plane image and its neighboring focal plane images. The optimal focal plane is determined by autofocus. The optimal focal plane image is divided into multiple initial patches. Each initial patch is input into a trained convolutional neural network patch sharpness classification model to obtain the probability that each patch belongs to one of three categories: sharp, blurry, or uncertain. Patches determined to be uncertain are recursively subdivided into multiple sub-patches, which are then re-input into the model for inference until the sub-patches reach a preset minimum patch size or the classification confidence of the sub-patches meets a preset confidence threshold, thus obtaining the final patch-level sharpness classification result. The patches that are classified as blurry in the sharpness classification are identified as patches to be repaired, while keeping the corresponding areas of the sharp patches in the best focal plane image unchanged; on the adjacent focal plane images in the multi-layer Z-axis image sequence, candidate patches with the same coordinate positions as the patch to be repaired are located, and the sharpness evaluation score of each candidate patch and its sharpness probability in the classification model are calculated respectively. The optimal candidate patch is selected as the replacement patch. The replacement patch is locally geometrically registered with the corresponding region in the optimal focal plane image to eliminate interlayer displacement deviation; a feathered fusion region with a preset overlap rate is set between the registered replacement patch and the adjacent retained region; pixel fusion is performed in the overlapping region using a gradient from the first weight to the second weight to generate the fused image of the XY scan point. The fused images of all XY scan points are stitched together to output a full-field fluorescence microscopy stitched image.

[0007] Furthermore, the autofocus specifically includes: Centered on the current Z-axis position, a coarse positioning scan is performed within a range of ±100μm with a step size of 20μm to acquire the first set of focal plane images. The multi-feature fusion sharpness score of each image is calculated to determine the peak position of the coarse positioning scan. Using the peak position determined by the coarse positioning scan as the center, a fine search scan is performed within a range of ±20μm with a step size of 2μm to acquire the second set of focal plane images. The multi-feature fusion sharpness score of each image is calculated, and the Z-axis coordinate with the highest score is determined as the optimal focal plane. The Z-axis range of the fine search scan is smaller than that of the coarse positioning scan, and the step size of the fine search scan is smaller than that of the coarse positioning scan.

[0008] Furthermore, the threshold for the image edge density feature used to distinguish between strong and weak edges is adaptively set based on the average gradient value of the image, and the specific calculation formula is as follows: ; In the calculation of the local contrast features, the standard deviation of the grayscale image is used for normalization. The specific calculation formula is as follows: ; In the calculation of the image edge gradient features, the Sobel operator is used to extract the gradient and calculate the average gradient value. The specific calculation formula is as follows: ; In the calculation of the Laplacian variance feature of the image, the Laplacian operator is used to calculate the standard deviation of the Laplacian response. The specific calculation formula is as follows: ; Final Sharpness Score ; Where edgeRatio is the proportion of pixels with strong edges, grayStdDev is the standard deviation of the grayscale image, meanGradient is the average gradient value, lapStdDev is the standard deviation of the Laplacian response, and a, b, c, d, e, f, and g are preset weight coefficients, and f+g=1.

[0009] Furthermore, the step of inputting each image patch into a trained convolutional neural network image patch sharpness classification model to obtain the sharpness classification result for each image patch specifically includes: The optimal focal plane image is divided into multiple initial patches of a preset size; Each initial patch is input into a ResNet18 tri-class classification model pre-trained on ImageNet, which outputs the probability that the patch belongs to one of three categories: clear, blurry, or uncertain. The sharpness status is determined based on the relationship between the classification probability and the preset probability threshold. For tiles that are determined to be uncertain, recursively subdivide them by dividing them into two equal parts along the length and width directions to generate four sub-tiles with their size halved. These sub-tiles are then re-input into the model for inference until the sub-tiles reach the preset minimum tile size of 56×56 pixels or the larger of the probability of the clear class and the probability of the fuzzy class in the sub-tiles meets the preset confidence threshold ≥ 0.85.

[0010] Furthermore, based on the sharpness classification results, a replacement patch is searched and selected from the multi-layer Z-axis image sequence for patches whose sharpness does not meet the preset conditions. Specifically, this includes: Blocks classified as blurry in the sharpness classification are identified as blocks whose sharpness does not meet the preset conditions; On the adjacent focal plane image in the multi-layer Z-axis image sequence, locate the candidate patch with the same coordinate position as the patch; Calculate the sharpness score and sharpness probability of each candidate patch in the convolutional neural network patch sharpness classification model, and then calculate the scores based on the comprehensive index. Sort the candidate tiles and select The largest one is used as the replacement tile; among them, The sharpness score is calculated based on the multi-feature fusion of candidate image patches. The clear probability output by the model. , For the preset weighting coefficients, and , .

[0011] Furthermore, the step of locally registering and feathering the replacement patch with the corresponding position in the optimal focal plane image specifically includes: Geometric registration is performed between the replacement patch and the corresponding region in the optimal focal plane image to eliminate positional deviations; A feathered blending region with a preset overlap rate is set between the registered replacement patch and the adjacent patch. The preset overlap rate is 20% of the patch side length. Pixel blending is performed in the overlapping region using a linear gradient from weight 0 to weight 1 to achieve natural connection.

[0012] A second aspect of the present invention provides a fluorescence cell multilayer microscopy imaging system based on multi-feature fusion, comprising: The image acquisition module is used to acquire a multi-layer Z-axis image sequence of fluorescent cell samples at XY scanning points. The multi-layer Z-axis image sequence includes an optimal focal plane image and its neighboring focal plane images. The optimal focal plane is determined by autofocus, which includes: calculating a multi-feature fusion sharpness score for each focal plane image and determining the optimal focal plane based on the score. The multi-feature fusion sharpness score is calculated by fusing at least two of the following image features: image edge gradient features, image Laplacian variance features, image edge density features, and image local contrast features. The tile segmentation and classification module is used to divide the optimal focal plane image into multiple initial tiles, and input each initial tile into a trained convolutional neural network tile sharpness classification model to obtain the probability that each tile belongs to three categories: sharp, blurry, and uncertain. For tiles that are determined to be uncertain, recursively binary subdivide them to generate multiple sub-tiles and re-input them into the model for inference until the sub-tiles reach a preset minimum tile size or the classification confidence of the sub-tiles meets a preset confidence threshold, so as to obtain the final tile-level sharpness classification result. The replacement tile search module is used to identify tiles with blurred sharpness classification results as tiles to be repaired, while keeping the corresponding areas of sharp tiles in the best focal plane image unchanged; on the adjacent focal plane images in the multi-layer Z-axis image sequence, candidate tiles with the same coordinate positions as the tile to be repaired are located, and the sharpness evaluation score and sharpness probability in the classification model of each candidate tile are calculated respectively. The optimal candidate tile is selected as the replacement tile. The local registration and fusion module is used to perform local geometric registration between the replacement patch and the corresponding region in the optimal focal plane image to eliminate interlayer displacement deviation; a feathered fusion region with a preset overlap rate is set between the registered replacement patch and the adjacent retained region; pixel fusion is performed in the overlapping region using a gradient from the first weight to the second weight to generate the fused image of the XY scan point. The image stitching module is used to stitch together the fused image of all XY scan points and output a full-field fluorescence microscopy stitched image.

[0013] A third aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps in the multi-feature fusion-based fluorescence cell multilayer microscopy imaging method as described in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the multi-feature fusion-based fluorescence cell multilayer microscopy imaging method as described in the first aspect of the present invention.

[0015] A fifth aspect of the present invention provides a computer program product comprising software code, wherein the program in the software code performs the steps of the multi-feature fusion-based fluorescence cell multilayer microscopy imaging method as described in the first aspect of the present invention.

[0016] Compared with existing technologies, this invention provides a multi-feature fusion-based method and system for fluorescence cell multilayer microscopy imaging. By pre-acquiring a multilayer image stack along the depth direction of the sample, a convolutional neural network is used to perform fine-grained sharpness discrimination on local regions of the optimal focal plane image. The system automatically selects sharper versions from neighboring focal planes to replace blurred areas, and finally achieves seamless stitching through geometric registration and feathering fusion. This enables full-field, high-resolution microscopic imaging of fluorescence samples with uneven thickness, significantly improving imaging integrity and local detail restoration capabilities in weak fluorescence scenes.

[0017] Compared with existing overall fusion techniques, this invention only performs cross-layer search and replacement on local patches that are determined to be blurry, while keeping the sharp areas in the optimal focal plane unchanged, thus avoiding artifacts and detail loss introduced by overall fusion in sharp areas. Compared with pixel-level fusion techniques based on depth map correction, this invention introduces a three-classification mechanism of sharp / blurry / uncertain and a recursive subdivision mechanism, and uses deep learning semantic features to distinguish the critical blur state, solving the problems of peak drift of a single sharpness index and misjudgment of local extrema in weak fluorescence scenes. Attached Figure Description

[0018] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0019] Figure 1 This is a flowchart of a fluorescence cell multilayer microscopy imaging method based on multi-feature fusion provided in Embodiment 1 of the present invention.

[0020] Figure 2 The flowchart of autofocus (coarse positioning scan and fine search scan) provided in Embodiment 1 of the present invention.

[0021] Figure 3 The flowchart is shown for the multi-feature fusion sharpness evaluation algorithm provided in Embodiment 1 of the present invention.

[0022] Figure 4 This is a schematic diagram of the XY serpentine scanning path provided in Embodiment 1 of the present invention.

[0023] Figure 5 This is a flowchart of ResNet18 tile sharpness classification and adaptive recursive block division provided in Embodiment 1 of the present invention.

[0024] Figure 6 This is a flowchart of cross-focal plane candidate search and replacement decision-making provided in Embodiment 1 of the present invention.

[0025] Figure 7 This is a schematic diagram of local registration and feathering fusion provided in Embodiment 1 of the present invention.

[0026] Figure 8 This is a schematic diagram of multi-layer Z-axis scanning acquisition provided in Embodiment 1 of the present invention.

[0027] Figure 9 This is a flowchart of the large field-of-view image stitching output provided in Embodiment 1 of the present invention.

[0028] Figure 10 This is an architectural diagram of a fluorescence cell multilayer microscopy imaging system based on multi-feature fusion, provided in Embodiment 2 of the present invention. Detailed Implementation

[0029] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0032] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0033] Example 1 like Figure 1 This embodiment provides a fluorescence cell multilayer microscopy imaging method based on multi-feature fusion, including: S1. Obtain a multi-layer Z-axis image sequence of fluorescent cell samples at XY scanning points. The multi-layer Z-axis image sequence includes the optimal focal plane image and its neighboring focal plane images. The optimal focal plane is determined by autofocus. The autofocus includes: calculating a multi-feature fusion sharpness score for each focal plane image and determining the optimal focal plane based on the score. The multi-feature fusion sharpness score is calculated by fusing at least two of the following image features: image edge gradient features, image Laplacian variance features, image edge density features, and image local contrast features.

[0034] Specifically, this step includes the following sub-steps: S11. Hardware Preparation and Image Acquisition: The prepared fluorescent cell slide samples are placed on a three-axis motorized displacement platform, and fluorescence images are acquired using a USB 3.0 industrial camera. The hardware configuration includes: a motorized displacement platform with a 100mm×100mm XY-axis travel, a 10mm Z-axis travel, and a repeatability of 1μm; an industrial camera with a resolution of 2440×2048 pixels, supporting fluorescence imaging mode; and a fluorescence illumination module with excitation wavelengths of 488nm / 561nm. The system controls the platform's movement and camera acquisition via a host computer to generate a sequence of microscopic images.

[0035] S12. Image Preprocessing: The acquired color fluorescence images are preprocessed. First, the Bgr2Gray method is used to convert the color images to grayscale images to reduce the interference of color channels on sharpness evaluation. Then, a 3×3 Gaussian kernel is used for smoothing and denoising, with σ=0.8 (in the OpenCV implementation, when σ=0, it is automatically calculated as 0.8 according to the kernel size) to suppress high-frequency noise and preserve edge information. The preprocessed images are used for subsequent sharpness evaluation and neural network classification.

[0036] S13, Autofocus (determine the optimal focal plane): Perform autofocus at each XY scan point, such as... Figure 2 As shown, it specifically includes: ① Coarse localization scan: With the current Z-axis position as the center, perform a uniform step scan within a range of 100μm above and below with a step size of 20μm, acquire 11 focal plane images, calculate the multi-feature fusion sharpness score of each image, and determine the peak position of the coarse localization scan.

[0037] ② Fine search scan: Using the peak position determined by the coarse positioning scan as the center, a fine scan is performed within a range of 20μm above and below with a step size of 2μm. 21 focal plane images are acquired, and the multi-feature fusion sharpness score of each image is calculated. The Z-axis coordinate with the highest score is determined as the optimal focal plane.

[0038] The calculation method for the multi-feature fusion sharpness score is as follows: like Figure 3 As shown, the acquired focal plane image is sequentially subjected to grayscale conversion, 3×3 Gaussian smoothing, Sobel gradient extraction, Laplacian variance calculation, edge density statistics, and local contrast calculation. The final sharpness score is then calculated by fusing the results using the following formula: Edge density weights:

[0039] in, This refers to the percentage of pixels with strong edges. The strong edge threshold is adaptively set to... , This represents the average gradient value.

[0040] Local contrast weight:

[0041] in, denoted as the standard deviation of the grayscale image.

[0042] Gradient score:

[0043] Laplacian variance score:

[0044] in, denoted as the standard deviation of the Laplace response.

[0045] Final score:

[0046] The Z-axis coordinate with the highest final score is selected as the optimal focal plane.

[0047] S14. Multi-layer Z-axis scanning: Centered on the optimal focal plane, acquire 5 layers of images both vertically and horizontally along the Z-axis, with a layer stride of 2 μm, for a total of 11 focal plane image sequences, forming a complete Z-stack dataset covering cellular structural information at different depths. This multi-layer Z-axis image sequence includes the optimal focal plane image and its neighboring focal plane images, used for subsequent cross-layer replacement. Figure 8 As shown, the scanning range is ±10μm, with the optimal focal plane Z_best as the center, 5 layers above and below, and a layer step size of 2μm, for a total of 11 layers.

[0048] S2. Divide the optimal focal plane image into multiple initial patches, and input each patch into a trained convolutional neural network patch sharpness classification model to obtain the probability that each patch belongs to the three categories of sharp, blurry, and uncertain; recursively divide the patches that are determined to be uncertain into multiple sub-patterns and re-input them into the model for inference until the sub-patterns reach the preset minimum patch size or the classification confidence of the sub-patterns meets the preset confidence threshold, and obtain the final patch-level sharpness classification result.

[0049] Specifically, such as Figure 5 As shown, this step includes the following sub-steps: S21. Initial segmentation: Divide the optimal focal plane image into multiple initial blocks with preset sizes. In this embodiment, the initial block size is 224×224 pixels.

[0050] S22, Model Inference: Each initial image patch is input into a ResNet18 three-class classification model pre-trained on ImageNet. This model retains the residual connection structure, and the output layer is modified to classify three categories (clear, blurry, uncertain), using the Softmax activation function. The input layer is fixed as a 224×224 pixel RGB patch. The model outputs the probability that the patch belongs to one of the three categories: clear, blurry, or uncertain.

[0051] S23. Classification Determination: The sharpness status is determined based on the relationship between the classification probability and a preset probability threshold. The determination rule is: a clear probability ≥ 0.7 is determined to be sharp, a blur probability ≥ 0.7 is determined to be blurry, and anything in between is determined to be uncertain.

[0052] S24. Recursive Binary Subdivision: For patches deemed uncertain, recursively subdivide them by dividing the current patch in half along both its length and width, generating four sub-patterns. Each sub-pattern is half the size of the original patch and is then re-inputted into the model for inference. This recursive subdivision process continues until the sub-patterns reach a preset minimum patch size (56×56 pixels in this example) or the larger of the sharpness class probability and the fuzzy class probability in the sub-pattern meets a preset confidence threshold (≥0.85 in this example). The final patch-level sharpness classification result is obtained. The minimum patch size of 56×56 pixels is set because: below this size, the ResNet18 input layer needs to be oversampled to 224×224, leading to feature distortion and insufficient spatial context information for cell substructures, significantly reducing classification reliability.

[0053] S3. Based on the clarity classification results, search for and select replacement blocks from the multi-layer Z-axis image sequence for blocks whose clarity does not meet the preset conditions.

[0054] like Figure 6 As shown, the input blur patch searches for the corresponding patch in the candidate image sequence of the neighboring layer (Z_best±5 layers, stride 2μm, total 11 layers). The local sharpness score (multi-feature fusion formula) and ResNet18 classification probability (clear probability) of each candidate patch are calculated. The candidate patch with the highest score is selected for replacement and sub-pixel bias is eliminated by local geometric registration.

[0055] Specifically: Patches classified as blurry in the sharpness classification are identified as patches to be repaired, while maintaining the corresponding regions of sharp patches in the optimal focal plane image unchanged. Candidate patches with the same coordinates as the identified patch are located in the neighboring focal plane images (i.e., image sequences with 5 layers above and below the optimal focal plane and an inter-layer stride of 2μm) within the multi-layer Z-axis image sequence. The local sharpness score (which can be the same multi-feature fusion sharpness score as in S13) and its sharpness probability in the convolutional neural network patch sharpness classification model are calculated for each candidate patch. The optimal candidate patch is then selected as the replacement patch.

[0056] When implementing the replacement decision, for each candidate map patch, its multi-feature fusion sharpness score is calculated. and the probability of clarity of ResNet18 output According to comprehensive indicators Sort the candidate tiles and select The largest one is used as the replacement tile; among them, The sharpness score is calculated based on the multi-feature fusion of candidate image patches. The clear probability output by the model. , The weighted coefficients are preset and all are greater than zero. This joint decision-making mechanism takes into account both statistical clarity and deep learning semantic discrimination, avoiding accidental misjudgments by a single indicator in weak fluorescence scenarios.

[0057] S4. Perform local geometric registration and feathering fusion of the replacement patch with the corresponding region in the optimal focal plane image to generate a fused image of the XY scan point.

[0058] like Figure 7 As shown, a 20% overlap area is set between the replaced tile and the adjacent tile, and a feathering blending formula is used. ,in A linear gradient from 0 to 1 is used to achieve a smooth transition at the boundary.

[0059] Specifically, this step includes the following sub-steps: S41. Local geometric registration: Perform geometric registration between the replacement block and the corresponding area in the optimal focal plane image to eliminate interlayer displacement deviation and ensure that the replacement position is aligned.

[0060] S42, Feathering Blending: A feathering blending region with a preset overlap rate is set between the registered replacement patch and the adjacent retained region. In this embodiment, the preset overlap rate is 20% of the patch side length. Pixel blending is performed in the overlapping region using a linear gradient from weight 0 to weight 1, that is, the linear gradient weight transitions from 0 to 1, achieving a natural connection between adjacent patches and reducing boundary artifacts.

[0061] S5. Stitch together the fused images of all XY scan points to output a full-field fluorescence microscopy stitched image.

[0062] Specifically, such as Figure 4 and Figure 9 As shown, a large-field-of-view image stitching is performed on the fusion results of all XY scan points. Geometric positioning is performed based on the physical coordinates (X, Y, Z) of each point, and brightness normalization is used to eliminate illumination differences between regions, outputting a high-resolution full-field fluorescence microscopy stitched image. The XY scan path adopts a serpentine scanning path: scanning line by line along the X-axis, odd-numbered lines move in the positive direction, and even-numbered lines move in the negative direction, with line transitions via Y-axis stepping to reduce platform idle travel and improve scanning efficiency.

[0063] In steps S1 to S5 above, the multi-feature fusion sharpness evaluation model and the convolutional neural network patch sharpness classification model form a coarse-fine collaborative mechanism: the multi-feature fusion sharpness evaluation model, based on interpretable image statistical features, performs millisecond-level real-time scoring on each candidate focal plane during the autofocus stage, quickly locating the optimal focal plane; the convolutional neural network patch sharpness classification model, based on deep semantic features, performs fine-grained quality arbitration on local patches during the multi-layer fusion stage, identifying critical blur states that are difficult to distinguish using traditional statistical methods. The two complement each other in the information, time, and spatial dimensions, jointly improving focusing stability and fusion accuracy in weak fluorescence scenes.

[0064] Example 2 like Figure 10 As shown, this embodiment provides a fluorescence cell multilayer microscopy imaging system based on multi-feature fusion, including: The image acquisition module is used to acquire multi-layer Z-axis image sequences of fluorescent cell samples at XY scanning points. These multi-layer Z-axis image sequences include the optimal focal plane image and its neighboring focal plane images. Specifically, this module includes: a hardware layer (for sample carrying, displacement control, and fluorescence image acquisition, including a positive motion three-axis motorized displacement platform, a USB 3.0 industrial camera, and a fluorescence illumination module); an autofocus module (for achieving autofocus through an improved hill-climbing search strategy and a multi-feature fusion sharpness evaluation method, where the coarse positioning scan range is ±100μm with a step size of 20μm, the fine search scan range is ±20μm with a step size of 2μm, and the multi-feature fusion sharpness evaluation method fuses edge gradient, Laplacian variance, edge density, and local contrast features, and calculates a score according to a formula); and a Z-stack acquisition module (for performing multi-layer Z-axis scanning acquisition near the optimal focal plane, with 5 layers above and below, a step size of 2μm, for a total of 11 layers).

[0065] The image patch segmentation and classification module divides the optimal focal plane image into multiple initial patches. Each patch is input into a trained convolutional neural network patch sharpness classification model to obtain the probability of each patch belonging to three categories: sharp, blurry, and uncertain. Patches classified as uncertain are recursively subdivided into two sub-patches, generating multiple sub-patches which are then re-inputted into the model for inference until the sub-patches reach a preset minimum patch size or the classification confidence of the sub-patches meets a preset confidence threshold, resulting in the final patch-level sharpness classification result. Specifically, this module implements adaptive recursive patching: the initial patch is 224×224 pixels, and inference is performed based on a ResNet18 three-class classification model (pre-trained on ImageNet, outputting sharp / blurry / uncertain probabilities). Uncertain patches are recursively subdivided into two sub-patches to a minimum of 56×56 pixels or the larger of the sharp and blurry probabilities is ≥0.85. The classification thresholds are sharp ≥0.7 and blurry ≥0.7.

[0066] The replacement tile search module is used to identify tiles with blurred sharpness classification results as tiles to be repaired, while keeping the corresponding areas of sharp tiles in the best focal plane image unchanged; from the neighboring focal plane images (5 layers above and below) in the multi-layer Z-axis image sequence, candidate tiles with the same coordinate positions as the tile to be repaired are located, and the sharpness evaluation score and sharpness probability in the classification model are calculated for each candidate tile, according to a comprehensive index. Sort, select The largest one is used as the replacement tile; among them , .

[0067] The local registration and fusion module is used to perform local geometric registration between the replacement patch and the corresponding region in the optimal focal plane image to eliminate interlayer displacement deviation. A feathered fusion region with a preset overlap rate is set between the registered replacement patch and the adjacent retained region. Within the overlapping region, pixel fusion is performed using a linear gradient from weight 0 to weight 1 to generate a fused image of the XY scan point. Specifically, geometric registration is first performed to eliminate interlayer displacement deviation, and then feathering fusion with a preset overlap rate of 20% of the patch side length is used. A linear gradient from weight 0 to weight 1 is used within the overlapping region to achieve a natural transition.

[0068] The image stitching module is used to stitch together the fused image of all XY scan points to output a full-field fluorescence microscopy stitched image. A serpentine scanning path is used (odd rows forward, even rows reverse), with geometric positioning and brightness normalization based on physical coordinates.

[0069] Example 3 Embodiment 3 of the present invention provides an electronic device.

[0070] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. The processor includes, but is not limited to, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a tensor processor (TPU), or an artificial intelligence acceleration chip. The program is used to execute the steps in the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion as described in Embodiment 1 of the present invention.

[0071] The detailed steps are the same as those of the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion provided in Example 1, and will not be repeated here.

[0072] Example 4 Embodiment 4 of the present invention provides a computer-readable storage medium.

[0073] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion as described in Embodiment 1 of the present invention.

[0074] The detailed steps are the same as those of the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion provided in Example 1, and will not be repeated here.

[0075] Example 5 Embodiment 5 of the present invention provides a computer program product.

[0076] A computer program product includes software code, wherein the program in the software code performs the steps of the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion as described in Embodiment 1 of the present invention.

[0077] The detailed steps are the same as those of the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion provided in Example 1, and will not be repeated here.

[0078] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages. For example, in one implementation, the methods and systems can be developed based on deep learning frameworks (such as TensorFlow, PyTorch, etc.) and using the Python language. Those skilled in the art will understand that other suitable programming languages ​​or tools can also be used for implementation without departing from the core ideas of the present invention.

[0079] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A multi-feature fusion-based fluorescence cell multilayer microscopy imaging method, characterized in that, include: Acquire a multi-layer Z-axis image sequence of fluorescent cell samples at XY scan points, wherein the multi-layer Z-axis image sequence includes the optimal focal plane image and its neighboring focal plane images; The optimal focal plane is determined by autofocus. The optimal focal plane image is divided into multiple initial patches. Each initial patch is input into a trained convolutional neural network patch sharpness classification model to obtain the probability that each patch belongs to one of the three categories: sharp, blurry, or uncertain. For tiles that are determined to be uncertain, recursively divide and subdivide them into multiple sub-tiles and re-input them into the model for inference until the sub-tiles reach the preset minimum tile size or the classification confidence of the sub-tiles meets the preset confidence threshold, and the final tile-level sharpness classification result is obtained. The patches that are classified as blurry in the sharpness classification are identified as patches to be repaired, while keeping the corresponding areas of the sharp patches in the best focal plane image unchanged; On the adjacent focal plane image in the multi-layer Z-axis image sequence, candidate patches with the same coordinate position as the patch to be repaired are located. The sharpness evaluation score of each candidate patch and its sharpness probability in the classification model are calculated respectively. The optimal candidate patch is selected as the replacement patch. The replacement patch is locally geometrically registered with the corresponding region in the optimal focal plane image to eliminate interlayer displacement deviation; a feathered fusion region with a preset overlap rate is set between the registered replacement patch and the adjacent retained region; pixel fusion is performed in the overlapping region using a gradient from the first weight to the second weight to generate the fused image of the XY scan point. The fused images of all XY scan points are stitched together to output a full-field fluorescence microscopy stitched image.

2. The method as described in claim 1, characterized in that, The autofocus specifically includes: Centered on the current Z-axis position, a coarse positioning scan is performed within a range of ±100μm with a step size of 20μm to acquire the first set of focal plane images. The multi-feature fusion sharpness score of each image is calculated to determine the peak position of the coarse positioning scan. Using the peak position determined by the coarse positioning scan as the center, a fine search scan is performed within a range of ±20μm with a step size of 2μm to acquire the second set of focal plane images. The multi-feature fusion sharpness score of each image is calculated, and the Z-axis coordinate with the highest score is determined as the optimal focal plane. The Z-axis range of the fine search scan is smaller than that of the coarse positioning scan, and the step size of the fine search scan is smaller than that of the coarse positioning scan.

3. The method as described in claim 2, characterized in that, The threshold for the image edge density feature used to distinguish between strong and weak edges is adaptively set based on the average gradient value of the image, and the specific calculation formula is as follows: ； In the calculation of the local contrast features, the standard deviation of the grayscale image is used for normalization. The specific calculation formula is as follows: ； In the calculation of the image edge gradient features, the Sobel operator is used to extract the gradient and calculate the average gradient value. The specific calculation formula is as follows: ; In the calculation of the Laplacian variance feature of the image, the Laplacian operator is used to calculate the standard deviation of the Laplacian response. The specific calculation formula is as follows: ; Final Sharpness Score ; in, For the percentage of pixels with strong edges, The standard deviation of the grayscale image. The average gradient value, The standard deviation of the Laplace response. , , , , , , For the preset weighting coefficients, and .

4. The method as described in claim 1, characterized in that, The step of inputting each image patch into a trained convolutional neural network image patch sharpness classification model to obtain the sharpness classification result for each image patch specifically includes: The optimal focal plane image is divided into multiple initial patches of a preset size; Each initial patch is input into a ResNet18 tri-class classification model pre-trained on ImageNet, which outputs the probability that the patch belongs to one of three categories: clear, blurry, or uncertain. The sharpness status is determined based on the relationship between the classification probability and the preset probability threshold. For tiles that are determined to be uncertain, recursively subdivide them by dividing them into two equal parts along the length and width directions to generate four sub-tiles with their size halved. These sub-tiles are then re-input into the model for inference until the sub-tiles reach the preset minimum tile size of 56×56 pixels or the larger of the probability of the clear class and the probability of the fuzzy class in the sub-tiles meets the preset confidence threshold ≥ 0.

85.

5. The method as described in claim 1, characterized in that, Based on the sharpness classification results, the process involves searching for and selecting replacement patches from the multi-layer Z-axis image sequence for patches whose sharpness does not meet the preset conditions. Specifically, this includes: Blocks classified as blurry in the sharpness classification are identified as blocks whose sharpness does not meet the preset conditions; On the adjacent focal plane image in the multi-layer Z-axis image sequence, locate the candidate patch with the same coordinate position as the patch; Calculate the sharpness score and sharpness probability of each candidate patch in the convolutional neural network patch sharpness classification model, and then calculate the scores based on the comprehensive index. Sort the candidate tiles and select The largest one is used as the replacement tile; among them, The sharpness score is calculated based on the multi-feature fusion of candidate image patches. The clear probability output by the model. , For the preset weighting coefficients, and , .

6. The method as described in claim 1, characterized in that, The step of locally registering and feathering the replacement patch with the corresponding position in the optimal focal plane image specifically includes: Geometric registration is performed between the replacement patch and the corresponding region in the optimal focal plane image to eliminate positional deviations; A feathered blending region with a preset overlap rate is set between the registered replacement patch and the adjacent patch. The preset overlap rate is 20% of the patch side length. Pixel blending is performed in the overlapping region using a linear gradient from weight 0 to weight 1 to achieve natural connection.

7. A fluorescence cell multilayer microscopy imaging system based on multi-feature fusion, characterized in that, include: The image acquisition module is used to acquire a multi-layer Z-axis image sequence of fluorescent cell samples at XY scanning points, wherein the multi-layer Z-axis image sequence includes the optimal focal plane image and its neighboring focal plane images; The optimal focal plane is determined by autofocus, which includes: calculating a multi-feature fusion sharpness score for each focal plane image and determining the optimal focal plane based on the score; the multi-feature fusion sharpness score is obtained by fusing at least two of the following image features: image edge gradient features, image Laplacian variance features, image edge density features, and image local contrast features; The tile segmentation and classification module is used to divide the optimal focal plane image into multiple initial tiles, and input each initial tile into a trained convolutional neural network tile sharpness classification model to obtain the probability that each tile belongs to three categories: sharp, blurry, and uncertain. For tiles that are determined to be uncertain, recursively binary subdivide them to generate multiple sub-tiles and re-input them into the model for inference until the sub-tiles reach a preset minimum tile size or the classification confidence of the sub-tiles meets a preset confidence threshold, so as to obtain the final tile-level sharpness classification result. The replacement tile search module is used to identify tiles with blurred sharpness classification results as tiles to be repaired, while keeping the corresponding areas of sharp tiles in the best focal plane image unchanged; on the adjacent focal plane images in the multi-layer Z-axis image sequence, candidate tiles with the same coordinate positions as the tile to be repaired are located, and the sharpness evaluation score and sharpness probability in the classification model of each candidate tile are calculated respectively. The optimal candidate tile is selected as the replacement tile. The local registration and fusion module is used to perform local geometric registration between the replacement patch and the corresponding region in the optimal focal plane image to eliminate interlayer displacement deviation; a feathered fusion region with a preset overlap rate is set between the registered replacement patch and the adjacent retained region; pixel fusion is performed in the overlapping region using a gradient from the first weight to the second weight to generate the fused image of the XY scan point. The image stitching module is used to stitch together the fused image of all XY scan points to output a full-field fluorescence microscopy stitched image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the fluorescence cell multilayer microscopy imaging method based on multi-feature fusion as described in any one of claims 1 to 6.

10. A computer program product, comprising software code, characterized in that, The program in the software code performs the steps of the fluorescence cell multilayer microscopy method based on multi-feature fusion as described in any one of claims 1 to 6.

Images