Deep learning driven sputum smear pathogen image recognition method and system thereof

Abstract

The present application relates to the technical field of medical image processing, and discloses a deep learning driven sputum smear pathogen image recognition method and system, which comprises five steps of dye normalization preprocessing, adaptive contrast enhancement, U-Net-based semantic segmentation, morphological feature analysis, and classification recognition and quantitative statistics. Color deconvolution is used to eliminate dye batch differences, and multi-scale adaptive contrast enhancement is used to improve the distinction between pathogens and background, so as to realize pixel-level accurate segmentation of gram-positive bacteria and gram-negative bacteria and automatic classification and recognition of coccus, bacillus and fungal spores, and the method is suitable for auxiliary diagnosis of respiratory infection etiology in a clinical microbiology laboratory.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a deep learning-driven method and system for pathogen image recognition of sputum smears. Background Technology

[0002] Respiratory tract infections are among the most prevalent infectious diseases in clinical practice, and accurate and rapid identification of pathogens is crucial for guiding clinical medication. Gram staining microscopy of sputum smears is one of the most widely used methods for etiological diagnosis of respiratory tract infections. This method involves preparing smears from patient sputum specimens, Gram staining them, and then observing them under an optical microscope. Laboratory personnel determine the type and severity of the pathogen based on its staining and morphological characteristics. However, traditional manual microscopy has several limitations. Firstly, pathogens in sputum smears are typically small; Gram-positive bacteria appear deep purple, while Gram-negative bacteria appear light red, resulting in low contrast with the background area. Even under high magnification, they are difficult to distinguish clearly, especially when the smear thickness is uneven or there are batch-to-batch variations in staining conditions, making the laboratory personnel's judgment susceptible to subjective influence. Secondly, in addition to pathogens, sputum specimens often contain a large number of interfering components such as epithelial cell fragments, leukocyte fragments, and mucus filaments. These non-pathogen components may morphologically resemble some pathogens, further increasing the difficulty of identification.

[0003] Chinese patent CN107464234A discloses a deep learning recognition system and method for lung nodule images based on an RGB channel overlay method. This method targets lung CT slice images. A data reading module reads the original lung CT image and converts it into a BMP image. A lung parenchyma extraction module enhances the lung parenchyma region of the image. A lung nodule extraction module extracts the coordinates of the lung nodule center from the lung parenchyma image sequence. An RGB channel overlay module cuts and overlays three consecutive lung parenchyma images into a pseudo-color image to enhance the difference between lung nodules and healthy tissue. Finally, an AlexNet deep convolutional neural network is used for learning, training, and prediction recognition. Although this scheme achieves automatic recognition of lung nodules in lung CT images using deep learning, its technical solution is mainly geared towards the three-dimensional continuous slice analysis scenario of CT tomographic images and is not suitable for the completely different imaging modality of sputum smear microscopic images. Specifically, this scheme has the following shortcomings: First, CT images and microscopic stained images differ fundamentally in imaging principles, contrast characteristics, and noise patterns. The RGB channel overlay method in this scheme relies on the spatial relationship between consecutive slices to enhance target features, while sputum smear microscopic images are single-layer planar images and do not possess this interlayer relationship of consecutive slices. Second, the AlexNet network used in this scheme can only output image-level classification results and cannot achieve pixel-level localization and segmentation of multiple scattered microscopic pathogens in microscopic images. Third, this scheme does not involve color normalization processing for stained smear images and cannot solve the image color shift problem caused by differences in staining conditions between different batches.

[0004] Therefore, clinical microbiology laboratories urgently need an image analysis method and system capable of correcting staining differences, accurately segmenting pathogens, performing quantitative morphological analysis, and automatically classifying multiple types of pathogens from Gram-stained sputum smear microscopic images, in order to improve the accuracy and efficiency of image-assisted diagnosis of pathogens. In recent years, although deep learning technology has made significant progress in the field of medical image analysis, especially semantic segmentation networks based on encoder-decoder structures, which have demonstrated excellent performance in cell and tissue segmentation tasks, applying these technologies to sputum smear pathogen identification scenarios still faces unique challenges such as staining inconsistencies, extremely small target sizes, class imbalance, and strong background interference. Therefore, a complete collaborative processing workflow from preprocessing to segmentation and recognition to quantitative statistics is urgently needed. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a deep learning-driven method and system for pathogen image recognition in sputum smears, aiming to solve the technical problems of pathogens being small in size and having low contrast with the background in sputum smear microscopic images, traditional image processing methods being sensitive to staining differences, and bacteria being easily confused with cell debris.

[0006] One aspect of the present invention provides a deep learning-driven method for pathogen image recognition in sputum smears, comprising the following steps: Step S1, staining normalization preprocessing step, acquiring a microscopic image of Gram-stained sputum smears, performing white balance correction on the microscopic image, and then using a color deconvolution algorithm based on optical density space transformation to decompose the microscopic image into a hematoxylin channel image and an eosin channel image, calculating the staining intensity normalization coefficient, and performing staining batch difference correction on the microscopic image, outputting a color-normalized image; Step S2, adaptive contrast enhancement step, converting the color-normalized image to the LAB color space, extracting the L luminance channel, performing multi-scale adaptive contrast enhancement processing, improving the contrast between pathogens and background through local region windowing and nonlinear mapping transformation that limits contrast, performing bilinear interpolation fusion on the enhancement results of adjacent windows to eliminate boundary jumps, and outputting an enhanced image; Step S3, semantic segmentation step, dividing the enhanced image into a semantic segmentation image, and then performing semantic segmentation on the enhanced image into a semantic segmentation image. The strong image input is based on a convolutional neural network segmentation model with a U-Net architecture. Multi-scale features are extracted layer by layer through the encoder path and upsampled layer by layer through the decoder path. High-resolution spatial information is fused using skip connections to achieve pixel-level multi-class classification. The output is a semantic segmentation mask containing Gram-positive bacteria regions, Gram-negative bacteria regions, and background regions. Step S4 is the morphological feature analysis step, which performs eight-neighbor connected component extraction on the pathogen regions in the segmentation mask. Morphological feature parameters such as area, major axis length, minor axis length, major-minor axis ratio, and circularity of each independent pathogen connected component are calculated by least bounding ellipse fitting. Step S5 is the classification, identification, and quantitative statistics step, which comprehensively utilizes staining category labels and morphological feature parameters to perform rule-based type determination on each pathogen, distinguishing between types such as cocci, bacilli, and fungal spores. The density distribution of each type of pathogen is statistically analyzed, and pseudo-color visualization result images and density heat maps are generated.

[0007] Another aspect of the present invention provides a deep learning-driven sputum smear pathogen image recognition system, comprising: a staining normalization module configured to perform white balance correction, color deconvolution, and staining batch difference correction on a microscopic image; an adaptive contrast enhancement module connected to the staining normalization module, configured to perform multi-scale adaptive contrast enhancement on the L channel of the color-normalized image in the LAB color space; a semantic segmentation module connected to the adaptive contrast enhancement module, configured to achieve pixel-level semantic segmentation through a pre-trained U-Net convolutional neural network; a morphological analysis module connected to the semantic segmentation module, configured to extract connected components and calculate morphological feature parameters through ellipse fitting; and a classification and quantitative statistics module connected to both the semantic segmentation module and the morphological analysis module, configured to perform pathogen type determination, density statistics, and pseudo-color visualization output.

[0008] The technical advantages of this invention are as follows: Staining differences between different batches of smears are eliminated through staining normalization preprocessing, ensuring that subsequent analysis results are unaffected by fluctuations in staining conditions; adaptive contrast enhancement significantly improves the grayscale difference between pathogens and the background, facilitating accurate extraction of pathogen regions by the segmentation model; pixel-level precise segmentation is achieved through a U-Net-based semantic segmentation network, enabling simultaneous differentiation between Gram-positive and Gram-negative bacteria; automatic identification and quantitative statistics of multiple pathogen types, such as cocci, bacilli, and fungal spores, are achieved through a combination of morphological feature analysis and classification recognition; and intuitive and efficient auxiliary diagnostic information is provided to clinicians through pseudo-color annotation and density heatmap generation. Attached Figure Description

[0009] Figure 1 This is a flowchart of a deep learning-driven pathogen image recognition method for sputum smears provided in an embodiment of the present invention.

[0010] Figure 2 This is an architecture diagram of the deep learning-driven sputum smear pathogen image recognition system provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0012] like Figure 1 As shown in the figure, this embodiment provides a deep learning-driven method for pathogen image recognition in sputum smears. This method comprises five core steps: staining normalization preprocessing (S1), adaptive contrast enhancement (S2), semantic segmentation (S3), morphological feature analysis (S4), and classification and quantitative statistics (S5). These steps form a sequential data flow and collaborative processing relationship. The specific implementation methods for each step will be described in detail below.

[0013] Step S1: Staining normalization preprocessing step. The core purpose of this step is to eliminate image color shifts caused by differences in staining solution concentration, staining time, and washing conditions between different batches of Gram-stained smears, ensuring that the input images processed in subsequent image analysis steps have consistent color distribution characteristics. In actual operation in clinical microbiology laboratories, even using the same staining procedure, smears prepared by different technicians or on different dates often show significant differences in hue and saturation under a microscope. If these batch differences are not corrected, they will directly affect the accuracy of the semantic segmentation model in distinguishing between Gram-positive and Gram-negative bacteria. Furthermore, the standard Gram staining procedure involves multiple steps, including crystal violet primary staining, iodine mordant staining, alcohol decolorization, and safranin counterstaining. Small variations in the operation time and reagent dosage at each step can cause shifts in the final image color characteristics. Therefore, staining normalization is a prerequisite for ensuring that the method of this invention can work stably under different laboratory conditions.

[0014] Specifically, the first step is to acquire a microscopic image of a Gram-stained sputum smear using a digital microscopic imaging device. In one embodiment of the present invention, the objective lens of the microscopic imaging device used has a magnification of 100x oil immersion lens, the image sensor resolution is 2048×1536 pixels, and the acquired microscopic image is an RGB three-channel color image, with the grayscale value of each channel ranging from 0 to 255.

[0015] After acquiring the microscopic image, white balance correction is performed first. White balance correction is the first step in the entire staining normalization process. Its purpose is to eliminate the overall color cast caused by variations in the microscope light source color temperature and differences in camera white balance settings, providing more accurate color input for subsequent color deconvolution. The white balance correction method involves automatically detecting background areas in the microscopic image. Preferably, this is done by statistically analyzing pixel areas with gray values ​​higher than 200, calculating the average gray values ​​of these background areas in the R, G, and B channels, and denoting them as follows: , and Then, the pixel values ​​of each channel are corrected as follows:

[0016] ,

[0017] ,

[0018] ,

[0019] in: , and The original image is in coordinates The gray values ​​of the R, G, and B channels at the specified location range from 0 to 255 and are dimensionless. , and This represents the corresponding channel's grayscale value after white balance correction; , and The preset reference white grayscale value is set to 240 in one embodiment of the present invention; , and These are the average gray levels of the background region in the R, G, and B channels of the current image, respectively. This white balance correction step can eliminate the overall color cast caused by inconsistent color temperature of the microscope light source.

[0020] After white balance correction, color deconvolution based on optical density space transformation is performed on the corrected image. The basic principle of color deconvolution is to convert the microscopic image from the RGB color space to the optical density space, and then decompose the mixed color signal into the contribution components of each individual dye according to the known dye color vector matrix. The optical density space transformation formula is as follows:

[0021] ,

[0022] in: For color channels Optical density value, Dimensionless; To correct the image in channels The grayscale value ranges from 1 to 255. The incident light intensity is taken as 255 in this embodiment. When the value is 0, to avoid infinite values ​​in logarithmic operations, the value will be... The cutoff value is 1. A higher optical density value indicates a deeper staining at that location and more light absorption.

[0023] In optical density space, a Gram staining image can be represented as a linear superposition of two staining components. The color deconvolution matrix used in this invention... It is a 2×3 matrix, where the first row is the standard color vector corresponding to the crystal violet staining channel. The first row represents the spectral absorption characteristics of Gram-positive bacteria staining; the second row is the standard color vector corresponding to the safranin counterstaining channel. This represents the spectral absorption characteristics of Gram-negative bacteria after counterstaining. Preferably, the above color vector can be calibrated using standard color chart samples according to the specific batch of staining reagents used in the laboratory. Optical density vector Vector of staining component concentration The relationship between them is:

[0024] ,

[0025] in: It is a 3×1 column vector composed of the optical density of each channel; Deconvolution matrix for color The transpose of has a dimension of 3×2; This is a 2×1 column vector consisting of the concentration values ​​of the two staining components. Corresponding to crystal violet concentration, The corresponding safranin concentrations are all dimensionless normalized values. By solving the overdetermined system of equations using the least squares method, the concentration values ​​of each staining component can be obtained, thereby separating the hematoxylin channel image and the eosin channel image.

[0026] After obtaining the staining separation results, the staining intensity normalization coefficient is further calculated. In one embodiment of the present invention, a set of reference images with standard staining quality is selected, and the mean value of the crystal violet channel concentration in the reference image set is calculated. and standard deviation and the mean concentration of sand and yellow channels and standard deviation For the current image to be processed, the mean value of its crystal violet channel concentration is also calculated. and standard deviation and the mean concentration of sand and yellow channels and standard deviation The transformation formula for staining normalization is:

[0027] ,

[0028] in: For the current image, the first The original concentration values ​​of the staining components, ; The concentration value is the normalized value; and The number of the current image The mean and standard deviation of the concentrations of the components; and These are the reference images in the set. The mean and standard deviation of the component concentrations are calculated. This normalization operation essentially aligns the color distribution of the current image to the color distribution of the reference image, thereby eliminating color differences between batches. Finally, the normalized concentration values ​​are inversely transformed back to the RGB color space using a color deconvolution matrix to obtain a color-normalized image, which serves as the input for step S2.

[0029] Step S2: Adaptive Contrast Enhancement Step. This step receives the color-normalized image output from Step S1 and performs multi-scale adaptive contrast enhancement processing on it to increase the gray-level difference between pathogen regions and background regions, enabling the subsequent semantic segmentation model to more accurately distinguish between pathogen and non-pathogen regions. In sputum smear microscopic images, the staining depth of Gram-positive and Gram-negative bacteria differs little from the gray-level value of the surrounding background, especially in areas where the smear is thicker or the staining is lighter, where pathogens almost blend into the background. Traditional global histogram equalization methods easily lead to over-enhancement or under-enhancement of local regions. Therefore, this step uses a contrast-limited adaptive histogram equalization method to solve this problem.

[0030] In practice, the color-normalized image is first converted from the RGB color space to the LAB color space. The L luminance channel is extracted for subsequent contrast enhancement processing, while the A and B channels remain unchanged. This is done to enhance only the luminance information without altering the image's hue and saturation, thus preserving the deep purple staining characteristics of Gram-positive bacteria and the light red staining characteristics of Gram-negative bacteria. The RGB to LAB color space conversion requires first converting the RGB values ​​to the XYZ color space as an intermediate transition, and then converting from XYZ to LAB space. In this embodiment, a D65 standard light source is used as the white point reference to ensure the accuracy of the color space conversion. After LAB space conversion, the L channel value ranges from 0 to 100, where 0 represents pure black and 100 represents pure white.

[0031] Then, the L-luminance channel image is divided into multiple non-overlapping local region windows according to a preset window size. Preferably, the window size is set to 64×64 pixels. This size selection takes into account two factors: on the one hand, the window size should not be too small, otherwise the number of pixels contained in a single window will be insufficient to construct a statistically significant gray-level histogram; on the other hand, the window size should not be too large, otherwise it will be impossible to effectively capture the local gray-level differences between different regions of the smear. In practical applications, the window size can be adjusted within the range of 16×16 pixels to 128×128 pixels according to the resolution of the microscopic image and the average size of the pathogen.

[0032] For each local region window, its grayscale histogram is calculated, and a contrast limiting operation is performed on this histogram. The core idea of ​​contrast limiting is to set a clipping threshold. When the frequency of a certain grayscale level in the histogram exceeds this clipping threshold, the excess is evenly distributed across the entire grayscale range. Then, a cumulative distribution function mapping is performed on the clipped histogram. The formula for calculating the clipping threshold is as follows:

[0033] ,

[0034] in: This is the cropping threshold, expressed in pixels. This represents the total number of pixels within the current window. For a 64x64 window, ; This represents the number of gray levels, with a value of 256. The contrast limiting factor has a value range of 0 to 1. In this embodiment... Take 0.5; The maximum gain slope parameter controls the maximum magnitude of local contrast enhancement. In this embodiment... A value of 3.0 corresponds to a contrast limit threshold of approximately 2.0. The basis for selecting this parameter range is: when... The enhancement effect is not obvious when the size is too small, but when the size is too small, the enhancement effect is not obvious. Excessive noise can be amplified, as experiments have shown. A good balance can be achieved between enhancement and noise suppression when the value is between 2.0 and 4.0.

[0035] After histogram equalization of each window, bilinear interpolation is performed to fuse the enhancement results between adjacent windows. For pixels located at the intersection of four adjacent windows, their final grayscale value is obtained by weighted fusion of the enhancement mapping results from the surrounding four windows through bilinear interpolation, thereby eliminating grayscale jumps at window boundaries and ensuring the visual continuity of the enhanced image. The specific interpolation formula is as follows:

[0036] ,

[0037] in: coordinates The brightness value of the pixel after enhancement, ranging from 0 to 255; For this pixel, respectively according to the adjacent number Line number The enhanced value obtained by the mapping function of the column window; and These are the normalized distances of the pixel in the width and height directions of the window, respectively, with values ​​ranging from 0 to 1. The enhanced L channel is merged with the original A and B channels and converted back to the RGB color space. The enhanced image is then passed to step S3.

[0038] Step S3: Semantic segmentation. This step is the core of the entire method. It receives the enhanced image output from Step S2 and classifies each pixel in the image using a convolutional neural network segmentation model based on the U-Net architecture, outputting a semantic segmentation mask. The semantic segmentation mask labels each pixel in the image as one of three categories: Gram-positive bacteria, Gram-negative bacteria, or background, providing accurate spatial localization for subsequent morphological analysis and classification.

[0039] The U-Net segmentation model used in this invention is an encoder-decoder symmetrical structure with an overall U-shaped topology. The input image size is preprocessed and uniformly adjusted to 256×256×3. The encoder path contains four downsampling stages, from input to bottleneck layer: Stage 1 contains two 3×3 convolutional layers, each outputting 64 feature maps, followed by a 2×2 max pooling layer to reduce the feature map size to 128×128; Stage 2 contains two 3×3 convolutional layers, outputting 128 feature maps, with a pooled size of 64×64; Stage 3 contains two 3×3 convolutional layers, outputting 256 feature maps, with a pooled size of 32×32; Stage 4 contains two 3×3 convolutional layers, outputting 512 feature maps, with a pooled size of 16×16. The bottleneck layer contains two 3×3 convolutional layers, outputting 1024 feature maps, with a feature map size of 16×16. Preferably, each convolutional layer in the encoder path is followed by a batch normalization layer and a ReLU activation function layer in sequence. The batch normalization layer accelerates training convergence and provides a regularization effect, while the ReLU activation function introduces non-linear expressive power by setting negative values ​​to zero.

[0040] The decoder path is symmetrical to the encoder path, also containing four upsampling stages. Each upsampling stage first doubles the feature map size using a 2×2 transposed convolutional layer, then merges it with the feature map from the corresponding stage in the encoder path via skip connections, performing channel-level concatenation. The merged feature map is then further processed by two 3×3 convolutional layers to extract fused features. The skip connection technique serves to transfer the high-resolution spatial detail information preserved in the encoder to the decoder, compensating for the spatial information loss caused by multiple pooling operations. This is crucial for accurately locating the boundaries of tiny pathogens. In sputum smear microscopic images, a single pathogen typically occupies only tens or even a dozen pixels. Without the fine spatial localization information provided by skip connections, relying solely on the decoder's own upsampling operations would be insufficient to recover the precise boundaries of these tiny targets. Preferably, before channel-level concatenation, the encoder feature map in the skip connection undergoes a center-clipping operation to ensure strict consistency in spatial dimensions, avoiding size mismatch issues caused by convolution operations. The number of feature map channels in each stage of the decoder are 512, 256, 128 and 64 respectively, and the feature map sizes are restored to 32×32, 64×64, 128×128 and 256×256 respectively.

[0041] At the final output of the decoder path, a 1×1 convolutional layer maps the 64-channel feature map into 3-channel outputs, corresponding to the three categories of Gram-positive bacteria, Gram-negative bacteria, and background. Then, a Softmax activation function converts the output values ​​of each channel into probability values. (For coordinates...) The pixel at that location belongs to the category The probability calculation formula is:

[0042] ,

[0043] in: coordinates The pixel belongs to the category The probability, These correspond to Gram-positive bacteria, Gram-negative bacteria, and background, respectively, with values ​​ranging from 0 to 1 and the sum of the probabilities of the three categories being 1. For 1×1 convolutional layers in category The output value, also known as the logit value, ranges from negative infinity to positive infinity. The final classification label for each pixel is the category with the highest probability. .

[0044] In one embodiment of the present invention, the U-Net segmentation model is trained using a weighted combination of the cross-entropy loss function and the Dice loss function as the total loss function, in order to balance pixel-level classification accuracy and region-level overlap. The total loss function is defined as follows:

[0045] ,

[0046] in: Cross-entropy loss measures the difference between the predicted probability distribution and the true label distribution. It is calculated by summing the negative logarithms of the predicted probabilities for all pixels and all categories and then averaging them. The Dice loss measures the degree of overlap between the predicted region and the ground truth labeled region, with a value ranging from 0 to 1, and 0 when there is complete overlap. and As the weighting coefficient, in this embodiment , The rationale for choosing an equal-weighted combination is that cross-entropy loss is sensitive to the classification accuracy of each pixel and is suitable for handling cases where class imbalance is not severe, while Dice loss is more sensitive to small target regions and helps improve the segmentation accuracy of small pathogen regions. The Adam optimizer is used during training, with an initial learning rate of [value missing]. The batch size was 8, and the training run consisted of 100 epochs, employing a cosine annealing strategy to gradually reduce the learning rate during training. The training dataset contained 800 labeled Gram-stained sputum smear microscopic images, with 640 images in the training set and 160 in the validation set. To enhance the model's generalization ability, various data augmentation operations were performed on the input images during training, including random horizontal flipping, random vertical flipping, random rotation angles within the range of -15 to +15 degrees, random scaling ratios between 0.9 and 1.1, and random brightness and contrast fine-tuning, with the augmentation magnitude controlled within ±10% of the original values. Furthermore, since the proportion of Gram-positive and Gram-negative bacteria in clinical sputum smear samples is typically much smaller than that of the background area, there is a significant class imbalance problem. In this embodiment, background pixels account for approximately 92% of the entire image, Gram-positive bacteria pixels account for approximately 5%, and Gram-negative bacteria pixels account for approximately 3%. To address this class imbalance problem, the loss function assigns higher weights to pathogen classes. Specifically, the background class has a weight of 1.0, Gram-positive bacteria have a weight of 5.0, and Gram-negative bacteria have a weight of 8.0. The weight values ​​are determined inversely proportional to the pixel proportion of each class in the training set.

[0047] After the semantic segmentation model completes inference, the output semantic segmentation mask is a single-channel label map of the same size as the input image, where pixel value 1 represents the Gram-positive bacteria region, pixel value 2 represents the Gram-negative bacteria region, and pixel value 0 represents the background region. This mask is used as the input for step S4.

[0048] Step S4: Morphological Feature Analysis. This step receives the semantic segmentation mask output from Step S3, extracts connected components from the regions marked as pathogens, and quantitatively calculates morphological feature parameters. This provides shape information for pathogen type determination in the subsequent Step S5. Although the semantic segmentation mask provides pixel-level staining category information, staining category alone is insufficient to distinguish between pathogens with different morphologies, such as cocci and bacilli. Therefore, further morphological analysis is needed to obtain the geometric shape features of each individual pathogen.

[0049] First, binary mask images are extracted from the semantic segmentation mask for Gram-positive bacteria regions with a pixel value of 1 and Gram-negative bacteria regions with a pixel value of 2, respectively. Then, the eight-neighbor connectivity labeling algorithm is used to extract connected components from each binary mask image. The core idea of ​​the eight-neighbor connectivity labeling algorithm is that for any foreground pixel in the binary image, it and its eight adjacent foreground pixels in the surrounding directions are considered to belong to the same connected component. A two-pass scanning method is used to aggregate all interconnected foreground pixels in the image into independent connected components and assign them unique labels. Preferably, a morphological opening operation (erosion followed by dilation) is performed on the binary mask before connected component extraction to eliminate possible small noise points and fine burrs in the segmentation result. The structuring element used in the opening operation is a disk with a radius of 2 pixels. The principle for selecting this size is that it is smaller than the equivalent radius of the smallest pathogen to avoid eliminating the real pathogen region.

[0050] After obtaining the connected components of each independent pathogen, the following morphological feature parameters are calculated for each connected component:

[0051] area The total number of foreground pixels contained in the connected component. In one embodiment of the present invention, under conditions of 100x oil immersion lens and 2048×1536 resolution, the actual physical size corresponding to a single pixel is approximately 0.065. Then the physical value of the area is Preferably, the lower limit threshold for area is set to 10 pixels, and the upper limit threshold for area is set to 5000 pixels. Connected components with an area smaller than the lower limit are considered noise and are removed. Connected components with an area larger than the upper limit may be cell debris or overlapping bacterial communities and need to be specially marked.

[0052] Major axis length and minor axis length The minimum bounding ellipse is obtained by fitting the connected component with its minimum bounding ellipse. The calculation of the minimum bounding ellipse is based on the second-order central moment matrix of the pixel coordinates within the connected component.

[0053] ,

[0054] in: For the pixel coordinates of the connected component in The second central moment in the direction, for The second central moment in the direction, for and The mixed second-order central moments of the direction. The major and minor axis lengths of the minimum circumscribed ellipse are respectively given by the matrix. Two eigenvalues and ( )Sure:

[0055] ,

[0056] in: and Each is a matrix The larger and smaller eigenvalues; This represents the area of ​​the connected region, which is the number of pixels. The units for the lengths of the major and minor axes are pixels.

[0057] Major-minor axis ratio Defined as the ratio of the major axis length to the minor axis length, i.e.:

[0058] ,

[0059] in: This is a dimensionless parameter; a larger value indicates a more elongated shape of the connected region. Cocci are typically approximately spherical. The value is close to 1.0; the bacilli are usually rod-shaped or club-shaped, and their The value is generally greater than 2.0.

[0060] Circularity The ratio of the area of ​​a connected region to the area of ​​its circumcircle measures how closely the shape of the connected region approximates an ideal circle. The formula is:

[0061] ,

[0062] in: The area of ​​the connected component is expressed in pixels. The perimeter of the connected region is obtained through chain code or boundary pixel statistics, and the unit is pixels. The value range is from 0 to 1, when it is a perfect circle. The more irregular or elongated the shape, the better. The smaller the value.

[0063] The above morphological feature parameters constitute the parameter set. The pathogen type is determined in step S5 along with the corresponding staining category label. It is worth noting that in actual clinical samples, some pathogens may be arranged in chains or clusters; for example, streptococci are arranged in chains, and staphylococci are clustered like grapes. For such aggregates, the connected component extraction algorithm will identify the entire aggregate as a large connected component. To accurately assess the number of pathogens within an aggregate, one embodiment of this invention introduces an aggregate splitting estimation mechanism: when the area of ​​a certain connected component... Exceeding the average area of ​​a single typical pathogen When the area ratio is three times that of the target area, the connected component is marked as an aggregate, and the number of pathogens contained therein is estimated based on the area ratio. ,in The area is obtained from the statistical analysis of individual pathogen samples known in the training set. This estimation mechanism can improve the accuracy of pathogen counting in subsequent quantitative statistical steps, and is particularly suitable for high-density infected samples.

[0064] Step S5: Classification and quantitative statistics. This step is the final step in the entire method process. It comprehensively utilizes the staining category information provided by the semantic segmentation mask output in step S3 and the morphological feature parameter set output in step S4 to perform type determination for each independent pathogen connected component, and performs quantitative statistics and visual annotation for various pathogens in the whole image.

[0065] The pathogen type determination adopts a rule-based classification decision method, and the determination rule is as follows: For connected regions labeled as Gram-positive, when their circularity... Greater than the roundness threshold And the ratio of major axis to minor axis Less than the major-minor axis ratio threshold When, it is determined to be a Gram-positive coccus; when its long axis to short axis ratio is... Greater than or equal to When the staining category is Gram-positive, it is determined to be Gram-positive bacilli. For connected regions labeled as Gram-negative, the same morphological determination logic is used to distinguish between Gram-negative cocci and Gram-negative bacilli. In one embodiment of the present invention, the roundness threshold is... Set the major-minor axis ratio to 0.75. A threshold of 2.0 was chosen, based on the optimal classification boundary determined through statistical analysis of known pathogen types in the clinical annotation dataset. Furthermore, regarding the area... Greater than the area threshold And roundness Within the preset range All connected components within the same region, regardless of their staining type, are classified as fungal spores. In this embodiment, Take 1000 pixels, Take 0.6, The value is 0.9. This determination is based on the fact that the diameter of fungal spores is typically 3 to 10 mm. They are significantly larger than the typical size of bacteria, by 0.5 to 2 mm. Therefore, the area of ​​the connected domain corresponding to fungal spores in the microscopic image is significantly larger than that of bacterial connected domains.

[0066] After determining the types of all pathogen connected components, quantitative statistics are performed on the entire image. The statistics include the total number, proportion, and density distribution of each type of pathogen. The density distribution is calculated as follows: the microscopic image is divided into multiple statistical regions according to a preset grid size (256×256 pixels in this embodiment). The number of Gram-positive cocci, Gram-positive bacilli, Gram-negative cocci, Gram-negative bacilli, and fungal spores within each statistical region is counted. Then, the region density value is calculated based on the number of each type of pathogen within each region and the actual area of ​​the statistical region. The formula for calculating the density value is:

[0067] ,

[0068] in: For the first Categories within each grid area Pathogen density, in units of individuals / ; Category within this region The number of pathogens, expressed in units; The actual physical area of ​​a single grid region is calculated from the grid pixel size and the pixel physical size. In this embodiment... .

[0069] Based on the density statistics above, a pseudo-color visualization image is generated. The pseudo-color annotation scheme is as follows: a semi-transparent colored overlay is superimposed on the original microscopic image, with different colors used to annotate different types of pathogens. Preferably, Gram-positive cocci are annotated in blue, Gram-positive bacilli in cyan, Gram-negative cocci in red, Gram-negative bacilli in orange, and fungal spores in yellow. Simultaneously, a density heat map is generated based on the total density value of each grid region, with high-density areas displayed in warm colors and low-density areas in cool colors, facilitating clinicians to quickly locate pathogen-rich areas. The final output visualization image and statistical data report together constitute the auxiliary diagnostic results of this method.

[0070] like Figure 2As shown, this embodiment provides a deep learning-driven sputum smear pathogen image recognition system. This system corresponds one-to-one with the steps of the above-described method embodiment, including a staining normalization module, an adaptive contrast enhancement module, a semantic segmentation module, a morphological analysis module, and a classification and quantitative statistics module. These modules are sequentially connected via data interfaces to form a complete image analysis and processing chain. The system can be deployed on a workstation equipped with a GPU computing accelerator card or on a cloud server to receive image data from a digital microscope via a network interface. After system startup, each module executes sequentially in a pipeline manner, with the output of one module automatically passed to the input port of the next module. This allows for fully automated analysis from raw images to diagnostic results without manual intervention.

[0071] The staining normalization module corresponds to step S1 in the method embodiment. This module is configured to receive the Gram-stained sputum smear microscopic image output by the digital microscopic imaging device and perform white balance correction and color deconvolution processing on it. This module internally includes a white balance correction subunit and a color deconvolution subunit. The white balance correction subunit automatically detects the background area in the image, calculates the average gray level of each channel in the background area, and normalizes the pixel values ​​of each channel according to the aforementioned white balance correction formula. The color deconvolution subunit stores a preset Gram staining standard color vector matrix, decomposes the corrected image into two independent staining component images—crystal violet and safranin—through optical density space transformation, and then performs staining intensity normalization according to the statistical parameters of the reference image set. The output of the staining normalization module is connected to the input of the adaptive contrast enhancement module, passing the color-normalized image to the next module.

[0072] The adaptive contrast enhancement module corresponds to step S2 in the method embodiment. This module is connected to the color normalization module and is configured to receive the color-normalized image and perform multi-scale adaptive contrast enhancement processing. This module first converts the input image to the LAB color space and extracts the L luminance channel. Then, it divides the L channel into multiple local region windows according to a preset window size and performs adaptive histogram equalization processing with contrast limitations on each region. Preferably, the module's window size, contrast limitation threshold, and other parameters are stored in a configuration file for flexible adjustment based on different laboratory conditions. The enhanced L channel is then merged with the original A and B channels to output the enhanced image.

[0073] The semantic segmentation module corresponds to step S3 in the method embodiment. This module is connected to the adaptive contrast enhancement module and is configured to input the enhanced image into a pre-trained U-Net convolutional neural network segmentation model. Internally, this module integrates an image preprocessing subunit, a model inference subunit, and a post-processing subunit. The image preprocessing subunit adjusts the input image to the required 256×256×3 size for the network; the model inference subunit loads the trained U-Net model weights, performs forward inference calculations, and outputs a three-class probability map for each pixel; the post-processing subunit performs an argmax operation on the probability map to generate a semantic segmentation mask and restores the mask to the original image size. The output of the semantic segmentation module is simultaneously connected to the morphological analysis module and the classification and quantitative statistics module. Preferably, the semantic segmentation module supports a model hot update mechanism. When new training data becomes available, updated model weight parameters can be loaded without stopping system operation, thereby continuously improving segmentation accuracy. In addition, this module also provides a confidence filtering function, which marks pixels with a classification probability lower than the preset confidence threshold as areas to be manually reviewed. The confidence threshold is set to 0.85 by default, and this threshold can be adjusted according to the clinical tolerance requirements for false positive and false negative rates.

[0074] The morphological analysis module corresponds to step S4 in the method embodiment. This module is connected to the semantic segmentation module and is configured to perform connected component extraction and morphological feature parameter calculation on the pathogen region in the semantic segmentation mask. This module extracts binary masks for Gram-positive and Gram-negative bacterial regions respectively, uses an eight-neighbor connectivity labeling algorithm to obtain each independent connected component, and calculates the area, major axis length, minor axis length, major-minor axis ratio, and circularity of each connected component through minimum bounding ellipse fitting, as described in detail in the aforementioned method embodiment. The output of the morphological analysis module is a structured morphological feature parameter table, containing a unique identifier, spatial coordinates, and all morphological feature parameters for each connected component.

[0075] The classification and quantitative statistics module corresponds to step S5 in the method embodiment. This module is connected to both the semantic segmentation module and the morphological analysis module. It comprehensively utilizes staining category information from the semantic segmentation module and morphological feature parameters from the morphological analysis module to perform rule-based type determination on each independent pathogen connected domain. This module maintains a configurable classification rule parameter table, including roundness threshold, aspect ratio threshold, and area threshold. The default values ​​of these parameters are determined based on statistical analysis of large-scale clinical annotation datasets, while also allowing users to customize adjustments according to the sample characteristics of specific laboratories. After determination, the module further performs full-image quantitative statistics, calculating the quantity, proportion, and regional density distribution of various pathogens, and generating a visualization result image including pseudo-color annotations and density heatmaps. Preferably, this module can also generate structured data reports, recording statistical information of various pathogens in tabular form, facilitating storage in the laboratory information management system for subsequent data management and trend analysis.

[0076] To verify the practical effectiveness of the deep learning-driven sputum smear pathogen image recognition method provided in this invention, a systematic comparative experiment was conducted in a clinical setting. The experimental dataset came from the microbiology laboratories of three hospitals of different levels, collecting a total of 1200 Gram-stained sputum smear microscopic images, including 280 Gram-positive cocci samples, 210 Gram-positive bacilli samples, 180 Gram-negative cocci samples, 260 Gram-negative bacilli samples, 70 fungal spore samples, and 200 mixed infection samples. All images were independently annotated by two senior laboratory technicians with more than 5 years of experience. The annotations included the location, boundaries, and type information of the pathogens. The Cohen's Kappa value between the two annotators was 0.91, indicating reliable annotation quality.

[0077] The experiment used 5-fold cross-validation to evaluate the model performance. Evaluation metrics for the semantic segmentation task included pixel-level accuracy, mean intersection-over-union (MIU) ratio, and Dice coefficients for each category. Experimental results showed that the method of this invention achieved a pixel-level accuracy of 95.3%, an MIU ratio of 82.7%, a Dice coefficient of 0.867 for Gram-positive bacteria regions, a Dice coefficient of 0.831 for Gram-negative bacteria regions, and a Dice coefficient of 0.962 for background regions. In the pathogen type classification task, using the annotation results of experienced laboratory technicians as the gold standard, the overall classification accuracy of this method reached 93.6%, with an accuracy of 95.8% for Gram-positive cocci, 94.2% for Gram-positive bacilli, 91.5% for Gram-negative cocci, 93.1% for Gram-negative bacilli, and 90.7% for fungal spores.

[0078] Compared to the control experiment without staining normalization preprocessing, the average intersection-union ratio (IUR) of semantic segmentation improved by 6.2 percentage points after adding staining normalization step S1, indicating that eliminating batch-specific staining differences significantly improves segmentation accuracy. Compared to the control experiment without adaptive contrast enhancement, the average IUR further improved by 3.8 percentage points after adding step S2, with a more significant improvement of 8.5 percentage points on samples with lighter staining. Furthermore, compared to the AlexNet-based image-level classification method disclosed in CN107464234A, the method of this invention can simultaneously output the precise location, category, and quantity information of pathogens, achieving a leap from single classification to comprehensive analysis. In terms of processing speed, the complete processing time for a single 2048×1536 resolution microscopic image is approximately 1.2 seconds, meeting the real-time auxiliary diagnostic needs of clinical laboratories. The process involves several steps: S1 staining normalization takes approximately 0.08 seconds; S2 contrast enhancement takes approximately 0.05 seconds; S3 semantic segmentation inference takes approximately 0.92 seconds (accelerated using an NVIDIA RTX 3080 GPU); S4 morphological analysis takes approximately 0.10 seconds; and S5 classification, statistics, and visualization output takes approximately 0.05 seconds. It is evident that semantic segmentation step S3 is the most computationally intensive step in the entire process. Using GPU acceleration effectively controls its inference time, ensuring the overall processing speed meets the clinical throughput requirement of processing over 50 microscopic images per minute.

[0079] The experimental results fully verify the synergistic effect between the steps of this invention. The staining normalization preprocessing step S1 provides a consistent color benchmark for subsequent steps; the adaptive contrast enhancement step S2 further amplifies the difference between the target and the background, enabling the semantic segmentation step S3 to obtain higher-quality input and produce more accurate segmentation results; the morphological feature analysis step S4 fully utilizes the segmentation results to extract quantitative shape information; and the classification, identification, and quantitative statistics step S5 organically integrates staining and morphological information to achieve comprehensive and accurate pathogen identification. These five steps are interconnected; the absence or replacement of any step will lead to a significant decrease in overall performance, demonstrating the indivisibility of this invention's technical solution as a complete and synergistic system.

[0080] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep learning-driven method for pathogen image recognition in sputum smears, characterized in that, Includes the following steps: Step S1, staining normalization preprocessing step: Obtain a microscopic image of Gram-stained sputum smear, perform staining separation processing on the microscopic image, use a color deconvolution algorithm based on optical density space transformation to decompose the microscopic image into a hematoxylin channel image and an eosin channel image, calculate the staining intensity normalization coefficient based on the hematoxylin channel image and the eosin channel image, perform staining batch difference correction on the microscopic image based on the staining intensity normalization coefficient, and output a color normalized image; Step S2, Adaptive Contrast Enhancement Step: Perform multi-scale adaptive contrast enhancement processing on the color normalized image, divide the color normalized image into multiple local region windows, calculate local contrast enhancement parameters for each local region window, perform nonlinear mapping transformation on the pixel grayscale values ​​in each local region window according to the local contrast enhancement parameters, perform boundary transition fusion processing on the enhancement results of adjacent local region windows, and output the enhanced image. Step S3, semantic segmentation step: The enhanced image is input into a convolutional neural network segmentation model based on the U-Net architecture. Multi-scale feature maps are extracted layer by layer from the enhanced image through the encoder path. The multi-scale feature maps are upsampled layer by layer through the decoder path and fused with the encoder feature maps of the corresponding layers through skip connections. Multi-class classification is performed on each pixel at the output layer of the decoder path to output a semantic segmentation mask. The semantic segmentation mask contains pixel-level annotations of Gram-positive bacteria regions, Gram-negative bacteria regions, and background regions. Step S4, Morphological Feature Analysis Step: Perform connected component extraction processing on the Gram-positive and Gram-negative bacterial regions in the semantic segmentation mask to obtain each independent pathogen connected component. Calculate a set of morphological feature parameters for each independent pathogen connected component. The set of morphological feature parameters includes area, major axis length, minor axis length, major-minor axis ratio, and circularity.

2. The method according to claim 1, characterized in that, In step S1, the color deconvolution algorithm uses a color deconvolution matrix of optical density spatial transformation as a preset Gram staining standard color vector matrix. The Gram staining standard color vector matrix includes a first color vector corresponding to the crystal violet staining channel and a second color vector corresponding to the safranin counterstaining channel. The staining intensity normalization coefficient is determined based on the ratio of optical density values ​​in each channel between the current batch image and the reference standard image.

3. The method according to claim 1, characterized in that, In step S2, the size of the local region window ranges from 16×16 pixels to 128×128 pixels. The nonlinear mapping transformation uses an adaptive histogram equalization algorithm with limited contrast. The contrast limit threshold ranges from 2.0 to 4.

0. The boundary transition blending process uses bilinear interpolation to eliminate grayscale jumps between adjacent windows.

4. The method according to claim 1, characterized in that, In step S3, the input image size of the U-Net-based convolutional neural network segmentation model is 256×256×3. The encoder path contains four downsampling stages, each of which contains two 3×3 convolutional layers and one 2×2 max pooling layer. The decoder path contains four upsampling stages, each of which contains one 2×2 transposed convolutional layer and two 3×3 convolutional layers. The output layer uses a 1×1 convolutional layer and a Softmax activation function to output a 3-class probability map.

5. The method according to claim 1, characterized in that, In step S1, before performing color separation, the color deconvolution algorithm first performs white balance correction on the microscopic image. The white balance correction normalizes the background gray mean of each color channel to a preset reference value based on the statistical gray mean and standard deviation of the background area of ​​the microscopic image.

6. The method according to claim 1, characterized in that, In step S3, each convolutional layer in the encoder path is followed by a batch normalization layer and a ReLU activation function layer in sequence. The skip connection fusion uses a channel-dimensional concatenation method to fuse the encoder feature map with the upsampled decoder feature map. The fused feature map is then processed by two 3×3 convolutional layers to extract fused features.

7. The method according to claim 1, characterized in that, In step S4, the connected component extraction process uses the eight-neighbor connected component labeling algorithm. The lengths of the major and minor axes in the morphological feature parameter set are determined by calculating the minimum circumscribed ellipse of the independent pathogen connected component. The circularity is calculated as the ratio of the area of ​​the independent pathogen connected component to the area of ​​its circumscribed circle.

8. The method according to claim 1, characterized in that, The method also includes step S5, a classification and quantitative statistical step: based on the staining category label in the semantic segmentation mask and the morphological feature parameter set, pathogen type determination is performed on each independent pathogen connected component to distinguish between cocci, bacilli, and fungal spores, and the quantity and density distribution of each type of pathogen are statistically analyzed to generate a pseudo-color visualization result image containing location markers and type labels for each type of pathogen; the pathogen type determination is based on the following rules: when the staining category is labeled as Gram-positive and the roundness is greater than the roundness threshold, it is determined to be a Gram-positive cocci; when the staining category is labeled as Gram-positive and the aspect ratio is greater than the aspect ratio threshold, it is determined to be a Gram-positive bacillus; when the staining category is labeled as Gram-negative and the roundness is greater than the roundness threshold, it is determined to be a Gram-negative cocci; when the staining category is labeled as Gram-negative and the aspect ratio is greater than the aspect ratio threshold, it is determined to be a Gram-negative bacillus; when the area is greater than the area threshold and the roundness is within a preset range, it is determined to be a fungal spore.

9. The method according to claim 8, characterized in that, In step S5, the statistical method for the density distribution is as follows: the microscopic image is divided into multiple statistical regions according to a preset grid size, the number of various pathogens in each statistical region is calculated, the regional density value of each type of pathogen is calculated based on the number of various pathogens in each statistical region and the area of ​​the statistical region, and a pathogen density thermal distribution map is generated based on the regional density value.

10. A deep learning-driven sputum smear pathogen image recognition system, used to implement the method of claim 9, characterized in that, include: The staining normalization module is configured to acquire a microscopic image of a Gram-stained sputum smear, perform color deconvolution processing based on optical density spatial transformation on the microscopic image, decompose the microscopic image into a hematoxylin channel image and an eosin channel image, calculate the staining intensity normalization coefficient, perform staining batch difference correction on the microscopic image, and output a color normalized image. An adaptive contrast enhancement module, connected to the color normalization module, is configured to perform multi-scale adaptive contrast enhancement processing on the color normalized image, divide the color normalized image into multiple local region windows, calculate contrast enhancement parameters for each local region window and perform nonlinear mapping transformation, perform boundary transition fusion on adjacent windows, and output an enhanced image. The semantic segmentation module, connected to the adaptive contrast enhancement module, is configured to input the enhanced image into a convolutional neural network segmentation model based on the U-Net architecture, perform pixel-level multi-class classification on the enhanced image through encoder and decoder paths, and output a semantic segmentation mask containing annotations of Gram-positive bacteria regions, Gram-negative bacteria regions, and background regions. The morphological analysis module, connected to the semantic segmentation module, is configured to perform connected component extraction and morphological feature parameter calculation on the pathogen region in the semantic segmentation mask. The morphological feature parameters include area, major axis length, minor axis length, major-minor axis ratio, and circularity. The classification and quantitative statistics module, connected to the semantic segmentation module and the morphological analysis module, is configured to perform type determination and statistical density distribution on each independent pathogen connected domain based on staining category labeling and morphological feature parameters, and output a pseudo-color visualization result image.

Images