A fast cell staining normalization method using learnable bilateral filtering

Abstract

The application discloses a fast cell staining normalization method using a learnable bilateral filter, and comprises the following steps: acquiring a full-resolution to-be-normalized image and a corresponding low-resolution target staining image; encoding the full-resolution to-be-normalized image and the target staining image to obtain feature hidden variables; making the full-resolution to-be-normalized image pass through full-resolution dynamic block multi-layer convolution to generate an index map; making the low-resolution to-be-normalized image and the target staining image pass through low-resolution dynamic block multi-layer convolution to generate a bilateral grid; in a weight generation and application module of linear transformation, using the index map to generate linear transformation weights on the bilateral grid, and applying the transformation weights on the full-resolution to-be-normalized image to generate a transformed image consistent with a standard picture style; and the application adopts a low-resolution high-resolution bilateral learning structure to reduce hardware overhead, provides the ability of multi-staining style conversion, and can be combined with a high-throughput scanner to perform real-time image processing.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a rapid cell staining normalization method using learnable bilateral filtering. Background Technology

[0002] Papanicolaou staining is commonly used for staining cell sections, especially cervical cell sections. However, subtle differences in staining agent formulations, staining preparation processes, scanner parameters, and backlighting can lead to different staining styles in digitized Papanicolaou-stained cell sections. These differences in staining styles result in different data distributions. Current computer-aided analysis and diagnostic algorithms rely on prior knowledge of the data distribution, and changes in the distribution can lead to a decrease in the performance of these algorithms. Therefore, staining normalization of the input images for analysis algorithms can transform the input data distribution into a known prior distribution, thereby improving the performance of subsequent algorithms.

[0003] Meanwhile, the digital pathological slides produced by Papanicolaou staining are large in size, with a single full slide typically ranging from 500 MBytes to 2 GBytes. Current staining normalization algorithms based on chromatin transformation and generative adversarial networks are slow and difficult to process in real time with high-throughput digital pathology scanners. Generally, the task of staining normalization is completed by first capturing and storing the image and then processing it offline. Chromatin transformation-based methods rely on the accurate extraction of chromatin basis vectors. Originally applied to H&E stained slides, they have good separation effects for hematoxylin (H) and eosin (E) staining, but they have errors in judging the eosinophilic and basophilic basis vectors of Papanicolaou stained slides, resulting in poor staining style normalization. At the same time, the extraction of chromatin basis vectors involves the optimization problem of solving matrix factorization, which cannot be processed in real time. Generative adversarial algorithms face the problems of artifacts and color crossover. If the cell density of the training data is inconsistent, especially the density of neutrophils, the normalized image generated by the algorithm will have extra cell artifacts due to the lower cell density input. Color crossover refers to the situation where the eosinophilic pink area and the basophilic blue-green area in the output image are inconsistent with the input. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by this invention is that existing cell staining methods have poor results due to differences in slide preparation processes, slow processing speed of staining normalization algorithms, and poor real-time performance.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a fast cell staining normalization method using learnable bilateral filtering, comprising: acquiring a full-resolution image to be normalized and a manually selected full-resolution image of the target staining from a scanner; obtaining low-resolution images corresponding to the full-resolution images to be normalized and the target staining from the full-resolution images through image scaling; encoding the full-resolution images to be normalized and the target staining from the full-resolution images to obtain encoding vectors; extracting features from the encoding vectors to obtain latent variables; inputting the full-resolution image to be normalized into a full-resolution dynamic block guided by the latent variables for multi-level convolution and generating an index map; performing multi-level convolution on the low-resolution images to be normalized and the target staining from the low-resolution images through multi-level convolution, spatial instance normalization, and the low-resolution dynamic block guided by the latent variables to generate a bilateral grid; in a linear transformation weight generation and application module, using the index map to index on the bilateral grid to generate linear transformation weights, and applying the transformation weights on the full-resolution image to be normalized to generate a transformed image consistent with the style of a standard image.

[0008] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, the acquisition of the full-resolution image to be normalized includes:

[0009] The effective image of a Papanicolaou-stained cell section is obtained by a scanner acquiring an effective field of view at high resolution. The effective image is the full-resolution image to be normalized.

[0010] The effective field of view includes a field of view containing analyzable cells under a microscope at a certain magnification, wherein the analyzable cells include at least one of eosinophilic cells and basophilic cells, and are precisely focused, unobstructed, and capable of analysis.

[0011] The target stained full-resolution image includes images selected by humans and of the same scale as the full-resolution image to be normalized;

[0012] The low-resolution image includes an image scaled from a full-resolution image to be normalized and target-colored, wherein the scaling factor is 4 by default, but can be changed according to the scale of the full-resolution image.

[0013] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, the acquisition of the encoding vector includes applying an encoding method to the full-resolution images to be normalized and the target staining to generate encoding vectors respectively.

[0014] The encoding method includes converting an RGB image into an HSV image, dividing the white background region, pink eosinophilic cell region, and blue-green basophilic cell region using a threshold segmentation method, and then encoding each region by calculating the mean and variance of different regions.

[0015] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, wherein: the acquisition of the latent feature variables includes,

[0016] An autoencoder consists of two parts: an encoder and a decoder. The calculation of the autoencoder is the compression of encoder features and the decompression of decoder features.

[0017] Under the premise of minimizing the changes in decoder output and encoder input information, the encoder output is the latent feature variable that guides the generation of full-resolution index map and low-resolution bilateral grid.

[0018] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, wherein: the generation of the index map includes,

[0019] The full-resolution content structure mapping module is composed of multiple dynamic blocks connected in series. The full-resolution image to be normalized is input into the full-resolution content structure mapping module.

[0020] The full-resolution image to be normalized undergoes a transformation after each dynamic block, and after multiple transformations, an index map that retains the structural information of the image to be normalized is generated.

[0021] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, the full-resolution dynamic block includes multi-layer parallel convolution and fully connected weight extraction.

[0022] The multi-layer parallel convolution involves performing a non-linear transformation on the input full-resolution image to be normalized or the output feature image of a dynamic block to satisfy the generation of the final index map. The number of branches of the parallel convolution is a set constant. Each parallel convolution does not change the scale or structural content of the image, but only changes the intensity of the pixels.

[0023] The fully connected weight extraction includes inputting the latent feature variables obtained by the autoencoder, and then obtaining the weights of the parallel convolution through multiple non-linear fully connected layers and the final Softmax activation function;

[0024] The calculation of the Softmax activation function includes,

[0025]

[0026] Among them, Z i Softmax(Z) represents the value of the i-th dimension of the latent feature variable. i Z represents the weights of the i-th parallel convolutional layer, P represents the number of parallel convolutions, and Z represents the weights of the i-th parallel convolutional layer. p This represents the value of the latent variable corresponding to P parallel convolutions.

[0027] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, wherein: the generation of the bilateral grid includes,

[0028] The low-resolution image to be normalized and the target coloring style image are subjected to multi-layer convolution for downsampling feature extraction, thereby obtaining feature images of three different scales;

[0029] The three feature images at different scales are input into three concatenated spatial adaptive instance normalization blocks to generate a feature map after coloring style transformation, thereby realizing the coloring style transformation of the feature images;

[0030] The global features of the transformed feature image and the local features of the transformed feature image under multiple contiguous dynamic blocks are extracted. The global features and local features are fused and then the features of multiple contiguous dynamic blocks are extracted to generate a stylized bilateral mesh.

[0031] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, the spatial adaptive instance normalization operation includes:

[0032] A pseudo-probability map is generated using threshold segmentation and weight balancing. Spatial region guidance is applied to the pseudo-probability map, and the coloring stylization of the feature image is completed through linear transformation of mean and variance.

[0033] The calculation of the linear transformation SpAdaIN(·) includes,

[0034]

[0035] Where, x ij y represents the value of the feature image to be normalized at position (i,j). ij This represents the value of the target stained feature image at position (i,j). This represents the variance of the region to which the feature image to be normalized belongs at position (i,j). This represents the variance of the region to which the target stained feature image belongs at position (i,j). This represents the mean value of the region to which the feature image to be normalized belongs at position (i,j). This represents the mean value of the target staining feature image in the region at position (i,j).

[0036] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, the linear transformation weight generation and application module includes:

[0037] The generation module includes performing trilinear interpolation on the length, width, and channel dimensions of the bilateral grid using the index map to obtain the amplification factor and bias of the linear transformation;

[0038] The application module includes applying the magnification factor and bias point-to-point to the full-size image to be normalized to complete the style conversion, and finally outputting an image similar to the target coloring style image.

[0039] As a preferred embodiment of the fast cell staining normalization method using learnable bilateral filtering described in this invention, it further includes:

[0040] To enable the storage of input and output images, the storage of module weights, and operation, a device is provided that applies a fast cell staining normalization method using learnable bilateral filtering.

[0041] The device includes a processor for computation, a memory, and an executable computation program and its weight parameters stored in the memory. The processor can load the computation program from the memory and initialize the computation program using the stored weight parameters.

[0042] The beneficial effects of this invention are as follows: This invention provides a fast coloring normalization method that can be used in real-time image processing with high-throughput scanners. It utilizes a learnable bilateral mesh for coloring style conversion combined with hardware parallelization, greatly accelerating the coloring normalization process. Simultaneously, it proposes a spatial adaptive normalization module that uses a pseudo-probability map estimated by threshold segmentation to guide computation in different semantic regions, resulting in better coloring normalization and avoiding artifact generation and color cross-contamination. The innovative use of style encoding and dynamic blocks enables this invention to perceive the coloring style of the input image, thereby achieving the function of multiple style conversions in a single model training. This invention employs a low-resolution, high-resolution bilateral learning structure to reduce hardware overhead, provides the ability to convert between multiple coloring styles, improves the coloring normalization effect, and thus enhances the performance of subsequent auxiliary analysis and diagnostic algorithms. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0044] Figure 1 This is a structural framework diagram of a fast cell staining normalization method using learnable bilateral filtering, provided in one embodiment of the present invention.

[0045] Figure 2 The second embodiment of the present invention provides a low-magnification panoramic image and an effective field of view of a digital whole section of Papanicolaou staining using a fast cell staining normalization method with learnable bilateral filtering.

[0046] Figure 3 A schematic diagram of the structure of an autoencoder for latent variable extraction in a fast cell staining normalization method using learnable bilateral filtering, provided in the second embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of a dynamic block in a fast cell staining normalization method using learnable bilateral filtering, provided in the second embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of a style sputtering block provided in a fast cell staining normalization method using learnable bilateral filtering, as provided in the second embodiment of the present invention. Detailed Implementation

[0049] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0051] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0052] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0053] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0054] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0055] Example 1

[0056] Reference Figure 1 As one embodiment of the present invention, a fast cell staining normalization method applying learnable bilateral filtering is provided, comprising:

[0057] S1: Input image processing procedure, acquire the full-resolution image to be normalized from the scanner's current field of view and the manually selected full-resolution image of the target staining conforming to the prior distribution of the auxiliary analysis, and obtain the corresponding low-resolution images of the two through image scaling. It should be noted that:

[0058] The normalized full-resolution image includes a valid image of a Papanicolaou-stained cell section acquired by a scanner at high resolution, capturing an effective field of view.

[0059] It should be noted that an effective field of view includes a field of view containing analyzable cells under a microscope at a certain magnification. The analyzable cells include at least one type of eosinophilic cells and basophilic cells, and must be in focus, unobstructed, and capable of analysis. If the image does not contain analyzable Papanicolaou cells, no processing is performed and feedback is sent to the scanner. Threshold segmentation can be used to detect whether there are foreground cells in the field of view, and histogram statistics can be used to determine whether it is an effective field of view.

[0060] Furthermore, the target staining full-resolution image is an image selected by humans with the same scale as the full-resolution image to be normalized, and it can be changed according to the changes in the prior distribution of subsequent auxiliary analysis;

[0061] Furthermore, the low-resolution image includes an image scaled from the full-resolution image to be normalized and the target stained, with a scaling ratio of 4 by default. This ratio can be changed according to the size of the acquired field of view, thereby adjusting the overhead of the low-resolution mesh generation operation according to the size of the full-resolution image.

[0062] S2: In the image coloring style encoding and feature extraction module, the full-resolution image to be normalized and the target coloring full-resolution image are encoded to obtain encoding vectors. Feature extraction is then performed on the encoding vectors to obtain latent variable features. It should be noted that:

[0063] Obtaining the encoded vector involves applying encoding methods to the full-resolution images to be normalized and the target stained images to generate encoded vectors respectively;

[0064] It should be noted that image staining style encoding is generated by concatenating the mean and variance of the same semantic region between the input image to be normalized and the target stained full-resolution image. The semantic region is generated by converting the input RGB image into an HSV image, performing threshold segmentation on the saturation channel (S channel) to separate foreground cells and white background, and performing threshold segmentation on the chroma channel (H channel) to separate pink eosinophilic cells and blue-green basophilic cells in the foreground region. According to the description of the effective field of view, the input image contains a background region and at least one type of foreground cell. The color style is encoded by calculating the mean and variance of the RGB three-channel pixels within the threshold segmentation region. If the input image does not contain all types of foreground cells, the corresponding encoding value is set to 0. The encoding vector dimension of both the image to be normalized and the target stained image is 18, so the final staining style encoding vector is 36-dimensional.

[0065] Furthermore, feature latent variables are obtained by extracting features from the encoded vector using an autoencoder;

[0066] It should be noted that the autoencoder consists of two parts: an encoder and a decoder. The calculation of the autoencoder is the compression of encoder features and the decompression of decoder. The encoder is a downsampled multilayer perceptron, whose input is the 36-dimensional coloring style encoding vector mentioned above. It extracts features through multiple nonlinear calculations to generate the dimensionality-reduced latent variables. The decoder is an upsampled multilayer perceptron, whose input is the latent variables. It samples the expected value through multiple nonlinear transformations to restore it to the input style encoding.

[0067] Furthermore, under the premise of minimizing the changes in decoder output and encoder input information, the encoder output is the latent feature variable that guides the generation of full-resolution index map and low-resolution bilateral grid. The vector after dimensionality reduction of latent variable feature extraction contains the coloring style of the image to be normalized and the target image, which can guide the weight transformation of other feature extraction modules.

[0068] S3: In the full-resolution content structure mapping module, the input is the full-resolution image to be normalized. After passing through multiple full-resolution dynamic block convolutions guided by latent variables, an index map is generated. It should be noted that:

[0069] The full-resolution content structure mapping module is composed of multiple dynamic blocks connected in series. When the full-resolution image to be normalized is input into the full-resolution content structure mapping module, the image will be transformed once after passing through each dynamic block. After multiple transformations, an index map that retains the structural information of the image to be normalized will be generated.

[0070] It should be noted that the number of concatenated dynamic blocks at full resolution can be changed according to the size of the input full-resolution image and the platform's computing power. The scale of the index map is consistent with the full-resolution image to be normalized, the number of channels is increased, and the value of each pixel is limited to 0 to 1. Furthermore, the number of concatenated dynamic blocks is set to a single digit. This is considered from two aspects: first, the task of converting the image to be normalized into a mapping map is not complicated and can be achieved with only a small number of nonlinear transformations; second, the scale of the full-resolution image is relatively large. If too many dynamic blocks are set, it will increase the algorithm complexity of this module, increase resource consumption, and reduce the efficiency of style normalization.

[0071] Furthermore, the full-resolution dynamic block includes multi-layer parallel convolution and fully connected weight extraction;

[0072] Multi-level parallel convolution involves performing a non-linear transformation on the input full-resolution image to be normalized or the output feature image of the previous dynamic block to satisfy the generation of the final index map. The number of branches of the parallel convolution is a set constant. Each parallel convolution does not change the scale or structural content of the image, but only changes the intensity of the pixels.

[0073] Fully connected weight extraction involves inputting the latent feature variables obtained from the autoencoder, and then obtaining the weights of the parallel convolutions through multiple non-linear fully connected layers and the final Softmax activation function.

[0074] The calculation of the Softmax activation function includes,

[0075]

[0076] Among them, Z i Softmax(Z) represents the value of the i-th dimension of the latent feature variable. i Z represents the weights of the i-th parallel convolutional layer, P represents the number of parallel convolutions, and Z represents the weights of the i-th parallel convolutional layer. p This represents the values ​​of the latent variables corresponding to P parallel convolutions;

[0077] It should be noted that the design of the full-resolution dynamic block makes the feature extraction process sensitive to the coloring style; input images with different coloring styles will change the weight changes of feature image extraction.

[0078] S4: In the low-resolution stylized bilateral mesh generation module, the low-resolution image to be normalized and the target stained image are input and then processed through multi-layer convolution, spatial instance normalization, and low-resolution dynamic block multi-layer convolution guided by latent feature variables to generate a bilateral mesh. It should be noted that:

[0079] The low-resolution stylized bilateral mesh generation module includes a pre-training feature extraction process, a spatial adaptive instance normalization process, a coloring style sputtering process, a global and local feature fusion process, and a dynamic block feature post-extraction process.

[0080] Furthermore, the low-resolution image to be normalized and the target saturation style image are input into the module, and three feature images of different scales are generated through a pre-trained feature extraction network. The three feature images of different scales are then input into three concatenated spatial adaptive instance normalization blocks to generate a feature map after saturation style conversion, thereby realizing the conversion of the saturation style of the feature image.

[0081] It should be noted that the Spatial Adaptive Instance Normalization (SpAdaIN) process provided by this invention adds spatial guidance compared to the single instance normalization (AdaIN) process. This part is completed by pseudo-probability region indexing. The pseudo-probability map is generated by region threshold filtering and probability map calculation. The difference between threshold filtering and threshold segmentation is that binary classification judgment is not required. The required channel map is subtracted from the threshold and weighted in the positive and negative intervals. The weighting design should ensure that the sum of the probabilities of the background, eosinophilic foreground, and basophilic foreground corresponding to each pixel after calculation is 1.

[0082] The specific calculation of spatial adaptive instance normalization includes,

[0083]

[0084] Where, x ij y represents the value of the feature image to be normalized at position (i,j). ij This represents the value of the target stained feature image at position (i,j). This represents the variance of the region to which the feature image to be normalized belongs at position (i,j). This represents the variance of the region to which the target stained feature image belongs at position (i,j). This represents the mean value of the region to which the feature image to be normalized belongs at position (i,j). This represents the mean value of the region to which the target stained feature image belongs at position (i,j);

[0085] The calculation of the mean includes,

[0086]

[0087]

[0088] Among them, P eos P represents the pseudoprobability of the acidophilicity of a pixel. bas P represents the pseudoprobability of the basophilicity of a pixel. bg This represents the pseudo-probability of a pixel's background. This represents the mean of the eosinophilic regions in the feature image to be normalized. This represents the mean of the basophilic regions in the feature image to be normalized. This represents the mean of the background region in the feature image to be normalized. This represents the mean of the eosinophilic regions in the target staining feature image. This represents the mean of the basophilic regions in the target staining feature image. This represents the mean value of the background region in the target stained feature image;

[0089] It should be noted that the pseudo-probability map is generated through region threshold filtering and probability map calculation. The difference between threshold filtering and threshold segmentation is that binary classification is not required; only the required channel map is subtracted from the threshold and weighted in the positive and negative intervals. The probability map calculation generates the pseudo-probability map by applying the Sigmoid function to the weighted result. The weighting design must ensure that the sum of the probabilities of the background, eosinophilic foreground, and basophilic foreground corresponding to each pixel after the Sigmoid function calculation is 1.

[0090] The specific calculations include,

[0091]

[0092]

[0093] P bg =Prob(s,θ)

[0094] P eos =(1-P bg )(1-Prob(h,η))

[0095] P bas =(1-P bg Prob(h,η)

[0096] Where c represents the input channel image, ψ represents the set threshold, f(x) represents the Sigmoid function, ε represents the linear scaling factor, s represents the saturation channel image, θ represents the corresponding threshold parameter, h represents the chroma channel image, and η represents the corresponding threshold parameter.

[0097] Furthermore, the global features of the transformed feature image and the local features of the transformed feature image under multiple concatenated dynamic blocks are extracted. The extracted global features and local features are then fused and the features of multiple concatenated dynamic blocks are extracted to generate a stylized bilateral mesh.

[0098] S5: In the linear transformation weight generation and application module, linear transformation weights are generated by indexing on a bilateral grid using an index map. These transformation weights are then applied to the full-resolution image to be normalized, generating a transformed image with a style consistent with the standard image. It should be noted that:

[0099] The linear transformation weight generation module includes obtaining the amplification factor and bias of the linear transformation by performing trilinear interpolation in the length, width, and channel dimensions of the bilateral grid through an index graph;

[0100] Furthermore, the linear transformation weighting module includes applying the magnification factor and bias point-to-point to the full-size image to be normalized to complete the style conversion, and finally outputs an image similar to the target coloring style image. This similar image can improve the accuracy of subsequent computer-aided analysis compared to the original image to be normalized.

[0101] Furthermore, to realize the storage of input and output images, the storage of module weights, and operation, a device is provided that applies a fast cell staining normalization method using learnable bilateral filtering.

[0102] It should be noted that the device includes a processor for computation, a memory, and an executable computation program and its weight parameters in the memory. The processor can load the computation program in the memory and initialize the computation program using the stored weight parameters.

[0103] This invention provides a rapid coloring normalization method that can be used in real-time image processing with high-throughput scanners. It utilizes a learnable bilateral mesh for coloring style transfer combined with hardware parallelization, significantly accelerating the coloring normalization process. Furthermore, it proposes a spatial adaptive normalization module that uses a pseudo-probability map estimated by threshold segmentation to guide computation in different semantic regions, resulting in better coloring normalization and avoiding artifact generation and color cross-referencing. The innovative use of style encoding and dynamic blocks enables the invention to perceive the coloring style of the input image, thus achieving the function of multiple style conversions in a single model training. This invention employs a low-resolution, high-resolution bilateral learning structure to reduce hardware overhead, providing the ability to convert between multiple coloring styles, improving the coloring normalization effect, and thus enhancing the performance of subsequent auxiliary analysis and diagnostic algorithms.

[0104] Example 2

[0105] Reference Figures 2-5 This is the second embodiment of the present invention. Unlike the first embodiment, this embodiment provides a verification test of a rapid cell staining normalization method using learnable bilateral filtering. To verify and illustrate the technical effects of the method, this embodiment compares the traditional technical solution with the method of the present invention, and compares the experimental results using scientific demonstration methods to verify the real effect of the method.

[0106] Taking cervical cell sections as an example, the size of a single digital whole slide is typically between 500 MBytes and 2 GBytes. Figure 2 The image shown is a low-magnification full view of a digital whole section of the cervix stained with Pap smear. It is generated by stitching together images taken from different fields of view at a certain magnification using a scanner. The full-resolution image to be normalized and the target stained image are determined by a high-throughput scanner. The default full-resolution image size is 1024*1024. The two are scaled to generate a low-resolution image with a size of 256*256.

[0107] In the image coloring style encoding and feature extraction module, the full-resolution image to be normalized and the target coloring full-resolution image are encoded to obtain encoding vectors. Feature extraction is then performed on the encoding vectors to obtain latent feature variables. Figure 3 The image shown is a mask image of three regions of an input color image after thresholding.

[0108] In the full-resolution content structure mapping module, the input is a full-resolution image to be normalized, and an index map is generated after passing through a multi-layer convolution of full-resolution dynamic blocks guided by latent variables.

[0109] In the low-resolution stylized bilateral mesh generation module, the input low-resolution image to be normalized and the target stained image are processed through multi-layer convolution, spatial instance normalization, and low-resolution dynamic block multi-layer convolution guided by latent feature variables to generate a bilateral mesh. Specific operational details are as follows: Figure 4 and Figure 5 As shown;

[0110] In the linear transformation weight generation and application module, the linear transformation weights are generated by indexing on the bilateral grid using an index map. The transformation weights are then applied to the full-resolution image to be normalized to generate a transformed image with the same style as the standard image.

[0111] Compared with existing staining normalization methods, SSIM and Gram Loss were used to quantitatively evaluate the effects of structure content preservation and staining style transformation, respectively. The results are shown in Table 1. Among them, the larger the SSIM, the better; the smaller the Gram Loss, the better; and the shorter the processing time, the better.

[0112] Table 1: Results of comparison with existing technologies.

[0113] Technology Name SSIM GramLoss Processing time (s) Reinhard 0.8991±0.0562 0.0098±0.0047 0.0481±0.0030 Macenko 0.9022±0.0565 0.0091±0.0048 1.1117±0.0375 Vahdane 0.9112±0.0484 0.0090±0.0046 3.0196±0.0248 This invention 0.9101±0.0080 0.0080±0.0040 0.0111±0.0003

[0114] As can be seen from Table 1, the present invention can better preserve the structural information of the image, obtain better coloring stylization, and significantly reduce the processing time; therefore, the present invention can be used in conjunction with a high-throughput scanner to complete coloring normalization in real-time image processing.

[0115] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A rapid cell staining normalization method using learnable bilateral filtering, characterized in that, include: The system acquires a full-resolution image to be normalized and a manually selected full-resolution image of the target staining sent by the scanner, and obtains low-resolution images corresponding to the full-resolution images to be normalized and the target staining full-resolution images by image scaling. The full-resolution image to be normalized and the full-resolution image of the target stained are encoded and the encoding vector is obtained. Feature extraction is performed on the encoding vector to obtain feature latent variables. The image to be normalized at full resolution is input into the latent variable-guided full resolution dynamic block for multi-layer convolution and an index map is generated. The low-resolution image to be normalized and the target stained image are subjected to multi-layer convolution, spatial instance normalization, and low-resolution dynamic blocks guided by the latent variables to generate a bilateral mesh. In the linear transformation weight generation and application module, the index map is used to index the bilateral grid to generate linear transformation weights, and the transformation weights are applied to the full-resolution image to be normalized to generate a transformed image with the same style as the standard image. The linear transformation weight generation module includes obtaining the amplification factor and bias of the linear transformation by performing trilinear interpolation in the length, width, and channel dimensions of the bilateral grid through the index graph; The linear transformation weighting module includes applying the magnification factor and bias point-to-point to the full-size image to be normalized to complete the style conversion, and finally outputting an image similar to the target coloring style image.

2. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 1, characterized in that: The acquisition of the full-resolution image to be normalized includes, The effective image of a Papanicolaou-stained cell section is obtained by a scanner acquiring an effective field of view at high resolution. The effective image is the full-resolution image to be normalized. The effective field of view includes a field of view containing analyzable cells under a microscope at a certain magnification, wherein the analyzable cells include at least one of eosinophilic cells and basophilic cells, and are precisely focused, unobstructed, and capable of analysis. The target stained full-resolution image includes images selected by humans and of the same scale as the full-resolution image to be normalized; The low-resolution image includes an image scaled from a full-resolution image to be normalized and target-colored, wherein the scaling factor is 4 by default, but can be changed according to the scale of the full-resolution image.

3. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 2, characterized in that: The acquisition of the encoding vectors involves applying encoding methods to the full-resolution images to be normalized and the target stained images to generate encoding vectors respectively. The encoding method includes converting an RGB image into an HSV image, dividing the white background region, pink eosinophilic cell region, and blue-green basophilic cell region using a threshold segmentation method, and then encoding each region by calculating the mean and variance of different regions.

4. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 3, characterized in that: The acquisition of the latent feature variables includes, An autoencoder consists of two parts: an encoder and a decoder. The calculation of the autoencoder is the compression of encoder features and the decompression of decoder features. Under the premise of minimizing the changes in decoder output and encoder input information, the encoder output is the latent feature variable that guides the generation of full-resolution index map and low-resolution bilateral grid.

5. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 4, characterized in that: The generation of the index graph includes, The full-resolution content structure mapping module is composed of multiple dynamic blocks connected in series. The full-resolution image to be normalized is input into the full-resolution content structure mapping module. The full-resolution image to be normalized undergoes a transformation after each dynamic block, and after multiple transformations, an index map that retains the structural information of the image to be normalized is generated.

6. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 5, characterized in that: The full-resolution dynamic block includes multi-layer parallel convolution and fully connected weight extraction; The multi-layer parallel convolution involves performing a non-linear transformation on the input full-resolution image to be normalized or the output feature image of a dynamic block to satisfy the generation of the final index map. The number of branches of the parallel convolution is a set constant. Each parallel convolution does not change the scale or structural content of the image, but only changes the intensity of the pixels. The fully connected weight extraction includes inputting the latent feature variables obtained by the autoencoder, and then obtaining the weights of the parallel convolution through multiple non-linear fully connected layers and the final Softmax activation function; The calculation of the Softmax activation function includes, ， in, Represents the latent variable of the feature. i Dimension value, Indicates the first i The weights of the parallel convolutional layers, where P represents the number of parallel convolutions. This represents the value of the latent variable corresponding to P parallel convolutions.

7. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 6, characterized in that: The generation of the bilateral mesh includes, The low-resolution image to be normalized and the target coloring style image are subjected to multi-layer convolution for downsampling feature extraction, thereby obtaining feature images of three different scales; The three feature images at different scales are input into three concatenated spatial adaptive instance normalization blocks to generate a feature map after coloring style transformation, thereby realizing the coloring style transformation of the feature images; The global features of the transformed feature image and the local features of the transformed feature image under multiple contiguous dynamic blocks are extracted. The global features and local features are fused and then the features of multiple contiguous dynamic blocks are extracted to generate a stylized bilateral mesh.

8. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 7, characterized in that: The spatial adaptive instance normalization operation includes, A pseudo-probability map is generated using threshold segmentation and weight balancing. Spatial region guidance is applied to the pseudo-probability map, and the coloring stylization of the feature image is completed through linear transformation of mean and variance. The linear transformation The calculations include, ， in, Indicates in The value of the feature image to be normalized at the location, Indicates in The value of the target staining feature image at the location. Indicates the feature image to be normalized in The variance of the region to which the location belongs. Indicates the target staining feature image in The variance of the region to which the location belongs. Indicates the feature image to be normalized in The mean of the region to which the location belongs. Indicates the target staining feature image in The mean of the region to which the location belongs.

9. The rapid cell staining normalization method using learnable bilateral filtering as described in claim 1, characterized in that: It also includes, To realize the storage of input and output images, the storage of module weights, and the operation, an apparatus is provided for a fast cell staining normalization method using learnable bilateral filtering. The apparatus includes a processor for computation, a memory, and an executable computation program and its weight parameters in the memory. The processor loads the computation program in the memory and initializes the computation program using the stored weight parameters.

Images