Systems and methods for pre-processing immunocytochemical images for machine learning image-to-image conversion

Abstract

Methods and systems are provided for pre-processing immunocytochemistry images for machine learning. The preprocessing method comprises the following steps: receiving paired positive and negative multi-protein images; and performing dyeing segmentation on the paired multi-protein image. The method further includes marking each image pixel corresponding to a cell of the paired multi-protein image; and converting image pixel information of the marked image pixels into a table form to generate a table of cell coordinates and geometric features for each of the paired multi-protein images. The method further includes, prior to inputting for the machine learning, generating two cell pairing tables for each of the paired multi-protein images according to Euclidean distance based pairing, the Euclidean distance-based pairing is based on cell coordinates in a table of the cell coordinates and geometric features of the multi-protein image for each of the pairs.

Description

Technical Field

[0001] The present invention relates generally to machine learning image-to-image conversion and, more particularly, to systems and methods for preprocessing immunocytochemistry images to improve the accuracy of machine learning image-to-image conversion. Background Art

[0002] Canine fine needle aspiration (FNA) based on immunocytochemistry is commonly used to anatomically observe the localization of specific proteins or antibodies in cells by using specific primary antibodies called biomarkers that bind to specific proteins or antigens. However, the large size of paired images (stained and unstained images, each image is about five gigabytes (~5GB) in size) obtained in a format that cannot be easily analyzed using machine learning and deep learning algorithms.

[0003] Currently, when paired images are involved, deep learning algorithms take longer to train and have high complexity; in addition, localization and labeling take too long to track and annotate each cell. In order to perform meaningful machine learning, converting image pixels used to depict cell characteristics and morphology into features for analysis requires well-defined preprocessing before machine learning, where preprocessing is the conversion of raw features into data that machine learning algorithms can understand and learn from.

[0004] Therefore, there is a need for methods and systems for preprocessing immunocytochemistry images for machine learning image-to-image conversion that overcome the shortcomings of existing systems and methods and provide simpler machine learning with greater accuracy and reduced computational time. Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and background of the present disclosure. Summary of the invention

[0005] According to at least one aspect of the present embodiment, a method for preprocessing immunocytochemistry images for machine learning is provided. The preprocessing method includes: receiving paired positive and negative polyprotein images; and performing staining segmentation on the paired polyprotein images. The method also includes: marking each image pixel corresponding to the cells of the paired polyprotein images; and converting the image pixel information of the marked image pixels into a table form to generate a table of cell coordinates and geometric features for each of the paired polyprotein images. The method also includes: before input for the machine learning, generating two cell pairing tables for each of the paired polyprotein images based on pairing based on Euclidean distance, wherein the pairing based on Euclidean distance is based on the cell coordinates in the table of cell coordinates and geometric features for each of the paired polyprotein images.

[0006] According to another aspect of the present embodiment, a system for preprocessing immunocytochemistry images for machine learning is provided. The system includes an image receiving module, an image segmentation module, a cell marking module, a table generation module, and a data output module. The image receiving module is configured to receive paired positive and negative polyprotein images, and the image segmentation module is configured to perform staining segmentation on the paired polyprotein images. The cell marking module is configured to mark each image pixel of the paired polyprotein image corresponding to the cells of the paired polyprotein image, and the table generation module is configured to convert the image pixel information of the marked image pixels into a table form to generate a table of cell coordinates and geometric features for each of the paired polyprotein images. The data output module is configured to generate two cell pairing tables for each of the paired polyprotein images according to pairing based on Euclidean distance when input for the machine learning, wherein the pairing based on Euclidean distance is based on the cell coordinates in the table of cell coordinates and geometric features for each of the paired polyprotein images. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are used to illustrate various embodiments and explain various principles and advantages according to the present embodiments. Like reference numerals in the accompanying drawings refer to the same or functionally similar elements in each view, and the accompanying drawings together with the following detailed description are incorporated into and form a part of the specification.

[0008] Figure 1 A framework diagram of the image preprocessing system according to this embodiment is shown.

[0009] Figure 2 includes Figure 2A , Figure 2B and Figure 2C , which shows a diagram of the separation of the hematoxylin staining image and the eosin staining image according to the present embodiment, wherein Figure 2A The hematoxylin-stained images show good separation, Figure 2B Hue, Saturation, Value (HSV) separation and Red, Green, Blue (RGB) separation are shown. Figure 2C Shows the use of Figure 2B The developed filter provides good separation for the eosin stained images.

[0010] Figure 3 includes Figure 3A and Figure 3B , shows that instance segmentation is performed on one of the cropped WG images according to the present embodiment, thereby generating normalized images and labeled images at different zoom levels, wherein Figure 3A shows the normalized WG-H images at eight different zoom levels, and Figure 3B Eight labeled images at corresponding zoom levels are shown.

[0011] Figure 4 includes 4A to 4D , shows a cropped and labeled image with a cell atlas and cell count according to this embodiment, wherein Figure 4A WG labeled images and their corresponding cell labels are shown. Figure 4B PAX5 labeling images and their corresponding cell markers are shown. Figure 4C The cell maps of both are shown. Figure 4D A bar graph of cell counts is shown.

[0012] Figure 5 Normalized, scaled, and rounded scatter plots of WG immune cells and CD3 immune-positive cells according to this example are shown.

[0013] Those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale, and that numbers in the figures may have been normalized for simplicity and clarity. DETAILED DESCRIPTION

[0014] The following detailed description is merely exemplary in nature and is not intended to limit the present invention or the application and use of the present invention. In addition, it is not intended to be bound by any theory proposed in the aforementioned background technology of the present invention or the following detailed description. The purpose of this embodiment is to provide a unique system and method for preprocessing immunocytochemistry images for machine learning image-to-image conversion. The preprocessing system and method according to this embodiment provide improved robustness and accuracy over traditional methods, and include preprocessing of protein images (e.g., Wright Giemsa (WG)-CD3 / PAX5 images), which converts the pixel information of each cell (even if the number of cells is equal to or greater than 4 million) into a tabular form with geometric features and simplifies it. In addition, the preprocessing system and method according to this embodiment converts and simplifies the pixel information of each cell in the image with cell arrangement among the paired image data sets, and its prediction accuracy is greater than 90%, and up to 92%, and only requires a very low minimum learning time for machine learning on the deep learning algorithm, for example, about 10 seconds or at most a few minutes compared to the conventional preprocessing time of several days. Therefore, the system and method according to this embodiment provide a preprocessing scheme that improves accuracy, reduces calculation time, and makes machine learning simple and understandable.

[0015] The image preprocessing according to the present embodiment obtains paired WG-CD3 / PAX5 positive or negative images, and the staining used to mark each cell is normalized and separated based on the non-maximum separation (NMS) convolutional neural network (CNN) of StartDist. The pixel information (i.e., cell information) of the marked image is converted into a tabular form together with the coordinates and geometric features for pairing CD3 / PAX5 positive or negative cells based on the minimum distance interval. The preprocessing according to the present embodiment generates two cell pairing tables for each pair of WG-CD3 / PAX5 images before being used as input for machine learning.

[0016] refer to Figure 1, Figure 100 shows the framework of the image preprocessing system according to the present embodiment. The framework according to the present embodiment receives as input a paired whole-side image (WSI) with WG-stained cells in the image, the image being CD3 positive or negative (+ / -) and PAX5 positive or positive (+ / -). Then, in the cropping and alignment steps 102a, 102b, the processing includes cropping and image registration. The registered pairs are divided into WG-CD3 / PAX5 positive pairs or negative pairs. Then, the registered pairs are normalized and filtered 104a, 104b to separate the hematoxylin and eosin staining into a normalized hematoxylin WG image 106, a normalized hematoxylin CD3 / PAX5 image 108, and a normalized eosin CD3 / PAS5 image 110. StarDist is a deep learning based 2D and 3D cell nucleus detection method that uses StartDist's NMS convolutional neural network 112 to perform segmentation of images 106, 108, 110, and cells are labeled as WG cells 114a, CD3 / PAX5 immunopositive cells 114b, and CD3 / PAX5 immunonegative cells 114c. With the help of the Regionprops tool in Python, cell counts, coordinates, and their geometric features such as (area, orientation, major axis length, diameter, orientation, solidity, and average intensity) are generated 116 in the form of tables (including overlapping tables) for paired WG-CD3 / PAX5 labeled images. Each table is sorted based on area and then normalized, scaled, and sorted 118. Once each table for paired WG-CD3 / PAX5 is sorted 118, all cells have been paired from WG with their corresponding stained cells in CD3 / PAX5 by the minimum Euclidean interval 120, which is indicated by the overlapping tables. According to the present embodiment, the pairing of cells from paired WG-CD3 / PAX5 images advantageously presents a simplified form of possible tabular data for machine learning based on only nine parameters (x-coordinate, y-coordinate, area, major axis length, minor axis length, orientation, equivalent diameter, average intensity, and solidity) for each counted cell desired for analysis. Thus, the framework shown in FIG. 100 preprocesses image information for machine learning image-to-image conversion and advantageously makes learning, prediction, and classification easier for machine learning and with sufficient accuracy in less learning time.

[0017] In the cropping steps 102a, 102b, the acquired data including the whole side image (WSI) of the WG paired with the stained CD3 / PAX5 having a rough size of about 5GB is cropped to a lower size so that each WG image is cropped 102a to a size of about 200MB to 250MB. Each WSI image is cropped 102a, 102b according to the normalization and marking limits of the StartDist algorithm by keeping the size of the WSI (in GB) divided by the number of cropped images less than or equal to about 200MB to 250MB. For example, if the image size of the WG image is 2GB, which is paired with the image size of the CD3 / PAX5 image of 1.5GB, the WG is kept standard and the paired images are cropped to 20 images with PNG format. QuPath software is used to read the WSI of the original NDPI scan.

[0018] As described above, the cropping steps 102a, 102b also include cell alignment. Cell alignment is used for image registration, and the first pass of image registration is adjusted using PTGui software and Python to pair-align the cropped WG image with the stained CD3 / PAX5 cropped image using a scale-invariant feature transform (SIFT) algorithm. To achieve this, fifteen to twenty-five cell coordinates are selected from each paired image, which have characteristics such as edges, scaling, rotation and translation.

[0019] Prior to image annotation using StartDist 112, each image pair is advantageously normalized and filtered 104a, 104b prior to color separation for nucleus segmentation to enable improved quantitative analysis. Figure 2A , Figure 200 shows a hematoxylin (H) stained image filtered 104a from a normalized WG / CD3 / PAX5 image 210 using conventional methods to derive a well-separated image 220, thereby deriving a normalized hematoxylin WG image 106 and a normalized hematoxylin CD3 / PAX5 image 108. It should be noted that the hematoxylin staining helps separate the blue cells from the WG image.

[0020] However, if Figure 2B As shown in FIG. 230 of FIG. 230 , the eosin (E) stained image is separated 104b from the CD3 / PAX5 image 240 by using hue, saturation, value (HSV) segmentation 250 and red, green, and blue (RGB) segmentation 255. The extracted segmentation values ​​of HSV 252 and the extracted segmentation values ​​of RGB 257 are shown in FIG. 230 . Based on the segmentation values ​​of HSV 252 and RGB 257, a filter is designed that sufficiently separates the eosin staining 270 from the CD3 / PAX5 image 260, as shown in FIG. Figure 2C As shown in Figure 260.

[0021] The framework according to the present embodiment uses StartDist to perform instance segmentation 112 for cell detection. StartDist uses trained per-pixel cell segmentation and subsequent pixel grouping for subsequent shape refinement. In order to avoid any errors in the cross-border or diffusion segments of cells, the framework 100 according to the present embodiment uses StarDist's non-maximum separation (NMS) convolutional neural network (CNN) to locate the cell nucleus via star-shaped convex polygons, which predicts the polygon of the cell instance at this position and reasonable shape for each pixel. NMS-CNN then marks and counts conflicting objects without considering color and separates them based on thresholds. By separating the color information of the image, cells can be counted and marked, and various quantitative analyses can be performed. Figure 2A Image 220 in FIG. 2 shows an example segmentation of one of the cropped WG images according to the present embodiment.

[0022] After instance segmentation 112 of the image, the image is labeled 114a, 114b, 114c ( Figure 1 ). Then use Regionprops to extract the label information. Regionprops is an image processing function that calculates various properties of pixel regions in an image, thereby measuring a set of properties for each label region in the label matrix or image. Figure 3A and 3B , Figure 3A shows normalized WG-H images at eight different zoom levels, and Figure 3B Eight labeled images at corresponding zoom levels are shown. Figure 3A In the first panel of FIG. 11 , it can be noted that there are “n” regions whose centroid, coordinates, area, diameter, major axis length, minor axis length, average intensity of pixels, and solidity can be easily measured. Therefore, a list or table with such information can be generated and extracted. According to the present embodiment, information of cell coordinates and geometric features, such as x-coordinate, y-coordinate, area, major axis length, minor axis length, orientation, equivalent diameter, average intensity, and solidity of each labeled cell or region of the paired WG-CD3 / PAX5 images used for registration is generated and extracted into table 116 ( Figure 1 ). Table 1 is Figure 3A Example of a table of cell coordinates and geometric features generated for a WG image.

[0023] Label x1 y1 area Equivalent diameter Average Strength Solidity 1 1572.241915 4293.429495 2319 54.338223 116.724450 0.979307 2 3206.334807 9831.170016 2488 56.283390 127.234727 0.977603 3 9738.976842 10023.875370 2375 54.990398 98.668632 0.979381 4 983.006436 3173.770919 3418 65.969180 118.709479 0.983597 5 1963.540326 798.415373 2641 57.988151 103.094283 0.982881

[0024] Table 1

[0025] After the marker information 114a, 114b, 114c is generated, cell counting and various analyses may be performed. Figures 4A to 4D , shows a cropped and labeled image with a cell map and cell count according to this embodiment. Figure 4A WG-labeled images with their corresponding cell markers are shown, while Figure 4B Images of PAX5 labeling with its corresponding cellular markers are shown. Figure 4C Cell diagrams of both are shown, and Figure 4D 4 shows a bar graph of cell counts, where bar 410 is the cell count of the WG-labeled image and bar 420 is the cell count in the PAX5-labeled image. Figure 4C and 4D As shown, a cell map (x, y) and a cell count can be generated. These cells in the WG-labeled image and the PAX5-labeled image can then be classified as immunopositive cells and immunonegative cells.

[0026] When stained with CD3 / PAX5, each WG image is stained with information of immunopositive and immunonegative cells, and when segmented 112, immunopositive cells are labeled 114b and immunonegative cells are labeled 114c. Cells 114b and 114c in the CD3 / PAX5 image both belong to the original WG image.

[0027] As part of normalization, scaling and sorting 118, the marked features are normalized because the geometric properties of each cell from the table 116 generated by Regionprops are normalized in their respective columns / matrices with min / max normalization. For example, scaling can scale the values ​​to six-bit integer values ​​before rounding. Rounding and sorting are performed using a Python script before pairing using the corresponding coordinates (x, y) 120 of the immunopositive and immunonegative cells. Figure 5 Normalized, scaled and rounded scatter plots are shown with WG immune cells in black and CD3 immunopositive cells in grey.

[0028] A sorting matrix is ​​defined for each (x, y) coordinate of WG-CD3 / PAX5 paired immunopositive cells and WG-CD3 / PAX5 paired immunonegative cells, with a sorting unit size of 50 units. Then, each sorting matrix is ​​labeled as a sorting label matrix.

[0029] After rounding and sorting 118, cells from the WG image paired with the stained CD3 / PAX5 image were paired by finding the center coordinates with the minimum Euclidean separation in the paired images. Prior to pairing, the WG cells and immunopositive / negative cells were sorted based on the x-values ​​of the cropped CD3 / PAX5 images.

[0030] After sorting, a script is formulated that conditions on finding cells in the WG image in proximity (x1, y1) to the corresponding immunopositive / negative cells in the approximate position of the CD3 / PAX5 image by using Euclidean interval separation for pairing. For example, the first row (x1, y1) of the sorted WG image is compared with all rows of the sorted (x2, y2) CD3 / PAX5 image. This creates an n×n dimensional distance matrix with distance information from each cell in the WG image to each cell in the CD3 / PAX5 image. From this matrix, WG cells are paired with CD3 / PAX5 cells based on the minimum distance between the cells. Finally, a table is generated with WG cells paired with CD3 / PAX5 cells and marked with a binary label (1 or 0). Immuno-negative cells are marked with 0, while immuno-positive cells are marked with 1.

[0031] According to the present embodiment, the preprocessing of immunocytochemical staining for improving the accuracy of machine learning involves normalization, staining separation, cell labeling based on non-maximum separation CNN, and segmentation methods, and can be performed in artificial intelligence-assisted pathology. For example, the preprocessing according to the present embodiment can be advantageously applied to artificial intelligence-assisted pathology for early diagnosis of suspected cancer, where cancer cells are usually identified by biomarkers such as hematoxylin and eosin staining cells before machine learning. More broadly, the preprocessing according to the present embodiment can be advantageously used in cytopathology, immunocytochemistry, staining separation by digital filtering, and cell localization.

[0032] One application that the global veterinary community will benefit from the pre-processing systems and methods according to this embodiment is the immunocytochemical image analysis for canine lymphoma. This will assist the fine needle aspiration (FNA) process by reducing the diagnostic time and converting the image data into a tabular form for clinical advice. In addition, by using the systems and methods according to this embodiment, medical, veterinary, scientific, and engineering personnel will benefit from training in AI research and development for their respective goals and analyses. In the future, the systems and methods according to this embodiment will be able to assist in the diagnosis of lymphoma, which can be used to help diagnostic laboratories, thereby improving diagnostic accuracy and improving the quality of life of cancer patients.

[0033] Therefore, it can be seen that the method and system according to the present embodiment provide a novel and effective image information preprocessing for machine learning image-to-image conversion, which makes machine learning simpler, faster and more efficient. The method and system according to the present embodiment improve the accuracy of machine learning of converted images, reduce machine learning calculation time, and make machine learning simple and understandable. At present, deep learning algorithms take a long time to train and have great complexity when applied to the positioning and labeling of paired images of stained cells, and such training requires too much time to track and annotate each cell. For meaningful machine learning, converting image pixel information of cell features and morphology into features requires well-defined preprocessing before machine learning. Preprocessing is the conversion of raw features into data that can be understood and learned by machine learning algorithms. This preprocessing not only makes learning simpler, but also has better accuracy and less computing time. According to this embodiment, multi-protein images (such as WG-CD3 / PAX5 images) are preprocessed to convert and simplify the pixel information of each cell (which can be a cell count of up to 4 million cells) in a tabular form. The pixel information has geometric features of cell arrangement in a paired image data set, and the prediction accuracy is 92%. The minimum learning time for machine learning is about 10 seconds to several minutes, while a deep learning algorithm using traditional preprocessing takes several days.

[0034] Although exemplary embodiments have been given in the foregoing detailed description of the present embodiment, it should be understood that there are a large number of variations. It should also be understood that the exemplary embodiments are merely examples and are not intended to limit the scope, applicability, operation or configuration of the present invention in any way. The foregoing detailed description will provide a convenient roadmap for those skilled in the art to implement the exemplary embodiments of the present invention, and it should be understood that various changes may be made to the functions and arrangements of the steps and operating methods described in the exemplary embodiments without departing from the scope of the present invention as described in the appended claims.

Claims

1. A method for preprocessing immunocytochemistry images for machine learning, the method comprising: receiving pairs of positive and negative polyprotein images; performing staining segmentation on the paired multi-protein images; labeling each image pixel corresponding to a cell of the paired multi-protein image; converting the image pixel information of the labeled image pixels into a table form for generating a table of cell coordinates and geometric features of each of the paired multi-protein images; as well as Prior to input for the machine learning, two cell pairing tables are generated for each of the paired multi-protein images based on Euclidean distance-based pairing, wherein the Euclidean distance-based pairing is based on the cell coordinates in the table of cell coordinates and geometric features for each of the paired multi-protein images.

2. The method according to claim 1, wherein: The paired multi-protein images include paired WG-CD3 / PAX5 images.

3. The method of claim 1, further comprising staining the received pairs of multi-protein images with hematoxylin and / or eosin prior to segmentation.

4. The method of claim 3, further comprising normalizing the staining of the paired multi-protein images prior to segmentation.

5. The method according to claim 1, wherein: The segmentation includes performing instance segmentation of the staining of the pair of multi-protein images using cell nucleus detection.

6. The method according to claim 5, wherein: The example segmentation of the staining of the paired multi-protein images using cell nucleus detection includes: separating the staining using the StarDist non-maximum separation convolutional neural network.

7. The method of claim 1, further comprising sorting the cells of the paired multi-protein images based on the cell coordinates in the table of cell coordinates and geometric features for each of the paired multi-protein images.

8. The method according to claim 7, wherein: Sorting the cells of the paired multi-protein images based on the cell coordinates comprises: Based on the cell coordinates, defining a sorting matrix including a plurality of sorting units; and Each of the cells is sorted into one of the plurality of sorting cells in response to the cell coordinates.

9. The method according to claim 1, wherein: The pairing based on Euclidean distance includes pairing corresponding cells in each of the paired multi-protein images based on a minimum Euclidean interval.

10. The method according to claim 1, wherein: Receiving the paired positive and negative multi-protein images includes: receiving whole-surface images of a plurality of protein cells, and cropping the whole-surface images to generate the paired positive and negative multi-protein images.

11. A system for preprocessing immunocytochemistry images for machine learning, the system comprising: an image receiving module configured to receive paired positive and negative multiprotein images; An image segmentation module, configured to perform staining segmentation on the paired multi-protein images; a cell labeling module configured to label each image pixel of the paired multi-protein image corresponding to a cell of the paired multi-protein image; a table generation module configured to convert the image pixel information of the labeled image pixels into a table form to generate a table of cell coordinates and geometric features for each of the paired multi-protein images; as well as A data output module is configured to generate two cell pairing tables for each of the paired multi-protein images based on Euclidean distance-based pairing when input for the machine learning, wherein the Euclidean distance-based pairing is based on cell coordinates in a table of cell coordinates and geometric features for each of the paired multi-protein images.

12. The system according to claim 11, wherein: The paired multi-protein images include paired WG-CD3 / PAX5 images.

13. The system according to claim 11, further comprising a cell staining module, wherein the cell staining module is configured to stain the received paired multi-protein images with hematoxylin and / or eosin, wherein: The cell staining module is connected to the image segmentation module to provide the stained image thereto.

14. The system according to claim 13, wherein: The cell staining module is further configured to normalize the staining of the paired multi-protein images before providing the stained images to the segmentation module.

15. The system according to claim 11, wherein: The segmentation module performs: instance segmentation of the staining of the paired multi-protein images using cell nucleus detection.

16. The system of claim 15, wherein: The segmentation module uses the StarDist non-maximum separation convolutional neural network to separate the staining to perform instance segmentation of the staining of the paired multi-protein images using cell nucleus detection.

17. The system according to claim 11 further comprises a sorting module configured to sort the cells of the paired multi-protein images based on the cell coordinates in the table of cell coordinates and geometric features for each of the paired multi-protein images.

18. The system of claim 17, wherein: The sorting module is configured to sort the cells of the paired multi-protein image based on the cell coordinates by defining a sorting matrix including a plurality of sorting units based on the cell coordinates, and sorting each of the cells into one of the plurality of sorting units in response to the cell coordinates.

19. The system of claim 11, wherein: The pairing based on Euclidean distance includes pairing corresponding cells in each of the paired multi-protein images based on a minimum Euclidean interval.

20. The system of claim 11, wherein: The image receiving module is configured to receive full-side images of a plurality of protein cells and crop the full-side images to generate the paired positive and negative multi-protein images.

Images