Three-stage diagnostic system for malignant gastric neoplasms based on analysis of pathomorphological specimens

Abstract

FIELD: image processing.SUBSTANCE: supporting medical decision-making based on the analysis of medical images. A system for diagnosing gastric pathologies is proposed, which includes the following components: an image acquisition unit intended for obtaining digital scans of full slides (WSI images) of gastric tissue; a WSI image preprocessing unit intended for dividing WSI images into fragments, their normalization and conversion of multichannel images into single-channel GrayScale, hereinafter referred to as slices. Slices are fragments of tissue, which is achieved by filtering fragments of the original WSI image based on their belonging to a cell mass mask obtained by binarizing the image using the Otsu algorithm; a slice analysis unit classifies each individual slice using an ensemble of neural networks; a results integration unit that combines the classification results of all slices to create a cumulative representation of pathologies and calculates the integral probability of WSI image belonging to ICD classes; a large-scale segmentation unit, based on the classifications of individual slices, generates XML markup of the original image, equivalent to the segmentation mask. The result visualization unit generates a thumbnail image displaying the original image and segmentation mask in compact form; the response transmission unit generates a docx output containing the ICD, an image, and XML markup.EFFECT: efficiency of systems and methods for diagnosing gastric pathologies using artificial intelligence.1 cl, 3 dwg, 2 tbl

Description

[0001] FIELD OF TECHNOLOGY

[0002] The present invention relates to the field of image processing, and more particularly to supporting medical decision making based on medical image analysis.

[0003] PRIOR ART

[0004] Gastrocnemia includes malignant and benign tumors, inflammatory processes, and precancerous conditions such as gastritis, ulcers, polyps, and metaplasia. Diagnosis of gastric pathologies is often based on a combination of clinical examinations and analysis of digital whole slide imaging (WSI).

[0005] Our product aims to create a system for detecting malignant gastric tumors based on step-by-step analysis of WSI images. The system involves dividing the WSI image into individual fragments, classifying each fragment using an ensemble of neural network classifiers, and then classifying the entire WSI image. This approach enables the analysis of large images where direct classification or segmentation is impossible.

[0006] Our product surpasses similar research and development in the following key parameters:

[0007] Multi-stage image analysis:

[0008] Our product includes multi-stage analysis, starting with WSI image preprocessing and ending with full-image integration of results. This approach allows for more accurate and reliable pathology detection than using a single model or single analysis step, as is often the case in other projects.

[0009] High classification accuracy:

[0010] We use an ensemble of neural networks to classify each WSI image fragment. This improves classification accuracy and reliability compared to using a single model, an advantage over projects considered, for example, at BMC Medicine and Frontiers in Oncology.

[0011] Integration of results:

[0012] Our product includes a results integration module that combines classification data from all slices to create a cumulative representation of pathologies across the entire WSI image. This allows for a more accurate assessment of the presence and distribution of pathologies, a feature often lacking in other studies.

[0013] Automatic report generation:

[0014] The system automatically generates detailed reports, including information on the presence of malignant tumors and the distribution of pathological patterns. This improves the convenience and speed of decision-making for physicians, unlike many existing solutions that require significant manual effort.

[0015] Large-scale segmentation:

[0016] Large-scale segmentation with XML markup generation has been implemented for the entire WSI image. This enables accurate visualization and analysis of pathological areas across the entire image, regardless of its resolution.

[0017] Reliability and speed of diagnostics:

[0018] Using the Otsu algorithm to highlight tissue areas in images speeds up the analysis process, as the algorithm quickly and efficiently identifies relevant areas. This facilitates rapid diagnosis of pathologies by excluding irrelevant areas and focusing on important tissue fragments, increasing the reliability of the results.

[0019] Taken together, these advantages make our project more accurate, reliable and convenient for use in clinical practice compared to existing solutions presented in the mentioned articles.

[0020] SUMMARY OF THE INVENTION

[0021] In order to improve the efficiency of systems and methods for diagnosing gastric pathologies using artificial intelligence, the present invention aims to develop an innovative system.

[0022] According to a first aspect of the present invention, a system for diagnosing gastric pathologies is provided, comprising the following components:

[0023] Imaging unit designed to obtain digital scans of full slides (WSI images) of gastric tissue.

[0024] The WSI image preprocessing block is designed to divide WSI images into fragments, normalize them, and convert multichannel images to single-channel GrayScale, hereinafter referred to as slices. Slices represent tissue fragments, achieved by filtering fragments of the original WSI image based on their membership in a cell mass mask obtained by binarizing the image using the Otsu algorithm.

[0025] The slice analysis unit classifies each individual slice using an ensemble of neural networks.

[0026] Result integration block, which combines the classification results of all slices to create a cumulative representation of pathologies, calculating the integrated probability of belonging of the WSI image to ICD classes.

[0027] The large-scale segmentation block, based on the classification of individual slices, generates XML markup of the original image, equivalent to the segmentation mask.

[0028] A result visualization block that generates a thumbnail image displaying the original image and segmentation mask in a compact form.

[0029] The response transmission block generates a count in docx format containing the ICD, an image, and markup in xml format.

[0030] The present invention improves the efficiency of systems and methods for diagnosing gastric pathologies. This ensures:

[0031] improving diagnostic accuracy;

[0032] increase the speed of diagnosis;

[0033] reducing the influence of the human factor (attentiveness, fatigue, responsibility);

[0034] Reducing the workload of medical personnel through automatic description of studies that do not contain pathologies;

[0035] These and other advantages of the present invention will become apparent from reading the following detailed description with reference to the accompanying drawings.

[0036] BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The claimed invention is illustrated by figures:

[0038] Fig. 1 - Example of annotation of a pathological specimen containing gastric tumors. Tubular adenocarcinoma G1 (red) and G2 (green), on the WSI image;

[0039] Fig. 2 - Block diagram of the system for diagnosing gastric pathologies;

[0040] Fig. 3 - Diagram of an ensemble of models performing pattern analysis.

[0041] EMBODIMENT OF THE INVENTION

[0042] Stomach cancer is one of the most common oncological diseases. It can develop in any part of the stomach and spread to other organs, especially the esophagus, lungs, and liver. Stomach cancer kills up to 800,000 people worldwide annually (as of 2008). Worldwide, stomach cancer is the fifth most common cancer and the third leading cause of cancer death, accounting for 7% of cases and 9% of deaths. Stomach cancer occurs more often in men.

[0043] Gastrocnemius diseases, such as tumors and inflammations, are typically diagnosed by physicians through manual image analysis. The present invention provides a method and system that mimic this diagnostic workflow by automatically determining the presence or absence of gastric cancer features in a pathological specimen and highlighting regions of interest in the pathological specimen for further evaluation by medical specialists.

[0044] WSI images play a key role in our project, enabling detailed analysis of gastric histological specimens. Using WSI images allows us to separate images into multiple independent fragments, which are then processed individually to improve diagnostic accuracy. This approach ensures more accurate and reliable detection of malignant tumors in histological images. The authors of this study compiled an anonymized database of patients with and without gastric pathologies. Each patient case consists of digital scans of full slides (WSI images) of gastric tissue. An experienced pathologist manually outlined the areas affected by the pathology on each digital image and classified the type of pathology.

[0045] The training data was labeled by medical specialists using the classes presented in the table below.

[0046] Table 1: Histological pattern classes for labeling pathological specimens.

[0047] SE squamous epithelium NG Normal glands of the mucous membrane F Fibrin IM Foci of intestinal metaplasia LT Lymphoid tissue GINL Low-grade glandular intraepithelial neoplasia GINH High-grade glandular intraepithelial neoplasia TACG1 G1 Tubular adenocarcinoma TACG2 G2 Tubular adenocarcinoma TACG3 G3 Tubular adenocarcinoma, solid variant PACG1 G1 Papillary adenocarcinoma PACG2 G2 Papillary adenocarcinoma MPAC G3 Micropapillary carcinoma PCC Loosely adherent cell carcinoma, signet ring cell subtype PCC-NOS Loosely adherent cell carcinoma, non-signet-ring cell subtype MAC1 Mucinous adenocarcinoma, type I MAC2 Mucinous adenocarcinoma, type II ACLS Adenocarcinoma with lymphoid stroma HAC Hepatoid adenocarcinoma ACFG Adenocarcinoma of the fundic glands SCC Squamous cell carcinoma of the stomach NDC Undifferentiated carcinoma NED Neuroendocrine differentiation

[0048] As shown in the block diagram in Fig. 2, the gastric pathology diagnostic system according to the present invention comprises an image acquisition unit, an image preprocessing unit, a slice analysis unit, a result integration unit, a large-scale segmentation unit, a result visualization unit, and a report transmission unit.

[0049] In one embodiment, the image acquisition unit may be a network card or other data receiving means. In another embodiment, the image acquisition unit may be a scanning apparatus itself.

[0050] The WSI image preprocessing block is responsible for generating slices from the WSI image, normalizing them, and converting them. The output slices are single-channel 256x256 images, each containing complete information about the corresponding region of the WSI image. These slices represent tissue fragments, achieved by obtaining a tissue mask by binarizing the image using the Otsu algorithm.

[0051] The slice analysis unit is responsible for classifying pathologies in individual stomach slices. It involves processing each slice with an ensemble of neural networks and generating a final class label for each slice.

[0052] The results integration block combines the classification results of all sections to create a cumulative representation of pathologies and calculate the integrated probability of belonging to a specific ICD code. It is represented by the adaboost classifier, which uses the prediction statistics for the sections obtained in the previous step as input.

[0053] The large-scale segmentation block uses the results of slice classification from the slice analysis block to generate XML markup of the original WSI image, which is essentially equivalent to the multi-class segmentation of the large WSI image.

[0054] The result visualization block generates a thumbnail of the original WSI image based on the results of the slice analysis block, using each slice and applying additional color coding to the thumbnail of the large WSI image.

[0055] The report transmission unit automatically generates a report based on the analysis of the WSI image, including the information of the presence of malignancy in the WSI image, as well as the predicted distribution of pathological patterns in the WSI image.

[0056] Stages of operation of the image acquisition unit:

[0057] Obtaining WSI images via network card or scanning device.

[0058] Stages of the WSI image preprocessing block:

[0059] Application of Otsu's algorithm for binarization, finding a tissue mask, and excluding non-tissue areas of the image from further system operation.

[0060] Split WSI image into single-channel slices of 256x256 pixels.

[0061] Set pixel values ​​to the range [0, 1].

[0062] Converting a three-channel image to a single-channel GreyScale

[0063] Stages of the slice analysis unit:

[0064] Processing each slice with an ensemble of neural networks.

[0065] Assign a final class label to each slice.

[0066] Stages of the results integration block:

[0067] Collecting statistics of slice classification predictions.

[0068] Using the AdaBoost classifier to process prediction statistics.

[0069] Large-scale segmentation unit operation stages:

[0070] Generate XML markup of the original WSI image based on the slice classification and the final AdaBoost classification results.

[0071] Stages of the result visualization block:

[0072] Generate a normal resolution WSI copy of the image in PNG format

[0073] Visualization of segmentation obtained from a large-scale segmentation block based on color coding applied to a thumbnail

[0074] Stages of the report transmission block:

[0075] Generate a report with information on malignant tumors and pathomorphological patterns.

[0076] Sending a report to the device that requested diagnostics.

[0077] Detailed description of each block operation is presented below

[0078] Image acquisition unit

[0079] The image acquisition unit in the gastric pathology diagnostic system is designed to collect and transmit digital full-slide scans (WSI images) of pathological specimens. Depending on the implementation, this unit may incorporate various hardware and technologies. In one embodiment, the unit may be a network card or other data acquisition device that integrates with hospital picture archiving and communication systems (PACS). In another embodiment, the image acquisition unit may be directly integrated into the scanning apparatus used to create high-quality, high-resolution images. The acquired images are then transmitted to the system for further processing and analysis.

[0080] Image preprocessing block

[0081] The image preprocessing block performs the key task of preprocessing acquired WSI images to prepare them for subsequent analysis. This block includes several important steps, beginning with tissue region extraction using the Otsu algorithm. This algorithm binarizes the image, enabling precise tissue mask generation. The resulting mask is used for further analysis, ensuring high performance and reliable data processing.

[0082] Next, slices are generated from the WSI image. During slicing, the entire image is divided into individual fragments measuring 256x256 pixels. After slicing, the images are normalized. Normalization is necessary to bring pixel values ​​into the [0, 1] range, which eliminates variations in illumination and contrast, making the images more uniform and suitable for further analysis. The images are then converted to single-channel formats, which simplifies their subsequent processing. Conversion to a single-channel format is also necessary to eliminate differences between tissues stained with different dyes and allows the neural network to analyze only the shape and distribution of cells.

[0083] The preprocessing unit prepares images for further analysis with minimal distortion and maximum accuracy. This ensures image compatibility with algorithms and models used in subsequent stages of gastric pathology diagnostics.

[0084] Slice Analysis Unit

[0085] Block analysis of slices representation by an ensemble of classifiers based on efficientnetb7.

[0086] Classifiers work in stages.

[0087] Step one: any slice is classified by model_1 (Fig. 3) as a slice containing normal gastric tissue or pathologically altered gastric tissue (not related to oncology) or elements of malignant tumors.

[0088] The second stage: According to the result of the high-level classifier, the slice is passed to one of the three classifiers Model_2A, Model_2B, Model_2C to set the output class.

[0089] Model 1 was trained on the entire training dataset, in which each image instance was labeled as "cancer," "sub-cancer," or "norm." When training this model, the data was annotated as follows. The "cancer" label was assigned to an image if it contained any of the initial annotation patterns (Table 1) related to malignant neoplasm classes. The "sub-cancer" label was assigned to an image if it contained any of the initial annotation patterns (Table 1) related to pathology classes but not related to malignant neoplasm classes. If an image lacked any pathological patterns, it was labeled as "norm."

[0090] Model_2A is designed to differentiate squamous epithelium, normal glandular tissue, intestinal metaplasia, and lymphoid tissue. Model_2B is designed to differentiate fibrin, granulation tissue, and glandular neoplasia. Model_2C is designed to differentiate squamous epithelium, normal glandular tissue, intestinal metaplasia, and lymphoid tissue. When training models 2A, 2B, and 2C, subsets of the entire training set related to cancer, sub-cancer, or normal were used. To train each of the models 2A, 2B, and 2C, the initial labeling was grouped as shown in Table 2.

[0091] The model ensemble works with the following classes:

[0092] Table 2: Classes for the model ensemble

[0093] Class Patterns Group Intestinal TACG1, TACG2, MAC1, PACG1, PACG2 Lauren Indeterminate TACG3, NDC Diffuse MAC2, PCC, PCC-NOS Other NED, ACLS, HAC, SCC, MPAC, ACFG squamous epithelium SE Normal Normal glandular tissue NG Lymphoid tissue LT Fibrin F Metaplastic and dysplastic Granulation tissue GT Glandular neoplasia (low-grade) GINL Glandular neoplasia (high-grade) GINH Foci of intestinal metaplasia IM

[0094] Results Integration Block

[0095] The input to the results integration block is the class prediction statistics for each slice, equivalent to the outputs of the ensemble of models 2A, 2B, and 2C from the slice analysis block. These statistics refer to the number of final predicted classes across the entire WSI image, determined by the ensemble for each slice. These statistics are fed to the adaboost classifier, whose output is the probability of the WSI image belonging to one of the following classes: K29, C16, D00, or D13, corresponding to the ICD code of the patient whose pathological specimen is displayed in the WSI image. This integrated probability takes into account the distribution and characteristics of identified pathologies across the entire image volume. The classes described above are ICD codes assigned by medical specialists to the entire WSI image.

[0096] Large-scale segmentation block

[0097] The Large-Scale Segmentation Block is responsible for generating XML markup for the original WSI image based on the data obtained from the Slice Analysis Block. This block takes the classification results from each individual slice where pathologies were identified and integrates them to create a holistic representation of pathologies across the entire WSI image.

[0098] The first step of the large-scale segmentation block is processing the results of the analysis of individual slices. These results include information on classification labels and the probabilities of the presence of various pathologies in each slice. Based on this data, the block creates segmentation maps that reflect the distribution of pathologies across the entire image.

[0099] Next, this block applies algorithms to combine these maps into a single XML markup. XML markup is a structured file that contains information about the location and types of pathologies in the entire WSI image. This markup is an important tool for further data analysis and interpretation, as it allows for visualization and exploration of the distribution of pathologies across the entire image.

[0100] The final output of the large-scale segmentation block is an XML file that can be used for further data processing and visualization, as well as for automatic report generation.

[0101] Result visualization block

[0102] The Result Visualization Block is responsible for creating a visualization block that is easy to understand and analyze based on the data obtained from the Large-Scale Segmentation Block. It generates a thumbnail of the original WSI image, onto which the analysis results are superimposed.

[0103] The visualization unit begins by obtaining data on the classification labels of each slice and segmented areas. Based on this data, a visualization is created, where each slice is marked on a thumbnail of the WSI image, and a mask of a fixed color and fixed transparency is applied to the corresponding areas based on the detected pathologies, ensuring optimal human perception of this information.

[0104] The visualization block utilizes the results of the slice analysis and large-scale segmentation blocks to accurately display the localization of pathological changes in the original image. This allows physicians and specialists to easily interpret the analysis results and make informed decisions based on the visualized data.

[0105] The final output of the visualization block is a miniature WSI image combined with a segmentation mask presented as a multi-color image, where each class of the slice analysis block is assigned its own color, with classes related to variations in normal gastric tissue displayed in shades of green, classes related to gastric pathologies excluding malignant neoplasms in shades of blue, and classes related to malignant neoplasms in shades of red.

[0106] Report Transmission Block

[0107] The report transfer unit in our system is designed to automatically generate and transfer reports based on WSI image analysis. This unit plays a key role in integrating results obtained from other system components and providing the physician with structured and comprehensive information on the condition of the histological specimen. After completing the image analysis and integrating the results, the report transfer unit generates a detailed report that includes several important elements. First, it records the slide number of the scanned histological specimen, ensuring precise specimen identification. This is essential for tracking and subsequent analysis in medical practice.

[0108] Next, the report includes a conclusion indicating the ICD code (C16, K29, D00, D13) as determined by the AI ​​and the probability that the entire sample belongs to a particular ICD code. Each ICD code is treated as a mutually exclusive integral class, and the task itself is considered equivalent to a multi-class drug classification problem. The automatically generated conclusion thus contains information about the presence of neoplasms in the scanned histological specimen and indicates the probability of pathology. The report transmission unit also generates a recommendation based on the analysis results. For example, if neoplasms are detected and there is a high probability of pathology, additional testing or treatment may be recommended. Recommendations are provided to support physicians in making informed clinical decisions.

[0109] The report generation block thus automates the report generation process, consolidating data from different analysis stages and presenting it in a structured format. This helps improve the accuracy and speed of diagnosis, providing physicians with complete and relevant information for decision-making.

[0110] EfficientNetB7 as the basis of basic classification

[0111] The EfficientNetB7 architecture is based on the principle of neural network scalability, effectively combining accuracy and performance. It uses a combination of augmentation, depth enhancement, and resolution enhancement techniques to achieve high feature representation with lower computational cost.

[0112] Feature Extraction: EfficientNetB7 begins by extracting features from the input image using basic convolution blocks. These blocks use convolutions with different kernel sizes and padding levels to capture information at different scale levels.

[0113] MBConv Blocks: The basic building blocks of EfficientNetB7, called MBConv (Mobile Inverted Bottleneck Convolution), include several steps:

[0114] Compression and Sparsification: Using 1x1 convolution layers to compress the input feature space, which reduces the number of channels.

[0115] Depthwise Convolutions: Apply depthwise convolutions that are performed separately for each channel, which reduces the computational complexity.

[0116] Expand and Pad: Expand the number of channels back to the original number using another 1x1 convolution, and add additional context using appropriate padding.

[0117] Nonlinear Activators: Applying an activation function such as ReLU or Swish to introduce nonlinearity.

[0118] Pooling Layers: Using global pooling layers to reduce spatial resolution while preserving important features. These layers help reduce the number of parameters and reduce the risk of overfitting.

[0119] Fully Connected Layers: Using fully connected layers, which process features at the pixel level and create a final feature representation for classification. These layers often include dropout to improve the model's resilience to overfitting.

[0120] Output Layer: The architecture is completed with an output layer with a softmax activation function, which transforms output values ​​into class probabilities. This allows the model to generate probabilistic predictions for each class.

[0121] Model Training: EfficientNetB7 is trained using the Adam optimizer with a low learning rate (e.g., 0.0002) and weight decay (e.g., 0.0001) to prevent overfitting and improve overall model performance. Early stopping is also important, stopping training if no improvement is observed on the validation set.

[0122] Application of Data Augmentation: During the training process, aggressive data augmentation is used, including rotation, shear, scaling, and brightness / contrast changes, to improve the generalization ability of the model.

[0123] The EfficientNetB7 architecture, with its thoughtful approach to scalability, is capable of effectively solving classification problems even on complex and diverse datasets.

[0124] AdaBoost as the basis of the ICD classifier

[0125] The AdaBoost (Adaptive Boosting) algorithm is used to create a highly effective classifier by combining multiple weak models (base classifiers), such as decision trees. This improves overall performance by increasing accuracy on difficult-to-classify objects.

[0126] Weight Initialization: At the beginning of each base classifier, all training samples are assigned equal weights. This allows each classifier to have an equal influence on training.

[0127] Training Base Classifiers: Several base classifiers (e.g., shallow decision trees) are trained sequentially, each focusing on correcting the mistakes made by previous classifiers.

[0128] Weighted Learning: Each classifier is trained based on the sample weights. Errors made in the current step increase the weights of the corresponding samples, making them more significant for the next classifier.

[0129] Weighted Voting: Each base classifier receives a weight proportional to its accuracy. The higher the accuracy of a classifier, the greater its voting weight.

[0130] Adaptive Weight Adjustment: After training each base classifier, the sample weights are adjusted:

[0131] Increase Misclassification Weights: The weights of samples that were misclassified are increased so that the next base classifiers can better focus on these difficult cases.

[0132] Reduce Weights of Correct Classifications: The weights of samples that have been classified correctly are reduced to reduce their influence on subsequent classifiers.

[0133] Classifier Combination: All base classifiers are combined to create a final ensemble classifier.

[0134] Weighted Merging: The final decision is made through a weighted vote of all base classifiers. Each classifier contributes to the final decision proportionally to its weight.

[0135] Output Layer: The output of the AdaBoost-based classifier is the combined decision of all base classifiers, represented as class probabilities. This allows the model to generate confident predictions for each class.

[0136] Model Training: AdaBoost is trained using various accuracy metrics (e.g., precision, F1-score) to evaluate the performance of each base classifier. The training process continues until a certain performance level is reached or until adding new base classifiers no longer improves the results.

[0137] Data Augmentation: Data augmentation, including various transformations such as translation, scaling, and rotation, can be used during the training process to improve the generalization ability of the model and make it more robust to various changes in the input data.

[0138] Using AdaBoost allows for the efficient combination of weak classifiers into a powerful ensemble classifier, resulting in improved model accuracy and robustness, especially in settings where classes have complex distributions or significant imbalance.

[0139] Focal Loss with weights for classes

[0140] Focal Loss is used to address class imbalance in the classification task for each neural network in the ensemble. Focal Loss was calculated using weights for positive and negative examples of each class, based on the frequency of occurrence of each class in the entire sample (hereinafter referred to as class-weight). After classifying each slice, the Focal Loss was calculated for each class. The value of the training loss function was multiplied by the weight of the corresponding class.

[0141] Otsu Algorithm

[0142] The Otsu algorithm is used to highlight tissue regions in WSI images, allowing analysis to focus only on significant fragments. The Otsu algorithm is applied in a manner that maximizes interclass variance, ensuring the best separation of image fragments containing cellular mass and coverslip defects and debris. Binary masks are created that highlight tissue regions. These masks are used to filter out slice fragments: only those fragments containing tissue are processed for further analysis. This approach allows resources to be focused on analyzing significant image regions, excluding background and irrelevant areas, increasing the efficiency and accuracy of subsequent stages of pathology classification and analysis.

Images