Gastric cancer pathological image classification method and system based on HER2 gene detection

Abstract

The present application belongs to the technical field of computer and image processing, and particularly relates to a gastric cancer pathological image classification method and system based on HER2 gene detection, aiming to solve the problem that the prior art cannot accurately classify gastric cancer pathological images, thereby failing to effectively assist doctors in predicting the curative effect of trastuzumab treatment for patients. The present application comprises: obtaining a gastric digital pathological section image of a gastric cancer patient; performing region extraction and segmentation operation to obtain a plurality of image blocks with a set pixel size and containing a set tissue type; performing data enhancement normalization and data division of the image blocks to obtain a training image package; performing network iterative training through the training image package to obtain a gastric cancer pathological image classification model; and classifying the real-time obtained gastric digital pathological section image of the patient through the gastric cancer pathological image classification model. The present application realizes accurate classification of gastric cancer pathological images, thereby assisting doctors in effectively predicting the curative effect of trastuzumab treatment for patients.

Description

Technical Field

[0001] The present invention belongs to the technical field of computers and image processing, and in particular relates to a gastric cancer pathology image classification method and system based on HER2 gene detection. Background Art

[0002] Gastric cancer is a common malignant tumor. Several important molecular pathogenic pathways of gastric cancer are currently known, including the PI3K / AKT / mTOR, MAPK signaling pathways ERK, JNK, p38, and Hippo pathways, which are closely related to apoptosis, autophagy, tumor size, invasion depth, and distant metastasis of gastric cancer cells. However, although targeted therapy for gastric cancer has shown considerable results in basic research, it is difficult to translate it into clinical practice. Currently, only human epithelial growth factor receptor 2 (HER2) has been approved for clinical targeted therapy of gastric cancer [1].

[0003] Trastuzumab is a specific anti-HER2 targeted drug. Its combination with chemotherapy has prolonged overall survival to 16 months compared to chemotherapy alone, significantly improving the average survival of less than one year for advanced gastric cancer treated with traditional chemotherapy. Multiple studies have demonstrated that anti-HER2 therapy (primarily monoclonal antibodies targeting the HER2 protein) has significant efficacy in both in vitro and in vivo gastric cancer models.

[0004] However, in actual clinical trials, the objective efficacy of trastuzumab is only 47.3%. How to develop a fast and accurate method to predict the efficacy of trastuzumab before treatment is of great clinical value. There is a high degree of heterogeneity in HER2 expression in gastric cancer, and different patients also show different degrees of heterogeneity in pathological images. The efficacy of trastuzumab treatment is related to the heterogeneity of HER2 expression in gastric cancer. Among patients receiving trastuzumab treatment, studies have observed that patients with gastric cancer with better homogeneity of HER2 expression have longer progression-free survival (PFS) and overall survival (OS) than those with higher heterogeneity [2][3].

[0005] Clinically, HER2 expression heterogeneity in gastric cancer is primarily assessed through immunohistochemistry and in situ hybridization on pathological sections. Immunohistochemistry can partially assess HER2 expression, while in situ hybridization is the gold standard for HER2 testing. However, even with relatively accurate HER2 assessment, accurate and effective classification of gastric cancer pathological images remains difficult, hindering physicians' ability to effectively predict the efficacy of trastuzumab treatment.

[0006] The following documents are technical background materials related to the present invention:

[0007] [1] Zhang Ruihao (reviewed by Zhang Ming). Research progress on anti-HER2 therapy for advanced gastric cancer[J]. Journal of Medical Postgraduates, 2022(035-002).

[0008] [2]Gravalos, C., & Jimeno, A. (2008). HER2 in gastric cancer: a new prognostic factor and a novel therapeutic target. Annals of Oncology, 19(9), 1523-1529.

[0009] [3]Huemer,F.,Weiss,L.,Regitnig,P.,Winder,T.,Hartmann,B.,Thaler,J.,...& E.(2020).Local and Central Evaluation of HER2 Positivity and Clinical Outcome in Advanced Gastric and Gastroesophageal Cancer—Results from the AGMT GASTRIC-5Registry.Journal of Clinical Medicine, 9(4),935. Summary of the Invention

[0010] In order to solve the above-mentioned problem in the prior art, namely, the inability of the prior art to accurately classify gastric cancer pathology images, and thus the inability to effectively assist doctors in predicting the efficacy of trastuzumab treatment for patients, the present invention provides a gastric cancer pathology image classification method based on HER2 gene detection, the gastric cancer pathology image classification method comprising:

[0011] Step S10, obtaining a digital pathological slice image of the stomach of a gastric cancer patient; the slice image includes the expression of the HER2 gene in the cancer tissue;

[0012] Step S20, performing region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type;

[0013] Step S30, performing data enhancement normalization and data partitioning on the plurality of image blocks having a set pixel size and containing a set tissue type to obtain a training image package;

[0014] Step S40, iteratively training the constructed ResNet convolutional neural network using the training image package to obtain a gastric cancer pathology image classification model;

[0015] Step S50 , based on the real-time acquired digital pathological slice image of the patient's stomach, classification is performed using a gastric cancer pathological image classification model to obtain an image classification result.

[0016] In some preferred embodiments, the digital pathological section image of the stomach of the gastric cancer patient is an image stained by hematoxylin-eosin staining;

[0017] The expression of the HER2 gene in the cancer tissue is obtained by immunohistochemistry, including negative information and positive information.

[0018] In some preferred embodiments, step S20 includes:

[0019] Step S21, connecting the cancer region boundary points in the annotation file to obtain the cancer region boundary, and converting the cancer region boundary into an image mask;

[0020] Step S22, downsampling the slice image and the image mask to a set level;

[0021] Step S23, extracting regions of the downsampled slice image using the downsampled image mask, and dividing the extracted sub-regions into image blocks of a set pixel size;

[0022] In step S24 , the ratio of the area of ​​each image block that overlaps with the ROI region to the area of ​​the image block is calculated, and image blocks with a size greater than a set threshold are extracted to obtain multiple image blocks with a set pixel size and containing a set tissue type.

[0023] In some preferred embodiments, the downsampling method is as follows:

[0024]

[0025] Where H, W are the height and width of the slice image and image mask after downsampling, height, width are the height and width of the slice image and image mask before downsampling, and level is the downsampling scale.

[0026] In some preferred embodiments, the data enhancement and normalization method is as follows:

[0027] Performing horizontal flipping, vertical flipping, and random rotation on each of a plurality of image blocks of a set pixel size and containing a set tissue type to obtain an enhanced image block set;

[0028] The brightness and contrast of each image block in the enhanced image block set are normalized to obtain an enhanced normalized image block set.

[0029] In some preferred embodiments, the data division method is as follows:

[0030] Randomly dividing the enhanced normalized image block set to obtain an image package set consisting of a set number of image blocks;

[0031] Determine the label of each image block and execute:

[0032] If an image bag contains at least one image block with a positive label, the image bag is marked as a positive multi-instance bag; otherwise, the image bag is marked as a negative multi-instance bag.

[0033] In some preferred embodiments, the method for determining the label of each image block is:

[0034] Extracting a feature map of the image block through a ResNet convolutional neural network;

[0035] A maximum pooling operation is performed on the feature map, and the probability of the image block belonging to the positive class and the negative class is obtained by calculating the softmax normalization function.

[0036] In some preferred embodiments, the gastric cancer pathology image classification model is trained by:

[0037] Step B10, extracting image texture and cell morphology of the image blocks in the training image package, and excluding gastric stromal cells and glandular cells to obtain a preprocessed training image package;

[0038] Step B20, constructing a ResNet convolutional neural network, and training a single-scale multi-instance learning network for predicting bag labels based on the preprocessed training image bag for each downsampling scale of the digital pathological section image of the stomach;

[0039] Step B30: Based on the weights of the single-scale multi-instance learning network, multi-instance learning is performed on the multi-scale pre-processed training image package to obtain a gastric cancer pathology image classification model.

[0040] In some preferred embodiments, the ResNet convolutional neural network includes a set number of residual connection blocks;

[0041] The residual connection block includes a 3×3 convolutional layer, a batch normalization layer, a ReLU activation function, a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

[0042] Another aspect of the present invention provides a gastric cancer pathology image classification system based on HER2 gene detection, the gastric cancer pathology image classification system comprising:

[0043] A data acquisition module is configured to obtain digital pathological slice images of the stomach of a gastric cancer patient; the slice images include expression of the HER2 gene in the cancer tissue;

[0044] a region extraction and segmentation module configured to perform region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type;

[0045] a data subpackaging module configured to perform data enhancement normalization and data partitioning on the plurality of image blocks of set pixel size and containing set tissue type to obtain a training image package;

[0046] A model training module is configured to iteratively train the constructed ResNet convolutional neural network using the training image package to obtain a gastric cancer pathology image classification model;

[0047] The classification module is configured to classify the digital pathological slice images of the patient's stomach obtained in real time through a gastric cancer pathological image classification model to obtain image classification results.

[0048] Beneficial effects of the present invention:

[0049] (1) The gastric cancer pathology image classification method based on HER2 gene detection of the present invention can analyze H&E-stained gastric cancer pathology sections and provide information on the patient's HER2 gene expression and predicted survival time, accurately classifying the digital pathology section images of the patient's stomach at various stages, thereby effectively assisting doctors in determining the pertinence of HER2 targeted therapy, improving the prognosis of some patients who need targeted therapy, and avoiding meaningless HER2 targeted therapy for some patients.

[0050] (2) The present invention provides a gastric cancer pathology image classification method based on HER2 gene detection. During model training, a single-scale multi-instance learning network for predicting bag labels is first trained. Then, based on the single-scale multi-instance learning network, multi-instance learning is performed to improve the performance of the trained model and further improve the accuracy of gastric cancer pathology image classification.

[0051] (3) The present invention adopts a gastric cancer pathology image classification method based on HER2 gene detection, and adds an attention mechanism to the model. The attention mechanism can help the model assign different weights to each part of the input, extract more critical and important information, and enable the model to make more accurate judgments. At the same time, it will not bring greater overhead to the calculation and storage of the model, further improving the accuracy of gastric cancer pathology image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0053] Figure 1 Schematic diagram of the process of the gastric cancer pathology image classification method based on HER2 gene detection of the present invention;

[0054] Figure 2 This is a schematic diagram of the multi-scale pathology image bag-of-class labels prediction process of the gastric cancer pathology image classification method based on HER2 gene detection of the present invention. DETAILED DESCRIPTION

[0055] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0056] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0057] The present invention provides a gastric cancer pathology image classification method based on HER2 gene detection. Developing an accurate and effective gastric cancer pathology image classification method, on this basis, assisting doctors in accurately, stably and conveniently predicting the efficacy of anti-HER2 targeted therapy has important clinical value and is also a clinical problem that needs to be solved urgently.

[0058] The present invention provides a gastric cancer pathology image classification method based on HER2 gene detection, the gastric cancer pathology image classification method comprising:

[0059] Step S10, obtaining a digital pathological slice image of the stomach of a gastric cancer patient; the slice image includes the expression of the HER2 gene in the cancer tissue;

[0060] Step S20, performing region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type;

[0061] Step S30, performing data enhancement normalization and data partitioning on the plurality of image blocks having a set pixel size and containing a set tissue type to obtain a training image package;

[0062] Step S40, iteratively training the constructed ResNet convolutional neural network using the training image package to obtain a gastric cancer pathology image classification model;

[0063] Step S50 , based on the real-time acquired digital pathological slice image of the patient's stomach, classification is performed using a gastric cancer pathological image classification model to obtain an image classification result.

[0064] In order to more clearly illustrate the gastric cancer pathology image classification method based on HER2 gene detection of the present invention, the following Figure 1 Each step in the embodiment of the present invention is described in detail.

[0065] The gastric cancer pathology image classification method based on HER2 gene detection according to the first embodiment of the present invention includes steps S10 to S50, each of which is described in detail as follows:

[0066] Step S10: obtaining a digital pathological slice image of the stomach of a gastric cancer patient; the slice image includes the expression of the HER2 gene in the cancer tissue.

[0067] The digital pathological section images of the stomach of gastric cancer patients are stained with hematoxylin-eosin. The expression of the HER2 gene in cancer tissue is obtained by immunohistochemistry, including negative and positive information.

[0068] In one embodiment of the present invention, the data set comes from 163 WSI (digital pathology slide images) images (H&E staining, hematoxylin-eosin staining) collected from a cancer hospital. The data set specifically includes four types of efficacy evaluation information: ORR (objective response rate), DCR (disease control rate), PFS (progression-free survival) and OS (overall survival).

[0069] Step S20 , performing region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type.

[0070] First, the digital pathology slide image is preprocessed, segmenting the entire large image into numerous 512×512 pixel images, each representing a specific tissue type. Because WSI images are a relatively specialized medical image, their width and height typically range from tens to hundreds of thousands of pixels, resulting in a relatively large image. This makes it impossible to simply process the entire image for network training like other medical images. Instead, a mask is used to extract the tumor region from the image, and then the image is segmented into small patches (patches) that can be used in the deep learning model.

[0071] In step S20, the tumor area in the image is first extracted using the labeled image mask, and then the image is divided into small blocks that can be used for the deep learning model. Specifically, the following steps are included:

[0072] Step S21 : Connecting the cancer region boundary points in the annotation file to obtain the cancer region boundary, and converting the cancer region boundary into an image mask.

[0073] The annotation file contains point-by-point information, which is converted into an image mask. Image annotation files are in XML format. This file stores the boundaries of each tissue type. The data is composed of a series of points, each representing the location where a medical professional clicked the mouse on the image. These points form a polygonal outline, delineating the area of ​​each tissue type.

[0074] Step S22: downsampling the slice image and the image mask to a set level.

[0075] The original WSI image is too large and needs to be downsampled. In one embodiment of the present invention, the downsampling method is shown in formula (1), where level represents the scale of downsampling. According to this formula, the image and the corresponding annotation points are downsampled to the corresponding level for further analysis:

[0076]

[0077] Among them, H, W are the height and width of the slice image and image mask after downsampling, respectively, and height, width are the height and width of the slice image and image mask before downsampling, respectively.

[0078] Step S23 , extracting regions of the downsampled slice image using the downsampled image mask, and dividing the extracted sub-regions into image blocks of a set pixel size.

[0079] For the boundary outline polygon extracted in step S21, the boundary of its outline can be extracted first, and the maximum value of the four upper, lower, left and right boundaries of the outline can be extracted to form a sub-region of the image waiting for preprocessing (to obtain a local image of the outlined area), and then the extracted image can be divided into patch-level images of size 512×512.

[0080] In step S24 , the ratio of the area of ​​each image block that overlaps with the ROI region to the area of ​​the image block is calculated, and image blocks with a size greater than a set threshold are extracted to obtain multiple image blocks with a set pixel size and containing a set tissue type.

[0081] For each segmented patch-level small image, the area that overlaps with the polygonal ROI extracted in step S21 is calculated. When the overlapping area reaches more than 75% of the area of ​​the patch-level small image, this patch is regarded as valid data and stored in the divided patch-level image.

[0082] Step S30 , performing data enhancement normalization and data partitioning on the plurality of image blocks having the set pixel size and containing the set tissue type to obtain a training image package.

[0083] For the training image package obtained in step S20, HER2-negative (parts labeled HER2=0 and HER=1) and HER2-positive (parts labeled HER2=2 and HER=3) patients are randomly divided into a training set and a test set at a ratio of 8:2.

[0084] Data enhancement normalization, the method is:

[0085] Performing horizontal flipping, vertical flipping, and random rotation on each of a plurality of image blocks of a set pixel size and containing a set tissue type to obtain an enhanced image block set;

[0086] The brightness and contrast of each image block in the enhanced image block set are normalized to obtain an enhanced normalized image block set.

[0087] In one embodiment of the present invention, in order to increase the number of data sets and avoid model overfitting, the data is flipped horizontally or vertically during data processing. In addition, the data set images are randomly rotated 45 degrees.

[0088] For data normalization, the dataset was normalized for parameters such as brightness and contrast. For the RGB color channels of the image, the means were normalized to [0.6209136, 0.39992052, 0.68346393] (the three values ​​represent the RGB channels in the image, the same below), and the variances were normalized to [0.26443535, 0.30418476, 0.19353978].

[0089] In cancer pathology, since tumor and non-tumor areas are mixed, a WSI often contains pathological tissues of HER2-negative and HER2-positive areas. Therefore, the present invention draws on the method of Multiple Instance Learning (MIL). Specifically, the present invention organizes different patch images from a WSI into one or several packages. In this image organization process, similar pathological images will be organized into a package so that the image texture and staining information in the package are relatively similar. If a package contains positive and negative pathological images, then the package will be marked as a positive example; if it is all negative images, it will be marked as a negative example. In addition, when pathologists diagnose patients, they observe slides at different scales. To simulate this, the present invention considers patches with multiple different scales.

[0090] Data partitioning method is:

[0091] Randomly dividing the enhanced normalized image block set to obtain an image package set consisting of a set number of image blocks;

[0092] Determine the label of each image block and execute:

[0093] If an image bag contains at least one image block with a positive label, the image bag is marked as a positive multi-instance bag; otherwise, the image bag is marked as a negative multi-instance bag.

[0094] The method for determining the label of each image block is:

[0095] Extracting a feature map of the image block through a ResNet convolutional neural network;

[0096] A maximum pooling operation is performed on the feature map, and the probability of the image block belonging to the positive class and the negative class is obtained by calculating the softmax normalization function.

[0097] Step S40: iteratively train the constructed ResNet convolutional neural network using the training image package to obtain a gastric cancer pathology image classification model.

[0098] Gastric cancer pathology image classification model, its training method is:

[0099] Step B10: extracting the image texture and cell morphology of the image blocks in the training image package, and excluding gastric stromal cells and glandular cells to obtain a pre-processed training image package.

[0100] Preprocess the patch image to be segmented, extract image texture, cell morphology and other information, and exclude gastric stromal cells, glandular cells, etc. that are significantly different from tumor tissue.

[0101] Step B20: construct a ResNet convolutional neural network, and train a single-scale multi-instance learning network for predicting bag labels based on the preprocessed training image bag for each downsampling scale of the digital pathological section image of the stomach.

[0102] The convolutional neural network used in this paper is ResNet18, whose main structure is the residual connection block. Each residual block first has two 3×3 convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Then, through a cross-layer direct path, these two convolution operations are skipped and the input is directly applied before the final ReLU activation function, as shown in Table 1:

[0103] Table 1

[0104]

[0105] Before model training, pre-training is performed on the model to improve the model effect.

[0106] Multi-instance learning for patch-level images of single-scale WSI:

[0107] For each WSI downsampling scale, a single-scale multi-instance learning network is trained to predict the bag label, where each bag only contains patches on the downsampled scale s image, and the optimized network obtains the bag class label prediction.

[0108] like Figure 2 As shown in FIG, a schematic diagram of the multi-scale pathology image bag-class label prediction process of the gastric cancer pathology image classification method based on HER2 gene detection of the present invention is shown. The training task of single-scale multi-instance learning is expressed as a minimization problem shown in formula (2):

[0109]

[0110] The term being minimized is the loss function for bag class label prediction, which is simply defined as the cross entropy between the true class label and the predicted class label. Here, through the attention mechanism, the final bag class label is predicted by only using examples with large attention.

[0111] The attention mechanism is a data processing method in machine learning. It exploits the fact that humans can extract global features from only a few important local features when observing an object. The attention mechanism helps the model assign different weights to each part of the input, extracting more critical and important information, enabling the model to make more accurate judgments without incurring additional computational and storage overhead. Leveraging the attention mechanism can make deep learning more targeted when observing objects, improving the accuracy of both object recognition and classification.

[0112] In step B21, the patch obtained from a gastric cancer pathological section image is processed into a 224*224*3 three-channel image and input into the ResNet18 module. After passing through four residual connection blocks (including batch normalization layer, convolution layer and activation function), a feature image Y1 of size 7*7*512 is extracted.

[0113] In step B22, the feature map Y1 is subjected to a maximum pooling operation, and the probability of the image belonging to the positive class or the negative class is calculated using a softmax normalization function.

[0114] In step B23, after completing steps B21-B22 for all images in a package, the label output P (negative or positive) of the package is obtained through the attention mechanism based on the different attention weights of each image in the package.

[0115] Step B24, calculate the cross entropy loss between the output P and the true label of the gastric cancer pathology image package using the cross entropy calculation formula, backpropagate the cross entropy loss to the convolutional neural network through the gradient descent algorithm, and update the network parameters of the convolutional neural network and the attention network.

[0116] Step B30: Based on the weights of the single-scale multi-instance learning network, multi-instance learning is performed on the multi-scale pre-processed training image package to obtain a gastric cancer pathology image classification model.

[0117] Multi-instance learning of patch images at multiple scales:

[0118] Pathology sections of different scales are included in the bag, and the trained weights are further fine-tuned. A multi-scale network is trained to predict bag labels of different scales, where each bag contains patches of different scales. The network is optimized to obtain bag label predictions.

[0119] The process of predicting bag-of-class labels for multi-scale pathology images is expressed as the function shown in formula (3):

[0120]

[0121] Among them, the feature extraction set s∈[S] has been preliminarily trained in the training process of single-scale pathological images, and the parameters of the previous step are directly used here.

[0122] Parameter Set The training of is expressed as a minimization problem as shown in formula (4):

[0123]

[0124] Step S50 , based on the real-time acquired digital pathological slice image of the patient's stomach, classification is performed using a gastric cancer pathological image classification model to obtain an image classification result.

[0125] After obtaining the classification results of the patient's gastric digital pathological section image, the patient's ORR label (complete response (CR) or partial response (PR)) can also be predicted based on the classification results.

[0126] Prediction of objective response rate (ORR):

[0127] The last fully connected layer of the gastric cancer pathology image classification model is changed to output the prediction of ORR, and the optimizer uses the cross entropy loss function as the loss function, as shown in formula (5):

[0128]

[0129] Among them, y represents the patient's actual ORR label, represents the predicted ORR signature.

[0130] Prediction of progression-free survival (PFS) (the time from the start of observation to death or data loss):

[0131] The last fully connected layer of the gastric cancer pathology image classification model is changed to output a numerical value, and the optimizer uses the C-index index to calculate the loss function, as shown in formula (6):

[0132]

[0133] Among them, T represents the survival time of two randomly selected patients i and j, l and They represent the true value and the value predicted by the model respectively. If the survival time of patient i is longer than that of patient j, the value of the function is 1, otherwise it is 0.

[0134] After evaluating and considering the model, the present invention further adjusted the loss function of the model. Similarly, the last fully connected layer of the gastric cancer pathology image classification model was changed to output a numerical value for outputting the predicted progression-free survival (PFS). The optimizer and regularization terms remained unchanged, as shown in formula (7):

[0135]

[0136] Among them, N E＝1 represents the number of patients who had a recorded clinical event (in this invention, death or disease progression), i =1 represents each specific case, Represents l2 regularization of the network, λ is the regularization parameter, Represents the output of the network, It refers to the set of patients who are still alive at time t and will die or have disease progression in the future.

[0137] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0138] The second embodiment of the present invention is a gastric cancer pathology image classification system based on HER2 gene detection, and the gastric cancer pathology image classification system includes:

[0139] A data acquisition module is configured to obtain digital pathological slice images of the stomach of a gastric cancer patient; the slice images include expression of the HER2 gene in the cancer tissue;

[0140] a region extraction and segmentation module configured to perform region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type;

[0141] a data subpackaging module configured to perform data enhancement normalization and data partitioning on the plurality of image blocks of set pixel size and containing set tissue type to obtain a training image package;

[0142] A model training module is configured to iteratively train the constructed ResNet convolutional neural network using the training image package to obtain a gastric cancer pathology image classification model;

[0143] The classification module is configured to classify the digital pathological slice images of the patient's stomach obtained in real time through a gastric cancer pathological image classification model to obtain image classification results.

[0144] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0145] It should be noted that the gastric cancer pathology image classification system based on HER2 gene detection provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into one module or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for the purpose of distinguishing the modules or steps and are not to be regarded as improper limitations of the present invention.

[0146] An electronic device according to a third embodiment of the present invention includes:

[0147] at least one processor; and

[0148] a memory communicatively connected to at least one of the processors; wherein,

[0149] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned gastric cancer pathology image classification method based on HER2 gene detection.

[0150] A fourth embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned gastric cancer pathology image classification method based on HER2 gene detection.

[0151] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0153] The intelligent prediction device for gastric cancer therapeutic effect according to the fifth embodiment of the present invention comprises a pathological image acquisition device, a prediction device, and a display device;

[0154] The pathological image acquisition device is used to obtain digital pathological slice images of the stomach of a gastric cancer patient; the slice images contain the expression status of the HER2 gene expressed by the cancer tissue.

[0155] The pathological image acquisition device can be an ordinary camera + scanner, a camera / camcorder + image acquisition card, a microscopic digital camera or a microscopic scanner, etc. The present invention will not describe them in detail here.

[0156] The prediction device includes an image processing module, a model training module, an image classification module and a prediction module:

[0157] The image processing module is used to perform region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type, perform data enhancement normalization and data partitioning on the plurality of image blocks of the set pixel size and containing the set tissue type, and obtain a training image package;

[0158] The model training module is used to iteratively train the constructed ResNet convolutional neural network through the training image package to obtain a gastric cancer pathology image classification model;

[0159] The image classification module is used to classify the digital pathological slice images of the patient's stomach acquired in real time using the gastric cancer pathology image classification model to obtain image classification results;

[0160] The prediction module is used to predict the objective response rate (ORR) and / or progression-free survival (PFS) based on the gastric cancer pathology image classification model and image classification results.

[0161] ORR includes complete response (CR) and partial response (PR).

[0162] The display module is used to display the prediction results output by the prediction device.

[0163] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0164] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0165] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A gastric cancer pathology image classification method based on HER2 gene detection, characterized in that: The gastric cancer pathology image classification method comprises: Step S10, obtaining a digital pathological slice image of the stomach of a gastric cancer patient; the slice image includes the expression of the HER2 gene in the cancer tissue; Step S20, performing region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type; Step S30, performing data enhancement normalization and data partitioning on the plurality of image blocks having a set pixel size and containing a set tissue type to obtain a training image package; Step S40, iteratively training the constructed ResNet convolutional neural network using the training image package to obtain a gastric cancer pathology image classification model; Step S50, based on the real-time acquired digital pathological slice image of the patient's stomach, classify it using a gastric cancer pathological image classification model to obtain an image classification result; The gastric cancer pathology image classification model, when predicting the objective response rate (ORR), changes the model's last fully connected layer to output the ORR prediction, and the optimizer uses the cross-entropy loss function as the loss function. When predicting progression-free survival (PFS), the model's last fully connected layer is changed to output a numerical value, and the optimizer uses the C-index exponent to calculate the loss function. The cross entropy loss function is: Among them, y represents the patient's actual ORR label, represents the predicted ORR signature; The C-index is: Among them, T represents the survival time of two randomly selected patients i and j, l and They represent the true value and the value predicted by the model respectively. If the survival time of patient i is longer than that of patient j, then C-Index = 1, otherwise C-Index = 0.

2. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 1, characterized in that: The digital pathological section image of the stomach of the gastric cancer patient is an image stained by hematoxylin-eosin staining; The expression of the HER2 gene in the cancer tissue is obtained by immunohistochemistry, including negative information and positive information.

3. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 1, characterized in that: Step S20 includes: Step S21, connecting the cancer region boundary points in the annotation file to obtain the cancer region boundary, and converting the cancer region boundary into an image mask; Step S22, downsampling the slice image and the image mask to a set level; Step S23, extracting regions of the downsampled slice image using the downsampled image mask, and dividing the extracted sub-regions into image blocks of a set pixel size; In step S24 , the ratio of the area of ​​each image block that overlaps with the ROI region to the area of ​​the image block is calculated, and image blocks with a size greater than a set threshold are extracted to obtain multiple image blocks with a set pixel size and containing a set tissue type.

4. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 3, characterized in that: The downsampling method is as follows: Where H, W are the height and width of the slice image and image mask after downsampling, height, width are the height and width of the slice image and image mask before downsampling, and level is the downsampling scale.

5. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 1, characterized in that: The data enhancement normalization method is as follows: Performing horizontal flipping, vertical flipping, and random rotation on each of a plurality of image blocks of a set pixel size and containing a set tissue type to obtain an enhanced image block set; The brightness and contrast of each image block in the enhanced image block set are normalized to obtain an enhanced normalized image block set.

6. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 5, characterized in that: The data division method is as follows: Randomly dividing the enhanced normalized image block set to obtain an image package set consisting of a set number of image blocks; Determine the label of each image block and execute: If an image bag contains at least one image block with a positive label, the image bag is marked as a positive multi-instance bag; otherwise, the image bag is marked as a negative multi-instance bag.

7. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 6, characterized in that: The method for determining the label of each image block is as follows: Extracting a feature map of the image block through a ResNet convolutional neural network; A maximum pooling operation is performed on the feature map, and the probability of the image block belonging to the positive class and the negative class is obtained by calculating the softmax normalization function.

8. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 1, characterized in that: The gastric cancer pathology image classification model is trained by: Step B10, extracting image texture and cell morphology of the image blocks in the training image package, and excluding gastric stromal cells and glandular cells to obtain a preprocessed training image package; Step B20, constructing a ResNet convolutional neural network, and training a single-scale multi-instance learning network for predicting bag labels based on the preprocessed training image bag for each downsampling scale of the digital pathological section image of the stomach; Step B30: Based on the weights of the single-scale multi-instance learning network, multi-instance learning is performed on the multi-scale pre-processed training image package to obtain a gastric cancer pathology image classification model.

9. The gastric cancer pathology image classification method based on HER2 gene detection according to claim 8, characterized in that: The ResNet convolutional neural network includes a set number of residual connection blocks; The residual connection block includes a 3×3 convolutional layer, a batch normalization layer, a ReLU activation function, a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

10. A gastric cancer pathology image classification system based on HER2 gene detection, used to implement the gastric cancer pathology image classification method based on HER2 gene detection according to any one of claims 1 to 9, characterized in that: The gastric cancer pathology image classification system comprises: A data acquisition module is configured to obtain digital pathological slice images of the stomach of a gastric cancer patient; the slice images include expression of the HER2 gene in the cancer tissue; a region extraction and segmentation module configured to perform region extraction and segmentation operations on the slice image to obtain a plurality of image blocks of a set pixel size and containing a set tissue type; a data subpackaging module configured to perform data enhancement normalization and data partitioning on the plurality of image blocks of set pixel size and containing set tissue type to obtain a training image package; A model training module is configured to iteratively train the constructed ResNet convolutional neural network using the training image package to obtain a gastric cancer pathology image classification model; The classification module is configured to classify the digital pathological slice images of the patient's stomach obtained in real time through a gastric cancer pathological image classification model to obtain image classification results.

Images