Gastric adenocarcinoma tumor image segmentation method with intestinal mother cell differentiation and storage medium
Through the tumor semantic segmentation model based on Swin-UNet, combined with self-attention structure and jump connection, the long-distance feature interaction problem in medical images is solved, and high-precision for gastric adenocarcinoma images with enteroblasts are achieved. Segmentation meets the segmentation accuracy requirements for medical applications.
Patent Information
- Application Number
- CN202410982891.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
AI Technical Summary
Existing convolutional neural networks cannot effectively overcome the long-distance feature interaction problem in medical image segmentation, and cannot meet the strict requirements of medical applications for segmentation accuracy.
Using the tumor semantic segmentation model based on Swin-UNet, the WSI images obtained and marked, and the preset sized case image blocks were obtained by sliding windows, and the training set, verification set and test set were divided according to the preset proportions to convolutional nerves. The network model is trained, and the self-attention structure and jump connection are used to achieve upsampling and segmentation at the feature level.
Accurate identification and segmentation of tumor areas in gastric adenocarcinoma images with enteroblasts were achieved, and pixel-level annotation was made to solve the problem of long-distance dependence, and segmentation accuracy and generalization ability were improved.
Smart Images

Figure CN118918328A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image segmentation, and in particular to a method and storage medium for segmenting a gastric adenocarcinoma tumor image with enteroblast differentiation based on a Swin-UNet tumor semantic segmentation model. Background Art
[0002] Gastric cancer is highly invasive and has a high mortality rate. The histomorphology of gastric cancer is complex and heterogeneous, especially gastric adenocarcinoma with enteroblastic differentiation. Gastric adenocarcinoma with enteroblastic differentiation is a subtype of gastric adenocarcinoma that expresses at least one primitive phenotypic marker (AFP, GPC3 and SALL4), has distinct pathological characteristics and highly invasive biological behavior, has a poor overall prognosis, and is prone to liver metastasis. Accurately identifying this type of potentially invasive gastric cancer is very important for the precise treatment and personalized postoperative management of patients. Artificial intelligence technology based on deep learning has been used for pathological diagnosis, tumor grading, immunohistochemical interpretation, molecular typing, etc. of cancer whole slide images (WSI), with good diagnostic sensitivity, reliability and consistency. Related studies, such as the potential for predicting lymph node metastasis of early colorectal cancer, predicting molecular typing of gastric cancer, predicting hormone receptor status of breast cancer, predicting tumor microsatellite status, etc., deep learning models have shown very high accuracy. At the same time, in the comparison study between AI and pathologists, it is shown that artificial intelligence is more accurate than humans. Compared with other medical images, WSI contains precise pixel-level image information and ultra-high resolution, which requires image segmentation technology with higher segmentation accuracy and generalization ability. Existing medical image segmentation methods mainly rely on convolutional neural networks, that is, using a symmetrical encoder-decoder structure and jump connections to obtain segmentation result images. In this method, multi-layer downsampling can achieve a larger image receptive field. Although CNN has achieved excellent performance in the field of medical image segmentation, it cannot overcome the problem of long-distance feature interaction and cannot fully meet the strict requirements of medical applications for segmentation accuracy. Summary of the invention
[0003] In view of the above problems, the present application provides a method and storage medium for gastric adenocarcinoma tumor image segmentation with enteroblastic differentiation based on the Swin-UNet tumor semantic segmentation model to solve the problem that existing convolutional neural networks cannot overcome the long-distance feature interaction problem in medical images and cannot fully meet the strict requirements of medical applications for segmentation accuracy.
[0004] To achieve the above object, the inventors provide a method for segmenting gastric adenocarcinoma tumor images with enteroblast differentiation based on the Swin-UNet tumor semantic segmentation model, comprising the following steps:
[0005] Obtain a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation;
[0006] Obtaining a region of interest drawn on the tumor region in the WSI image, as well as pixel-level annotations, wherein the annotations include the category to which they belong and coordinate information of each category;
[0007] Combining WSI images and annotations, case image blocks of preset sizes are obtained through sliding window traversal;
[0008] The WSI images are divided into training set, validation set and test set according to the preset ratio;
[0009] The convolutional neural network model Swin-Unet was trained according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model.
[0010] In some embodiments, acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps:
[0011] Several paraffin tissue blocks were obtained from confirmed cases of gastric adenocarcinoma with enteroblastic differentiation;
[0012] After the paraffin tissue blocks were sliced, standardized HE staining was performed to make tissue sections;
[0013] The tissue sections were scanned by a slice scanner and converted into WSI images.
[0014] In some embodiments, acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps:
[0015] WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation were obtained from the TCGA database.
[0016] In some embodiments, the annotation is in XML format;
[0017] The method of combining the WSI image and the annotation to obtain a case image block of a preset size by sliding window traversal specifically includes the following steps:
[0018] Convert the annotations in XML format to binary mask images in TIF format;
[0019] Combining the WSI image and the binary mask image, the case image block of a preset size is obtained by sliding window traversal.
[0020] In some embodiments, the following steps are also included:
[0021] During the training process of the convolutional neural network model Swin-Unet, the training of the convolutional neural network model Swin-Unet is optimized by the AdamW optimizer.
[0022] Another technical solution is provided, a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are performed:
[0023] Obtain a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation;
[0024] Obtaining a region of interest drawn on the tumor region in the WSI image, as well as pixel-level annotations, wherein the annotations include the category to which they belong and coordinate information of each category;
[0025] Combining WSI images and annotations, case image blocks of preset sizes are obtained through sliding window traversal;
[0026] The WSI images are divided into training set, validation set and test set according to the preset ratio;
[0027] The convolutional neural network model Swin-Unet was trained according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model.
[0028] In some embodiments, acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps:
[0029] Several paraffin tissue blocks were obtained from confirmed cases of gastric adenocarcinoma with enteroblastic differentiation;
[0030] After the paraffin tissue blocks were sliced, standardized HE staining was performed to make tissue sections;
[0031] The tissue sections were scanned by a slice scanner and converted into WSI images.
[0032] In some embodiments, acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps:
[0033] WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation were obtained from the TCGA database.
[0034] In some embodiments, the annotation is in XML format;
[0035] The method of combining the WSI image and the annotation to obtain a case image block of a preset size by sliding window traversal specifically includes the following steps:
[0036] Convert the annotations in XML format to binary mask images in TIF format;
[0037] Combining the WSI image and the binary mask image, the case image block of a preset size is obtained by sliding window traversal.
[0038] In some embodiments, the following steps are also included:
[0039] During the training process of the convolutional neural network model Swin-Unet, the training of the convolutional neural network model Swin-Unet is optimized by the AdamW optimizer.
[0040] Different from the prior art, the above technical scheme obtains a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation, and then obtains the region of interest drawn on the tumor area in the WSI image, as well as pixel-level annotations, wherein the annotations include the category to which it belongs and the coordinate information of each category; then, the WSI image and the annotations are combined to obtain case image blocks of a preset size by sliding window traversal, and then the WSI image is divided according to a preset ratio to obtain a training set, a validation set and a test set; then, the convolutional neural network model Swin-Unet is trained according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model, and the tumor semantic segmentation model can be used to recognize and segment the tumor area in the gastric adenocarcinoma image with enteroblastic differentiation. The convolutional neural network model Swin-Unet is used to train the tumor semantic segmentation model. Since the convolutional neural network model Swin-Unet first divides the input image into non-overlapping small image blocks, each image block constitutes a network input, and obtains the features of other image blocks in the same image as keys and values, performs attention calculations, and obtains the attention response of the image block to all image areas. Then, the feature expansion layer is used to implement feature level upsampling, and the primary features generated by the encoder are obtained by skipping the sampling area. Finally, the feature representation with the same resolution as the original image is obtained as the final segmentation result. This self-attention structure can better solve the problem of long-distance feature dependency, and realizes pixel-level segmentation of the image without using interpolation and convolution. It makes full use of the pixel-level annotation of the image, gives play to the advantages of the self-attention structure in obtaining global features, and uses skip connections to reduce information loss during data transmission.
[0041] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and shall not be considered as limitations of the present application.
[0043] In the drawings of the specification:
[0044] Figure 1 A schematic diagram of a flow chart of a method for segmenting a gastric adenocarcinoma tumor image with enteroblast differentiation based on the Swin-UNet tumor semantic segmentation model described in a specific implementation method;
[0045] Figure 2 A schematic diagram of an example of case image block data described in a specific implementation manner;
[0046] Figure 3 is a schematic diagram of the tumor semantic segmentation model described in the specific implementation method;
[0047] Figure 4 is a schematic diagram of the segmentation result described in the specific implementation manner;
[0048] Figure 5 A schematic diagram of the structure of the storage medium described in the specific implementation method.
[0049] The reference numerals in the above drawings are described as follows:
[0050] 510. Storage medium,
[0051] 520. Processor. DETAILED DESCRIPTION
[0052] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.
[0053] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.
[0054] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.
[0055] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.
[0056] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.
[0057] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.
[0058] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.
[0059] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.
[0060] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.
[0061] See also Figure 1 This embodiment provides a method for segmenting a gastric adenocarcinoma tumor image with enteroblast differentiation based on a Swin-UNet tumor semantic segmentation model, comprising the following steps:
[0062] Step S110: acquiring a certain number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation;
[0063] Step S120: the annotation includes the category to which it belongs and the region of interest drawn for the tumor region in each acquired WSI image, and the coordinate information of the pixel-level annotation category;
[0064] Step S130: combining the WSI image and the annotations, and obtaining a case image block of a preset size by means of a sliding window traversal;
[0065] Step S140: Divide the WSI image into a training set, a validation set and a test set according to a preset ratio;
[0066] Step S150: training the convolutional neural network model Swin-Unet according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model.
[0067] By obtaining a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation, and then obtaining the region of interest drawn on the tumor area in the WSI image, and the pixel-level annotation, wherein the annotation includes the category to which it belongs and the coordinate information of each category; then combining the WSI image and the annotation, obtaining the case image block of a preset size by sliding window traversal, and then dividing the WSI image according to a preset ratio to obtain a training set, a validation set and a test set; then training the convolutional neural network model Swin-Unet according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model, through which the tumor semantic segmentation model can be used to recognize and segment the tumor area in the gastric adenocarcinoma image with enteroblastic differentiation. The convolutional neural network model Swin-Unet is used to train the tumor semantic segmentation model. Since the convolutional neural network model Swin-Unet first divides the input image into non-overlapping small image blocks, each image block constitutes a network input, and obtains the image block features calculated by different weight matrices as keys and values, performs attention calculations, and obtains the attention response of the image block to all image areas. Then, the feature expansion layer is used to implement feature level upsampling, and the primary features generated by the encoder are obtained by skipping the sampling area. Finally, the feature representation with the same resolution as the original image is obtained as the final segmentation result. This self-attention structure can better solve the problem of long-distance feature dependency, and realizes pixel-level segmentation of the image without using interpolation and convolution. It makes full use of the pixel-level annotation of the image, gives play to the advantages of the self-attention structure in obtaining global features, and uses skip connections to reduce information loss during data transmission.
[0068] In some embodiments, acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps:
[0069] Several paraffin tissue blocks were obtained from confirmed cases of gastric adenocarcinoma with enteroblastic differentiation;
[0070] After the paraffin tissue blocks were sliced, standardized HE staining was performed to make tissue sections;
[0071] The tissue sections were scanned by a slice scanner and converted into WSI images.
[0072] WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation can be obtained through local cases, wherein confirmed cases of gastric adenocarcinoma with enteroblastic differentiation that have been surgically resected without preoperative neoadjuvant treatment are collected. After reviewing the pathological sections of all cases, a corresponding number of cases are selected for inclusion in the study, such as collecting confirmed cases of gastric adenocarcinoma with enteroblastic differentiation that have been surgically resected without preoperative neoadjuvant treatment in the Department of Pathology of Fujian Provincial Hospital from January 2014 to January 2023. After reviewing the pathological sections of all cases, 131 WSIs were finally included in the study. A preset number of representative paraffin tissue blocks of the selected cases were selected, sliced at 4 μm, and standardized HE staining was made into tissue sections, and then scanned 40 times by a slice scanner and converted into WSI images in SVS format. In other embodiments, the WSI images are randomly assigned, and the pathologist draws the region of interest around the tumor area on the WSI image, and then the slices are annotated at the pixel level; then a data set for training the segmentation of gastric adenocarcinoma with enteroblastic differentiation is produced by preprocessing, which is used to train the semantic segmentation model of this tumor.
[0073] In some embodiments, acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps:
[0074] WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation were obtained from the TCGA database.
[0075] The cBioPortal database (http: / / www.cbioportal.org) was used to retrieve the mRNA expression data of gastric cancer (TCGA, Nature 2014) dataset. Cases were initially screened based on the mRNA expression levels of the original phenotypic genes (SALL4, GPC3, and AFP). The screening strategy was to use the Onco Query Language to query samples with specific expression values. For mRNA (RNASeq V2 RSEM) expression, z-score thresholds = ±1 were used. At the same time, unlabeled gastric cancer WSI files were downloaded through the Genomic Data Commons portal (https: / / portal.gdc.cancer.gov / ). After reviewing the TCGA case WSI and combining it with the mRNA expression results, confirmed cases of gastric adenocarcinoma with enteroblastic differentiation were finally confirmed. After screening the WSI display quality and pathological preparation type, the qualified WSI images were selected and finally included in the external validation analysis of the model. The WSI images downloaded from TCGA are all in SVS format, providing four levels of resolution sets, which are scan sets of tissue pathology sections at 5x, 10x, 20x, and 40x views of the WSI scanner.
[0076] In some embodiments, the annotation is in XML format;
[0077] The method of combining the WSI image and the annotation to obtain a case image block of a preset size by sliding window traversal specifically includes the following steps:
[0078] The annotations in XML format are converted into binary mask images in TIF format; the binary mask image is used to mark the target area of interest in the tissue area, where the area with a value of 1 indicates a tissue area with pathological significance, where pathologists perform subtype annotation and analysis, and the area with a value of 0 indicates an area without typical characteristics, where pathologists do not perform subtype annotation.
[0079] Combining the WSI image and the binary mask image, a pathological image block of a preset size is obtained by sliding window traversal.
[0080] By saving the annotations in an XML format file and converting the XML format annotations into a binary mask image in a TIF format, the binary mask image generated by combining the WSI image and the XML format annotations is sampled using a sliding window traversal method to obtain a pathological image block with a size of 768*768 at a 20x magnification as the network input sample.
[0081] In some embodiments, the following steps are also included:
[0082] During the training process of the convolutional neural network model Swin-Unet, the training of the convolutional neural network model Swin-Unet is optimized by the AdamW optimizer.
[0083] The established tumor semantic segmentation model of gastric adenocarcinoma with enteroblastic differentiation was trained for a total of 80 rounds, and the AdamW optimizer was used to optimize the training of the model and then adjust the parameters of each layer of the neural network.
[0084] In order to improve the training speed, the pre-trained weights on ImageNet are used to initialize the model parameters. For the setting of training hyperparameters, the batch size is set to 8 and the initial learning rate (lr) is set to 0.1; in order to prevent the model from overfitting, the Dropout method is used to reduce the overfitting phenomenon, and the Dropout biological factor is set to 0.3.
[0085] In some embodiments, a method for segmenting gastric adenocarcinoma tumor images with enteroblastic differentiation based on the Swin-UNet tumor semantic segmentation model includes case selection and data set construction, data set production, and establishment of a tumor semantic segmentation model during the actual training process of the tumor semantic segmentation model.
[0086] In case selection and dataset construction, training datasets were constructed by using local cases and TCGA cases. Among them, the local dataset was collected by collecting confirmed cases of gastric adenocarcinoma with enteroblastic differentiation who underwent surgical resection without preoperative neoadjuvant treatment in the Department of Pathology of Fujian Provincial Hospital from January 2014 to January 2023. After reviewing the pathological sections of all cases, 131 WSIs were finally included in the study. 131 representative paraffin tissue blocks of the selected cases were selected, sliced at 4μm, and made into tissue sections with standardized HE staining. And scanned by a slice scanner 40 times and converted into WSI files in SVS format. After the WSI files were randomly assigned, the pathologists of Fujian Provincial Hospital drew the region of interest around the tumor area and annotated the slices at the pixel level. Then, a dataset for training tumor segmentation of gastric adenocarcinoma with enteroblastic differentiation was produced by preprocessing, which was used to train this tumor semantic segmentation model. TCGA cases were obtained by using cBioPortal (http: / / www.cbioportal.org) to retrieve the mRNA expression data of gastric cancer (TCGA, Nature 2014) dataset. Cases were initially screened based on the mRNA expression levels of the original phenotypic genes (SALL4, GPC3, and AFP). The screening strategy was to use Onco QueryLanguage to query samples with specific expression values. For mRNA (RNA Seq V2 RSEM) expression, z-scorethresholds=±1 was used. At the same time, unannotated gastric cancer WSI files were downloaded through the Genomic Data Commons portal (https: / / portal.gdc.cancer.gov / ). After reviewing the TCGA case WSI and combining it with the mRNA expression results, 31 cases of gastric adenocarcinoma with enteroblastic differentiation were finally diagnosed. After screening the WSI display quality and pathological preparation type, a total of 25 WSIs of 25 cases of gastric adenocarcinoma with enteroblastic differentiation were finally included in the external validation analysis of the model. The WSI images downloaded from TCGA are all in SVS format, providing four levels of resolution sets, which are scan sets of tissue pathology sections at 5x, 10x, 20x, and 40x views of the WSI scanner.
[0087] The dataset production includes dataset annotation and transformation as well as dataset preprocessing. The dataset annotation and transformation process is as follows: WSI images of gastric adenocarcinoma cases with enteroblastic differentiation confirmed by the Pathology Department of Fujian Provincial Hospital and WSI of gastric adenocarcinoma cases with enteroblastic differentiation downloaded from the TCGA public database are annotated by pathologists of the Pathology Department of Fujian Provincial Hospital using the Motic Gallery data platform (http: / / med.motic.com). All annotators have undergone WSI annotation training in advance, and the annotators have 3-10 years of work experience. After all WSI files are randomly assigned, the annotators first draw the region of interest in the tumor area, and then annotate the slices at the pixel level. All annotations are finally reviewed and modified by pathologists with 10 and 20 years of work experience in the specialty field of the Pathology Department of Fujian Provincial Hospital. The annotations created by the pathologists serve as the final reference standard for tumor detection. Finally, the annotations of the dataset are saved in the form of xml files. The xml annotation file contains relevant information such as the category to which it belongs and the coordinates of each category. The annotation file in XML format is converted into a binary mask image in TIF format for use in the training of the semantic segmentation model.
[0088] The preprocessing process of the dataset is as follows: for the 131 annotated WSI data, combine the WSI image and the xml annotation file to generate a binary mask image. The above WSI contains a total of 331 marked ROI areas, and the sliding window traversal method is used to sample the pathological image blocks of size 768*768 at 20 times magnification as network input samples. In order to verify the effectiveness of the algorithm, the dataset is divided by case division, and the dataset is divided into training set, validation set and test set at a ratio of 70%, 10% and 20%, respectively, containing 91, 13 and 27 full-slice images. In the end, the training set contains 55,948 image blocks, the test set contains 14,650 image blocks, and the validation set includes 6,795 image blocks. The distribution of the dataset is shown in the figure below:
[0089]
[0090] Examples of pathological image block data are as follows: Figure 2 As shown, (a) is the ROI area, (b) is the ROI annotated image, (c) is the original image of the image block, (d) is the image block annotated image, and (e) is the image block MASK.
[0091] The semantic segmentation model of gastric adenocarcinoma with enteroblastic differentiation was trained for 80 rounds. The AdamW optimizer was used to optimize the training of the model and then adjust the parameters of each layer of the neural network. In order to increase the training speed, the weights pre-trained on ImageNet were used to initialize the model parameters. For the setting of training hyperparameters, the batch size was set to 8 and the initial learning rate (lr) was set to 0.1; in order to prevent the model from overfitting, the Dropout method was used to reduce the overfitting phenomenon, and the Dropout biological factor was set to 0.3.
[0092] The establishment of the tumor semantic segmentation model, because the data used has accurate pixel-level labels, the convolutional neural network model Swin-Unet is used for training. The network first divides the input image into small non-overlapping image blocks, and each image block constitutes an input token of the network. Specifically, the image block is used as the query (Query, Q) input, and the features of other image blocks in the same image are obtained as the key (Key, K) and value (Value, V). Attention calculation is performed to obtain the attention response of the image block to all image areas. Then, the designed feature expansion layer is used to achieve upsampling at the feature level, and the primary features generated by the encoder are obtained by skip connection. Finally, the feature representation with the same resolution as the original image is obtained as the final segmentation result. The self-attention structure can better solve the problem of long-distance dependency of features and realize pixel-level segmentation of the image without using interpolation and convolution. This method makes full use of the pixel-level annotation of the image, gives full play to the advantages of the self-attention module in obtaining global features, and uses skip connection to reduce information loss during data transmission. During the training process, Cross entrop, Dice loss and Lovasz loss functions are used for training respectively.
[0093] The overall structure of the established Swin-UNet tumor semantic segmentation model is as follows: Figure 3As shown in the figure, it is mainly composed of an encoder, a decoder and a jump connection, and the basic unit is a Swin Transformer 2 module. Both the encoder and the decoder are divided into four stages: the first stage of the encoder consists of a Patch embedding layer and two Swin Transformer 2 modules, and the Patch embedding layer is replaced by a Patch fusion layer in other stages. The role of the Patch fusion layer is to downsample the image and then splice it in the depth direction, which is similar to the pooling layer operation in CNN. In the decoder, except for the third stage, each stage consists of two Swin Transformer 2 modules and one Patch expansion layer. The Patch expansion layer is used for upsampling operations to reshape the feature maps of adjacent dimensions into large feature maps with 2 times the resolution upsampling. Among them, the third stage of the encoder and decoder has 6 Swin Transformer 2 modules. The multi-scale features from the encoder are fused through jump connections between each stage of the encoding and decoding parts to supplement the spatial information loss caused by downsampling. Finally, the Patch expansion layer performs 4x upsampling to restore the resolution of the feature map to the input resolution, and then a linear projection layer is applied to these upsampled features to output pixel-level segmentation predictions.
[0094] The main improvements of the Swin Transformer 2 module are: placing the LN layer after the attention mechanism layer and the feedforward neural network layer so that the output value of each layer is stable; replacing the original dot product attention mechanism with a scaled cosine attention mechanism. Since the calculation of self-attention is independent of the input amplitude, a softer attention weight can be generated; replacing the previous parameter normalization method with a continuous relative position bias method in logarithmic space. By applying a small meta-network to the position coordinates in logarithmic space, position biases in any coordinate range can be generated, achieving smooth migration of pre-trained models between data sets of different resolutions. The Swin Transformer 2 block is also constructed based on a moving window. Figure 3 Two consecutive Swin Transformer 2 blocks are shown in Figure 2. Each Swin Transformer2 module consists of an LN layer, a (S)W-MSA module, a residual connection, and a two-layer MLP with a nonlinear activation function.
[0095] Different loss functions are used to train the established Swin UNet model. Different loss function combinations are also used in the experiment to train the Swin UNet model. The effects of different loss functions and their combinations on the model segmentation performance are compared. The experimental results show that the network segmentation performance is significant under the combination of Dice loss and CE loss. The experimental results are shown in the following table:
[0096]
[0097] The cross entropy loss function (Cross entropy) and the Dice loss function (Diceloss) are used in the network training process. The Adam optimizer is used to train 80 rounds to achieve the best performance. The dice coefficient and Hausdorff_95 of the image segmentation test are shown in the following table:
[0098]
[0099] The segmentation results are as follows Figure 4 As shown, Figure 4 In the figure, (a) is the original image, (b) is the labeled image, and (c) is the predicted image heat map.
[0100] This study innovatively used the convolutional neural network model Swin-Unet based on the self-attention mechanism (Transformer) for image segmentation training for the first time, confirming that the deep learning model can identify gastric adenocarcinoma with enteroblastic differentiation, and has high performance. The above shows that the deep learning model can be used for auxiliary diagnosis of gastric adenocarcinoma with enteroblastic differentiation and is a promising biomarker for gastric adenocarcinoma with enteroblastic differentiation, but further prospective clinical trials are needed to confirm its effectiveness.
[0101] In recent years, most pathological image segmentation uses the U-Net model based on convolutional neural networks, that is, a symmetrical Encoder-Decoder structure and a jump connection method are used to obtain the segmentation result map. In this method, multi-layer downsampling can achieve a larger image receptive field, but the current segmentation methods based on convolutional networks cannot overcome the problem of long-distance feature interaction. This experiment uses the convolutional neural network model Swin-Unet for training. The network first divides the input image into small non-overlapping image blocks. Each image block constitutes an input token of the network. Specifically, the image block is used as the query (Query, Q) input, and the features of other image blocks in the same image are obtained as the key (Key, K) and value (Value, V). Attention calculation is performed to obtain the attention response of the image block to all image areas. Then, the designed feature expansion layer is used to achieve upsampling at the feature level, and the primary features generated by the encoder are obtained by jump connection. Finally, the feature representation with the same resolution as the original image is obtained as the final segmentation result. The self-attention structure can better solve the problem of long-distance dependency of features and realize pixel-level segmentation of images without using interpolation and convolution. This method makes full use of the pixel-level annotation of the image, takes advantage of the self-attention module in obtaining global features, and uses skip connections to reduce information loss during data transmission.
[0102] See also Figure 5 In another embodiment, a storage medium 510 stores a computer program, and the computer program is executed by a processor 520 to perform the following steps:
[0103] Obtain a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation;
[0104] Obtaining a region of interest drawn on the tumor region in the WSI image, as well as pixel-level annotations, wherein the annotations include the category to which they belong and coordinate information of each category;
[0105] Combining WSI images and annotations, case image blocks of preset sizes are obtained through sliding window traversal;
[0106] The WSI images are divided into training set, validation set and test set according to the preset ratio;
[0107] The convolutional neural network model Swin-Unet was trained according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model.
[0108] WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation can be obtained through local cases, wherein confirmed cases of gastric adenocarcinoma with enteroblastic differentiation that have been surgically resected without preoperative neoadjuvant treatment are collected. After reviewing the pathological sections of all cases, a corresponding number of cases are selected for inclusion in the study, such as collecting confirmed cases of gastric adenocarcinoma with enteroblastic differentiation that have been surgically resected without preoperative neoadjuvant treatment in the Department of Pathology of Fujian Provincial Hospital from January 2014 to January 2023. After reviewing the pathological sections of all cases, 131 WSIs were finally included in the study. A preset number of representative paraffin tissue blocks of the selected cases were selected, sliced at 4um, and standardized HE staining was made into tissue sections, and then scanned 40 times by a slice scanner and converted into WSI images in SVS format. In other embodiments, the WSI images are randomly assigned, and the pathologist draws the region of interest around the tumor area on the WSI image, and then the slices are annotated at the pixel level; then a data set for training the segmentation of gastric adenocarcinoma tumors with enteroblastic differentiation is produced by preprocessing, which is used to train the semantic segmentation model of this tumor.
[0109] In some embodiments, acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps:
[0110] WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation were obtained from the TCGA database.
[0111] The cBioPortal database (http: / / www.cbioportal.org) was used to retrieve the mRNA expression data of gastric cancer (TCGA, Nature 2014) dataset. Cases were initially screened based on the mRNA expression levels of the original phenotypic genes (SALL4, GPC3, and AFP). The screening strategy was: use Onco Query Language to query samples with specific expression values. For mRNA (RNASeq V2 RSEM) expression, z-score thresholds = ±1 were used. At the same time, unlabeled gastric cancer WSI files were downloaded from the Genomic Data Commons portal (https: / / portal.gdc.cancer.gov / ). After reviewing the TCGA case WSI and combining it with the mRNA expression results, the confirmed cases of GAED were finally confirmed. After screening the WSI display quality and pathological preparation type, the qualified WSI images were selected and finally included in the external validation analysis of the model. The WSI images downloaded from TCGA are all in SVS format, providing four layers of resolution sets, which are the scan sets of tissue pathology sections under the 5x, 10x, 20x, and 40x views of the WSI scanner.
[0112] In some embodiments, the annotation is in XML format;
[0113] The method of combining the WSI image and the annotation to obtain a case image block of a preset size by sliding window traversal specifically includes the following steps:
[0114] The annotations in XML format are converted into binary mask images in TIF format; the binary mask image is used to mark the target area of interest in the tissue area, where the area with a value of 1 indicates a tissue area with pathological significance, where pathologists perform subtype annotation and analysis, and the area with a value of 0 indicates an area without typical characteristics, where pathologists do not perform subtype annotation.
[0115] Combining the WSI image and the binary mask image, the case image block of a preset size is obtained by sliding window traversal.
[0116] By saving the annotations in an XML format file and converting the XML format annotations into a binary mask image in a TIF format, the binary mask image generated by combining the WSI image and the XML format annotations is sampled using a sliding window traversal method to obtain a pathological image block with a size of 768*768 at a 20x magnification as the network input sample.
[0117] In some embodiments, the following steps are also included:
[0118] During the training process of the convolutional neural network model Swin-Unet, the training of the convolutional neural network model Swin-Unet is optimized by the AdamW optimizer.
[0119] The established tumor semantic segmentation model of gastric adenocarcinoma with enteroblastic differentiation was trained for a total of 80 rounds, and the AdamW optimizer was used to optimize the training of the model and then adjust the parameters of each layer of the neural network.
[0120] In order to improve the training speed, the pre-trained weights on ImageNet are used to initialize the model parameters. For the setting of training hyperparameters, the batch size is set to 8 and the initial learning rate (lr) is set to 0.1; in order to prevent the model from overfitting, the Dropout method is used to reduce the overfitting phenomenon, and the Dropout biological factor is set to 0.3.
[0121] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.
Claims
1. A method for segmenting gastric adenocarcinoma tumor images with enteroblastic differentiation based on the Swin-UNet tumor semantic segmentation model, characterized in that: The following steps are involved: Acquire a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation; Obtaining a region of interest drawn on the tumor region in the WSI image, as well as pixel-level annotations, wherein the annotations include the category to which they belong and coordinate information of each category; Combining the WSI image and the annotations, the pathological image blocks of the preset size are obtained by sliding window traversal; The WSI images are divided into training set, validation set and test set according to the preset ratio; The convolutional neural network model Swin-Unet was trained according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model.
2. The method for segmenting gastric adenocarcinoma tumor images with enteroblastic differentiation based on the Swin-UNet tumor semantic segmentation model according to claim 1, characterized in that: The method of acquiring a plurality of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps: Several paraffin tissue blocks were obtained from confirmed cases of gastric adenocarcinoma with enteroblastic differentiation; After the paraffin tissue blocks were sliced, standardized HE staining was performed to make tissue sections; The tissue sections were scanned by a slice scanner and converted into WSI images.
3. The method for segmenting gastric adenocarcinoma tumor images with enteroblastic differentiation based on the Swin-UNet tumor semantic segmentation model according to claim 1, characterized in that: The method of acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps: WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation were obtained from the TCGA Gastric Cancer Database.
4. The method for segmenting gastric adenocarcinoma tumor images with enteroblastic differentiation based on the Swin-UNet tumor semantic segmentation model according to claim 1, characterized in that: The annotation is in XML format; The method of combining the WSI image and the annotation to obtain a case image block of a preset size by sliding window traversal specifically includes the following steps: Convert the annotations in XML format to binary mask images in TIF format; Combining the WSI image and the binary mask image, a pathological image block of a preset size is obtained by sliding window traversal.
5. The method for segmenting gastric adenocarcinoma tumor images with enteroblastic differentiation based on the Swin-UNet tumor semantic segmentation model according to claim 1, characterized in that: The following steps are also included: During the training process of the convolutional neural network model Swin-Unet, the training of the convolutional neural network model Swin-Unet is optimized by the AdamW optimizer.
6. A storage medium storing a computer program, characterized in that: The computer program is executed by a processor to perform the following steps: Obtain a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation; Obtaining a region of interest drawn on the tumor region in the WSI image, as well as pixel-level annotations, wherein the annotations include the category to which they belong and coordinate information of each category; Combining WSI images and annotations, case image blocks of preset sizes are obtained through sliding window traversal; The WSI images are divided into training set, validation set and test set according to the preset ratio; The convolutional neural network model Swin-Unet was trained according to the divided training set, validation set and test set to obtain the final tumor semantic segmentation model.
7. The storage medium according to claim 6, characterized in that The method of acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps: Several paraffin tissue blocks were obtained from confirmed cases of gastric adenocarcinoma with enteroblastic differentiation; After the paraffin tissue blocks were sliced, standardized HE staining was performed to make tissue sections; The tissue sections were scanned by a slice scanner and converted into WSI images.
8. The storage medium according to claim 6, characterized in that The method of acquiring a number of WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation comprises the following steps: WSI images of confirmed cases of gastric adenocarcinoma with enteroblastic differentiation were obtained from the TCGA database.
9. The storage medium according to claim 6, characterized in that: The annotation is in XML format; The method of combining the WSI image and the annotation to obtain a case image block of a preset size by sliding window traversal specifically includes the following steps: Convert the annotations in XML format to binary mask images in TIF format; Combining the WSI image and the binary mask image, the case image block of a preset size is obtained by sliding window traversal.
10. The storage medium according to claim 6, characterized in that The following steps are also included: During the training process of the convolutional neural network model Swin-Unet, the training of the convolutional neural network model Swin-Unet is optimized by the AdamW optimizer.
Citation Information
Cited By
Intelligent identification method and system for liver single-cell necrosis based on deep learning, and storage medium
CN121563996A
Intelligent recognition method and system for liver single cell necrosis based on deep learning, and storage medium
CN121563996B
