Gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid

Abstract

The application discloses a gastric cancer pathological image segmentation method combining adaptive attention and feature pyramids, realizes automatic segmentation of the gastric cancer pathological image, and can be used for clinical auxiliary diagnosis of gastric cancer. The method firstly acquires a gastric cancer pathological image and a doctor's annotation result, and expands data through an image enhancement method; then the pathological image is input into a semantic segmentation network combining adaptive attention and feature pyramids, the cancerous area is accurately positioned, and the lesion edge is finely segmented; similarity loss and cross-entropy loss are adopted to relieve the problem of uneven samples in the data set; and a repeated training strategy is adopted to make the model adaptively converge to the optimum. The application can effectively reduce the misjudgment of the cancerous area, accurately capture the edge information of the cancerous area, realize smooth and accurate segmentation of the lesion edge, and provide reliable support for subsequent treatment of patients.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and computer vision, in particular to a gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid. BACKGROUND

[0002] Gastric cancer is one of the malignant tumors with the highest mortality rate in China. The diagnosis of gastric cancer needs to be differentiated from false lymphoma, gastric mucosa prolapse and other gastric cancer. Early diagnosis of gastric cancer is crucial to improve the prognosis of patients.

[0003] The diagnosis of gastric cancer requires the collection of pathological sections of gastric tissue, which relies on the experience of doctors to determine whether canceration occurs. The pathological manifestations of canceration area are diverse, and the judgment of canceration by doctors is subjective. At the same time, the manual lesion segmentation method is time-consuming and laborious. The treatment method of gastric cancer in clinic is usually surgical resection, and accurate judgment of the scope of canceration can significantly reduce the pain of patients.

[0004] At present, the semantic segmentation network for gastric cancer pathological image segmentation is an important research direction. The semantic segmentation network obtains specific network parameters through data-driven training to fit the process of doctor reading film decision, so that the network has the same analysis and learning ability as people. In the semantic segmentation network, the PSPNet based on pyramid structure and the DeepLab network based on atrous convolution show good effect in natural segmentation and street scene segmentation, but they are far less widely used than UNet in medical image segmentation. First, medical image data is more difficult to obtain, which is difficult to support the training of complex networks, and second, the semantic information of medical images is simpler and does not need too much down-sampling structure to construct high-dimensional semantics, but the network needs to pay more attention to the understanding of low-dimensional texture information. UNet structure is simple, and the operation space is large according to the task specificity, which can avoid the loss of low-dimensional information caused by multi-layer down-sampling and atrous convolution. The gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid is proposed, which can realize accurate positioning of canceration area and accurate segmentation of canceration area edge, and has certain clinical value. SUMMARY

[0005] In view of this, the purpose of the present application is to propose a gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid. The method first constructs a gastric cancer dataset and expands the data through an image enhancement method. In the training stage, the pathological image is input into the semantic segmentation network combining adaptive attention and feature pyramid, and the cancerous area is segmented. In the training process, the similarity loss and cross-entropy loss are combined to alleviate the problem of uneven sample distribution in the dataset. The repeated training strategy is adopted, and after one training is completed, the model parameters are reloaded for retraining, so that the model adaptively converges to the optimal. The present application can effectively reduce the misjudgment of the cancerous area and accurately capture the edge information of the cancerous area, realize the smooth and accurate segmentation of the lesion edge, provide reliable support for the subsequent treatment of patients, and has clinical practical value.

[0006] The present application adopts the following scheme:

[0007] A gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid, comprising the following steps:

[0008] Step S1: obtain gastric cancer pathological images and cancerous area medical annotation results, and construct a gastric cancer dataset;

[0009] Step S2: expand the gastric cancer dataset using an image enhancement method;

[0010] Step S3: use the expanded gastric cancer dataset to train a segmentation model combining adaptive attention and feature pyramid, segment the cancerous area in the gastric cancer dataset by the segmentation model, and obtain the preliminary trained model parameters and the cancerous area segmentation result corresponding to each gastric cancer pathological image;

[0011] Step S4: repeat step S3, and use the repeated training strategy to obtain the final parameters of the segmentation model;

[0012] Step S5, use the trained segmentation model to process the gastric cancer pathological image to be segmented to obtain the cancerous area segmentation result.

[0013] Further, step S1 specifically comprises the following steps:

[0014] Step S11: obtain gastric cancer pathological images;

[0015] Step S12: experts draw the cancerous area in the pathological image according to experience;

[0016] Step S13: generate a cancerous area annotation image according to the expert drawing result, and the mask image only has white and black colors, the white area represents the cancerous area, and the black area represents the normal tissue area;

[0017] Step S14: Each gastric cancer pathological image and the corresponding cancerous region annotation image constitute an image-label pair, and all image-label pairs constitute a gastric cancer dataset.

[0018] Further, step S2 specifically comprises the following steps:

[0019] Step S21: Perform horizontal flip on part of the image-label pairs in the dataset;

[0020] Step S22: Perform vertical flip on part of the image-label pairs in the dataset;

[0021] Step S23: Randomly add Gaussian noise to all images in the dataset, and randomly perform tone transformation.

[0022] Step S24: Randomly crop all image-label pairs in the dataset.

[0023] Further, the segmentation model combining adaptive attention and feature pyramid in step S3 comprises:

[0024] A model encoder for characterizing the characteristics of the cancerous region in the pathological image to obtain a pathological image feature map;

[0025] A model decoder for reconstructing a cancerous region prediction mask image according to the pathological image feature map obtained by the model encoder;

[0026] A connection module for connecting corresponding layers of the model encoder and the model decoder to realize information interaction between the model encoder and the model decoder.

[0027] Further, the feature encoder module comprises a plurality of encoding layers composed of adaptive attention networks and residual networks, and a feature pyramid network, the input of the first encoding layer being the gastric cancer pathological image, the input of the subsequent encoding layer being the output of the previous encoding layer; the output of the last encoding layer is input into the model decoder after passing through the feature pyramid network.

[0028] Further, the input of the adaptive attention network in each encoding layer is the input of the current encoding layer, the input of the residual network in each encoding layer is the output of the adaptive attention network in the current encoding layer, and the output of the residual network in each encoding layer is the output of the current encoding layer.

[0029] Further, the model decoder comprises a plurality of decoding layers composed of an up-sampling network and a feature fusion network, and the first decoding layer does not contain the feature fusion network; the number of decoding layers is the same as that of the encoding layers; the input of the first decoding layer is the output of the feature pyramid network in the feature encoder module, the input of the subsequent decoding layer is the output of the previous decoding layer and the output of the corresponding encoding layer, and the output of the last decoding layer is the output of the model decoder.

[0030] Further, the input of the feature fusion network in each decoding layer is the output of the previous decoding layer and the output of the corresponding encoding layer, the input of the up-sampling network in each decoding layer after the second decoding layer is the output of the feature fusion network in the current decoding layer, and the output of the up-sampling network in each decoding layer is the output of the current decoding layer.

[0031] Further, the step S4 specifically comprises the following steps:

[0032] Step S41: re-loading and training the model parameters obtained after the model training;

[0033] Step S42: repeating the training until the model accuracy no longer improves, and obtaining the final model parameters.

[0034] Further, when training the segmentation model combined with adaptive attention and feature pyramid, a similarity loss and a cross-entropy loss are used.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] The application discloses a gastric cancer pathological image segmentation method combined with adaptive attention and a feature pyramid, which can automatically segment a gastric cancer pathological image and solves the problem of time-consuming and labor-intensive manual segmentation. The method constructs a gastric cancer data set, then inputs a pathological image into a semantic segmentation network combined with adaptive attention and a feature pyramid, and enables the network to adaptively learn accurate positioning of a cancerous region, can capture characteristics that cannot be observed by the naked eye in a medical image, and eliminates the subjectivity of a doctor's judgment of cancer. The method uses a similarity loss and a cross-entropy loss and a repeated training strategy, so that the model adaptively converges to an optimum, realizes accurate segmentation of the cancerous region, can accurately guide surgical operation, effectively alleviates the pain of patients, and can be used for clinical auxiliary diagnosis of gastric cancer and provides reliable support for subsequent treatment of patients. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a gastric cancer pathological image segmentation method flowchart of the embodiment of the application.

[0038] Figure 2 It is an image enhancement flowchart of the embodiment of the application.

[0039] Figure 3 FIG. 1 is a schematic diagram of a segmentation model structure combining adaptive attention and feature pyramid for gastric cancer pathological image segmentation according to an embodiment of the present application.

[0040] Figure 4 FIG. 2 is a schematic diagram of a repeated training strategy for gastric cancer pathological image segmentation according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, any other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0042] Reference Signs List Figure 1 FIG. 1 is a schematic diagram of a segmentation model structure combining adaptive attention and feature pyramid for gastric cancer pathological image segmentation according to an embodiment of the present application.

[0043] Step S101: Constructing a gastric cancer dataset.

[0044] In clinic, doctors first obtain a patient's gastric tissue section, generate a gastric pathological image using a digital slide scanner after staining treatment. The purpose of the segmentation network is to imitate the doctor reading the film to find the cancerous area, so it is necessary for an expert doctor to delineate the cancerous area on the gastric pathological image according to experience, and generate a cancerous area annotation image surrounded thereby. The annotation image is composed of white and black colors, wherein the white color represents the cancerous area, and the black color represents the normal tissue area. Each pathological image and its corresponding annotation image constitute a group of image-label pairs, and all the image-label pairs constitute a gastric cancer dataset.

[0045] Step S102: Using an image enhancement method to expand the dataset.

[0046] The image enhancement method can generate more images from limited images without substantially increasing data. The network should be able to accurately distinguish the same target under different angles, light intensity, local shielding, position movement, and different shooting distances. The image enhancement method simulates this process. Image enhancement methods such as shielding part of the image, adjusting image brightness, adding noise or local blur to the image, etc. can reduce the sensitivity of the model to the image, force the model to learn more complex semantic information in the image, improve the invariance of the model, and thus effectively avoid the interference of information unrelated to the target in the image on the model judgment. Image enhancement can also to some extent avoid sample imbalance. By image enhancement, the number of samples with less sample is expanded, the proportion of imbalance is reduced, and the training of the network is positively affected.

[0047] The image enhancement process used in this embodiment is shown in Figure 2 First, the input image is horizontally flipped with a probability of 0.5, then vertically flipped with a probability of 0.5, and then Gaussian noise, random color change and random cropping are added to the image. The flipping operation and the cropping operation need to operate on the image and the label at the same time, and the noise and color change operation only needs to operate on the image. Through the above image enhancement means, the data set can be expanded by 20 times, effectively alleviating the model overfitting, and improving the model generalization and model robustness.

[0048] Step S103: training using a segmentation model combining adaptive attention and feature pyramid.

[0049] In one embodiment, the structure diagram of the segmentation model combining adaptive attention and feature pyramid is shown in Figure 3 , which includes:

[0050] The model encoder is used to characterize the characteristics of the cancerous region in the pathological image, measure the importance of the features in the spatial dimension and channel dimension, and adjust the weight of the features according to the importance. The cancerous region features are often more important and have larger feature weights, and the model pays more attention to these features. The non-cancerous region features are often less important and have smaller feature weights, and the model selectively ignores these features. The encoder adaptively adjusts the representation of image features, improves the feature similarity of the cancerous region, and tries to separate the features of the cancerous region and the non-cancerous region as far as possible to avoid the interference of the features of the non-cancerous region on the model judgment, and obtains the encoded pathological image feature map.

[0051] The model decoder is used to adaptively establish the relationship between the feature map and the target output result according to the pathological image feature map obtained by the encoder, so that the network prediction result is as close to the label result as possible. The input of the decoder is the output of the encoder, and the decoder is composed of multiple layers of up-sampling. Each layer of up-sampling reconstructs the output image of the previous layer, expands the image size, and the output of the last layer of up-sampling is a predicted mask image with the same size as the original pathological image. The predicted mask image is composed of black and white colors. Each pixel point on the image corresponds to the pixel point at the same position of the original pathological image. When the pixel color is white, it represents that the pixel point belongs to the cancerous area, and vice versa, if it is black, it represents that the pixel point belongs to the normal tissue area.

[0052] The connection module between the encoder and the decoder is used for information interaction between the encoder and the decoder. The encoder and the decoder are hierarchical structures with the same number of layers. The output of the last layer of the encoder is used as the input of the decoder after passing through the feature pyramid module. The last layer feature map of the encoder is a high-level modeling of the overall semantics of the image, but the texture details of the image are lost. In the decoding process, in order to compensate for the lost information, it will affect the judgment of the texture details by the decoder, and it is difficult to accurately identify the edge of the cancerous area. In order to solve this problem, the connection module is used to connect the corresponding encoding layer and decoding layer of the hierarchical structure, and the texture information is directly introduced into the corresponding decoding layer, which effectively improves the accuracy of the model.

[0053] In this step, the gastric cancer pathological image is first input into the model encoder combined with adaptive attention and feature pyramid, and the first to fifth layer feature maps are obtained by sequentially passing through five adaptive attention modules and residual modules. Specifically, the input of the first adaptive attention module is the pathological image, the input of the first residual module is the output of the first adaptive attention module, and the output of the first residual module is the first layer feature map. The input of the next adaptive attention module is the output of the previous residual module, the input of each residual module is the output of the adaptive attention module of the layer, and each residual module outputs a feature map.

[0054] The fifth layer feature map output by the encoder combined with adaptive attention and feature pyramid is input into the feature pyramid module, and the output result of the feature pyramid module is input into the decoder combined with adaptive attention and feature pyramid. Once through the five layers of up-sampling module, the first to fifth intermediate results are obtained. Specifically, the input of the first up-sampling module is the output of the feature pyramid module, and the output of the first up-sampling module is the fifth layer intermediate result. The input of the second up-sampling module is the fusion result of the fifth layer intermediate result and the fourth layer feature map, which is realized by the feature fusion module, and the output is the fourth layer intermediate result. In this way, after the fusion result of the first layer feature map and the second layer intermediate result passes through the fifth layer up-sampling module, the predicted mask image of the gastric cancer pathological image is obtained.

[0055] In a specific implementation of the present application, the feature encoder structure diagram of the segmentation model combining adaptive attention and feature pyramid is as shown in Figure 3

[0056] The adaptive attention module is used to strengthen the network's attention to important features in the channel dimension and the spatial dimension. The adaptive attention module first reduces the model size through downsampling, extracts features through a convolutional layer, and then sequentially passes through a channel attention module and a spatial attention mechanism module to adaptively assign weights to different channel features and different spatial position features, respectively, to realize channel and spatial attention. In this embodiment, the channel attention module can use the existing Squeeze and Excitation module (SENet), and the spatial attention mechanism module can use the existing Spatial Transformer Network (STN).

[0057] The residual module is used to directly map shallow information to a deep network, which can effectively break the symmetry of the network, increase the amount of information represented by a high-dimensional feature vector, improve the representation ability of the network, increase the nonlinearity of the network, and alleviate the information decay caused by the deepening of the network. In this embodiment, the residual module can use the existing residual connection method (ResNet).

[0058] The feature pyramid module is used to obtain multi-scale image features and fuse shallow and deep features, which can effectively integrate features of different receptive fields, integrate context information, improve the expression ability of the model, and strengthen the guiding role of deep semantic information on segmentation. In this embodiment, the feature pyramid module can use the existing Feature Pyramid Network (FPN).

[0059] Step S104: Obtain the model parameters and the cancerous region segmentation result.

[0060] In this embodiment, the obtained model parameters and the cancerous region segmentation result are the model parameters and the predicted mask image generated by the first trained model. The model parameters are reloaded and retrained again using the repeated training strategy, the similarity loss and the cross-entropy loss are calculated according to the label and the predicted mask image, the problem of uneven sample distribution in the data set is alleviated, and the accuracy of the model after training is no longer improved until the training is stopped. The trained segmentation model can process the gastric cancer pathological image to be segmented to obtain the cancerous region segmentation result.

[0061] As shown in Figure 4 ​Fig. 1 shows the segmentation result obtained by the present embodiment, wherein (a) is a schematic diagram of a gastric cancer pathological image, the gray closed curve is the boundary of the cancerous region drawn by a doctor, and the curve contains the cancerous region; (b) is a schematic diagram of the final segmentation mask result of the gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid, white is the cancerous region, and black is the normal tissue region; and (c) is a schematic diagram of the local detail enlargement of the final segmentation mask result.

[0062] The above only lists specific embodiments of the present application. Obviously, the present application is not limited to the above embodiments, and there can be many variations. All variations that can be directly derived or inferred from the disclosure of the present application by those of ordinary skill in the art should be considered as falling within the scope of the present application.

Claims

1. A gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid, characterized in that, The method comprises the following steps: Step S1: acquiring gastric cancer pathological images and cancerous region medical annotation results, and constructing a gastric cancer dataset; Step S2: using an image enhancement method to expand the gastric cancer dataset; Step S3: using the expanded gastric cancer dataset to train a segmentation model combining adaptive attention and a feature pyramid, segmenting the cancerous region in the gastric cancer dataset by the segmentation model, and obtaining preliminary trained model parameters and a cancerous region segmentation result corresponding to each gastric cancer pathological image; The segmentation model combining adaptive attention and a feature pyramid in step S3 comprises: a model encoder, which is used to characterize the characteristics of the cancerous region in the pathological image, and obtain a pathological image feature map; the model encoder module comprises a plurality of encoding layers composed of an adaptive attention network and a residual network, and a feature pyramid network, the input of the first encoding layer is the gastric cancer pathological image, the input of the subsequent encoding layer is the output of the previous encoding layer; the output of the last encoding layer is input into the model decoder after passing through the feature pyramid network; a model decoder, which is used to reconstruct a cancerous region prediction mask image according to the pathological image feature map obtained by the model encoder; the model decoder comprises a plurality of decoding layers composed of an up-sampling network and a feature fusion network, and the first decoding layer does not contain a feature fusion network; the number of decoding layers is the same as that of encoding layers; the input of the first decoding layer is the output of the feature pyramid network in the feature encoder module, the input of the subsequent decoding layer is the output of the previous decoding layer and the output of the corresponding encoding layer, and the output of the last decoding layer is the output of the model decoder; a connection module, which is used to connect the corresponding layers of the model encoder and the model decoder, and realize information interaction between the model encoder and the model decoder; Step S4: repeating step S3, using a repeated training strategy to obtain the final parameters of the segmentation model; Step S5: using the trained segmentation model to process the gastric cancer pathological image to be segmented, and obtaining a cancerous region segmentation result.

2. The gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid according to claim 1, characterized in that, Step S1 specifically comprises the following steps: Step S11: acquiring a gastric cancer pathological image; Step S12: an expert draws the cancerous region in the pathological image according to experience; Step S13: generating a cancerous region annotation image according to the expert drawing result, wherein the mask image only has white and black colors, the white region represents the cancerous region, and the black region represents the normal tissue region; Step S14: each gastric cancer pathological image and the corresponding cancerous region annotation image constitute an image-label pair, and all the image-label pairs constitute a gastric cancer dataset.

3. The gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid according to claim 2, characterized in that, Step S2 specifically comprises the following steps: Step S21: implementing horizontal flipping on part of the image-label pairs in the dataset; Step S22: implementing vertical flipping on part of the image-label pairs in the dataset; Step S23: randomly adding Gaussian noise and random tone transformation to all images in the dataset; Step S24: randomly cropping all image-label pairs in the dataset.

4. The gastric cancer pathological image segmentation method combining adaptive attention and feature pyramid according to claim 1, characterized in that, The input of the adaptive attention network in each encoding layer is the input of the current encoding layer, the input of the residual network in each encoding layer is the output of the adaptive attention network in the current encoding layer, and the output of the residual network in each encoding layer is the output of the current encoding layer.

5. The gastric cancer pathological image segmentation method of claim 1, wherein, The input of the feature fusion network in each decoding layer is the output of the previous decoding layer and the output of the corresponding encoding layer, the input of the up-sampling network in the second decoding layer and each decoding layer thereafter is the output of the feature fusion network in the current decoding layer, and the output of the up-sampling network in each decoding layer is the output of the current decoding layer.

6. The gastric cancer pathological image segmentation method of claim 1, wherein, The step S4 specifically comprises the following steps: Step S41: re-loading and training the model parameters obtained after the model is trained; Step S42: repeating the training until the model accuracy no longer improves, and obtaining the final model parameters.

7. The gastric cancer pathological image segmentation method of claim 1, wherein, When the segmentation model combining the adaptive attention and the feature pyramid is trained, a similarity loss and a cross-entropy loss are used.

Images