An esophageal squamous cell carcinoma image segmentation method and system based on attention mechanism

Abstract

The present invention proposes a segmentation method for esophageal squamous cell carcinoma images based on an attention mechanism, which belongs to the field of image processing technology. It includes: obtaining the original esophageal squamous cell carcinoma CT image sequence; preprocessing the acquired images; building an attention network model, inputting the preprocessed images into the model for training; using the trained model to output evaluation indicators to obtain the esophageal squamous cell carcinoma segmentation results. The present invention introduces a channel attention module and a spatial attention module to enhance important features and suppress unimportant features. On the premise of improving segmentation accuracy, it greatly improves the generalization ability of the deep neural network model. At the same time, it uses a visualization method to explain the image area that the deep neural network pays attention to during the feature extraction process, thereby explaining the internal operation mechanism of the neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an esophageal squamous cell carcinoma image segmentation method and system based on an attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Esophageal squamous cell carcinoma, also known as esophageal squamous cell carcinoma, is a type of cancer with a high incidence. Since the World Health Organization clearly proposed the "three early" strategy of early detection, early diagnosis, and early treatment of cancer, cancer screening and early diagnosis and treatment have been recognized as the most effective way to prevent and control cancer.

[0004] Clinical studies have shown that computed tomography (CT) is a commonly used technology for screening tumors. Doctors use CT to find the patient's tumor area and develop a corresponding treatment plan. Since esophageal squamous cell carcinoma is highly sensitive to radiation, radiotherapy is usually the first choice of treatment. However, the special location of esophageal squamous cell carcinoma requires high accuracy of radiotherapy, so it is very important to quickly and accurately outline the tumor target area in the CT image.

[0005] Fully convolutional neural networks represented by U-Net were first used for medical image segmentation tasks. However, in the process of model feature extraction, neural networks usually extract many features that are irrelevant to the classification target, resulting in a waste of computing resources and reduced computing efficiency. At the same time, since the tumor area only occupies a very small area of ​​the CT image, this will further reduce the segmentation accuracy of the image deep learning algorithm.

[0006] At the same time, deep neural networks are a black box to people, and their internal operating mechanisms are unknown. We only know that they can recognize images with a high accuracy, but we have no idea how they do it and what features they use to distinguish images. Summary of the invention

[0007] In order to solve the above problems, the present invention provides an esophageal squamous cell carcinoma segmentation method and system based on the attention mechanism, introduces a channel attention module and a spatial attention module, enhances important features and suppresses unimportant features, and greatly improves the generalization ability of the deep neural network model while improving the segmentation accuracy. At the same time, a visualization method is used to explain the image area that the deep neural network pays attention to during the feature extraction process, so as to explain the internal operation mechanism of the neural network.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of the present invention provides a segmentation method for esophageal squamous cell carcinoma images based on an attention mechanism, comprising:

[0010] Obtain the original CT image sequence of esophageal squamous cell carcinoma;

[0011] Preprocess the acquired images;

[0012] Build an attention network model and input the preprocessed images into the model for training;

[0013] The trained model is used to output evaluation indicators and obtain the esophageal squamous cell carcinoma segmentation results.

[0014] A second aspect of the present invention provides an esophageal squamous cell carcinoma image segmentation system based on an attention mechanism, comprising:

[0015] The data acquisition module is configured to: acquire original esophageal squamous cell carcinoma CT image sequence;

[0016] The data processing module is configured to: pre-process the acquired image;

[0017] The model building module is configured to: build an attention network model;

[0018] The model training module is configured to: input the preprocessed image into the model for training;

[0019] The result output module is configured to: use the trained model to output evaluation indicators to obtain esophageal squamous cell carcinoma segmentation results.

[0020] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the method for segmenting esophageal squamous cell carcinoma images based on an attention mechanism as described in the first aspect of the present invention.

[0021] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for segmenting esophageal squamous cell carcinoma images based on the attention mechanism as described in the first aspect of the present invention are implemented.

[0022] One or more of the above technical solutions have the following beneficial effects:

[0023] 1. The present invention obtains a CT image sequence and performs a rough segmentation preprocessing operation on the image based on anatomical knowledge, which alleviates the problem of category imbalance to a certain extent, saves computing resources, improves computing efficiency, and improves segmentation accuracy.

[0024] 2. The present invention inputs the preprocessed image into the U-Net model, and by jump-connecting the features of the encoding part with the features of the decoding part, fuses the features of the low-level and high-level layers, supplements the details in the decoding process, and makes the information after decoding richer.

[0025] 3. The present invention also adds an attention module to the jump connection, selects important features that are conducive to semantic expression in the channel domain and the spatial domain respectively and focuses on the target area, thereby improving the segmentation accuracy of the model.

[0026] 4. The present invention adopts the score-class activation mapping method (Score-Calss Activation Map) to generate a visual thermal image to distinguish the area that the deep neural network focuses on during the feature extraction process and the area after the feature is accurate by the attention module, so as to explain the working process of the deep neural network and the effectiveness of the attention module.

[0027] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 This is a flowchart of the esophageal squamous cell carcinoma image segmentation method based on the attention mechanism;

[0030] Figure 2 This is a flow chart of the channel attention module of the esophageal squamous cell carcinoma image segmentation method based on the attention mechanism;

[0031] Figure 3 This is a flowchart of the spatial attention module of the esophageal squamous cell carcinoma image segmentation method based on the attention mechanism. DETAILED DESCRIPTION

[0032] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0033] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0035] Embodiment 1

[0036] This embodiment discloses a segmentation method for esophageal squamous cell carcinoma images based on an attention mechanism.

[0037] like Figure 1 As shown, this embodiment provides a segmentation method for esophageal squamous cell carcinoma images based on an attention mechanism, comprising the following steps:

[0038] Obtain the original CT image sequence of esophageal squamous cell carcinoma;

[0039] Preprocess the acquired images;

[0040] Build an attention network model and input the preprocessed images into the model for training;

[0041] The trained model is used to output evaluation indicators and obtain the esophageal squamous cell carcinoma segmentation results.

[0042] In step 1, a CT image sequence of esophageal squamous cell carcinoma is obtained, and the size of each image is adjusted to 512×512×3 pixels.

[0043] In step 2, the image acquired in step 1 is preprocessed, specifically including preliminary positioning of the approximate area of ​​the esophagus based on anatomical knowledge and data enhancement.

[0044] Step 2-1, in the plane image of esophageal squamous cell carcinoma, the spine and esophagus are roughly on the same vertical line and within a certain offset range. At the same time, the esophagus is between the two lungs. Therefore, the cross formed by the horizontal tangent of the spine as the X-axis, the vertical dividing line between the two lungs as the Y-axis, and the intersection as the origin O can preliminarily locate the approximate area where the esophagus is located, and the esophageal area can be roughly segmented based on this. The image size after removing the redundant background is 101×101×3 pixels.

[0045] Step 2-2: Data enhancement is performed on the roughly segmented esophageal squamous cell carcinoma image, mainly using geometric transformation, including rotation and flipping.

[0046] Step 3: Use the training set obtained by preprocessing to train the convolutional neural network model. The attention network model is based on the U-Net network, including a normalization layer, a downsampling layer, an upsampling layer, a pooling layer, an activation layer and an attention layer.

[0047] The specific structure of the attention network model will be described in detail below.

[0048] Normalization layer: The normalization layer is to normalize the image pixel values ​​of the acquired esophageal squamous cell carcinoma image set according to the maximum and minimum values ​​of the image pixel values ​​to accelerate the convergence of the model during the training process. The normalization formula is:

[0049]

[0050] Downsampling stage: The normalized image slices are input into the downsampling stage, where the downsampling stage follows the classic architecture of convolutional neural networks. Each downsampling layer consists of two repeated applications of 3*3 convolutions. Each convolution contains an activation layer, and a 2*2 maximum pooling operation with a step size of 2 is performed between the downsampling layers. In each downsampling step, the number of feature channels is doubled, reducing the feature dimension while extracting the spatial information of the image.

[0051] Upsampling stage: Each step in the upsampling stage includes upsampling the feature map, followed by halving the number of feature channels through deconvolution, concatenating the feature map of the same size from the corresponding downsampling stage after being filtered by the attention layer and cropped accordingly, and each layer undergoes two 3*3 convolution operations, each followed by an activation layer. In the last layer of the upsampling stage, a 1*1 convolution is used to map the feature vector to the required number of classes.

[0052] Activation layer: The activation layer enhances the network's expressive power through nonlinear mapping. It maps the input information to a nonlinear space through a nonlinear activation function to extract different features. The activation layer uses the ReLU function. The function formula is:

[0053] f(x) = max(0, x)

[0054] Attention layer: including channel attention module and spatial attention module. The channel attention module includes a global average pooling layer using SE blocks, a first fully connected layer, a ReLU activation function layer, a second fully connected layer, and a sigmoid activation function layer, and the spatial attention module includes a global average pooling layer, a segmentation patch layer, a self-attention calculation layer, a secondary weight calculation layer, and a sigmoid activation function layer.

[0055] Channel attention layer: The channel attention layer specifically uses SE blocks to enhance important features and suppress unimportant features through feature selection.

[0056] like Figure 2 As shown, the SE block first uses global average pooling for each channel independently. This stage is called the squeeze stage, or feature aggregation stage, and its formula is:

[0057]

[0058] Among them, W is the width of the feature channel, H is the height of the feature channel, and the generated vector dimension is 1×1×C, where C is the corresponding number of feature channels.

[0059] Then, two nonlinear fully connected layers and a Sigmoid function are used to generate channel weights. This process is called excitation, or the feature recalibration phase, and its formula is:

[0060] s=F ex =σ(W 2 δ(W 1 g(x))),

[0061] Among them, W 1 is the first fully connected layer, and the output dimension is r is a scaling parameter, the purpose of this parameter is to control the complexity of the model by reducing the dimension; W 2 It is the second fully connected layer with an output dimension of 1×1×C. δ is the ReLU activation function and σ is the sigmoid activation function.

[0062] Finally, all channels of the feature channel are multiplied by the corresponding weight s i ,i∈(1,C).

[0063] Spatial Attention Layer: The spatial attention layer selects important spatial regions in the features and suppresses unimportant spatial regions through calculation.

[0064] like Figure 3 As shown in the figure, the process is divided into two calculation stages. In the first calculation stage, global average pooling is used in the channel direction to generate a weighted activation map for each pixel that describes the importance of the pixel. The formula is:

[0065]

[0066] Among them, C is the number of channels of the feature map, and the size of the generated activation map is W×H×1.

[0067] After that, the generated activation map is divided into patches of equal size, and the size of each patch is The value of r is different in each downsampling stage.

[0068] Then, the self-attention between any two patches is calculated. This process uses the contextual relationship between patches to calculate the relevance between contexts. In the process of calculating self-attention, the influence of distance on the relevance between any two patches is considered. The calculation formula is:

[0069]

[0070] The dist function is used to calculate the Manhattan distance from any Patch to the current Patch. The calculation formula is:

[0071] dist(Pi ,P j )=|x i -x j |+y i -y j |, (x, y) is the coordinate of the corresponding Patch.

[0072] The second calculation stage is used to further highlight the patches with higher weights in the first calculation stage.

[0073] First, calculate the proportion of each patch weight to the total weight:

[0074]

[0075] Then multiply the proportional coefficient to the corresponding Patch:

[0076]

[0077] After completing the second calculation stage, the adjusted activation map is normalized.

[0078] Finally, multiply the positions corresponding to all feature maps of the feature channel by the weights of the corresponding positions of the activation map

[0079] The visualization process is specifically as follows:

[0080] The features from the downsampling and the features calibrated by the attention module are compared visually. The feature maps obtained by downsampling and the feature maps calibrated by the attention module are input into the score-class activation mapping function to generate the corresponding heat map, which visually explains the area of ​​the image that the convolution operation focuses on in the downsampling, and the area that is focused on after the attention module is calibrated.

[0081] Specifically, given a U-Net model Y = f(X), Score-CAM first obtains the activations of the k channels of the input sample X of the lth convolutional layer in the model. For a known baseline input X b ,activation The contribution to Y is defined as The Up(·) function will The operation of upsampling to the same size as the input X, s(·) is the regularization function, which maps the input matrix to the range [0,1], is the Hadamard product, and its formula is:

[0082]

[0083] min(·) is to find the minimum value of the input matrix, and max(·) is to find the maximum value of the input matrix.

[0084] Consider a convolutional layer l in the U-Net model. Given a class of interest c, the Score-CAM calculation formula is:

[0085] in

[0086] Based on this, the heat map of the input sample X is obtained.

[0087] The U-Net model is trained using the pre-processed training set and validation set data: the U-Net model is randomly initialized, the learning rate is set to 0.0001, the Adam optimizer is selected, and the loss function uses the Dice loss function. The formula of the Dice loss function is:

[0088]

[0089] During the training process, forward propagation is first used to calculate the output of the model, and then backpropagation is used to update the parameters of the model. By reducing the loss function, the expected output is continuously approached, the error is reduced, and finally a trained U-Net model is obtained.

[0090] Step 4: Test the U-Net model.

[0091] Step 4-1: Use the test set to test the optimal model in the training process. Use the accuracy, precision and recall rate to evaluate the model. The evaluation index formula is defined as follows:

[0092]

[0093]

[0094]

[0095] Among them, TP is a true positive example, which is predicted by the model as a positive example and is actually a positive example; FP is a false positive example, which is predicted by the model as a positive example and is actually a false example; FN is a false negative example, which is predicted by the model as a negative example and is actually a positive example; TN is a true negative example, which is predicted by the model as a negative example and is actually a negative example.

[0096] Step 4-2: Use the trained model to outline the exact area of ​​esophageal squamous cell carcinoma.

[0097] Embodiment 2

[0098] This embodiment discloses a segmentation system for esophageal squamous cell carcinoma images based on an attention mechanism, including:

[0099] The data acquisition module is configured to: acquire original esophageal squamous cell carcinoma CT image sequence;

[0100] The data processing module is configured to: pre-process the acquired image;

[0101] The model building module is configured to: build an attention network model;

[0102] The model training module is configured to: input the preprocessed image into the model for training;

[0103] The result output module is configured to: use the trained model to output evaluation indicators to obtain esophageal squamous cell carcinoma segmentation results.

[0104] Embodiment 3

[0105] The purpose of this embodiment is to provide a computer-readable storage medium.

[0106] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for segmenting esophageal squamous cell carcinoma images based on an attention mechanism as described in Example 1 of the present disclosure.

[0107] Embodiment 4

[0108] The purpose of this embodiment is to provide an electronic device.

[0109] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for segmenting esophageal squamous cell carcinoma images based on the attention mechanism as described in Example 1 of the present disclosure are implemented.

[0110] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0111] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0112] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A segmentation method for esophageal squamous cell carcinoma images based on an attention mechanism, characterized in that: include: Obtain the original CT image sequence of esophageal squamous cell carcinoma; Preprocess the acquired images; Build an attention network model and input the preprocessed images into the model for training; The attention network model is based on the U-Net network, including a normalization layer, a downsampling stage, an upsampling stage, an activation layer, a pooling layer and an attention layer; the attention layer includes a channel attention module and a spatial attention module; The channel attention module includes a global average pooling layer using SE blocks, a first fully connected layer, a ReLU activation function layer, a second fully connected layer, and a sigmoid activation function layer; the spatial attention module includes a global average pooling layer, a segmentation patch layer, a self-attention calculation layer, a secondary weight calculation layer, and a sigmoid activation function layer; The SE block in the channel attention layer applies global average pooling to each channel independently, and its formula is: Among them, W is the width of the feature channel, H is the height of the feature channel, and the generated vector dimension is 1×1×C, where C is the number of corresponding feature channels; Two nonlinear fully connected layers and a sigmoid function are used to generate channel weights, and the formula is: s=F ex =σ(W2δ(W1g1(x))), Among them, W1 is the first fully connected layer, and the output dimension is r is a scaling parameter that controls the complexity of the model by reducing dimensionality; W2 is the second fully connected layer with an output dimension of 1×1×C; δ is the ReLU activation function, and σ is the sigmoid activation function; Multiply all channels of the feature channel by the corresponding channel weight s i , i∈(1,C); The spatial attention layer is divided into two calculation stages. The first calculation stage uses global average pooling in the channel direction to generate a weighted activation map for each pixel that describes the importance of the pixel. The formula is: Among them, C is the number of channels of the feature map, and the size of the generated activation map is W×H×1; The generated activation map is divided into patches of equal size, and the self-attention between any two patches is calculated. In the process of calculating the self-attention, the influence of the distance on the correlation between any two patches is considered. The calculation formula is: The dist function is used to calculate the Manhattan distance from any Patch to the current Patch. The calculation formula is: dist(P i ,P j )=|x i -x j |+|y i -y j | (x,y) is the coordinate of the corresponding Patch; The second calculation stage is used to further highlight the patches with high weights in the first calculation stage, and calculate the proportion of each patch's weight to the total weight: Multiply the proportional coefficient to the corresponding Patch. After completing the second calculation stage, normalize the adjusted activation map and multiply the corresponding positions of all feature maps of the feature channel by the weight of the corresponding position of the activation map. It also includes the visualization process, specifically: Visually compare the features after downsampling and the features calibrated by the attention module; input the feature map obtained by downsampling and the feature map calibrated by the attention module into the score-class activation mapping function to generate the corresponding heat map; The trained model is used to output evaluation indicators and obtain the esophageal squamous cell carcinoma segmentation results.

2. The method for segmenting esophageal squamous cell carcinoma images based on the attention mechanism according to claim 1, characterized in that: The preprocessing of the acquired image specifically includes: The esophageal region in the acquired esophageal squamous cell carcinoma image is located and roughly segmented, the data set is expanded, and the data set is divided into a training set, a validation set, and a test set according to a set ratio.

3. The method for segmenting esophageal squamous cell carcinoma images based on the attention mechanism according to claim 1, characterized in that: The formula of the attention network model loss function is: Among them, TP is a true positive example, FP is a false positive example, and FN is a false negative example.

4. The method for segmenting esophageal squamous cell carcinoma images based on the attention mechanism according to claim 1, characterized in that: The model is evaluated using three evaluation indicators: accuracy, precision and recall. The evaluation indicator formula is: Among them, TP is a true positive example, which is predicted by the model as a positive example and is actually a positive example; FP is a false positive example, which is predicted by the model as a positive example and is actually a false example; FN is a false negative example, which is predicted by the model as a negative example and is actually a positive example; TN is a true negative example, which is predicted by the model as a negative example and is actually a negative example.

5. A system for segmenting esophageal squamous cell carcinoma images based on an attention mechanism, using the method for segmenting esophageal squamous cell carcinoma images based on an attention mechanism according to claim 1, characterized in that: include: The data acquisition module is configured to: acquire original esophageal squamous cell carcinoma CT image sequence; The data processing module is configured to: pre-process the acquired image; The model building module is configured to: build an attention network model; The model training module is configured to: input the preprocessed image into the model for training; The result output module is configured to: use the trained model to output evaluation indicators to obtain esophageal squamous cell carcinoma segmentation results.

6. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the esophageal squamous cell carcinoma image segmentation method based on the attention mechanism as described in any one of claims 1 to 4 are implemented.

7. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the esophageal squamous cell carcinoma image segmentation method based on the attention mechanism as described in any one of claims 1 to 4 are implemented.

Images