Multi-head self-attention introduced three-dimensional kidney tumor image segmentation method

By introducing multi-head self-attention module and three-dimensional U2-Net into the renal and renal tumor image segmentation algorithm, the problems of large amount of parameters and low segmentation accuracy are solved, and efficient renal and renal tumor segmentation is achieved, which is suitable for clinical practice.

CN119027657APending Publication Date: 2024-11-26TIANJIN UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310603694.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing kidney and renal tumor image segmentation algorithm has large parameters and low segmentation accuracy, overfitting the model, insufficient sample data, uneven category distribution, and the inability to effectively capture global feature information in the convolutional neural network.

Method used

The three-dimensional kidney tumor image segmentation algorithm for multi-head self-attention is introduced, combining three-dimensional U2-Net and multi-head self-attention module to reduce overfitting through preprocessing and data augmentation, and the multi-head self-attention module extracts global feature information and performs 26 Connecting domain post-processing to improve segmentation accuracy.

Benefits of technology

The segmentation accuracy of kidney and renal tumors is improved in the case of small parameters, and is suitable for clinical medical scenarios such as small embedded devices, which significantly improves the model's ability to extract context information of the medical image of kidney and renal tumors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027657A_ABST
    Figure CN119027657A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-head self-attention introduced three-dimensional kidney tumor image segmentation method, which comprises the following steps of: 1, pre-processing a kidney and kidney tumor image data set, re-sampling all three-dimensional data to the same voxel pitch, carrying out intensity cutting, carrying out de-noising processing on an image, and carrying out multi-head self-attention on the de-noised image; expanding a training data set by using technologies such as rotation and zooming; an image segmentation network introducing multi-head self-attention is constructed by using a Pytorch framework, and a main body structure of a network model is divided into two parts: a three-dimensional U2-Net trunk model used for feature extraction and recovery and based on a coding and decoding structure, and a multi-head self-attention module used for extracting a global feature dependency relationship of a three-dimensional image. The two parts are integrated together for training; and step 3, testing the algorithm model and retaining optimal parameters, and performing kidney tumor three-dimensional medical image segmentation by using the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a three-dimensional renal tumor CT image segmentation method introducing multi-head self-attention. Background Art

[0002] Renal cell carcinoma, also known as kidney cancer, is a highly harmful cancer. It is one of the most common cancers of the genitourinary system, and the main patient population is the middle-aged and elderly. The basic function of the kidney is to help humans clean up metabolic waste and retain useful substances, such as proteins and sugars. If the kidney is damaged, the human body environment will be seriously unbalanced, and the overall function will collapse. With the rapid development of modern society and the accelerated pace of life, many middle-aged people and even teenagers are facing the hazards and risks of kidney cancer. In clinical practice, kidney and kidney tumor images are mainly obtained through medical imaging methods such as electronic computed tomography and magnetic resonance imaging. Doctors manually read the film to determine the patient's lesion area, divide the tumor size, analyze the severity of kidney cancer, and then give treatment advice. However, this process requires a lot of labor costs, because radiologists may need to mark the target area in hundreds of slices for a patient. Therefore, in order to reduce the labor cost burden caused by manual segmentation, it is of great significance to realize the automated segmentation of medical images and help patients obtain diagnosis results, receive treatment and recover in the later stage as soon as possible.

[0003] In recent years, computer vision technology has developed rapidly, and deep learning methods with convolutional neural networks as the backbone network have performed well in tasks such as target detection, image classification, semantic segmentation, and instance segmentation. For kidney and renal tumor segmentation tasks, segmenting the foreground area from three-dimensional medical images belongs to semantic segmentation. Methods based on convolutional neural networks have achieved relatively good results. Methods based on the Transformer architecture can capture the contextual information of three-dimensional medical images and establish long-term dependencies between sequences. They perform well in many medical image processing tasks. However, pure Transformers have poor ability to extract local features and have a large number of parameters.

[0004] Although the existing methods can achieve the segmentation purpose well, there are still the following problems: (1) There is less sample data of kidney tumors. Since medical images involve privacy issues, hospitals need to keep patient information confidential and provide less data for training and verification. When training the network model, less sample data can easily lead to model overfitting, making the model less robust and generalizable; (2) The category distribution in kidney and kidney tumor images is uneven. Since CT images include many abdominal organs, the kidney and kidney tumor occupy a smaller image area, and there is a serious imbalance in the distribution of foreground and background, which brings great difficulties to the accurate segmentation of kidneys and tumors. (3) The method based on convolutional neural network cannot capture the global information of the feature map due to the limited receptive field of convolution. For the three-dimensional medical images of kidney and kidney tumors with rich context information, it is impossible to make good use of their spatial characteristics, resulting in poor segmentation accuracy. At the same time, most convolutional neural network algorithm models are large and have a large number of parameters, which cannot be widely used in clinical practice. Summary of the invention

[0005] In view of the problem that the current kidney and renal tumor image segmentation algorithm has a large number of parameters and low segmentation accuracy, the present invention combines the advantages of convolutional neural networks and Transformer architectures to propose a three-dimensional renal tumor image segmentation algorithm that introduces multi-head self-attention. The technical solution of the present invention is as follows:

[0006] A 3D renal tumor image segmentation algorithm introducing multi-head self-attention includes the following steps:

[0007] The first step is to preprocess the kidney and renal tumor images. The present invention uses the KiTS2019 official challenge dataset KiTS19 for training, verification, and reasoning. The CT image data in the dataset is resampled so that all images have the same voxel spacing, and normalization is used for denoising so that the images can better adapt to the processing of the three-dimensional convolutional neural network. In order to prevent overfitting in network training, the image data is enhanced by random rotation and random scaling to expand the training set;

[0008] In the second step, the Pytorch framework was used to build a 3D renal tumor image segmentation network with multi-head self-attention and train and verify it. The overall structure of the network of the present invention is divided into two parts: 3D U for feature extraction and recovery 2 -Net, a multi-head self-attention module for global feature information extraction. The specific method is as follows:

[0009] (1) Improved 3D U 2 -Net:

[0010] 3D U 2 -Net consists of two-dimensional U 2-Net is upgraded in dimension. The external structure of the network refers to U-Net, which is an encoding and decoding structure. Each level of encoding and decoding module is a nested sub-U-Net structure. The two-level nested structure strengthens the extraction of feature map information. 2 -Net converts the two-dimensional U 2 -Net's five-level downsampling is reduced to four levels, the convolution kernel and pooling kernel are upgraded, LeakyReLU and instance normalization strategies are used to alleviate the gradient vanishing problem under small sample input, a deep supervision strategy is adopted to obtain multi-scale output at the decoding end, and a loss function composed of the Dice loss function and the cross entropy loss function is used for training to solve the category imbalance of the input and output samples;

[0011] (2) Introducing the multi-head self-attention module:

[0012] The multi-head self-attention module was originally a functional module in the Transformer field for extracting long-term dependencies between sequences. This paper uses it in the task of 3D renal tumor medical image segmentation and adds the module to the 3D U 2 After the last encoder of -Net, the feature map is extracted through the encoding path, and the 3D feature map is evenly divided into sub-image blocks and flattened into a one-dimensional sequence. After combining the position encoding, the correlation between each sub-sequence is calculated through the multi-head self-attention module, and then the dependency between the global feature information is obtained. The output sequence is reshaped into a 3D feature map and then converted into a 3D U 2 -Net’s decoding module performs upsampling and output;

[0013] (3) Perform 26-connected domain post-processing:

[0014] The CT images of kidneys and renal tumors contain other abdominal organs, and segmenting the kidneys and renal tumors from them is susceptible to noise interference. The present invention performs post-processing analysis on the segmentation results after reasoning based on 26 connected domains, converts foreground pixels that obviously do not belong to the kidneys and renal tumors into background pixels, and improves the segmentation accuracy of the kidneys and renal tumors.

[0015] The third step is to test the network and save the best network model parameters, send the KiTS19 test set to the network for test verification, and upload the segmentation results to the KiTS2019 official online server to evaluate the segmentation performance.

[0016] This paper applies deep learning technology to medical image segmentation tasks. Aiming at the problems of low segmentation accuracy and large number of parameters in the current renal tumor segmentation methods, a method based on 3D U 2-Net and introduces a multi-head self-attention image segmentation network, which extracts image features multiple times through the backbone network and the nested sub-network, and reduces the number of channels to reduce the model parameters. A multi-head self-attention module is introduced at the end of the encoding, and the deep feature map is divided into sub-image blocks of equal size and flattened into a one-dimensional sequence. After combining position encoding, multi-head self-attention calculation is performed to obtain the dependency of global feature information, improve the model's ability to extract contextual information of kidney and renal tumor medical images, and improve the model accuracy. The algorithm proposed in the present invention achieves good segmentation accuracy for kidneys and renal tumors with a small number of parameters, and has good application value in clinical medical scenarios such as small embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A three-dimensional segmentation network structure diagram of the embodiment of the present invention using a multi-head self-attention module

[0019] Figure 2 The three-dimensional U provided in the embodiment of the present invention 2 -Net structure diagram

[0020] Figure 3 Flowchart of calculation of multi-head self-attention module provided for an embodiment of the present invention

[0021] Figure 4 Data processing flow chart of multi-head self-attention module provided for an embodiment of the present invention

[0022] Figure 5 A comparison visualization diagram of the segmentation results provided in an embodiment of the present invention DETAILED DESCRIPTION

[0023] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0024] The first step is to preprocess kidney and renal tumor images

[0025] In the experiment, the present invention uses the official data set KiTS19 of the KiTS2019 Kidney and Renal Tumor Segmentation Challenge to train and verify the network. The data set contains 300 three-dimensional CT images of renal cancer patients. Each image contains a modality and two types of labels: kidney and renal tumor. The task completed by the present invention is to segment the foreground information of two categories from the background, which belongs to the category of semantic segmentation. Among the 300 CT images, 210 contain label information, which are manually divided by experts with an accuracy rate of more than 99.5%. The 210 data are divided into training sets and validation sets at a ratio of 8:2 to assist model training. The remaining 90 cases do not contain label information and are test sets. The trained network model performs forward reasoning on the test set images to obtain the segmentation results and submit them to the challenge online server to obtain the segmentation Dice value to evaluate the model performance.

[0026] Due to the shooting environment factors, the pixel spacing of each image in the KiTS19 data set is different, which often destroys the advantages of three-dimensional convolution. The present invention resamples the data for preprocessing, resamples all images to the same voxel spacing, and sets the window position and window width according to the CT value of the kidney to remove the influence of the noise of other organs on the segmentation task. In order to alleviate the overfitting of the model training, the image is randomly rotated 15 degrees in the positive and negative directions and scaled to 0.85 times or 1.25 times the original image to complete the data enhancement. The preprocessed images are single-channel images with different sizes, which are cropped into 80×160×160 image blocks to adapt to the unified processing of the network model.

[0027] In the second step, the present invention uses the Pytorch framework to build a three-dimensional image segmentation network that introduces multi-head self-attention for feature learning and network training. Figure 1 The overall structure of the 3D segmentation network with the multi-head self-attention module proposed in the present invention is shown. In the figure, Trans-Block is the processing process of the multi-head self-attention module on the serialized image. The specific details of the network are divided into two parts: the backbone network for feature extraction and restoration, and the 3D U 2 -Net, a multi-head self-attention module for extracting global feature information, which is described as follows:

[0028] (1) 3D U 2 -Net:

[0029] The present invention aims at three-dimensional image data and constructs an improved three-dimensional U 2 -Net network model, Figure 2 Shows a three-dimensional U 2 -Net network structure, the basic architecture is the same as the two-dimensional U 2 -Net, a two-level nested U-shaped structure, 2 -Net is different in that the improved 3D U 2-Net draws on the design of encoding and decoding layers of 3D U-Net and V-Net, and reduces one layer of the original six-level encoder and five-level decoder, resulting in five-level encoders from En_1 to En_5 and four-level encoders from De_1 to De_4, to avoid excessive downsampling that makes the feature map size too small. At the same time, the number of downsampling inside a single RSU (Residual_U-block) is also reduced, that is, the internal nested layer of En_1 after dimensionality increase contains 4 downsamplings, En_2 to En_4 decreases, the internal nested layer of De_1 contains 4 downsamplings, De_2 to De_4 decreases, and the En_5 nested layer does not contain downsampling, and dilated convolution is used to expand the receptive field. In addition, the three-dimensional U 2 -Net upgrades the convolution kernel, pooling kernel, standardization, and activation function layers inside RSU to three dimensions, replaces the original 3×3 convolution with 3×3×3 convolution, and uses a 2×2×2 pooling kernel for the maximum pooling used in the downsampling process. Trilinear interpolation is used between En_5, De_4, De_3, De_2, and De_1 to achieve upsampling.

[0030] The improved three-dimensional network of the present invention sets the input image patch size to 1×80×160×160, the input channel is 1, 160×160 is the image length and width, 80 is the depth dimension, and in the external downsampling process, the dimensions of the three dimensions are reduced to 1 / 2 of the original feature map size. A jump connection is added between the symmetrical encoder and decoder to fuse the shallow and deep feature information. The encoder En_1 adjusts the number of input image channels to 64. The RSU structure and external downsampling do not change the number of channels. The number of channels at the decoding end increases to 128 after the jump connection, and then adjusted to 64 after convolution. The present invention improves the three-dimensional U 2 -Net uses the LeakyReLU function and instance normalization strategy to alleviate the gradient vanishing problem under the premise of small samples and improve the model fitting ability. It uses a loss function composed of the Dice loss function and the cross entropy loss function for training to speed up the convergence of the model and alleviate the category imbalance problem of the input data and output data.

[0031] (2) Introducing the multi-head self-attention module

[0032] In order to enhance the model's extraction of global feature information of kidney and renal tumor images, this paper proposes a global feature-based multi-head self-attention module (GF-MSA), which is combined with sequence flattening, position encoding, and normalization operations and added to the 3D U 2 -Net’s encoding end establishes long-term dependencies for the subsequences generated by the 3D feature graph. Figure 3The detailed calculation process of multi-head self-attention is shown. Since the feature map is a three-dimensional image, and the data processed by the multi-head self-attention module is in the form of a sequence, the present invention divides the feature map obtained at the encoding end into N sub-image blocks, and then flattens each image block into a sequence, that is, X p , use linear projection E to map each flattened sequence to a C-dimensional embedding space, and add position information E to each sequence pos , the specific formula is:

[0033]

[0034] In the formula is all the sequence data after the serialization process, N is the number of image blocks, also represents the sequence length, C is the number of channels, (l-1) is Figure 3 The number of levels in . The process of self-attention calculation is as follows Figure 3 , through multiple groups of W to be learned q , W k and W v The matrix is ​​multiplied with a single sequence to obtain multiple sets of Q (Query), K (Key), and V (Value) vectors of the sequence, and then the attention coefficient matrix of the remaining sequences for the current sequence is calculated as follows:

[0035]

[0036] Where d k represents the sequence dimension, K and V come from the same sequence, and Q comes from another sequence. After the current sequence is linearly represented by the remaining sequences, the current sequence contains global information.

[0037] exist Figure 3 In the figure, MLP is a multi-layer perceptron, which includes two fully connected layers, two Dropout layers and an activation function layer (GELU), and performs global relationship processing on sequence data. Each GF-MSA module is connected to MLP through residual connection and layer normalization is performed. In order to reflect the different parameters of different GF-MSA modules, the present invention combines two-level GF-MSA into a Transformer Layer, and the final multi-head self-attention calculation formula is:

[0038]

[0039]

[0040]

[0041]

[0042] Flatten the encoding end feature map into a sequence for self-attention calculation, and reshape it into the input feature map data processing steps such as Figure 4 As shown, it is named Trans-Block, whose input is the output of the last layer encoder, and the output is based on the three-dimensional U 2 -Net decoder, the present invention selects 4 layers of Transformer Layers (8 levels of GF-MSA in total) and adjusts the input sequence channels to 128 through 1×1×1 convolution, and then adjusts to the number of input feature map channels after the attention calculation.

[0043] (3) Connected domain post-processing

[0044] The present invention adopts a post-processing analysis method based on connected domains, and post-processes the reasoning results of the trained network on the validation set and the test set to remove the false positive point area, so as to improve the segmentation accuracy. Since the three-dimensional image contains spatial information, the present invention performs a 26-neighborhood analysis on a single voxel point to determine whether the voxel values ​​of the 26 neighboring points around it belong to the same category as it, and uses topology and optimized decision tree methods to find the optimal judgment path, avoid similarity judgments on all neighborhoods, and reduce the search time for similar pixels. Connected domain post-processing attributes a small number of foreground voxel points to the background, which significantly improves the segmentation accuracy of kidneys and renal tumors.

[0045] The third step is to test the network and save the best network model parameters.

[0046] The KiTS19 test set was used to verify the network. The 90 unlabeled images in the test set were resampled and preprocessed, and cropped into image blocks of size 80×160×160 as network input. The network performed semantic segmentation on them, and the results after segmentation were processed by connected domain to obtain the final segmentation result map. The results were uploaded to the official server of the KiTS2019 Kidney and Renal Tumor Challenge to obtain the Dice score coefficient to evaluate the model performance. After the segmentation, the best parameter model was retained.

[0047] The segmentation algorithm proposed in this invention has Dice values ​​of 0.9685 and 0.8290 for kidney and renal tumor segmentation on the KiTS19 test set, respectively. Figure 5 The visual segmentation results randomly selected from the validation set are shown, including label results, 3D U 2 -Net segmentation results and 3D U 2 -Net segmentation result. The white dotted lines in the image delineate the kidney and tumor areas. It can be seen from the figure that the segmentation algorithm proposed in the present invention can better segment the kidney and renal tumor from the kidney 3D image, and the 3D U 2 -Net is more 3D U 2-Net has better segmentation effect, and the segmentation results are smoother and closer to the label results.

[0048] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A three-dimensional renal tumor image segmentation method based on multi-head self-attention is introduced, comprising the following steps: The first step is to preprocess the kidney and renal tumor images. The present invention uses the KiTS2019 official challenge dataset KiTS19 for training, verification, and reasoning. The CT image data in the dataset is resampled so that all images have the same voxel spacing, and normalization is used for denoising so that the images can better adapt to the processing of the three-dimensional convolutional neural network. In order to prevent overfitting in network training, the image data is enhanced by random rotation and random scaling to expand the training set; In the second step, the Pytorch framework was used to build a 3D renal tumor image segmentation network with multi-head self-attention. The overall structure of the network was divided into two parts: 3D U 2 -Net, a multi-head self-attention module for global feature information extraction. The specific method is as follows: (1) 3D U 2 -Net: 3D U 2 -Net through 2D U 2 -Net is obtained by dimensionality upgrade. The external structure of the network refers to U-Net, which is an encoding and decoding structure. Each level of encoding and decoding module is a nested sub-U-Net structure. The two-level nested structure strengthens the extraction of feature map information. 2 -Net converts the two-dimensional U 2 -Net's five-level downsampling is reduced to four levels, the convolution kernel and pooling kernel are upgraded, LeakyReLU and instance normalization strategies are used to alleviate the gradient vanishing problem under small sample input, a deep supervision strategy is adopted to obtain multi-scale output at the decoding end, and a loss function composed of the Dice loss function and the cross entropy loss function is used for training to solve the category imbalance of the input and output samples; (2) Introducing the multi-head self-attention module: The multi-head self-attention module was originally a functional module in the Transformer field for extracting long-term dependencies between sequences. This paper uses it in the task of 3D renal tumor medical image segmentation and adds the module to the 3D U 2 After the last encoder of -Net, the feature map is extracted through the encoding path, and the 3D feature map is evenly divided into sub-image blocks and flattened into a one-dimensional sequence. After combining the position encoding, the correlation between each sub-sequence is calculated through the multi-head self-attention module, and then the dependency between the global feature information is obtained. The output sequence is reshaped into a 3D feature map and then converted into a 3D U 2 -Net’s decoding module performs upsampling and output; (3) Perform 26-connected domain post-processing: The CT images of kidneys and renal tumors contain other abdominal organs, and segmenting the kidneys and renal tumors from them is susceptible to noise interference. The present invention performs post-processing analysis on the segmentation results after reasoning based on 26 connected domains, converts foreground pixels that obviously do not belong to the kidneys and renal tumors into background pixels, and improves the segmentation accuracy of the kidneys and renal tumors. The third step is to test the network and save the best network model parameters, send the KiTS19 test set to the network for test verification, and upload the segmentation results to the KiTS2019 official online server to evaluate the segmentation performance.

Citation Information

Cited By

  • Railway tunnel construction water leakage risk identification and disposal method and device

    CN120877202A