A spatial attention method and device for pet-ct multimodal tumor segmentation

Abstract

The present application relates to a kind of spatial attention method and device for PET-CT multimodal tumor segmentation.The method and device set up separately encoded PET input channel and CT input channel in multiscale convolution spatial attention network, PET image in PET-CT multimodal image is input to PET input channel to extract PET characteristic information, CT image in PET-CT multimodal image is input to CT input channel to extract CT characteristic information;PET decoding result and CT decoding result are fused.The method and device are based on PET-CT multimodal image, using the method of spatial attention, consider the characteristics of PET and CT respectively, first use network to extract the characteristic information of both, then the characteristic information is fused, especially the application of multiscale convolution makes that network can highlight tumor area, suppress non-tumor area, obtain more accurate tumor segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image segmentation, in particular to a spatial attention method and device for PET-CT multi-modal tumor segmentation. BACKGROUND

[0002] Medical image segmentation is mainly based on computer tomography (CT), positron emission tomography (PET) or magnetic resonance imaging (MRI) to delineate the tumor from the image, which is a basic task of medical image analysis. Although CT images have high resolution and can accurately depict internal structural information of objects, they are not clear enough for imaging of soft tissue structures such as tumors. PET is a new type of molecular imaging technology based on radioactive tracers, which is quite sensitive to soft tissue regions such as tumors, so it is quite effective for tumor detection. However, the resolution of PET imaging is poor, and the tumor boundary detected is not clear enough, which is not conducive to determining the tumor boundary in the tumor segmentation task. The emergence of combined PET-CT imaging solves the above problems, which integrates the advantages of CT and PET, and has high accuracy of CT for structural information and high sensitivity of PET for tumor detection. However, current tumor segmentation is manually completed by experienced experts, which is a time-consuming, labor-intensive and expensive task. Therefore, it is of great scientific significance and application prospect to develop an automatic tumor segmentation method based on PET-CT multi-modal for tumor diagnosis and reducing the workload of doctors.

[0003] Most of the current automatic tumor segmentation methods are based on CT or PET or MRI alone. Some are based on PET-CT or PET-MRI multi-modal, but they only simply fuse the two images, and rarely consider the differences between the two.

[0004] Chinese patent application No. 202210499262.X proposes a "spatial attention network for PET-CT multi-modal tumor segmentation", which extracts the feature information of PET and CT by designing a separate spatial attention module.

[0005] Xiaohang Fu et al. published an article titled "Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation" in IEEE Journal of Biomedical and Health Informatics in 2021. The network is divided into PET and CT two sub-networks (where the network of the PET channel is the multimodal spatial attention module proposed by the author), which adopts the U-Net structure. The network contains 5 layers, each layer contains two convolutions, and the number of convolution kernels in each layer is 64, 128, 256, 512, and 1024 respectively. Finally, a convolution kernel is used for one convolution and the softmax activation function is used to generate the final spatial attention result. The author samples the output attention result map of the PET sub-network and fuses it as weight information into the up-sampling stage of the CT, and generates the final segmentation result.

[0006] However, the above prior art has the following technical defects:

[0007] 1. The tumor segmentation effect using single modality data is poor;

[0008] 2. Most algorithms are designed for special tumors, and the generalization of the algorithm is poor;

[0009] 3. The use of multi-modal data does not consider the difference between different modalities. SUMMARY

[0010] The embodiment of the application provides a spatial attention method and device for PET-CT multi-modal tumor segmentation, so as to at least solve the technical problem of poor precision of existing medical image segmentation.

[0011] According to an embodiment of the application, a spatial attention method for PET-CT multi-modal tumor segmentation is provided, comprising the following steps:

[0012] A separate encoded PET input channel and a CT input channel are arranged in the multi-scale convolution spatial attention network, a PET image in a PET-CT multi-modal image is input into the PET input channel to extract PET feature information, and a CT image in the PET-CT multi-modal image is input into the CT input channel to extract CT feature information.

[0013] The PET decoding result and the CT decoding result are fused.

[0014] Further, the PET input channel and the CT input channel are two symmetric but weight-unshared input channels.

[0015] Further, before the PET-CT multi-modal image is input into the channel, the method further comprises: resampling, cropping and normalizing the PET-CT multi-modal image, specifically:

[0016] First, the PET and CT multi-modal images are resampled to a resolution of 1mm x 1mm x 1mm, and then cropped to a size of 144 x 144 x 144; then the CT multi-modal image is normalized to [-1, 1] using the maximum and minimum value, and the PET multi-modal image is normalized using the mean and variance.

[0017] Further, before the PET decoding result and the CT decoding result are fused, the method further comprises:

[0018] A channel weight mask module is arranged in the PET input channel and the CT input channel, and the channel weight mask module includes a global average pooling layer, two fully connected layers and a sigmoid layer;

[0019] The global average pooling maps an input with a shape format of [B, C, H, W, D] to [B, C, 1, 1, 1], compresses the last three dimensions, and obtains a weight vector with a shape of [B, C, 1]; wherein B represents the batch size; C represents the number of input channels; H, W and D represent the three-dimensional image size of the input;

[0020] After the global average pooling, there are two fully connected layers, wherein the first fully connected layer has C input nodes and 2*C output nodes, and the second fully connected layer has 2*C input nodes and C output nodes; the second fully connected layer uses a sigmoid activation function to obtain the final channel weight mask;

[0021] A short connection is added between the encoding and the decoding, the input is multiplied by the channel weight mask, and the channel dimension feature information is compressed and expanded.

[0022] Further, the input data size used is [B, C, H, W, D], and the input is first passed through a channel weight mask module to complete the compression and expansion of the channel to obtain a result X with a size of [B, C, H, W, D];

[0023] Then the output result of the channel weight mask module is subjected to multi-scale convolution, three different sizes of convolution kernels are adopted, 1x1x1, 3x3x3 and 5x5x5, the convolution step length is 1, the padding is 0, 1 and 2 respectively, and the activation function is relu; the convolution results of 3x3x3 and 5x5x5 are multiplied in matrix form, and the result U with the size of [B, C, H, W, D] is flattened into a one-dimensional vector S with the size of [B, C, (H*W*D), 1], and the weight information T with the size of [B, C, (H*W*D), 1] is generated through the softmax activation function, and then it is reconstructed into the original size W with the size of [B, C, H, W, D]; the dot product of the result W and the 1x1x1 convolution result is obtained to obtain the multi-scale spatial attention result; a short connection is used to splice the input and output of the multi-scale convolution in the channel and normalize them;

[0024] Finally, a 1x1x1 convolution is used to adjust the number of channels to the same as the input, and the size of the result V is [B, C, H, W, D].

[0025] Further, an automatic expansion residual module is arranged on the original residual network; the automatic expansion residual module is composed of two branches, one branch has a convolution kernel size of 3x3x3, a convolution step length of 1, a padding number of 1 and a relu activation function; the other branch automatically changes according to the input channel and the set output channel to determine;

[0026] If the input channel and the set output channel are equal, the branch does not perform any operation, and the automatic expansion residual module degenerates into a standard residual network; if the input channel and the set output channel are not equal, the branch performs convolution operation, the convolution kernel size is 1x1x1, and the activation function is relu; finally, the output is the sum of the results of the two branches;

[0027] The two automatic expansion residual modules are used in series; the first automatic expansion residual module changes the number of channels, and the second automatic expansion residual module does not change the number of channels.

[0028] Further, an encoding module and a decoding module are arranged in the multi-scale convolution spatial attention network.

[0029] The encoding module includes a residual layer, a spatial attention layer and a pooling layer, which are connected in series; the residual layer is composed of two automatic expansion residual modules; the input channels of each stage are 2, 16, 32, 64 and 128 respectively, and the output channels are 16, 32, 64, 128 and 256 respectively; the input channel and the output channel of the spatial attention layer remain the same and the feature image size does not change; the convolution kernel size of the pooling layer is 2x2x2 and the convolution step length is 2.

[0030] The decoding module comprises an inverse convolution layer, a residual layer and a spatial attention layer, which are connected in series; the convolution kernel size of the inverse convolution layer is 3*3*3, the convolution step is 2, and the padding is 1; the residual layer and the spatial attention layer are both composed of two automatic expansion residual modules, and the input and output channels and the size remain unchanged; the input channels of each stage are 256, 128, 64 and 32 respectively, and the output channels are 128, 64, 32 and 16 respectively.

[0031] Further, the PET input channel and the CT input channel both adopt the U-Net structure, comprising five encoding modules and four decoding modules.

[0032] The convolution kernel size of the first input convolution layer of the multi-scale convolution spatial attention network is 5*5*5, the convolution kernel size of the last layer is 1*1*1, and the activation function is sigmoid; the activation functions of the remaining layers are all relu.

[0033] Further, the Dice similarity coefficient DSC and the Hausdorff distance HD are used as evaluation indexes; wherein:

[0034]

[0035] HD(X,Y)=max(h(X,Y),h(Y,X))

[0036]

[0037] The loss function is the Dice loss:

[0038]

[0039] The Focal loss function is also added in the loss function:

[0040]

[0041] The final loss function is:

[0042] Loss(X,Y)=Loss D (X,Y)+Loss F (X,Y)

[0043] Wherein y and represent the tumor segmentation label and the tumor segmentation prediction result respectively; alpha is a balance weight factor, and the value is set to 0.5; gamma is a weight decay factor, and is set to 2;

[0044] The training process uses an Adam optimizer, two momentum factors are set to 0.9 and 0.99 respectively, a total of 300 times of training, and the learning rate is attenuated in the training process by using a simulated annealing algorithm, and the learning rate is restarted every 25 times of training; the learning rate is initialized to 3e-4, and the minimum is 1e-6; finally, a neural network mapping model capable of realizing automatic segmentation of three-dimensional tumors is obtained.

[0045] According to another embodiment of the application, a spatial attention device for PET-CT multi-modal tumor segmentation is provided, comprising:

[0046] A feature information extraction unit is configured to set separate encoded PET input channels and CT input channels in a multi-scale convolutional spatial attention network, input a PET image in a PET-CT multi-modal image to the PET input channels to extract PET feature information, and input a CT image in the PET-CT multi-modal image to the CT input channels to extract CT feature information.

[0047] A fusion unit is configured to fuse the PET decoding result and the CT decoding result.

[0048] A storage medium stores a program file capable of implementing any one of the above spatial attention methods for PET-CT multi-modal tumor segmentation.

[0049] A processor is configured to run a program, wherein the program performs any one of the above spatial attention methods for PET-CT multi-modal tumor segmentation when running.

[0050] The spatial attention method and device for PET-CT multi-modal tumor segmentation in the embodiments of the application are based on PET-CT multi-modal images, use the method of spatial attention, consider the characteristics of PET and CT respectively, separately extract the feature information of PET and CT using the network, fuse the feature information, and especially the application of multi-scale convolution enables the network to highlight the tumor region, suppress the non-tumor region, and obtain a more accurate tumor segmentation result. In the encoding stage, the feature information of PET and CT is extracted respectively, and in the decoding stage, the feature information extracted by the application is fused, which can use the high sensitivity of PET for tumor detection to locate the tumor position, and use the high accuracy of CT for structural information to determine the tumor boundary. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate certain illustrative embodiments of the application and together with the description serve to explain the application. In the drawings:

[0052] Figure 1A module diagram of the multiscale convolutional spatial attention network of the present application;

[0053] Figure 2 A whole framework diagram of the multiscale convolutional spatial attention network of the present application;

[0054] Figure 3 A segmentation result diagram of the multiscale convolutional spatial attention network of the present application;

[0055] Figure 4 A boundary segmentation result diagram of the multiscale convolutional spatial attention network of the present application. DETAILED DESCRIPTION

[0056] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the protection scope of the present application.

[0057] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0058] Embodiment 1

[0059] According to an embodiment of the present application, a spatial attention method for PET-CT multi-modal tumor segmentation is provided, comprising the following steps:

[0060] In the multiscale convolutional spatial attention network, a separately encoded PET input channel and a CT input channel are arranged, a PET image in a PET-CT multi-modal image is input to the PET input channel to extract PET feature information, and a CT image in the PET-CT multi-modal image is input to the CT input channel to extract CT feature information;

[0061] The PET decoding result and the CT decoding result are fused.

[0062] The spatial attention method for PET-CT multi-modal tumor segmentation in the embodiment of the application is based on PET-CT multi-modal images, uses the method of spatial attention, respectively considers the characteristics of PET and CT, separately extracts the feature information of both by using a network, then fuses the feature information, and especially the application of multi-scale convolution enables the network to highlight the tumor region, suppresses the non-tumor region, and obtains a more accurate tumor segmentation result. In the encoding stage, the application is respectively used to extract the feature information of PET and CT, in the decoding stage, the application fuses the feature information extracted by both, which can respectively use the high sensitivity of PET for tumor detection to locate the tumor position, and simultaneously use the high accuracy of CT for structural information to determine the tumor boundary.

[0063] The PET input channel and the CT input channel are two symmetrical but weight-unshared input channels.

[0064] Before the PET-CT multi-modal image is input into the channel, the method further comprises: resampling, cropping and normalizing the PET-CT multi-modal image, specifically:

[0065] First, the PET and CT multi-modal images are resampled to a resolution of 1mm*1mm*1mm, and then cropped to a size of 144*144*144; then the CT multi-modal image is normalized to [-1, 1] using the maximum and minimum value, and the PET multi-modal image is normalized using the mean and variance.

[0066] Before the PET decoding result and the CT decoding result are fused, the method further comprises:

[0067] A channel weight mask module is arranged in the PET input channel and the CT input channel, and the channel weight mask module comprises a global average pooling layer, two fully connected layers and a sigmoid layer;

[0068] The global average pooling maps the input with a shape format of [B, C, H, W, D] to [B, C, 1, 1, 1], compresses the last three dimensions, and obtains a weight vector with a shape of [B, C, 1]; wherein B represents the batch size; C represents the number of input channels; H, W and D represent the three-dimensional image size of the input;

[0069] After the global average pooling, there are two fully connected layers, wherein the input nodes of the first fully connected layer are C, and the output nodes are 2*C; the input nodes of the second fully connected layer are 2*C, and the output nodes are C; the second fully connected layer uses a sigmoid activation function to obtain the final channel weight mask;

[0070] A short connection is added between encoding and decoding, the input is multiplied by a channel weight mask, and the channel dimension feature information is compressed and expanded.

[0071] Wherein, the input data size used is [B, C, H, W, D], the input is first passed through a channel weight mask module to complete the squeezing and expansion on the channel to obtain a result X with a size of [B, C, H, W, D];

[0072] Then, multi-scale convolution is performed on the output result of the channel weight mask module, three different size convolution kernels are used, which are 1×1×1, 3×3×3 and 5×5×5, the convolution step is 1, the padding is 0, 1 and 2 respectively, and the activation function is relu; the convolution results of 3×3×3 and 5×5×5 are multiplied by matrix, and the result U with a size of [B, C, H, W, D] is flattened into a one-dimensional vector S with a size of [B, C, (H*W*D), 1], and the weight information T with a size of [B, C, (H*W*D), 1] is generated through the softmax activation function, and then it is reconstructed into the original size W with a size of [B, C, H, W, D]; the dot product of the result W and the 1×1×1 convolution result is obtained to obtain the multi-scale spatial attention result; a short connection is used to concatenate the input and output of the multi-scale convolution by channel and to normalize them;

[0073] Finally, a 1×1×1 convolution is used to adjust the number of channels to the same as the input, and the size of the result V is [B, C, H, W, D].

[0074] Wherein, an automatic expansion residual module is arranged on the original residual network; the automatic expansion residual module is composed of two branches, one branch has a convolution kernel size of 3×3×3, a convolution step of 1, a padding number of 1, and a relu activation function; the other branch automatically changes according to the input channel and the set output channel to determine;

[0075] If the input channel and the set output channel are equal, the branch does not perform any operation, and the automatic expansion residual module degenerates into a standard residual network; if the input channel and the set output channel are not equal, the branch performs convolution operation, the convolution kernel size is 1×1×1, and the activation function is relu; the final output is the sum of the results of the two branches;

[0076] The two automatic expansion residual modules are used in series; the first automatic expansion residual module changes the number of channels, and the second automatic expansion residual module does not change the number of channels.

[0077] Wherein, an encoding module and a decoding module are arranged in the multi-scale convolution spatial attention network.

[0078] The encoding module comprises a residual layer, a spatial attention layer and a pooling layer, which are connected in series; the residual layer is composed of two automatic expansion residual modules; the input channels of each stage are 2, 16, 32, 64 and 128 respectively, and the output channels are 16, 32, 64 and 128 respectively; the input channel and the output channel of the spatial attention layer remain consistent and the feature image size is unchanged; the convolution kernel size of the pooling layer is 2*2*2, and the convolution step is 2;

[0079] The decoding module comprises an inverse convolution layer, a residual layer and a spatial attention layer, which are connected in series; the convolution kernel size of the inverse convolution layer is 3*3*3, the convolution step is 2, and the padding is 1; the residual layer and the spatial attention layer are both composed of two automatic expansion residual modules connected in series, and the input and output channels and the size remain unchanged; the input channels of each stage are 256, 128, 64 and 32 respectively, and the output channels are 128, 64, 32 and 16 respectively.

[0080] The PET input channel and the CT input channel both adopt the U-Net structure, comprising five encoding modules and four decoding modules.

[0081] The convolution kernel size of the first input convolution layer of the multi-scale convolution spatial attention network is 5*5*5, the convolution kernel size of the last layer is 1*1*1, and the activation function is sigmoid; the activation functions of the remaining layers are all relu.

[0082] The Dice similarity coefficient DSC (Dice Similarity Coefficient, DSC) and the Hausdorff distance HD (Hausdorff distance, HD) are used as evaluation indexes; wherein:

[0083]

[0084] HD(X,Y)=max(h(X,Y),h(Y,X))

[0085]

[0086] The loss function is the Dice loss:

[0087]

[0088] The Focal loss function is also added to the loss function:

[0089]

[0090] The final loss function is:

[0091] Loss(X,Y)=Loss D(X, Y) + Loss F (X, Y)

[0092] where y and represent the tumor segmentation label and the segmentation prediction result of the tumor respectively; alpha is a balance weight factor, which is set to 0.5; gamma is a weight decay factor, which is set to 2;

[0093] The training process uses the Adam optimizer, the two momentum factors are set to 0.9 and 0.99 respectively, a total of 300 times of training, and the learning rate decay adopts the simulated annealing algorithm during the training process, and the learning rate is restarted every 25 times of training; the learning rate is initialized to 3e-4, and the minimum is 1e-6; finally, the neural network mapping model capable of realizing three-dimensional tumor automatic segmentation is obtained.

[0094] The spatial attention method for PET-CT multi-modal tumor segmentation of the application will be described in detail below with specific embodiments:

[0095] To solve the problem that the tumor automatic segmentation algorithm has poor effect, the application considers the difference and complementarity between various modal data, different input channels are designed, a specific spatial attention mechanism is combined to distinguish the feature information of different modal data, and the final segmentation result is obtained through feature fusion, thereby improving the segmentation performance.

[0096] The application is based on PET-CT multi-modal images, uses the method of spatial attention, considers the characteristics of PET and CT respectively, first extracts the feature information of the two by using the network alone, then fuses the feature information, especially the application of multi-scale convolution makes the network can highlight the tumor area, suppress the non-tumor area, and obtain more accurate tumor segmentation result. In the encoding stage, the network is divided into two symmetrical but non-shared weight input channels, which are used to extract the feature information of PET and CT respectively, and in the decoding stage, the application fuses the feature information extracted by the two, which can respectively use the high sensitivity of PET for tumor detection to locate the tumor position, and use the high accuracy of CT for structural information to determine the tumor boundary. In addition, in order to establish the connection of context information, the application also adds a skip connection between the encoding and decoding. The application significantly improves the network performance without significantly increasing the network complexity, and greatly improves the segmentation accuracy.

[0097] The specific operation steps of the application are as follows:

[0098] Step one: data preprocessing

[0099] According to the needs, the application resamples, crops and normalizes the inputted multi-modal data, first resamples the PET and CT to the resolution of 1mmx1mmx1mm, and then crops them to the size of 144x144x144. Before training the network, the CT image is normalized to [-1, 1] using the maximum and minimum value, and the PET image is normalized using the mean and variance.

[0100] Step two: design channel weight mask module

[0101] The channel weight mask (CWM) module includes a global average pooling layer, two fully connected layers and a sigmoid layer. The global average pooling maps the input with the shape format of [B, C, H, W, D] to [B, C, 1, 1, 1], compresses the last three dimensions, and the application obtains a weight vector with the shape of [B, C, 1]. Wherein B represents the batch size; C represents the input channel number; H, W and D represent the three-dimensional image size of the input. After the global average pooling, there are two fully connected layers, wherein the first fully connected layer has C input nodes and 2*C output nodes, and the second fully connected layer has 2*C input nodes and C output nodes. The second fully connected layer uses a sigmoid activation function to obtain the final channel weight mask. In order to apply the weight mask to the original input, the application adds a short connection to multiply the input and the channel weight mask, which completes the compression and expansion of the channel dimension feature information.

[0102] Step three: design multi-scale convolution spatial attention module

[0103] In the application, the input data used by the application is 3D, with the size of [B, C, H, W, D], and the designed spatial attention module is as follows Figure 1The input is first passed through a channel weight mask (CWM) module to complete the squeezing and expansion on the channel to obtain result X (size [B, C, H, W, D]). Then, multi-scale convolution is performed on the output result of the CWM, three different sizes of convolution kernels are used, 1x1x1, 3x3x3 and 5x5x5, the convolution step is 1, the padding is 0, 1 and 2 respectively, and the activation function is relu, so the size of the convolution result is unchanged. The convolution results of 3x3x3 and 5x5x5 are multiplied by matrix and the result U (size [B, C, H, W, D]) is flattened into a one-dimensional vector S (size [B, C, (H*W*D), 1]). After the softmax activation function generates weight information T (size [B, C, (H*W*D), 1]), it is reconstructed into the original size W (size [B, C, H, W, D]). The dot product of the result W and the 1x1x1 convolution result is obtained. In order to preserve the context information, a short connection is used to concatenate the input and output of the multi-scale convolution by channel and perform normalization (Batch Normalization, BN). Finally, a 1x1x1 convolution is used to adjust the number of channels to the same as the input, without using the activation function. The size of the result V is [B, C, H, W, D].

[0104] Step four: design an automatic expansion residual module

[0105] In the designed neural network, in order to automatically change the number of convolution kernels at different stages and prevent model degradation, the present application designs an automatic expansion residual module based on the original residual network. The module consists of two branches, one branch has a convolution kernel size of 3x3x3, a convolution step of 1, a padding number of 1, and a relu activation function. The other branch automatically changes according to the input channel and the set output channel. If the input channel and the set output channel are equal, the branch does not perform any operation, which is equivalent to a short connection, and the module degenerates into a standard residual network. If the input channel and the set output channel are not equal, the branch performs convolution operation, the convolution kernel size is 1x1x1, and the activation function is relu. The final output is the sum of the results of the two branches. In the experiment, the present application uses two automatic expansion residual modules in series. The first automatic expansion residual module changes the number of channels, and the second automatic expansion residual module does not change the number of channels.

[0106] Step five: design a multi-scale spatial attention encoding and decoding module

[0107] To achieve better results, the application also separately designs corresponding encoding and decoding modules. The encoding module includes a residual layer (ResNet), a spatial attention layer (ISA-Net), and a pooling layer (Pooling), which are connected in series (RIPM). The residual layer is composed of two automatically expanded residual modules; the input channels of each stage are 2, 16, 32, 64, and 128, respectively, and the output channels are 16, 32, 64, 128, and 256, respectively. The input and output channels of the spatial attention layer remain the same, and the feature image size remains unchanged; the convolution kernel size of the pooling layer is 2x2x2, and the convolution step is 2. The decoding module includes a deconvolution layer (DeConv), a residual layer (ResNet), and a spatial attention layer (ISA-Net), which are also connected in series (DRIM). The convolution kernel size of the deconvolution layer is 3x3x3, the convolution step is 2, and the padding is 1. The residual layer is also composed of two automatically expanded residual modules connected in series, and the input and output channels and sizes remain unchanged. The spatial attention layer is also the same. The input channels of each stage are 256, 128, 64, and 32, respectively, and the output channels are 128, 64, 32, and 16, respectively.

[0108] Step six: design the overall network structure

[0109] Based on the RIPM and DRIM encoding and decoding modules, the application designs a network structure as shown in Figure 2 The network input is divided into PET input channels and CT input channels. Each channel uses a U-Net structure, including 5 encoding modules and 4 decoding modules, and the structures of the two channels are symmetrical but do not share weight parameters. The two channels are separately encoded, and in the decoding stage, the results of each decoding module of the PET channel are respectively fused with the CT channel as the final output result. The convolution kernel size of the last layer of the network is 1x1x1, and the activation function is sigmoid. The activation functions of the remaining layers are all relu. The convolution kernel size of the first input convolution layer of the network is 5x5x5, which is convenient for obtaining a larger receptive field. The network parameter settings are shown in Table 1.

[0110]

[0111] Table 1

[0112] Step seven: design the loss function and evaluation index

[0113] In the present application, the Dice similarity coefficient DSC (Dice similarity coefficient, DSC) and Hausdorff distance HD (Hausdorff distance, HD) are used as evaluation indexes. Among them:

[0114]

[0115] HD(X,Y)=max(h(X,Y),h(Y,X))

[0116]

[0117] The loss function is the Dice loss:

[0118]

[0119] In order to solve the problem of sample imbalance, the Focal loss function is also added in the loss function:

[0120]

[0121] The final loss function is:

[0122] Loss(X,Y)=Loss D (X,Y)+Loss F (X,Y)

[0123] Where y and represent the tumor segmentation label and the segmentation prediction result of the tumor respectively. Alpha is a balance weight factor, and the value is set to 0.5; Gamma is a weight decay factor, and is set to 2. Referring to Figure 3 is the segmentation result image of the network, wherein the first row and the first two columns are CT and PET inputs respectively, and the third and fourth columns are tumor labels and segmentation results of the network of the application respectively; the second row is the segmentation results of four comparison algorithms. Referring to Figure 4 is the boundary segmentation result image of the tumor, wherein the circles from the outside to the inside represent the segmentation results of V-Net, 3D U-Net, SE-Net, SK-Net, the network of the application, and the tumor label in turn.

[0124] Step eight: training the network

[0125] The training process uses the Adam optimizer, and the two momentum factors are set to 0.9 and 0.99 respectively, and a total of 300 times are trained, and the learning rate is attenuated in the training process by using the simulated annealing algorithm, and the learning rate is restarted every 25 times of training. The learning rate is initialized to 3e-4, and the minimum is 1e-6. Finally, a neural network mapping model capable of realizing three-dimensional tumor automatic segmentation is obtained.

[0126] Embodiment 2

[0127] According to another embodiment of the application, a spatial attention device for PET-CT multi-modal tumor segmentation is provided, comprising:

[0128] The feature information extraction unit is configured to set separate encoded PET input channels and CT input channels in the multi-scale convolution space attention network, input a PET image in a PET-CT multi-modal image to the PET input channels to extract PET feature information, and input a CT image in the PET-CT multi-modal image to the CT input channels to extract CT feature information.

[0129] The fusion unit is configured to fuse the PET decoding result and the CT decoding result.

[0130] The spatial attention device for PET-CT multi-modal tumor segmentation in the embodiment of the application is based on a PET-CT multi-modal image, uses a spatial attention method, considers the characteristics of PET and CT respectively, separately extracts the feature information of PET and CT by using a network, fuses the feature information, and especially the application of multi-scale convolution enables the network to highlight a tumor region and suppress a non-tumor region, so that a more accurate tumor segmentation result is obtained.

[0131] The spatial attention device for PET-CT multi-modal tumor segmentation in the embodiment of the application is based on a PET-CT multi-modal image, uses a spatial attention method, considers the characteristics of PET and CT respectively, separately extracts the feature information of PET and CT by using a network, fuses the feature information, and especially the application of multi-scale convolution enables the network to highlight a tumor region and suppress a non-tumor region, so that a more accurate tumor segmentation result is obtained.

[0132] To solve the problem of poor effect of a tumor automatic segmentation algorithm, the application considers the difference and complementarity between various modal data, different input channels are designed, a specific spatial attention mechanism is combined to distinguish the feature information of different modal data, and a final segmentation result is obtained through feature fusion, so that the performance of segmentation is improved.

[0133] The spatial attention device for PET-CT multi-modal tumor segmentation in the embodiment of the application is based on a PET-CT multi-modal image, uses a spatial attention method, considers the characteristics of PET and CT respectively, separately extracts the feature information of PET and CT by using a network, fuses the feature information, and especially the application of multi-scale convolution enables the network to highlight a tumor region and suppress a non-tumor region, so that a more accurate tumor segmentation result is obtained.

[0131] The spatial attention device for PET-CT multi-modal tumor segmentation in the embodiment of the application is based on a PET-CT multi-modal image, uses a spatial attention method, considers the characteristics of PET and CT respectively, separately extracts the feature information of PET and CT by using a network, fuses the feature information, and especially the application of multi-scale convolution enables the network to highlight a tumor region and suppress a non-tumor region, so that a more accurate tumor segmentation result is obtained.

[0132] To solve the problem of poor effect of a tumor automatic segmentation algorithm, the application considers the difference and complementarity between various modal data, different input channels are designed, a specific spatial attention mechanism is combined to distinguish the feature information of different modal data, and a final segmentation result is obtained through feature fusion, so that the performance of segmentation is improved.

[0133] The spatial attention device for PET-CT multi-modal tumor segmentation in the embodiment of the application is based on a PET-CT multi-modal image, uses a spatial attention method, considers the characteristics of PET and CT respectively, separately extracts the feature information of PET and CT by using a network, fuses the feature information, and especially the application of multi-scale convolution enables the network to highlight a tumor region and suppress a non-tumor region, so that a more accurate tumor segmentation result is obtained.

[0134] The specific operation steps of the present application are as follows:

[0135] Step one: data preprocessing

[0136] According to the needs, the present application resamples, crops and normalizes the input multi-modal data, first resamples the PET and CT to a resolution of 1mmx1mmx1mm, and then crops them to a size of 144x144x144. Before training the network, the CT image is normalized to [-1, 1] using the maximum and minimum value, and the PET image is normalized using the mean and variance.

[0137] Step two: design channel weight mask module

[0138] The channel weight mask (CWM) module includes a global average pooling layer, two fully connected layers and a sigmoid layer. The global average pooling maps the input with shape format [B, C, H, W, D] to [B, C, 1, 1, 1], compressing the last three dimensions, and the present application obtains a weight vector with shape [B, C, 1]. Wherein B represents batch size; C represents the number of input channels; H, W, D represent the size of the three-dimensional image input. After global average pooling, there are two fully connected layers, of which the first fully connected layer has C input nodes and 2*C output nodes, and the second fully connected layer has 2*C input nodes and C output nodes. The second fully connected layer uses a sigmoid activation function to obtain the final channel weight mask. In order to apply the weight mask to the original input, the present application adds a short connection to multiply the input and the channel weight mask, which completes the compression and expansion of the channel dimension feature information.

[0139] Step three: design multi-scale convolution spatial attention module

[0140] In the present application, the input data used in the present application are all 3D, with a size of [B, C, H, W, D], and the spatial attention module designed is as follows: Figure 1The input is first passed through a channel weight mask (CWM) module to complete the squeezing and expansion on the channel to obtain result X (size [B, C, H, W, D]). Then, multi-scale convolution is performed on the output result of the CWM, three different sizes of convolution kernels are used, 1x1x1, 3x3x3 and 5x5x5, the convolution step is 1, the padding is 0, 1 and 2 respectively, and the activation function is relu, so the size of the convolution result is unchanged. The convolution results of 3x3x3 and 5x5x5 are multiplied by matrix and the result U (size [B, C, H, W, D]) is flattened into a one-dimensional vector S (size [B, C, (H*W*D), 1]). After the softmax activation function generates weight information T (size [B, C, (H*W*D), 1]), it is reconstructed into the original size W (size [B, C, H, W, D]). The dot product of the result W and the 1x1x1 convolution result is obtained. In order to preserve the context information, a short connection is used to concatenate the input and output of the multi-scale convolution by channel and perform normalization (Batch Normalization, BN). Finally, a 1x1x1 convolution is used to adjust the number of channels to the same as the input, without using the activation function. The size of the result V is [B, C, H, W, D].

[0141] Step four: design an automatic expansion residual module

[0142] In the designed neural network, in order to automatically change the number of convolution kernels at different stages and prevent model degradation, the present application designs an automatic expansion residual module based on the original residual network. The module consists of two branches, one branch has a convolution kernel size of 3x3x3, a convolution step of 1, a padding number of 1, and a relu activation function. The other branch automatically changes according to the input channel and the set output channel. If the input channel and the set output channel are equal, the branch does not perform any operation, which is equivalent to a short connection, and the module degenerates into a standard residual network. If the input channel and the set output channel are not equal, the branch performs convolution operation, the convolution kernel size is 1x1x1, and the activation function is relu. The final output is the sum of the results of the two branches. In the experiment, the present application uses two automatic expansion residual modules in series. The first automatic expansion residual module changes the number of channels, and the second automatic expansion residual module does not change the number of channels.

[0143] Step five: design a multi-scale spatial attention encoding and decoding module

[0144] To achieve better results, the application also separately designs corresponding encoding and decoding modules. The encoding module includes a residual layer (ResNet), a spatial attention layer (ISA-Net), and a pooling layer (Pooling), which are connected in series (RIPM). The residual layer is composed of two automatically expanded residual modules; the input channels of each stage are 2, 16, 32, 64, and 128, respectively, and the output channels are 16, 32, 64, 128, and 256, respectively. The input and output channels of the spatial attention layer remain the same, and the feature image size remains unchanged; the convolution kernel size of the pooling layer is 2x2x2, and the convolution step is 2. The decoding module includes a deconvolution layer (DeConv), a residual layer (ResNet), and a spatial attention layer (ISA-Net), which are also connected in series (DRIM). The convolution kernel size of the deconvolution layer is 3x3x3, the convolution step is 2, and the padding is 1. The residual layer is also composed of two automatically expanded residual modules connected in series, and the input and output channels and sizes remain unchanged. The spatial attention layer is also the same. The input channels of each stage are 256, 128, 64, and 32, respectively, and the output channels are 128, 64, 32, and 16, respectively.

[0145] Step six: design the overall network structure

[0146] Based on the RIPM and DRIM encoding and decoding modules, the application designs a network structure as shown in Figure 2 The network input is divided into PET input channels and CT input channels. Each channel uses a U-Net structure, which includes 5 encoding modules and 4 decoding modules, and the structures of the two channels are symmetrical but do not share weight parameters. The two channels are separately encoded, and in the decoding stage, the results of each decoding module of the PET channel are respectively fused with the CT channel as the final output result. The convolution kernel size of the last layer of the network is 1x1x1, and the activation function is sigmoid. The activation functions of the remaining layers are all relu. The convolution kernel size of the first input convolution layer of the network is 5x5x5, which is convenient for obtaining a larger receptive field. The network parameter settings are shown in Table 1.

[0147]

[0148] Table 1

[0149] Step seven: design the loss function and evaluation index

[0150] In the present application, the Dice similarity coefficient DSC (Dice similarity coefficient, DSC) and Hausdorff distance HD (Hausdorff distance, HD) are used as evaluation indexes. Among them:

[0151]

[0152] HD(X,Y) = max(h(X,Y),h(Y,X))

[0153]

[0154] The loss function is the Dice loss:

[0155]

[0156] To solve the problem of sample imbalance, the Focal loss function is also added to the loss function:

[0157]

[0158] The final loss function is:

[0159] Loss(X,Y) = Loss D (X,Y) + Loss F (X,Y)

[0160] Where y and represent the tumor segmentation label and the segmentation prediction result of the tumor respectively. Alpha is a balance weight factor, and the value is set to 0.5; Gamma is a weight decay factor, and is set to 2. See Figure 3 is the segmentation result of the network, where the first row, the first and second columns are CT and PET inputs, the third and fourth columns are tumor labels and segmentation results of the network of the application respectively; the second row is the segmentation results of four comparison algorithms. See Figure 4 is the boundary segmentation result of the tumor, where the circles from the outside to the inside represent the segmentation results of V-Net, 3D U-Net, SE-Net, SK-Net, the network of the application, and the tumor label in turn.

[0161] Step eight: training the network

[0162] The training process uses the Adam optimizer, and the two momentum factors are set to 0.9 and 0.99 respectively, and a total of 300 times are trained. The learning rate decay in the training process adopts the simulated annealing algorithm, and the learning rate is restarted every 25 times of training. The learning rate is initialized to 3e-4, and the minimum is 1e-6. Finally, the neural network mapping model capable of realizing three-dimensional tumor automatic segmentation is obtained.

[0163] Example 3

[0164] A storage medium, the storage medium stores a program file capable of realizing the above-mentioned any one of the spatial attention methods for PET-CT multi-modal tumor segmentation.

[0165] Example 4

[0166] A processor for running a program, wherein the program performs the spatial attention method for PET-CT multi-modal tumor segmentation of any one of the above when running.

[0167] The automatic residual expansion module of the application completes feature channel conversion and prevents model degradation; the multi-scale convolution spatial attention module extracts feature information, enriching feature diversity and accuracy; the RIPM module and the DRIM module respectively complete original feature encoding and fused feature decoding; the double-channel non-complete symmetric network respectively realizes PET and CT feature extraction and fusion, improving the accuracy of the segmentation result.

[0168] The application fuses the features extracted by the PET image and the CT image by considering the difference and complementarity of the two, the PET channel mainly extracts the position information of the tumor, and the CT channel mainly extracts the boundary information of the tumor. Combined with the multi-scale convolution-based spatial attention module, the tumor region can be effectively highlighted, and the non-tumor region can be suppressed, thereby improving the accuracy of tumor segmentation. In addition to being applied to PET-CT multi-modal tumor segmentation, the application can also be applied to PET-MRI multi-modal tumor segmentation, and after appropriate deformation, the application can also be used for single CT or PET or MRI tumor segmentation.

[0169] The above-mentioned embodiment numbers of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0170] In the above-mentioned embodiments of the application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0171] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the system embodiments described above are only schematic, for example, the division of units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0172] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0173] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0174] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0175] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A spatial attention method for PET-CT multimodal tumor segmentation, characterized in that, The method comprises the following steps: The PET input channel and the CT input channel are two symmetric but non-shared weight input channels. Before the PET-CT multi-modal image is input into the input channel, the method further comprises: The PET-CT multi-modal image is resampled, cropped and normalized, specifically: First, the PET and CT multi-modal images are resampled to a resolution of 1mm*1mm*1mm, and then cropped to a size of 144*144*144; then the CT multi-modal image is normalized to [-1, 1] using the maximum and minimum value, and the PET multi-modal image is normalized using the mean and variance. Before the PET decoding result and the CT decoding result are fused, the method further comprises:

2. The spatial attention method for PET-CT multimodality tumor segmentation according to claim 1, wherein, A channel weight mask module is arranged in the PET input channel and the CT input channel, and the channel weight mask module comprises a global average pooling layer, two fully connected layers and a sigmoid layer; 3. The spatial attention method for PET-CT multimodality tumor segmentation according to claim 1, wherein, The global average pooling maps the input with a shape format of [B, C, H, W, D] to [B, C, 1, 1, 1], compresses the last three dimensions, and obtains a weight vector with a shape of [B, C, 1]; wherein B represents the batch size; C represents the number of input channels; H, W and D represent the three-dimensional image size of the input; After the global average pooling, there are two fully connected layers, wherein the first fully connected layer has C input nodes and 2*C output nodes, and the second fully connected layer has 2*C input nodes and C output nodes; the second fully connected layer uses a sigmoid activation function to obtain the final channel weight mask; 4. The spatial attention method for PET-CT multimodality tumor segmentation according to claim 3, wherein, ​ ​ ​ ​ A short connection is added between encoding and decoding, and the input is multiplied by a channel weight mask to compress and expand the channel dimension feature information.

5. The spatial attention method for PET-CT multimodality tumor segmentation according to claim 4, wherein, The input data size used is [B, C, H, W, D], and the input is first passed through a channel weight mask module to complete the squeezing and expansion on the channel to obtain a result X with a size of [B, C, H, W, D]; Then, multi-scale convolution is performed on the output result of the channel weight mask module, using three different size convolution kernels, 1×1×1, 3×3×3 and 5×5×5, with a convolution step size of 1 and padding of 0, 1 and 2 respectively, and the activation function is relu; the convolution results of 3×3×3 and 5×5×5 are multiplied by matrix, and the result U with a size of [B, C, H, W, D] is flattened into a one-dimensional vector S with a size of [B, C, (H*W*D), 1], and the weight information T with a size of [B, C, (H*W*D), 1] is generated through the softmax activation function, and then reconstructed into the original size W with a size of [B, C, H, W, D]; the dot product of the result W and the 1×1×1 convolution result is obtained to obtain the multi-scale spatial attention result; A short connection is used to concatenate the input and output of the multi-scale convolution by channel and to normalize it; Finally, a 1×1×1 convolution is used to adjust the number of channels to the same as the input, and the size of the result V is [B, C, H, W, D].

6. The spatial attention method for PET-CT multimodality tumor segmentation according to claim 5, wherein, An automatic expansion residual module is set on the original residual network; the automatic expansion residual module consists of two branches, one branch with a convolution kernel size of 3×3×3, a convolution step size of 1, a padding number of 1, and a relu activation function; the other branch automatically changes according to the input channel and the set output channel; If the input channel and the set output channel are equal, the branch does not perform any operation, and the automatic expansion residual module degenerates into a standard residual network; if the input channel and the set output channel are not equal, the branch performs convolution operation with a convolution kernel size of 1×1×1 and a relu activation function; the final output is the sum of the results of the two branches; The two automatic expansion residual modules are used in series; the first automatic expansion residual module changes the number of channels, and the second automatic expansion residual module does not change the number of channels.

7. The spatial attention method for PET-CT multimodality tumor segmentation according to claim 6, characterized in that, The PET input channel and the CT input channel both use U-Net structure, which includes 5 encoding modules and 4 decoding modules; The convolution kernel size of the first input convolution layer of the multi-scale convolution spatial attention network is 5×5×5, and the convolution kernel size of the last layer is 1×1×1, and the activation function is sigmoid; the activation functions of the remaining layers are all relu.

8. The spatial attention method for PET-CT multimodality tumor segmentation according to claim 7, wherein, The Dice similarity coefficient DSC and the Hausdorff distance HD are used as evaluation indexes; wherein: HD(X, Y) = max(h(X, Y), h(Y, X)) The loss function is the Dice loss: The Focal loss function is also added to the loss function: The final loss function is: Loss(X,Y) = Loss D (X,Y) + Loss F (X,Y) where y and represent the tumor segmentation label and the segmentation prediction result of the tumor, respectively; a is a balance weight factor, which is set to 0.5; and g is a weight decay factor, which is set to 2. The training process uses an Adam optimizer, two momentum factors are set to 0.9 and 0.99 respectively, a total of 300 times of training, and the learning rate is decayed in the training process by using a simulated annealing algorithm, and the learning rate is restarted every 25 times of training; the learning rate is initialized to 3e-4, and the minimum is 1e-6; finally, a neural network mapping model capable of realizing automatic segmentation of three-dimensional tumors is obtained.

9. A spatial attention device for PET-CT multi-modal tumor segmentation using the spatial attention method for PET-CT multi-modal tumor segmentation of claim 1, characterized in that, Comprise: The feature information extraction unit is used for setting separate encoded PET input channels and CT input channels in the multi-scale convolutional spatial attention network, inputting the PET image in the PET-CT multi-modal image to the PET input channel to extract the PET feature information, and inputting the CT image in the PET-CT multi-modal image to the CT input channel to extract the CT feature information; The fusion unit is used for fusing the PET decoding result and the CT decoding result.

Images