Liver Tumor Identification Method and Device Based on Local-Global Spatiotemporal Feature Fusion

Abstract

The present invention provides a method and device for liver tumor discrimination based on the fusion of local-global spatio-temporal features. The method includes: segmenting an abdominal region image from an input medical image; inputting the abdominal region image into a convolutional neural network for local feature extraction; inputting the extracted local features into an attention mechanism module for global feature extraction; encoding the extracted global features into a pulse sequence and then inputting the pulse sequence into a spiking neural network for spatio-temporal feature extraction; fusing the extracted local features, global features and spatio-temporal features, and outputting the liver tumor category based on the fused features. The present invention uses a spiking neural network for spatio-temporal feature extraction and discriminates the liver tumor category based on the fusion of local features, global features and spatio-temporal features, which significantly improves the discrimination accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical imaging, and particularly relates to a method and device for differentiating liver tumors based on the fusion of local-global spatio-temporal features. Background Art

[0002] Malignant liver tumors include primary tumors and metastatic tumors, etc. In imaging, primary liver cancer can be single or multiple lesions with an enhancement feature of "fast in and fast out"; while metastatic liver cancer mostly presents as multiple, round or oval-shaped lesions, often with unclear enhancement. Clinically, the treatment strategies for different liver malignant tumors vary greatly, and it is crucial for doctors to accurately diagnose the tumor type of patients before treatments such as surgery, targeted therapy, and immunotherapy. With the rapid development of deep learning technology, the field of computer-aided diagnosis (CAD) has developed rapidly, especially in the fields of medical image segmentation and classification. The disadvantage of single-phase CT scans is that it is difficult to accurately locate the contour of organs, and referring to different phases is an effective strategy to identify the organ boundary as completely as possible. For example, many guidelines clearly recommend using a three-phase (arterial phase, portal venous phase, and equilibrium phase) liver protocol for contrast-enhanced CT imaging.

[0003] Currently, there are two difficulties in liver tumor classification. Firstly, for commonly used convolutional neural networks, due to the locality of convolutional operations, they cannot handle long-distance relationships well. And multi-phase liver CT has explicit and important long-distance dependencies compared with natural images. On the other hand, although existing Transformer-based network architectures can supplement the global information lost by convolutional operations, it is still difficult to capture the spatio-temporal information of medical images. To address the above problems, the present invention utilizes the advantages of spiking neural networks and uses a brand-new architecture for tumor classification. This model uses a convolutional neural network to extract local features of multi-phase CT images, inputs the local features into an attention-based network to extract global features in the sequence relationship, then inputs the global features into a spiking neural network to extract features containing spatio-temporal information, and differentiates primary liver cancer and metastatic liver cancer based on the fusion of the three types of features. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a method and device for differentiating liver tumors based on the fusion of local-global spatio-temporal features.

[0005] To achieve the above object, the present invention adopts the following technical solutions.

[0006] In the first aspect, the present invention provides a method for differentiating liver tumors based on the fusion of local-global spatio-temporal features, including the following steps:

[0007] Segment the abdominal region image from the input medical image;

[0008] Input the abdominal region image into a convolutional neural network for local feature extraction;

[0009] Input the extracted local features into an attention mechanism module for global feature extraction;

[0010] Encode the extracted global features into a pulse sequence and then input them into a spiking neural network for spatio-temporal feature extraction;

[0011] Fuse the extracted local features, global features, and spatio-temporal features, and output the liver tumor categories based on the fused features: primary cancer and metastatic cancer.

[0012] Furthermore, the input medical image is a multi-phase CT image including an arterial phase, a portal venous phase, and a equilibrium phase.

[0013] Furthermore, the method for segmenting the abdominal region image from the input medical image includes: optimizing the abdominal region boundary using reinforcement learning.

[0014] Furthermore, the attention mechanism module is a Transformer module.

[0015] Furthermore, the fused features are obtained by weighted summation of the extracted local features, global features, and spatio-temporal features; the weights of the local features, global features, and spatio-temporal features are determined through model training.

[0016] In a second aspect, the present invention provides a liver tumor discrimination device based on local-global-spatio-temporal feature fusion, including:

[0017] An abdominal image segmentation module for segmenting an abdominal region image from the input medical image;

[0018] A local feature extraction module for inputting the abdominal region image into a convolutional neural network for local feature extraction;

[0019] A global feature extraction module for inputting the extracted local features into an attention mechanism module for global feature extraction;

[0020] A spatio-temporal feature extraction module for encoding the extracted global features into a pulse sequence and then inputting them into a spiking neural network for spatio-temporal feature extraction;

[0021] A tumor category output module for fusing the extracted local features, global features, and spatio-temporal features and outputting the liver tumor categories based on the fused features: primary cancer and metastatic cancer.

[0022] Furthermore, the input medical image is a multi-phase CT image including an arterial phase, a portal venous phase, and a equilibrium phase.

[0023] Further, the abdominal image segmentation module is specifically configured to optimize the boundary of the abdominal region by using reinforcement learning.

[0024] Further, the attention mechanism module is a Transformer module.

[0025] Further, the fused feature is obtained by weighted summation of the extracted local features, global features, and spatio-temporal features; the weights of the local features, global features, and spatio-temporal features are determined through model training.

[0026] Compared with the prior art, the present invention has the following beneficial effects.

[0027] In the present invention, the abdominal region image is segmented from the input medical image, the abdominal region image is input into a convolutional neural network for local feature extraction, the extracted local features are input into an attention mechanism module for global feature extraction, the extracted global features are encoded into a pulse sequence and then input into a spiking neural network for spatio-temporal feature extraction, and the liver tumor category is output based on the fusion of the extracted local features, global features, and spatio-temporal features, realizing the automatic identification of liver tumors. The present invention uses a spiking neural network for spatio-temporal feature extraction and discriminates the liver tumor category based on the fusion of local features, global features, and spatio-temporal features, significantly improving the discrimination accuracy. Description of the Drawings

[0028] Figure 1 It is a flowchart of a method for identifying liver tumors based on the fusion of local, global, and spatio-temporal features according to an embodiment of the present invention.

[0029] Figure 2 It is a schematic flowchart of another embodiment of the present invention.

[0030] Figure 3 It is a schematic structural diagram of the Transformer module.

[0031] Figure 4 It is a block diagram of a device for identifying liver tumors based on the fusion of local, global, and spatio-temporal features according to an embodiment of the present invention. Detailed Embodiments

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0033] Figure 1The following is a flowchart of a method for differentiating liver tumors based on the fusion of local-global spatio-temporal features according to an embodiment of the present invention, including the following steps:

[0034] Step 101: Segment the abdominal region image from the input medical image;

[0035] Step 102: Input the abdominal region image into a convolutional neural network for local feature extraction;

[0036] Step 103: Input the extracted local features into an attention mechanism module for global feature extraction;

[0037] Step 104: Encode the extracted global features into a pulse sequence and then input them into a spiking neural network for spatio-temporal feature extraction;

[0038] Step 105: Fuse the extracted local features, global features, and spatio-temporal features, and output the liver tumor categories based on the fused features: primary cancer and metastatic cancer.

[0039] In this embodiment, step 101 is mainly used to segment the abdominal region image from the input medical image. There are many types of images included in medical images. The medical images in this embodiment include, but are not limited to, CT (Computed Tomography) images and MRI (Magnetic Resonance Imaging) images. In order to reduce the processing burden on the image, it is necessary to first segment and crop the original input image to crop out the abdominal region image including the liver. The segmentation of the abdominal region image can be obtained by inputting the original input image into a trained image segmentation model. The targets determined by the segmentation model are the boundaries between the chest and abdomen, and the boundaries between the abdomen and pelvis. The labels of the training data of the segmentation model are the slice numbers of the boundaries between the chest and abdomen, and the boundaries between the abdomen and pelvis of the input medical image.

[0040] In this embodiment, step 102 is mainly used for local feature extraction. In this embodiment, the abdominal region image obtained in step 101 is input into a convolutional neural network CNN (Convolutional Neural Networks) to extract local features. In the visual nervous system, the receptive field of a neuron refers to a specific area on the retina, and only the stimuli within this area can activate the neuron. CNN is proposed based on this receptive field mechanism in biology. CNN is a feedforward neural network, but different from the general fully connected feedforward neural network, its convolutional layer has the characteristics of local connection and weight sharing, so it can greatly reduce the number of weight parameters, thereby reducing the complexity of the model and improving the running speed. A typical CNN is composed of convolutional layers, pooling layers (or downsampling layers), and fully connected layers stacked crosswise. The role of the convolutional layer is to extract the features of a local area through the convolution operation of the convolution kernel and the input image. Different convolution kernels are equivalent to different feature extractors. The role of the pooling layer is to perform feature selection and reduce the number of features, thereby further reducing the number of parameters. Generally, the maximum pooling method and the average pooling method are adopted. The fully connected layer is used to fuse the different features obtained. General CNNs are very suitable for the extraction of local detail features.

[0041] In this embodiment, step 103 is mainly used for global feature extraction. In this embodiment, an attention mechanism module is used for global feature extraction. A single attention mechanism module is suitable for extracting regions of interest, and the combination of multiple attention mechanism modules can be used to extract global features containing context information. For example, the multi-head attention mechanism module or the combination of the self-attention mechanism module and the multi-head attention mechanism module is often used to extract global features. The attention mechanism, under the limited computer capacity, draws on the attention mechanism of the human brain and only focuses on some key information inputs for processing to improve the efficiency of the neural network. The calculation of the attention mechanism can be divided into two steps: one is to calculate the attention distribution on all input information; the other is to calculate the weighted sum of the input information according to the attention distribution. The weighting coefficient is the attention distribution α i = softmax(f i W att q), indicating the degree of correlation between the input vector f i and the query vector q. Therefore, the attention mechanism can extract global features from the input image. The self-attention mechanism is a variant of the attention mechanism. It has only one input and calculates the attention between each unit in the sequence and all units in the sequence, reducing the dependence on external information and being more adept at capturing the internal correlation of data or features. The self-attention mechanism linearly maps the input into K, Q, and V using three learnable variables, then multiplies Q and K to obtain the attention weight of K to Q, and then uses the normalized weight and V to generate the context information of the input. The calculation formula is

[0042] In this embodiment, step 104 is mainly used to extract spatio-temporal features using a Spiking Neural Network (SNN). Due to the locality of common convolutional neural network operations, it cannot handle long-range relationships well, that is, it is only suitable for extracting local feature information; while some existing network architectures (such as Transformer) can supplement the global information lost by convolutional operations, but it is still difficult to capture the spatio-temporal information of medical images. To address the above problems, this embodiment uses a spiking neural network to extract spatio-temporal feature information. The spiking neural network is a new generation of artificial neural network model inspired by biology, belonging to the subset of deep learning, and has strong biological foundation support. The design idea of the spiking neural network is that neurons in the dynamic neural network are not activated in every iteration propagation, but are activated when their membrane potential reaches a certain specific value. When a neuron is activated, it generates a signal to be transmitted to other neurons, increasing or decreasing their membrane potential. The neuron model of the spiking neural network is Izhikevich. Izhikevich proposed a two-dimensional spiking neuron, and its calculation is as follows:

[0043]

[0044]

[0045] if v>v peak ,then v←c,u←u+d

[0046] The meanings of the parameters in the formula are as follows in the table:

[0047] Parameter Meaning Typical value Unit a The time constant of u, used for the slow recovery of the membrane potential 0.02 <![CDATA[ms -1 > b The dependence parameter of u on v 0.2 <![CDATA[10 -9 Ω -1 > c The value of reset after the pulse -65 mv d The increase degree of u after reset 2 S Stimulation, including excitatory, inhibitory, internal, noise, etc pA v Membrane potential mv u Recovery variable pA <![CDATA[v t > Instantaneous threshold potential mv <![CDATA[v r > Resting potential mv

[0048] The spiking neural network requires an input pulse sequence, so it is necessary to first encode the extracted global features into a pulse sequence. There are many encoding methods for image features, and the commonly used ones include threshold encoding method, Gaussian difference method, frequency encoding method, etc.

[0049] In this embodiment, step 105 is mainly used to output the liver tumor category. In this embodiment, by fusing the extracted local features, global features and spatio-temporal features, the liver tumor category is output based on the fused features, as Figure 3 shown. The liver tumor categories output in this embodiment include primary cancer and metastatic cancer, so the liver tumor category can be obtained by inputting the fused features into a trained binary classifier.

[0050] In this embodiment, local features and global features are respectively extracted from the input image. In particular, spatio-temporal features are extracted through a spiking neural network, and the liver tumor category is identified based on the fused features of the three features, which can significantly improve the accuracy of liver tumor category identification.

[0051] As an alternative embodiment, the input medical image is a multi-phase CT image including an arterial phase, a portal venous phase, and a equilibrium phase.

[0052] This embodiment defines the input medical image. The medical image in this embodiment is a CT image. Considering that the information volume of a single-phase CT image is small and it is difficult to accurately locate the contour of an organ; while multi-phase CT images of different phases have the advantages of complementary advantages and large information volume, and it is easy to obtain a complete recognition of the organ boundary. The medical image in this embodiment uses a multi-phase CT image including an arterial phase, a portal venous phase, and a equilibrium phase. In fact, many guidelines clearly recommend using a three-phase (arterial phase, portal venous phase, and equilibrium phase) liver protocol for CT contrast-enhanced imaging. The information of the three phases is important in clinical diagnosis, especially for the diagnosis of cancer categories. Among them, the arterial phase helps to detect tumors, and the portal venous phase helps to show the invasion of surrounding structures. The information of the three phases can provide a reliable imaging basis for clinical diagnosis.

[0053] As an alternative embodiment, the method for segmenting the abdominal region image from the input medical image includes: optimizing the boundary of the abdominal region by using reinforcement learning.

[0054] This embodiment provides a technical solution for segmenting the abdominal region image. In this embodiment, the image segmentation model for segmenting the abdominal region image uses a reinforcement learning framework to let the network learn how to recommend the dividing line. The network that has completed the learning can automatically determine the optimal dividing line. Reinforcement learning includes 5 core components: Environment, Agent, State, Action, Reward. Reinforcement learning regards learning as a trial and evaluation process. The agent selects an action to be used in the environment. After the environment accepts the action, the state changes, and at the same time, a reinforcement signal (reward value) is generated and fed back to the agent. The agent then selects the next action based on the reinforcement signal and the current state of the environment. The selection principle is to maximize the positive reward value.

[0055] As an alternative embodiment, the attention mechanism module is a Transformer module.

[0056] This embodiment provides a specific network architecture for extracting global features. In this embodiment, the main Transformer module is used to extract global features. The Transformer module is stacked by multiple Transformer blocks as Figure 3As shown in the figure. The Transformer abandons the traditional CNN and RNN (Recurrent Neural Network) structures, and the entire network structure is almost entirely composed of attention mechanisms. Compared with the traditional self-attention mechanism, the multi-head self-attention mechanism is added, further improving the performance of the self-attention mechanism. The schematic diagram of the Transformer structure is as Figure 3 shown, including layer normalization (LN), multi-head self-attention (MSA), and multi-layer perception (MLP).

[0057] As an alternative embodiment, the fused feature is obtained by weighted summation of the extracted local features, global features, and spatio-temporal features; the weights of the local features, global features, and spatio-temporal features are determined through model training.

[0058] This embodiment provides a technical solution for fusing local features, global features, and spatio-temporal features. In this embodiment, feature fusion is achieved by weighted summation of the three features. A fully connected layer can be used to limit the final feature dimensions of the three features to the same size, and the learnable weights w1, w2, and w3 are assigned to the three features, and the contribution degree of each feature is adaptively adjusted through network training.

[0059] Figure 4 This is a schematic diagram of the composition of a liver tumor discrimination device based on local-global-spatio-temporal feature fusion according to an embodiment of the present invention. The device includes:

[0060] An abdominal image segmentation module 11, configured to segment an abdominal region image from the input medical image;

[0061] A local feature extraction module 12, configured to input the abdominal region image into a convolutional neural network for local feature extraction;

[0062] A global feature extraction module 13, configured to input the extracted local features into an attention mechanism module for global feature extraction;

[0063] A spatio-temporal feature extraction module 14, configured to encode the extracted global features into a pulse sequence and then input it into a spiking neural network for spatio-temporal feature extraction;

[0064] A tumor category output module 15, configured to fuse the extracted local features, global features, and spatio-temporal features, and output the liver tumor categories: primary cancer and metastatic cancer, based on the fused features.

[0065] The device of this embodiment can be used to execute Figure 1 the technical solution of the method embodiment shown. The implementation principle and technical effect are similar, and will not be elaborated here. The same is true for the subsequent embodiments, and no further explanation will be given.

[0066] As an optional embodiment, the input medical image is a multi-phase CT image including an arterial phase, a portal venous phase, and a equilibrium phase.

[0067] As an optional embodiment, the abdominal image segmentation module 11 is specifically configured to: optimize the boundary of the abdominal region by using reinforcement learning.

[0068] As an optional embodiment, the attention mechanism module is a Transformer module.

[0069] As an optional embodiment, the fused feature is obtained by weighted summation of the extracted local feature, global feature, and spatio-temporal feature; the weights of the local feature, global feature, and spatio-temporal feature are determined through model training.

[0070] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for differentiating liver tumors based on the fusion of local-global spatio-temporal features, characterized in that, It includes the following steps: Segment the abdominal region image from the input medical image; Input the abdominal region image into a convolutional neural network for local feature extraction; Input the extracted local features into an attention mechanism module for global feature extraction; Encode the extracted global features into a pulse sequence and then input it into a spiking neural network for spatio-temporal feature extraction; Fuse the extracted local features, global features, and spatio-temporal features; the weights of the local features, global features, and spatio-temporal features are determined through model training. Use a fully connected layer to limit the final feature dimensions of these local features, global features, and spatio-temporal features to the same size, and assign learnable weights to the local features, global features, and spatio-temporal features. Obtain the fused features by performing weighted summation on the extracted local features, global features, and spatio-temporal features; Output the liver tumor categories based on the fused features: primary cancer and metastatic cancer.

2. The liver tumor discrimination method based on local-global spatio-temporal feature fusion according to claim 1, wherein The input medical image is a multi-phase CT image including the arterial phase, portal venous phase, and equilibrium phase.

3. The liver tumor discrimination method based on local-global spatio-temporal feature fusion according to claim 1, characterized in that The method for segmenting the abdominal region image from the input medical image includes: optimizing the abdominal region boundary using reinforcement learning.

4. The method for differentiating liver tumors based on local-global spatio-temporal feature fusion according to claim 1, wherein The attention mechanism module is a Transformer module.

5. A liver tumor discrimination device based on local-global spatio-temporal feature fusion, characterized in that, It includes: An abdominal image segmentation module for segmenting the abdominal region image from the input medical image; A local feature extraction module for inputting the abdominal region image into a convolutional neural network for local feature extraction; A global feature extraction module for inputting the extracted local features into an attention mechanism module for global feature extraction; A spatio-temporal feature extraction module for encoding the extracted global features into a pulse sequence and then inputting it into a spiking neural network for spatio-temporal feature extraction; A tumor category output module for fusing the extracted local features, global features, and spatio-temporal features; the weights of the local features, global features, and spatio-temporal features are determined through model training. Use a fully connected layer to limit the final feature dimensions of these local features, global features, and spatio-temporal features to the same size, and assign learnable weights to the local features, global features, and spatio-temporal features. Obtain the fused features by performing weighted summation on the extracted local features, global features, and spatio-temporal features; Output the liver tumor categories based on the fused features: primary cancer and metastatic cancer.

6. The liver tumor discrimination device based on local-global spatio-temporal feature fusion according to claim 5, wherein The input medical image is a multi-phase CT image including the arterial phase, portal venous phase, and equilibrium phase.

7. The liver tumor discrimination device based on local-global spatio-temporal feature fusion according to claim 5, characterized in that The abdominal image segmentation module is specifically used for: optimizing the abdominal region boundary using reinforcement learning.

8. The liver tumor discrimination device based on local-global spatio-temporal feature fusion according to claim 5, characterized in that The attention mechanism module is a Transformer module.

Images