A lymphoma lesion segmentation method based on convolutional neural network multi-modal fusion

Abstract

A kind of lymphoma lesion segmentation method based on convolutional neural network multimodal fusion, the method comprises the following steps: step 1. Constructing double-branch convolutional neural network model containing feature fusion module;Step 2. PET and CT images are respectively input into the PET branch and CT branch of convolutional neural network, and intermediate feature map is obtained;Step 3. PET and CT intermediate feature maps with the same size and channel number are input into the fusion module after concatenate, and the fused feature map is obtained;Step 4. The fused feature map is input into the corresponding decoder layer, and segmentation prediction is carried out.The performance of lymphoma lesion automatic identification segmentation is improved by the additional fusion module, which fuses the intermediate feature maps of PET and CT.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a lymphoma lesion segmentation method based on convolutional neural network multimodal fusion. Background Art

[0002] Lymphoma is a cancer involving lymphatic tissue and the lymphatic system. It originates in lymphocytes and can occur in lymph nodes, spleen, bone marrow, tonsils, tonsillar rings, and other lymphatic tissues. Lymphomas are widely distributed throughout the body, with lesions varying in size and shape. Furthermore, due to the inherently low resolution of PET images and the presence of partial volume effect (PVE), large lesions appear blurred and small lesions appear blurry and dim. This makes it difficult for doctors to accurately delineate the entire lymphoma lesion area with the naked eye.

[0003] Medical image multimodal fusion fuses data from different medical imaging modalities to obtain more comprehensive, accurate, and rich information, thereby improving the effectiveness of medical diagnosis, treatment, and research. This is because different modalities can provide specific types of anatomical, physiological, or functional information, and there is information complementarity between different medical imaging modalities. For example, structural images (such as CT) can provide the anatomical structure of organs or tissues, while functional images (such as PET) can reveal the metabolic activity and functional state of tissues. Multimodal fusion can integrate this information together to obtain a more comprehensive understanding. At the same time, compared with early feature fusion (direct fusion of PET-CT original images), late feature fusion (fusion of intermediate features of PET and CT extracted by convolutional neural networks) will achieve better performance in segmentation. Summary of the Invention

[0004] To address the low accuracy of existing lymphoma lesion segmentation methods, this paper proposes a lymphoma lesion segmentation method based on multimodal fusion using convolutional neural networks. This method extracts features from both PET and CT modalities, then fuses these features using a feature fusion module. Finally, the fused features are input into a decoder for segmentation prediction. This feature fusion module is designed to improve the performance of automatic lymphoma lesion identification and segmentation.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A lymphoma lesion segmentation method based on convolutional neural network multimodal fusion includes the following steps:

[0007] Step 1. Build a two-branch convolutional neural network model including a feature fusion module;

[0008] Step 2. Input the PET and CT images into the PET branch and CT branch of the convolutional neural network respectively to obtain the intermediate feature maps. The process is as follows:

[0009] 2.1. Input the PET and CT images into the PET branch and CT branch respectively. After the first convolution group of the PET branch, the intermediate feature map pet1 is generated. After the first convolution group of the CT branch, the intermediate feature map ct1 is generated. The formula is as follows;

[0010] pet1=f pet_0 (pet0)ct1=f ct_0 (ct0)

[0011] Where pet1 and ct1 represent the PET and CT intermediate feature maps generated after convolution, respectively, and f pet_0 and f ct_0 Represent the first convolution group operation of the PET and CT branches respectively, pet0 and ct0 represent the input original PET and CT images respectively;

[0012] 2.2、pet the intermediate feature map i and ct i Perform 2 times downsampling respectively to obtain pet i ′ and ct i ′ , and then pet i ′ and ct i ′ Input to the i+1th convolution group of PET and CT branches respectively to obtain their respective intermediate feature maps pet i+1 and ct i+1 , the formula is as follows:

[0013] pet i ′ =D pet_i (pet i )ct i ′ =D ct_i (ct i )

[0014] pet i+1 =f pet_i (pet i ′ )ct i+1 =f ct_i (ct i ′ )

[0015] In the formula, pet i and ct iRepresents the intermediate feature maps generated by the previous convolution group of the PET and CT branches, D pet_i and D ct_i Represents the downsampling operations of PET and CT branches respectively, pet i ′ and ct i ′ Represent the intermediate feature maps after downsampling, f pet_i and f ct_i Represents the i+1th convolution group of PET and CT branches respectively, pet i+1 and ct i+1 Represent the intermediate feature maps generated by the i+1th convolution group of the PET and CT branches, respectively, i=1,2,...,4;

[0016] Step 3. The PET and CT intermediate feature maps with the same size and number of channels are concatenated and input into the feature fusion module to obtain the fused feature map. The process is as follows:

[0017] 3.1、pet j and ct j After concatenating by channel dimension, we get Mid_f j Input feature fusion module, the formula is as follows:

[0018] Mid_f j =C(pet j ,ct j )

[0019] Where Mid_f j Represents a temporary feature map, C represents the concatenate operation, pet j and ct j Represent the intermediate feature maps of the j-th PET and CT branches, j = 1, 2, ..., 5;

[0020] 3.2、Mid_f j Re-split into pet j and ct j , and each is split into half according to the channel dimension to obtain pet j_0 and ct j_0 , pet j_1 and ct j_1 , j=1,2,......,5, and then pet j_0 and ct j_0 After concatenation according to the channel dimension, the intermediate fusion feature map Mix_f is obtained by fusion convolution j , the formula is as follows;

[0021] Mix_f j =F j_0 (C(pet j_0 ,ct j_0 ))

[0022] Where, Mix_f j Represents the intermediate fusion feature map, F j_0 Represents the fused convolution operation, C represents the concatenate operation, pet j_0 and ct j_0 Represents the first half of the intermediate feature map of the j-th PET and CT branches, j = 1, 2, ..., 5;

[0023] 3.3、Mix_f j and pet j_1 and ct j_1 After concatenation according to the channel dimension, the fused feature map Mixed_f is obtained by convolution. j , the formula is as follows;

[0024] Mixed_f j =F j_1 (C(Mix_f j ,pet j_1 ,ct j_1 ))

[0025] Where, Mixed_f j Represents the fused feature map, F j_1 Represents the fused convolution operation, C represents the concatenate operation, pet j_1 and ct j_1 Represent the second half of the intermediate feature map of the j-th PET and CT branches, j = 1, 2, ..., 5;

[0026] Step 4. Input the fused feature map into the corresponding decoder layer for segmentation prediction.

[0027] Compared with the prior art, the present invention has the following beneficial effects: the method optimizes and fuses the intermediate feature maps of PET and CT through a feature fusion module, thereby improving the performance of lymphoma lesion segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the overall framework diagram of the dual-branch convolutional neural network model including the feature fusion module of the present invention;

[0029] Figure 2 This is a structural diagram of the feature fusion module of the present invention. DETAILED DESCRIPTION

[0030] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0031] Reference Figure 1 and Figure 2 A lymphoma lesion segmentation method based on convolutional neural network multimodal fusion, the method comprising the following steps:

[0032] Step 1. Build a two-branch convolutional neural network model including a feature fusion module;

[0033] Step 2. Input the PET and CT images into the PET branch and CT branch of the convolutional neural network respectively to obtain the intermediate feature maps. The process is as follows:

[0034] 2.1. Input the PET and CT images into the PET branch and CT branch respectively. After the first convolution group of the PET branch, the intermediate feature map pet1 is generated. After the first convolution group of the CT branch, the intermediate feature map ct1 is generated. The formula is as follows;

[0035] pet1=f pet_0 (pet0)ct1=f ct_0 (ct0)

[0036] Where pet1 and ct1 represent the PET and CT intermediate feature maps generated after convolution, respectively, and f pet_0 and f ct_0 Represent the first convolution group operation of the PET and CT branches respectively, pet0 and ct0 represent the input original PET and CT images respectively;

[0037] 2.2、pet the intermediate feature map i and ct i Perform 2 times downsampling respectively to obtain pet i ′ and ct i ′ , and then pet i ′ and ct i ′ Input to the i+1th convolution group of PET and CT branches respectively to obtain their respective intermediate feature maps pet i+1 and ct i+1 , the formula is as follows:

[0038] pet i ′ =D pet_i (pet i )ct i ′ =D ct_i(ct i )

[0039] pet i+1 =f pet_i (pet i ′ )ct i+1 =f ct_i (ct i ′ )

[0040] In the formula, pet i and ct i Represents the intermediate feature maps generated by the previous convolution group of the PET and CT branches, D pet_i and D ct_i Represents the downsampling operations of PET and CT branches respectively, pet i ′ and ct i ′ Represent the intermediate feature maps after downsampling, f pet_i and f ct_i Represents the i+1th convolution group of PET and CT branches respectively, pet i+1 and ct i+1 Represent the intermediate feature maps generated by the i+1th convolution group of the PET and CT branches, respectively, i=1,2,...,4;

[0041] Step 3. The PET and CT intermediate feature maps with the same size and number of channels are concatenated and input into the feature fusion module to obtain the fused feature map. The process is as follows:

[0042] 3.1、pet j and ct j After concatenating by channel dimension, we get Mid_f j Input feature fusion module, the formula is as follows:

[0043] Mid_f j =C(pet j ,ct j )

[0044] Where Mid_f j Represents a temporary feature map, C represents the concatenate operation, pet j and ct j Represent the intermediate feature maps of the j-th PET and CT branches, j = 1, 2, ..., 5;

[0045] 3.2、Mid_f j Re-split into pet j and ctj , and each is split into half according to the channel dimension to obtain pet j_0 and ct j_0 , pet j_1 and ct j_1 , j=1,2,......,5, and then pet j_0 and ct j_0 After concatenation according to the channel dimension, the intermediate fusion feature map Mix_f is obtained by fusion convolution j , the formula is as follows;

[0046] Mix_f j =F j_0 (C(pet j_0 ,ct j_0 ))

[0047] Where, Mix_f j Represents the intermediate fusion feature map, F j_0 Represents the fused convolution operation, C represents the concatenate operation, pet j_0 and ct j_0 Represents the first half of the intermediate feature map of the j-th PET and CT branches, j = 1, 2, ..., 5;

[0048] 3.3、Mix_f j and pet j_1 and ct j_1 After concatenation according to the channel dimension, the fused feature map Mixed_f is obtained by convolution. j , the formula is as follows;

[0049] Mixed_f j =F j_1 (C(Mix_f j ,pet j_1 ,ct j_1 ))

[0050] Where, Mixed_f j Represents the fused feature map, F j_1 Represents the fusion reshape convolution operation, C represents the concatenate operation, pet j_1 and ct j_1 Represent the second half of the intermediate feature map of the j-th PET and CT branches, j = 1, 2, ..., 5;

[0051] Step 4. Input the fused feature map into the corresponding decoder layer for segmentation prediction.

[0052] This embodiment extracts features from both PET and CT modalities, then fuses the extracted features using a feature fusion module. Finally, the fused features are fed into a decoder for segmentation prediction. This feature fusion module is designed to improve the performance of automatic lymphoma lesion identification and segmentation.

[0053] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A lymphoma lesion segmentation method based on multimodal fusion of convolutional neural networks, characterized by: The method comprises the following steps: Step 1. Build a two-branch convolutional neural network model including a feature fusion module; Step 2. Input the PET and CT images into the PET branch and CT branch of the convolutional neural network respectively to obtain the intermediate feature maps. The process is as follows; 2. Step 1: Input the PET and CT images into the PET branch and CT branch respectively. After the first convolution group of the PET branch, the intermediate feature map pet1 is generated. After the first convolution group of the CT branch, the intermediate feature map ct1 is generated. The formula is as follows; pet1=f pet_0 (pet0)ct1=f ct_0 (ct0) Where pet1 and ct1 represent the PET and CT intermediate feature maps generated after convolution, respectively, and f pet_0 and f ct_0 Represent the first convolution group operation of the PET and CT branches respectively, pet0 and ct0 represent the input original PET and CT images respectively; Step 2: pet the intermediate feature map i and ct i Perform 2 times downsampling respectively to obtain pet i ′ and ct i ′ , and then pet i ′ and ct i ′ Input to the i+1th convolution group of PET and CT branches respectively to obtain their respective intermediate feature maps pet i+1 And, the formula is as follows: pet i ′ =D pet_i (pet i )ct i ′ =D ct_i (ct i ) pet i+1 =f pet_i (pet i ′ )ct i+1 =f ct_i (ct i ′ ) In the formula, pet i and ct i Represents the intermediate feature maps generated by the previous convolution group of the PET and CT branches, D pet_i and D ct_i Represents the downsampling operations of PET and CT branches respectively, pet i ′ and ct i ′ Represent the intermediate feature maps after downsampling, f pet_i and f ct_i Represents the i+1th convolution group of PET and CT branches respectively, pet i+1 and ct i+1 Represent the intermediate feature maps generated by the i+1th convolution group of the PET and CT branches, respectively, i=1,2,...,5; Step 3. The PET and CT intermediate feature maps with the same size and number of channels are concatenated and input into the fusion module to obtain the fused feature map. The process is as follows:

3. Step 1, pet j and ct j After concatenating by channel dimension, we get Mid_f j Input feature fusion module, the formula is as follows: Mid_f j =C(pet j ,ct j ) Where Mid_f j Represents a temporary feature map, C represents the concatenate operation, pet j and ct j Represent the intermediate feature maps of the j-th PET and CT branches, j = 1, 2, ..., 5; Step 2: Mid_f j Re-split into pet j and ct j , and each is split into half according to the channel dimension to obtain pet j_0 and ct j_0 , pet j_1 and ct j_1 , j=1,2,......,5, and then pet j_0 and ct j_0 After concatenation according to the channel dimension, the intermediate fusion feature map Mix_f is obtained by fusion convolution j , the formula is as follows; Mix_f j =F j_0 (C(pet j_0 ,ct j_0 )) Where, Mix_f j Represents the intermediate fusion feature map, F j_0 Represents the fused convolution operation, C represents the concatenate operation, pet j_0 and ct j_0 Represents the first half of the intermediate feature map of the j-th PET and CT branches, j = 1, 2, ..., 5; Step 3, then Mix_f j and pet j_1 and ct j_1 After concatenation according to the channel dimension, the fused feature map Mixed_f is obtained by convolution. j , the formula is as follows; Mixed_f j =F j_1 (C(Mix_f j ,pet j_1 ,ct j_1 )) Where, Mixed_f j Represents the fused feature map, F j_1 Represents the fused convolution operation, C represents the concatenate operation, pet j_1 and ct j_1 Represent the second half of the intermediate feature map of the j-th PET and CT branches, j = 1, 2, ..., 5; Step 4. Input the fused feature map into the corresponding decoder layer for segmentation prediction.

Images