Liver image segmentation method based on multi-head attention feature fusion

Abstract

The invention relates to the technical field of liver segmentation, in particular to a multi-head attention feature fusion-based liver image segmentation method, which comprises the following steps of: establishing a liver image segmentation model based on a Unet network structure, and collecting a liver CT image and a corresponding labeled region image; performing edge detection operator processing on the liver CT image to obtain a liver region contour image; carrying out binarization processing on the liver CT image to obtain a liver region binarization image; processing the liver CT image through a segmentation network to obtain a liver region coarse segmentation image; adjusting the liver CT image, the marked region image, the liver region contour image, the liver region binarization image and the liver region coarse segmentation image to the same size and then merging the images to obtain a merged image; training a liver image segmentation model by adopting the merged image set, calculating loss by adopting a loss function, and optimizing model parameters according to the loss until convergence; obtaining a to-be-processed liver CT image, and inputting the to-be-processed liver CT image into the trained liver image segmentation model to obtain a liver segmentation image; according to the invention, more accurate segmented images can be provided.

Description

Technical Field

[0001] The present invention relates to the technical field of liver segmentation, and in particular to a liver image segmentation method based on multi-head attention feature fusion. Background Art

[0002] Liver segmentation is the process of identifying and extracting the liver region in medical images. Liver segmentation is of great significance in clinical practice, as it can help doctors accurately assess the size, location, and morphology of liver lesions and facilitate the diagnosis, treatment, and monitoring of diseases.

[0003] Since the morphology and grayscale information of the liver in the image vary greatly, and may have similar features with surrounding tissues, it is easy to cause segmentation errors; at the same time, medical imaging data is usually more complex, with problems such as noise and uneven brightness, which will also affect the accuracy of the segmentation results; traditional segmentation methods have major problems. With the continuous development of deep learning technology, liver segmentation methods based on deep learning have also made significant progress. Traditional deep learning-based segmentation methods usually use convolutional neural network (CNN) architectures, such as U-Net, SegNet, DeepLab, etc. to build models. Through the model, a large amount of medical imaging data is learned, feature representation and matching rules are automatically learned, and different loss functions and optimization methods are combined for training, so as to achieve accurate segmentation of liver images.

[0004] Some scholars proposed to apply the region-based convolutional neural network (R-CNN) in the field of image segmentation detection. Through experiments on the PASCAL VOC-2007 dataset, R-CNN increased the detection accuracy from 29.2% of the traditional method to 66.0%. This method integrates the three steps of target feature extraction, feature selection and feature classification into the same model, thereby greatly optimizing performance and efficiency. Based on R-CNN, some scholars proposed the pyramid pooling network model (SPPNet), and some scholars proposed the Fast R-CNN algorithm. These two algorithms effectively improved the speed of image detection and increased the accuracy on the PASCAL VOC 2007 dataset to 70%. Other scholars launched the Faster R-CNN algorithm based on Fast R-CNN, introduced the candidate window grid technology (RPN), greatly accelerated the calculation speed, and increased the detection accuracy on PASCAL VOC 2007 to 73.2%. However, the segmentation effect is overly dependent on the fineness of the label area, while ignoring the information of the original image contour area, and the traditional segmentation method may not be able to fully utilize the multi-scale features in the image, which limits the model's understanding and adaptability to complex scenes. Summary of the invention

[0005] To solve the above problem, the present invention provides a liver image segmentation method based on multi-head attention feature fusion, comprising the following steps:

[0006] S1. Establishing a liver image segmentation model based on the Unet network structure, the liver image segmentation model includes a feature extraction module, a feature fusion module, an attention screening module and an upsampling module; the feature extraction module includes 4 downsampling layers, and the upsampling module includes 4 upsampling layers;

[0007] S2. Acquire liver CT images and their corresponding annotated area images;

[0008] S3. Performing edge detection operator processing on the liver CT image of step S2 to obtain a liver region contour image;

[0009] S4. Binarizing the liver CT image of step S2 to obtain a binary image of the liver region;

[0010] S5. The liver CT image of step S2 is passed through a segmentation network to obtain a rough segmentation image of the liver region;

[0011] S6. adjusting the liver CT image, the annotated region image, the liver region contour image, the liver region binary image, and the liver region coarse segmentation image to the same size and merging them to obtain a merged image;

[0012] S7. Use the merged image set to train the liver image segmentation model, use the loss function to calculate the loss, and optimize the model parameters according to the loss until convergence; specifically including:

[0013] S11. Input the merged image into the downsampling module to obtain a downsampled image group, which includes a 256×256 image X 1,0 , 128×128 image X 2,0 , 64×64 image X 3,0 and a 32×32 image X 4,0 ;

[0014] S12. Input the downsampled image group and the merged image into the feature fusion module to obtain a feature fusion image group, which includes a 64×64 image 128×128 image 256×256 image and 512×512 images

[0015] S13. Input the feature fusion image group into the attention screening module to obtain an attention image group, which includes a 64×64 image O 1 , 128×128 image O 2 , 256×256 image O 3and a 512×512 image O 4 ;

[0016] S14. Input the 32×32 images in the attention image group and the down-sampled image group into the up-sampling module to obtain an output image group, where the output image group includes image out1, image out2, image out3, and image out4;

[0017] S15. Perform convolution transformation on each image in the output image group to obtain a corresponding prediction image, and calculate the loss according to the prediction image;

[0018] S8. Obtain the liver CT image to be processed and input it into the trained liver image segmentation model to obtain a liver segmentation image.

[0019] Beneficial effects of the present invention:

[0020] The present invention uses a multi-head attention mechanism to capture features at different scales, thereby improving the recognition accuracy of the liver area; obtains the liver area contour image through edge detection operator processing, and combines it with other image information to enhance the utilization of liver contour information; designs a specific feature fusion module to better combine features at different resolution levels to make up for the limitations of the expression ability of single-scale features; and reduces the amount of network calculations and improves the network processing speed by structural pruning while ensuring the fusion between features. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the method of the present invention;

[0022] Figure 2 It is a schematic diagram of the structure of the model of the present invention;

[0023] Figure 3 It is a schematic diagram of the feature fusion module of the present invention;

[0024] Figure 4 This is a schematic diagram of the fusion block flow of the present invention;

[0025] Figure 5 This is an example of image acquisition and processing in the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The present invention provides a liver image segmentation method based on multi-head attention feature fusion, such as Figure 1 As shown, the following steps are included:

[0028] S1. Establishing a liver image segmentation model based on the Unet network structure, the liver image segmentation model includes a feature extraction module, a feature fusion module, an attention screening module and an upsampling module; the feature extraction module includes 4 downsampling layers, and the upsampling module includes 4 upsampling layers;

[0029] S2. Acquire liver CT images and their corresponding annotated area images;

[0030] S3. Performing edge detection operator processing on the liver CT image of step S2 to obtain a liver region contour image; wherein the edge detection operator is preferably a Sobel operator;

[0031] S4. Binarizing the liver CT image of step S2 to obtain a binary image of the liver region;

[0032] S5. The liver CT image of step S2 is passed through a segmentation network to obtain a rough segmentation image of the liver region; wherein the segmentation network adopts a Unet_2D network, which uses an ImageNet pre-trained model and unet as a backbone network;

[0033] S6. The liver CT image, the annotated region image, the liver region contour image, the liver region binarization image and the liver region coarse segmentation image are adjusted to the same size and then merged to obtain a merged image; the liver CT image, the liver region binarization image, the liver region contour image, the annotated region image and the liver region coarse segmentation image are respectively as follows Figure 5 As shown in (a)-(e);

[0034] S7. training a liver image segmentation model using the merged image set, calculating the loss using a loss function, and optimizing model parameters according to the loss until convergence;

[0035] S8. Obtain the liver CT image to be processed and input it into the trained liver image segmentation model to obtain a liver segmentation image.

[0036] Preferably, step S7 inputs the 512×512 merged image into the liver image segmentation model for training, such as Figure 2 As shown, including:

[0037] S11. Merge the image X 0,0 Input the downsampling module to get the downsampled image group, which includes 256×256 images X 1,0 , 128×128 image X 2,0 , 64×64 image X 3,0 and a 32×32 image X 4,0 ;

[0038] S12. Input the downsampled image group and the merged image into the feature fusion module to obtain a feature fusion image group, which includes a 64×64 image 128×128 image 256×256 image and 512×512 images

[0039] S13. Input the feature fusion image group into the attention screening module to obtain an attention image group, which includes a 64×64 image O 1 , 128×128 image O 2 , 256×256 image O 3 and a 512×512 image O 4 ;

[0040] S14. Input the 32×32 images in the attention image group and the down-sampled image group into the up-sampling module to obtain an output image group, where the output image group includes image out1, image out2, image out3, and image out4;

[0041] S15. Perform convolution transformation on each image in the output image group to obtain a corresponding prediction image, and calculate the loss based on the prediction image.

[0042] Specifically, Figure 3 As shown in FIG. 1 , the processing flow of the feature fusion module specifically includes:

[0043] The 512×512 merged image X 0,0 With a 256×256 image X 1,0 After the fusion block, we get a 512×512 image X 0,1 ;

[0044] Convert the 256×256 image X 1,0 With 128×128 image X 2,0 After the fusion block, we get a 256×256 image X 1 ,1 ;

[0045] The 512×512 merged image X 0,0 With a 512×512 image X 0,1 Fusion, and then the fusion result and 256 × 256 image X 1,1 After the fusion block, we get a 512×512 image X 0,2 ;

[0046] Convert the 128×128 image to 2,0 With 64×64 image X3,0 After the fusion block, we get a 128×128 image X 2,1 ;

[0047] Convert the 256×256 image X 1,0 With a 256×256 image X 1,1 Fusion, and then the fusion result and 128 × 128 image X 2,1 After the fusion block, we get a 256×256 image X 1,2 ;

[0048] The 512×512 merged image X 0,0 , 512×512 image X 0,1 and a 512×512 image X 0,2 Fusion, and then the fusion result and 256 × 256 image X 1,2 After the fusion block, we get a 512×512 image X 0,3 ;

[0049] Convert the 64×64 image to 3,0 With a 32×32 image X 4,0 After the fusion block, we get a 64×64 image X 3,1 ; Image X 3,1 It's the image

[0050] Convert the 128×128 image to 2,0 With 128×128 image X 2,1 Fusion, and then the fusion result 64×64 image X 3 ,1 After the fusion block, we get a 128×128 image X 2,2 ; Image X 2,2 It's the image

[0051] Convert the 256×256 image X 1,0 , 256×256 image X 1,1 and a 256×256 image X 1,2 Fusion, and then the fusion result 128 × 128 image X 2,2 After the fusion block, we get a 256×256 image X 1,3 ; Image X 1,3 It's the image

[0052] The 512×512 merged image X 0,0 , 512×512 image X 0,1 , 512×512 image X 0,2 and a 512×512 image X 0,3Fusion, and then the fusion result 256 × 256 image X 1,3 After the fusion block, we get a 512×512 image X 0,4 ; Image X 0,4 It's the image

[0053] In order to improve the network speed and reduce the amount of network calculation, the structure of the feature fusion module can be pruned during the training process. For example, the 32×32 image X in the downsampled image group 4,0 Do not participate in feature fusion, Figure 3 The red part is trimmed off to image X 0,3 As an image ImageX 1,2 As an image ImageX 2,1 As an image ImageX 3,0 As an image Output; after pruning, the structure is simplified and the fusion of features is guaranteed.

[0054] Specifically, using image F i and image G i As the input of the fusion block, the image G i The scale ratio image F i Half as small as Figure 3 As shown, Figure 4 As shown in the figure, the specific processing of the fusion block includes:

[0055] S111. Image F i Reshape into first local information Represents image F i where j=1,2,..,H Fi ×W Fi The characteristics of the elements, H Fi , W Fi Represents image F i The height and width of the image G i Compressed into the first global feature g i , expressed as:

[0056] g i =Flatten(AvgPool(G i ))

[0057] Among them, AvgPool(·) represents average pooling, and Flatten(·) represents compression operation; the first local information is concatenated with the first global feature and then input into the multi-layer perceptron to obtain the fusion feature It is expressed as:

[0058]

[0059] Fusion features Mixed features after head disassembly

[0060] S112. Image G i Reshape into second local information Represents image G i where j=1,2,..,H Gi ×W Gi The characteristics of the elements, H Gi , W Gi Represents the image G i The height and width of image F i Compressed into the second global feature f i , expressed as:

[0061]

[0062] The second local information is concatenated with the second global feature and then input into the multi-layer perceptron to obtain the fusion feature. It is expressed as:

[0063]

[0064] Fusion features Mixed features after head disassembly

[0065] S113. Mixed features After upsampling, the upsampling result is combined with the mixed feature Output after fusion.

[0066] Specifically, the attention screening module processes the four images in the feature fusion image group respectively, where the processed images Get image O i The process includes:

[0067] S131. Image Divide into N image blocks of the same size to obtain a region set in Represents the n=1, 2, ..., N image blocks; calculate the query matrix corresponding to each image block Key Matrix Value Matrix W Q Represented as the query weight matrix, W K Expressed as a key weight matrix, W V Represented as a value-weight matrix;

[0068] S132. Calculate the attention weight matrix by one-to-one correspondence between the key matrix and the value matrix, expressed as:

[0069]

[0070] represents the n=1,2,…,Nth attention weight matrix, Represents the scaling factor; Arrange all attention weight matrices in descending order of size, and extract the first k attention weight matrices to integrate into an attention weight matrix A i ; In the embodiment of the present invention, k=12;

[0071] S133. Integrate all value matrices to obtain a value matrix V i , according to the attention weight matrix A i Sum value matrix V i Calculate the attention matrix X i =A i ×V i , the attention matrix X i and value matrix V i Add together to get image O i =X i +V i .

[0072] Specifically, Figure 2 As shown, the specific process of step S14 includes:

[0073] S141. The 32×32 image in the downsampled image group is passed through the first upsample layer to obtain a first upsampled image, and the first upsampled image is combined with the image O 1 The fusion results in image out1;

[0074] S142. The image out1 is passed through the second upsampling layer to obtain a second upsampling image, and the second upsampling image is combined with the image O 2 The fusion results in image out2;

[0075] S143. The image out2 is passed through the third upsampling layer to obtain a third upsampling image, and the third upsampling image is combined with the image O 3 The fusion results in image out3;

[0076] S144. The image out3 is passed through the fourth upsampling layer to obtain a fourth upsampling image, and the fourth upsampling image is combined with the image O 4 The fusion results in image out4.

[0077] Specifically, Figure 2 As shown, each image in the output image group is subjected to convolution transformation to obtain a corresponding prediction image, including:

[0078] The image is passed through the LN layer and the 3×3 convolution layer to obtain the convolution image, and the convolution image is passed through the sigmoid activation function layer and then upsampled to obtain the prediction image.

[0079] Specifically, the loss function L used in the present invention is seg It is expressed as:

[0080] L seg =L BCE +L dice

[0081] Among them, L BCE represents the cross entropy loss, L dice Represents the Dice coefficient loss, the specific formula is:

[0082]

[0083]

[0084] y m represents the true label of the mth sample, represents the predicted value of the mth sample, M represents the number of samples, R represents the total number of pixels in each image, and y m,r represents the true label of the rth pixel of the mth sample, Represents the predicted value of the rth pixel of the mth sample.

[0085] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0086] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A liver image segmentation method based on multi-head attention feature fusion, characterized in that: The following steps are involved: S1. Establishing a liver image segmentation model based on the Unet network structure, the liver image segmentation model includes a feature extraction module, a feature fusion module, an attention screening module and an upsampling module; the feature extraction module includes 4 downsampling layers, and the upsampling module includes 4 upsampling layers; S2. Acquire liver CT images and their corresponding annotated area images; S3. Performing edge detection operator processing on the liver CT image of step S2 to obtain a liver region contour image; S4. Binarizing the liver CT image of step S2 to obtain a binary image of the liver region; S5. The liver CT image of step S2 is passed through a segmentation network to obtain a rough segmentation image of the liver region; S6. adjusting the liver CT image, the annotated region image, the liver region contour image, the liver region binary image, and the liver region coarse segmentation image to the same size and merging them to obtain a merged image; S7. Use the merged image set to train the liver image segmentation model, use the loss function to calculate the loss, and optimize the model parameters according to the loss until convergence; specifically including: S11. Input the merged image into the downsampling module to obtain a downsampled image group, which includes a 256×256 image X 1,0 , 128×128 image X 2,0 , 64×64 image X 3,0 and a 32×32 image X 4,0 ; S12. Input the downsampled image group and the merged image into the feature fusion module to obtain a feature fusion image group, which includes a 64×64 image 128×128 image 256×256 image and 512×512 images S13. Input the feature fusion image group into the attention screening module to obtain an attention image group, which includes a 64×64 image O 1 , 128×128 image O 2 , 256×256 image O 3 and a 512×512 image O 4 ; S14. Input the 32×32 images in the attention image group and the down-sampled image group into the up-sampling module to obtain an output image group, where the output image group includes image out1, image out2, image out3, and image out4; S15. Perform convolution transformation on each image in the output image group to obtain a corresponding prediction image, and calculate the loss according to the prediction image; S8. Obtain the liver CT image to be processed and input it into the trained liver image segmentation model to obtain a liver segmentation image.

2. A liver image segmentation method based on multi-head attention feature fusion according to claim 1, characterized in that: The processing of the feature fusion module specifically includes: Combine the 512×512 image with the 256×256 image X 1,0 After the fusion block, we get a 512×512 image X 0,1 ; Convert the 256×256 image X 1,0 With 128×128 image X 2,0 After the fusion block, we get a 256×256 image X 1,1 ; Combine the 512×512 merged image with the 512×512 image X 0,1 Fusion, and then the fusion result and 256 × 256 image X 1,1 After the fusion block, we get a 512×512 image X 0,2 ; Convert the 128×128 image to 2,0 With 64×64 image X 3,0 After the fusion block, we get a 128×128 image X 2,1 ; Convert the 256×256 image X 1,0 With a 256×256 image X 1,1 Fusion, and then the fusion result and 128 × 128 image X 2 ,1 After the fusion block, we get a 256×256 image X 1,2 ; The 512×512 merged image X 0,0 , 512×512 image X 0,1 and a 512×512 image X 0,2 Fusion, and then the fusion result and 256 × 256 image X 1,2 After the fusion block, we get a 512×512 image X 0,3 ; Convert the 64×64 image to 3,0 With a 32×32 image X 4,0 After the fusion block, a 64×64 image is obtained Convert the 128×128 image to 2,0 With 128×128 image X 2,1 Fusion, and then the fusion result 64 × 64 image X 3,1 After the fusion block, a 128×128 image is obtained Convert the 256×256 image X 1,0 , 256×256 image X 1,1 and a 256×256 image X 1,2 Fusion, and then the fusion result 128 × 128 image X 2,2 After the fusion block, a 256×256 image is obtained The 512×512 merged image and the 512×512 image X 0,1 , 512×512 image X 0,2 and a 512×512 image X 0,3 Fusion, and then the fusion result 256 × 256 image X 1,3 After the fusion block, a 512×512 image is obtained 3. The method for liver image segmentation based on multi-head attention feature fusion according to claim 2, characterized in that: Using image F i and image G i As the input of the fusion block, the image G i The scale ratio image F i Half as small; the specific processing of the fusion block includes: S111. Image F i Reshape into first local information Represents image F i where j=1,2,..,H Fi ×W Fi The characteristics of the elements, H Fi , W Fi Represents image F i The height and width of the image G i Compressed into the first global feature g i ; The first local information is concatenated with the first global feature and then input into the multi-layer perceptron to obtain the fusion feature Fusion features Mixed features after head disassembly S112. Image G i Reshape into second local information Represents the second input G i where j=1,2,..,H Gi ×W Gi The characteristics of the elements, H Gi , W Gi Represents the image G i The height and width of image F i Compressed into the second global feature f i ; The second local information is spliced ​​with the second global feature and then input into the multi-layer perceptron to obtain the fusion feature Fusion features Mixed features after head disassembly S113. Mixed features After upsampling, the upsampling result is combined with the mixed feature Output after fusion.

4. The liver image segmentation method based on multi-head attention feature fusion according to claim 1 is characterized in that: Attention filter module processes images Get image O i The process includes: S131. Image Divide into N image blocks of the same size to obtain a region set in Represents the n=1, 2, ..., N image blocks; calculate the query matrix corresponding to each image block Key Matrix Value Matrix W Q Represented as the query weight matrix, W K Expressed as a key weight matrix, W V Represented as a value-weight matrix; S132. Calculate the attention weight matrix by one-to-one correspondence between the key matrix and the value matrix, sort all the attention weight matrices in descending order of size, and extract the first k attention weight matrices to integrate into an attention weight matrix A i ; S133. Integrate all value matrices to obtain a value matrix V i , according to the attention weight matrix A i Sum value matrix V i Calculate the attention matrix X i , the attention matrix X i and value matrix V i Add together to get image O i .

5. The liver image segmentation method based on multi-head attention feature fusion according to claim 1, characterized in that: The specific process of step S14 includes: S141. The 32×32 image in the downsampled image group is passed through the first upsample layer to obtain a first upsampled image, and the first upsampled image is combined with the image O 1 The fusion results in image out1; S142. The image out1 is passed through the second upsampling layer to obtain a second upsampling image, and the second upsampling image is combined with the image O 2 The fusion results in image out2; S143. The image out2 is passed through the third upsampling layer to obtain a third upsampling image, and the third upsampling image is combined with the image O 3 The fusion results in image out3; S144. The image out3 is passed through the fourth upsampling layer to obtain a fourth upsampling image, and the fourth upsampling image is combined with the image O 4 The fusion results in image out4.

6. The method for liver image segmentation based on multi-head attention feature fusion according to claim 1, characterized in that: Perform convolution transformation on each image in the output image group to obtain the corresponding prediction image, including: The image is passed through the LN layer and the 3×3 convolution layer to obtain the convolution image, and the convolution image is passed through the sigmoid activation function layer and then upsampled to obtain the prediction image.

7. The method for liver image segmentation based on multi-head attention feature fusion according to claim 1, characterized in that: The loss function L used in step S8 is seg for: L seg =L BCE +L dice Among them, L BCE represents the cross entropy loss, L dice represents the Dice coefficient loss, y m represents the true label of the mth sample, represents the predicted value of the mth sample, M represents the number of samples, R represents the total number of pixels in each image, and y m,r represents the true label of the rth pixel of the mth sample, Represents the predicted value of the rth pixel of the mth sample.

Images