An image segmentation method introducing spatial information and attention mechanism

Abstract

The application relates to an image segmentation method introducing spatial information and an attention mechanism and belongs to the image segmentation field. Specifically, the method comprises the following steps: S1, a data preparation stage: preprocessing a brain medical image and cutting out an image block from the preprocessed image; S2, a feature coding stage: extracting image features through pre-activated 3D convolution; and S3, a feature decoding stage: restoring the size of an original image through deconvolution and an attention mechanism with position coding, so that the image segmentation process is completed. The application pays attention to spatial information through the attention mechanism, and the network segmentation performance is improved.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation and relates to an image segmentation method that incorporates spatial information and attention mechanisms. Background Technology

[0002] With the rapid development of medical imaging technologies such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), medical images are playing an increasingly important role in clinical medical diagnosis. Medical image segmentation can provide better scientific reference for medical personnel in judging and diagnosing diseases and their causes, thereby greatly reducing the misdiagnosis rate caused by insufficient human visual acuity or the lack of subjective clinical experience of medical personnel, and further improving the utilization rate of medical images. U-Net has proven to be very effective in medical image processing tasks. In medical images, there are 3D images with complex anatomical structures, where a purely convolutional encoding and decoding structure cannot fully utilize the spatial information of 3D images. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide an image segmentation method that incorporates spatial information and an attention mechanism. By using 3D relative position encoding and an attention mechanism, the spatial information of the 3D image is fully extracted, and the semantic information of the image is more accurately recovered on the decoding path, thereby improving the model segmentation accuracy.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] An image segmentation method incorporating spatial information and an attention mechanism is proposed. This method designs a self-attention segmentation network assisted by relative position encoding. In the encoding stage, image features are extracted by pre-activated 3D convolution. In the decoding stage, the image size is gradually restored by deconvolution, and image features are restored by embedding a NonLocal self-attention module with relative position encoding. The method includes the following steps:

[0006] S1: Data preparation stage: Preprocessing brain medical images and cropping image patches from the preprocessed images;

[0007] S2: Feature encoding stage: Extracting image features through pre-activated 3D convolution;

[0008] S3: Feature Decoding Stage: The feature maps obtained in the encoding stage are deconvolutioned and an attention mechanism with positional encoding are used to restore the original image size, thus completing the image segmentation process.

[0009] Furthermore, the step S1 includes the following steps:

[0010] S11: Cropping a 3D medical image by removing the background area with a gray value of 0 along the plane formed by any two axes;

[0011] S12: Apply Z-Score normalization to the cropped image to make the mean of the image grayscale distribution 0 and the standard deviation 1, so that it follows a normal distribution;

[0012] S13: Cut the cropped image into 32×32×32 image blocks, and randomly select one block as the input for the feature encoding stage in step S2; if it is multimodal data, stitch all the data of all modes together along the channel dimension to form a multi-channel image as the input for the feature encoding stage in step S2.

[0013] Furthermore, step S2 includes the following steps:

[0014] S21: Use ordinary 3D convolution extraction to extract features from the cropped 3D image to obtain a 32×32×32 feature map;

[0015] S22: Use a 3D convolution with a stride of 2 to downsample the feature map of S21;

[0016] S23: Repeat operations S21 and S22 to finally obtain a 4×4×4 feature map.

[0017] Furthermore, step S3 specifically includes the following steps:

[0018] S31: For the 3D feature map obtained in the encoding stage, calculate the relative positions of other pixels with all pixels as the origin;

[0019] S32: Embed the positional encoding generated in S31 into the NonLocal self-attention mechanism to perform feature fusion on the feature map in the encoding stage;

[0020] S33: Upsample the feature map using deconvolution, then repeat S32;

[0021] S34: Repeat step S33 twice to achieve image segmentation.

[0022] The beneficial effects of this invention are as follows: by introducing spatial information through relative position encoding and embedding it into the self-attention mechanism, the learning of weights in the attention mechanism depends not only on grayscale information but also on positional information. By focusing on spatial information through the attention mechanism, the network segmentation performance is improved.

[0023] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0025] Figure 1 This is a network structure diagram of the image segmentation model in this invention;

[0026] Figure 2 This is a schematic diagram of the relative position encoding structure in this invention;

[0027] Figure 3 This is the upsampling module of the network in this invention. Detailed Implementation

[0028] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0029] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0030] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0031] Please see the appendix Figures 1-3 This invention provides an image segmentation method that incorporates spatial information and an attention mechanism. In this embodiment, it is assumed to be used for segmenting brain tissue images. This method designs a self-attention segmentation network assisted by relative position encoding. The network structure diagram is attached. Figure 1 As shown. In the encoding stage, image features are extracted by pre-activated 3D convolution. In the decoding stage, the image size is gradually restored by deconvolution, and a NonLocal self-attention module with embedded relative position encoding (as shown in the attached diagram) is used. Figure 3 (As shown) Image features are recovered. This method includes the following steps:

[0032] Step 1: Preprocess brain medical images, and crop out 32×32×32 image blocks from the preprocessed images;

[0033] Step 101: Crop the 3D medical image by removing the background area with a grayscale value of 0 along the plane formed by any two axes;

[0034] Step 102: Apply Z-Score normalization to the cropped image to make the mean of the image grayscale distribution 0 and the standard deviation 1, so that it follows a normal distribution;

[0035] Step 103: Cut the cropped image into 32×32×32 image blocks, and randomly select one block as the input to the model; if it is multimodal data, then stitch all the data of all modalities together along the channel dimension to form a multi-channel image as the network input.

[0036] Step 2: Feature Encoding Stage: Extract image features through pre-activated 3D convolution;

[0037] Step 201: Use ordinary 3D convolution extraction to extract features from the 3D image to obtain a 32×32×32 feature map;

[0038] Step 202: Downsample the feature map from Step 201 using a 3D convolution with a stride of 2;

[0039] Step 203: Repeat steps 201 and 202 to finally obtain a 4×4×4 feature map.

[0040] Step S3: Feature Decoding Stage: The feature map obtained in the encoding stage is deconvolutioned and an attention mechanism with positional encoding is used to restore the original image size, thus completing the image segmentation process.

[0041] Step 301: For the 3D feature map obtained in the encoding stage, calculate the relative positions of other pixels with all pixels as the origin, such as... Figure 2 As shown;

[0042] Step 302: As Figure 3 As shown, the positional encoding generated in step 301 is embedded into the NonLocal self-attention mechanism to perform feature fusion on the feature map in the encoding stage.

[0043] Step 303: Upsample the feature map using deconvolution, and then repeat step 302.

[0044] Step 304: Repeat step 303 twice to achieve image segmentation.

[0045] Spatial information is introduced through relative position encoding and embedded into the self-attention mechanism, making the learning of weights in the attention mechanism dependent not only on grayscale information but also on positional information. By focusing on spatial information through the attention mechanism, the network's segmentation performance is improved.

[0046] To verify the effectiveness of the present invention, the following experiments were conducted:

[0047] Based on this image segmentation method incorporating spatial information and attention mechanisms, it was tested on the IBSR18 dataset. The IBSR18 dataset contains 18 training samples, and the testing objective is to segment brain tissue MRI images into gray matter (GM), white matter (WM), cerebrospinal fluid (CSF), and background. Fourteen data samples were used as the training set, and the remaining one was used as the validation set. The method was compared with Method 1, which uses both spatial and channel attention; Method 2, which uses a self-attention mechanism; Method 3, which uses axial attention; and Method 4 of this invention. The Dice coefficient was used as the evaluation metric, and its formula is as follows:

[0048]

[0049] Where A represents the result of neural network segmentation, and B represents the gold standard given by the dataset.

[0050] Table 1 presents the test results on the dataset. It can be seen that, in terms of the Dice coefficient, the neural network based on this invention performs better in each segmentation result.

[0051] Table 1

[0052] CSF GM WM AVG Method 1 85.88 95.30 95.05 92.08 Method 2 86.12 95.38 95.04 92.17 Method 3 86.20 95.37 95.01 92.19 Method 4 86.77 95.48 95.06 92.44

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An image segmentation method incorporating spatial information and attention mechanism, characterized in that: Comprising the following steps: S1: data preparation stage: preprocessing brain medical images, cropping image blocks from the preprocessed images; S2: feature encoding stage: extracting image features through pre-activated 3D convolution; comprising the following steps: S21: using a general 3D convolution to extract features of the cropped three-dimensional image to obtain a feature map of ; S22: using 3D convolution with a step size of 2 to down-sample the feature map of S21; S23: repeating operation S21 and S22 to finally obtain a 4x4x4 feature map; S3: feature decoding stage: restoring the original image size through deconvolution and attention mechanism with position encoding of the feature map obtained in the encoding stage, completing the image segmentation process; comprising the following steps: S31: for the 3D feature map obtained in the encoding stage, the relative positions of other pixel points are calculated respectively with all pixel points as the origin; S32: embedding the position encoding generated in S31 into the NonLocal self-attention mechanism to perform feature fusion on the feature map in the encoding stage; S33: using deconvolution to up-sample the feature map, and then repeating S32; S34: repeating step S33 twice to realize image segmentation.

2. The image segmentation method of claim 1, wherein, The step S1 comprises the following steps: S11: cropping the three-dimensional medical image, and cropping the background area with a gray value of 0 along the plane formed by any two axes; S12: using Z-Score normalization on the cropped image to make the mean value of the image gray scale distribution 0 and the standard deviation 1, so that it conforms to the normal distribution; S13: Cut the cropped image into multiple... The image is divided into blocks of different sizes, and one block is randomly selected as the input for the feature encoding stage in step S2. If it is multimodal data, all modal data are stitched together along the channel dimension to form a multi-channel image as the input for the feature encoding stage in step S2.

Images