A brain image data processing method based on improved Unet++

Abstract

The application belongs to the technical field of brain image recognition, and particularly relates to a brain image data processing method based on an improved Unet++. The application first divides the 3D image of the brain MRI into three 2D planes, namely, the axial plane (transverse plane), coronal plane and sagittal plane, establishes an AM-Unet++ (attention module supplementarily-Unet++) network, extracts features of the region of interest in the brain, introduces attention mechanism analysis, finally performs information fusion through a 3D-CNN fusion network, and finally obtains a 3D lesion segmentation result. The application not only retains the spatial context coherence of the three-dimensional image, but also avoids the problem of high 3D model operation cost.

Description

Technical Field

[0001] This invention belongs to the field of brain image recognition technology, and in particular relates to a brain image data processing method based on an improved Unet++. Background Technology

[0002] Stroke is an acute cerebrovascular disease caused by the sudden rupture or blockage of blood vessels in the brain, resulting in brain tissue damage. It includes ischemic and hemorrhagic strokes. Severe strokes can cause permanent neurological damage, and if not diagnosed and treated promptly in the acute phase, they can lead to serious complications. Due to the rapid onset and high risk of complications associated with stroke, timely and accurate radiological imaging models can quickly screen high-risk patients, better guide departmental treatment decisions, and help physicians select more targeted rehabilitation programs for patients in the recovery phase.

[0003] Chinese patent application CN202110674662.7 discloses an automatic detection system for cerebral hemorrhage based on an improved U-Net network. This system uses an improved U-Net network to mark the location of the hemorrhage region in brain CT images from which brain parenchyma is extracted. It includes an RCSP convolutional module, a CBL4 convolutional module, a feature pyramid attention mechanism module, a multi-scale feature skip connection module, and an output module. The paper "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation" addresses the shortcomings of 2D U-Net in segmentation due to discontinuous segmentation and loss of spatial context information. It proposes a new 3D U-Net method based on 2D U-Net, replacing all 2D operations with 3D operations and employing a smooth, dense deformation field to enhance the network's segmentation performance.

[0004] The aforementioned traditional methods suffer from the following problems: 2D-Unet methods for processing medical images often decompose the 3D MRI volume into many 2D slices, and then use a segmentation model to output the segmentation results for each slice. Traditional 2D convolutional Unet models ignore the spatial characteristics of medical image data, affecting the final result. To address the shortcomings of 2D Unet networks, 3D Unet networks have emerged, extending the Unet network's application from 2D medicine to 3D images. However, while 3D Unet networks can achieve stereoscopic processing of brain tissue in 3D MRI images, the network has numerous learning parameters, resulting in slow processing speed. It is also prone to overfitting on small datasets and suffers from high computational resource requirements. Summary of the Invention

[0005] The purpose of this invention is to provide a method for processing brain image data. First, the 3D image of a brain MRI is divided into three 2D planes: axial (transverse), coronal, and sagittal. An AM-Unet++ (attention modules supplementally-Unet++) network is then established to extract features from regions of interest within the brain. Attention mechanism analysis is introduced, and finally, information is fused using a 3D-CNN fusion network to obtain the final 3D lesion segmentation result. This invention preserves the spatial contextual coherence of the 3D image while avoiding the high computational cost of 3D models.

[0006] The technical solution of this invention is as follows:

[0007] A brain imaging data processing method based on an improved Unet++ includes the following steps:

[0008] S1. Creating training data:

[0009] After preprocessing and normalization, the acquired raw MRI images were divided into three 2D image sequences: axial, coronal, and sagittal images.

[0010] S2. Construct a brain imaging data processing model, including an improved Unet++ network and a 3D fusion network:

[0011] An improved Unet++ network is obtained by adding an attention module to the Unet++ network. The input of the attention module is the encoder x. i-1，0 and decoder x i,4-i For any i ≤ 4, the attention module concatenates the two inputs to obtain the feature map F. i :

[0012] F i = <x i-1，0 x i，4-i >

[0013] Then, for feature map F i Downsampling is performed, followed by max pooling and average pooling to generate two 2D attention maps. and

[0014]

[0015]

[0016] Where D(·) represents the downsampling operation, and AvgPool(·) and MaxPool(·) represent the average pooling operation and the max pooling operation, respectively; then, a standard convolutional layer is used for concatenation and convolution operations to obtain the concatenated and feature-learned two-dimensional attention map, and A(·) represents the attention gate operation:

[0017]

[0018] Where σ(·) is the sigmoid function and f(·) is the convolution operation;

[0019] Each decoder node X i,j The formula for calculating the feature map stack is as follows:

[0020]

[0021] Where i represents the i-th layer downsampling along the encoder index, j represents the j-th convolutional layer skipping the connected index dense block, and U(·) represents the upsampling operation.<a,b> This means concatenating and splicing a and b together;

[0022] A 3D fusion network is constructed, comprising a 3D fusion main network, a 3D fusion sub-network, and a 3D fusion output network. The 3D fusion main network includes a first 3D convolutional layer, a first BN normalization layer, and a first ReLU activation function layer. The inputs to the first 3D convolutional layer are the predicted values ​​output from the improved Unet++ network. The predicted outputs extracted from the three 2D image sequences are defined as follows: In addition, the set weights ω1, ω2, and ω3 constitute the input of the first 3D convolutional layer. The output of the first 3D convolutional layer is passed through the first BN normalization layer and the first ReLU activation function layer to obtain the output of the 3D fusion main network. The 3D fusion sub-network includes a second 3D convolutional layer and a second BN normalization layer. The inputs of the second 3D convolutional layer are the outputs of the lowest encoder nodes of each branch of the improved Unet++ network, which, combined with the set weights, are defined as follows: The output of the second 3D convolutional layer is passed through the second BN normalization layer to obtain the output of the 3D fusion sub-network. The outputs of the 3D fusion main network and the 3D fusion sub-network are concatenated and then input into the 3D fusion output network. The 3D fusion output network includes a third 3D convolutional layer, a second ReLU activation function layer, and a softmax layer. The outputs of the 3D fusion main network and the 3D fusion sub-network are concatenated and then passed through the third 3D convolutional layer, the second ReLU activation function layer, and the softmax layer to finally obtain the softmax probability. The probability predicts the voxel value of each region. The loss function used is:

[0023]

[0024] Where si is the predicted voxel output, si > t means that the voxel is predicted to have a lesion, otherwise it is predicted that there is no lesion, t is a hyperparameter that is adjusted according to the model training situation; ti is the true lesion value, ti = 1 means that the voxel actually has a lesion, otherwise it actually has no lesion.

[0025] S3. Use the training data obtained in S1 to train the brain image data processing model constructed in S2 to obtain a trained brain image data processing model.

[0026] S4. Obtain brain imaging data, process it according to the method in S1, and then input it into the trained brain imaging data processing model to obtain the processing results.

[0027] The beneficial effects of this invention are as follows:

[0028] Compared with existing technologies, the technical solution proposed in this invention overcomes the problems of high computational cost of 3D models and loss of detail in 2D models in medical image models. Our model avoids the high resource usage of 3D models by slicing 3D images into 2D sequences and then processing the 2D images separately. Finally, a 3D fusion network is used to fuse and reconstruct the continuous sequence of 2D images into a 3D image, thus preserving the spatial contextual coherence of the 3D image. This invention cleverly solves existing problems, reduces resource usage while preserving spatial features, saves costs, and is more accurate. Attached Figure Description

[0029] Figure 1 This is the overall flowchart of the present invention.

[0030] Figure 2 This is a diagram showing the overall structure of the brain imaging data processing model in this invention.

[0031] Figure 3 This is a schematic diagram of the 2D plane being divided.

[0032] Figure 4 This is an AM-U-net++ network diagram.

[0033] Figure 5 It is attention module A i picture.

[0034] Figure 6 It is a 3D CNN fusion network diagram. Detailed Implementation

[0035] The present invention will now be described in detail with reference to the accompanying drawings.

[0036] like Figure 1 The diagram shows the overall process of this invention, including image slicing and preprocessing, feature extraction, and 3D fusion steps. The specific methods for each step are as follows:

[0037] Image preprocessing:

[0038] First, the original MRI images were registered using the SPM12 tool in MATLAB to avoid brain subject shift. The pixel data of the brain images were preprocessed. The original pixel values ​​were between 0 and 1000 and were normalized to the range of [0, 1].

[0039] The complete 3D image is divided into three 2D image sequences: the complete 3D MRI image is denoted as R. H×W×D Use NumPy slicing methods, such as Figure 3 As shown, it is divided into three 2D planes, denoted as: axial plane R1, coronal plane R2 and sagittal plane R3, where H×W×D represents the characteristic dimensions of length, width and depth.

[0040] Construct the AM-Unet++ network to extract features from 2D image sequences:

[0041] The improved U-net++ network proposed in this invention is an improvement on the symmetrical network structure U-net++. U-net++ is essentially a deep monitoring encoder-decoder network, where the encoder and decoder subnetworks are connected through a series of nested, dense skip paths. The aim is to use symmetrical upsampling to compensate for the information lost during downsampling, reduce the semantic gap between the encoder and decoder feature maps, and improve data utilization efficiency. The final output obtained by extracting features through the improved network is the predicted lesion region image.

[0042] The improved U-net++ network in this invention is defined as the AM-UNet++ network, and its structure is as follows: Figure 4 As shown. Using X i,j Let X represent the node of the j-th convolutional layer in the i-th layer, where i is the index of the downsampling layers along the encoder, and j is the index of the convolutional layers of dense blocks along the jump path. i,j The feature map stack computation can be categorized into the following types, where x i,j X represents i,j The output of the node, i max =4, 0≤i+j≤4:

[0043] 1. Encoder node in AM-Unet++ network

[0044] Each encoder node X i,0 The formula for calculating the feature map stack is as follows, x i,0 This is represented as the output of each encoder node:

[0045] x i，0 =f(D(x) i-1，0))

[0046] Where f(·) represents the convolution operation and D(·) represents the downsampling operation.

[0047] f(·) uses a 3×3 convolutional block with a stride of 1 and adds ReLU and BN layers to speed up network convergence and alleviate gradient explosion and vanishing problems in the network.

[0048] 2. Dense block convolutional layer node X of AM-Unet++ network i,j :

[0049]

[0050] Where f(·) represents the convolution operation, U(·) represents the upsampling operation, and the concatenated layers... Right now <x i ，0 , ..., x i,k , ..., x i，j >,<a,b> This means connecting a and b in series.

[0051] The purpose of dense block convolutional layers is to reduce the semantic gap between the feature maps of the encoder and decoder subnetworks, and to optimize the problem more directly.

[0052] 3. Use the attention module in the AM-Unet++ network to capture local details and preserve texture information.

[0053] like Figure 5 As shown, the spatial attention module of this model receives two inputs: one from the corresponding encoder x, which contains all contextual and spatial information from the (i-1)th layer of the network. i-1，0 The other input is from the decoder layer below it, x. i,4-i The two inputs are concatenated to obtain the feature map F. i as follows:

[0054] F i = <x i-1，0 x i，4-i >

[0055] The feature map is downsampled, and then max pooling and average pooling are performed respectively to generate two 2D attention maps. and They are represented as follows:

[0056]

[0057]

[0058] Where D(·) represents the downsampling operation, AvgPool(·) and MaxPool(·) represent the average pooling operation and the max pooling operation, respectively. Then, a standard convolutional layer is used for concatenation and convolution operations to obtain a concatenated and feature-learned two-dimensional attention map. A(·) represents the attention gate operation, which is calculated as follows:

[0059]

[0060] Where σ(·) is the sigmoid function and f(·) is the convolution operation.

[0061] The attention module of this model introduces max pooling and average pooling methods after feature map downsampling. Max pooling can reduce the deviation of the estimated mean caused by convolutional layer parameter errors and retain more texture information of brain images. Average pooling reduces the problem of increased variance of the estimated value caused by the limited neighborhood size and reduces the error caused by background information. The combination of the two helps to capture low-dimensional local detail features.

[0062] 4. Combine the decoder node of the attention module

[0063] Each decoder node X i,j The formula for calculating the feature map stack for (i≥0, j=4-i) is as follows:

[0064]

[0065] i represents the i-th layer downsampled along the encoder index, and j represents the j-th convolutional layer that skips the dense block of connected indexes;

[0066] The f(·) function represents the convolution operation, U(·) represents the upsampling operation, and A(·) represents the attention gate operation.<a,b> This means connecting a and b in series.

[0067] The final result is obtained by fusing the data using a 3D CNN fusion network:

[0068] The 3D CNN fusion network in this invention includes a main network and a sub-network structure. The main network captures the local appearance of the lesion structure, while the sub-network learns higher-level features, namely the relative position of the lesion area in the brain. The main and sub-networks are then fused to obtain the final prediction result. This fusion network structure can take into account the context and improve the prediction accuracy.

[0069] like Figure 6 As shown, the predicted lesion output images are extracted by processing the axial plane R1, coronal plane R2, and sagittal plane R3 of each path using the AM-Unet++ network. The input images are stacked together, and the weights for the three branches are designed as ω1, ω2, and ω3, respectively. Then the input of the node to be fused is: The parameter set W = {ω1, ω2, ω3}, where ω1 + ω2 + ω3 = 1 and 0 < ω1 < 1, 0 < ω2 < 1, 0 < ω3 < 1. By adjusting the parameter values ​​of each path, the influence of the three different 2D paths on the final prediction can be compared, thus obtaining the optimal prediction result and the corresponding parameter set.

[0070] The 3D stacked volume of the main network input is 3×(H×W×D). The main network of the combined input 3D-CNN fusion network is fused through a 3D CNN convolutional layer with a 3×3×3 kernel, a BN normalization layer, and a ReLU activation function layer to obtain the predicted lesion area. The ReLU+BN layer can reduce the possibility of gradient explosion and gradient vanishing problems and prevent overfitting.

[0071] Simultaneously, by combining the lowest encoder node of each branch of the AM-Unet++ network (i.e., the output of the lowest downsampled node), the weights of each branch are designed, and the input of the node to be fused is: The 3D stack volume is 3×(H×W×D). The input is to the sub-network, which passes through a 3D CNN convolutional layer with a 3×3×3 kernel and a BN normalization layer. The fused output is used as a supplement to the contextual location information of the predicted lesion region.

[0072] After processing by the main and auxiliary networks, the output is 2×(H×W×D), with two channels, each with a size of H×W×D. One channel is obtained from the fusion of the main network; the second channel is obtained from the auxiliary network. The two channel outputs are concatenated and then passed through a 3D CNN convolutional layer with a 3×3×3 kernel, a ReLU activation function layer, and a softmax normalized exponential function. The final output is a softmax probability, which predicts the voxel value of each region, with a prediction range of [0,1]. The voxel value of the background (i.e., the normal region) is 0. Therefore, the dice loss function L is used. f The calculation is as follows:

[0073]

[0074] Where si is the predicted voxel output, si>t means that the voxel is predicted to have a lesion, otherwise it is predicted that there is no lesion, t is a hyperparameter that is adjusted according to the model training situation; ti is the true lesion value, ti=1 means that the voxel actually has a lesion, otherwise it actually has no lesion.

[0075] After training the above network model, a model for processing brain imaging data can be obtained. Therefore, this invention provides a method for processing brain imaging data of stroke patients. Based on the neuroimaging data (head MRI images) of the patient provided by the hospital, the images are preprocessed according to the characteristics of neuroimaging. The 3D image is divided into three 2D planar images, which are respectively input into an AM-Unet++ (attention modules supplementally-Unet++) convolutional neural network with a multi-level attention mechanism to extract and select imaging features. Then, the images are input into a feature fusion network to generate the patient's total feature representation. The processed and predicted lesion area of ​​the patient is obtained, which facilitates the physician to carry out active symptomatic intervention. High-risk patients can be screened for key attention based on this.

Claims

1. A brain imaging data processing method based on an improved Unet++, characterized in that, Includes the following steps: S1. Creating training data: After preprocessing and normalization, the acquired raw MRI images were divided into three 2D image sequences: axial, coronal, and sagittal images. S2. Construct a brain imaging data processing model, including an improved Unet++ network and a 3D fusion network: An improved Unet++ network is obtained by adding an attention module to the Unet++ network. Let represent the node of the j-th convolutional layer in the i-th layer, where i is the index of the downsampling layers along the encoder, and j is the index of the convolutional layers of the dense blocks along the jump path. This represents the encoder nodes of the improved Unet++ network. The formula for calculating the feature map stack is as follows: ， in This represents the convolution operation. This indicates a downsampling operation. Use convolutional blocks with a stride of 1 and a kernel size of 3×3, and add ReLU and BN layers; Dense block convolutional layer nodes of the improved Unet++ network : ， ， in, The function represents the convolution operation. Indicates upsampling operation, concatenated layer Right now , This means concatenating and splicing a and b together; The input to the attention module is the encoder. and decoder , The attention module concatenates the two inputs to obtain the feature map. : = ， Then examine the feature map Downsampling is performed, followed by max pooling and average pooling to generate two 2D attention maps. and : ， ， in, This indicates a downsampling operation. and These represent the average pooling operation and the max pooling operation, respectively; then, a standard convolutional layer is used for concatenation and convolution operations to obtain a concatenated and feature-learned two-dimensional attention map, which is then used... Indicate that you should pay attention to door operation: ， in, ( () is the sigmoid function. This is a convolution operation; Each decoder node The formula for calculating the feature map stack is as follows: ， Where i represents the i-th layer downsampled along the encoder index, and j represents the j-th convolutional layer skipping the connected index dense block. Indicates an upsampling operation. This means concatenating and splicing a and b together; A 3D fusion network is constructed, comprising a 3D fusion main network, a 3D fusion sub-network, and a 3D fusion output network. The 3D fusion main network includes a first 3D convolutional layer, a first BN normalization layer, and a first ReLU activation function layer. The inputs to the first 3D convolutional layer are the predicted values ​​output from the improved Unet++ network. The predicted outputs extracted from the three 2D image sequences are defined as follows: , In addition to the set weights , , The inputs that constitute the first 3D convolutional layer are respectively , , The output of the first 3D convolutional layer is passed through the first BN normalization layer and the first ReLU activation function layer to obtain the output of the 3D fusion main network. The 3D fusion sub-network includes a second 3D convolutional layer and a second BN normalization layer. The inputs of the second 3D convolutional layer are the outputs of the lowest encoder node of each branch of the improved Unet++ network, which, combined with the set weights, are defined as follows: , , The output of the second 3D convolutional layer is passed through the second BN normalization layer to obtain the output of the 3D fusion sub-network. The outputs of the 3D fusion main network and the 3D fusion sub-network are concatenated and then input into the 3D fusion output network. The 3D fusion output network includes a third 3D convolutional layer, a second ReLU activation function layer, and a softmax layer. The outputs of the 3D fusion main network and the 3D fusion sub-network are concatenated and then passed through the third 3D convolutional layer, the second ReLU activation function layer, and the softmax layer to finally obtain the softmax probability. The probability predicts the voxel value of each region. The loss function used is: ， Where si is the predicted voxel output, si>t means that the voxel is predicted to have a lesion, otherwise it is predicted that there is no lesion, t is a hyperparameter that is adjusted according to the model training situation; ti is the true lesion value, ti=1 means that the voxel actually has a lesion, otherwise it actually has no lesion. S3. Use the training data obtained in S1 to train the brain image data processing model constructed in S2 to obtain a trained brain image data processing model. S4. Obtain brain imaging data, process it according to the method in S1, and then input it into the trained brain imaging data processing model to obtain the processing results.

Images