A brain tissue segmentation method based on image semantic information fusion

Abstract

The application relates to a brain tissue segmentation method based on image semantic information fusion, and belongs to the technical field of digital image processing, and comprises the following steps: S1: acquiring a multi-modal brain tissue MR image dataset, dividing the dataset into a training set and a test set; the data of the training set and the test set are blocked, that is, three-dimensional data are cropped and pretreated; S2: T1 weighted mode and T2 weighted mode MR images in the training set are respectively processed through two encoding streams, features of the obtained two different modes are fused, and fused features are obtained; S3: the fused features are used to calculate the similarity between the cut pieces in a position transformation mode; S4: feature mapping is selected from the previous two layers of encoders, boundary feature mapping with fine-grained constraints is provided in a deep supervision mode to assist segmentation; and S5: after training is completed, the brain tissue segmentation model constructed by steps S2-S4 is tested by using the test set.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology and relates to a brain tissue segmentation method based on image semantic information fusion. Background Technology

[0002] Magnetic resonance imaging (MRI) is a non-invasive imaging technique that does not cause ionizing radiation damage to the human body and is the primary diagnostic tool for brain diseases in infants and young children. Brain tissue segmentation in MRI is a crucial step in quantifying brain structural volume, studying infant brain development patterns, and treating adult diseases. Accurately segmenting infant brain MRI images into white matter, gray matter, and cerebrospinal fluid is very helpful in studying early brain development and assessing its progression. In multimodal MR images, T1-weighted modality enhances the contrast between white and gray matter, while T2-weighted modality highlights lesion areas in brain tissue. Different modalities can provide different intensity distributions of brain tissue regions, which helps clinicians observe differences between brain tissue regions. Therefore, complementary information between multimodal MR images can guide the brain tissue segmentation process, resulting in more accurate segmentation results. However, due to the blurred boundaries, low contrast, and complex anatomical relationships of brain tissue regions, manual segmentation of brain tissue regions is very difficult, especially since many brain structures are formed in the neonatal period and have similar intensity distributions in MR images. Therefore, segmenting brain structures based on intensity thresholds is quite challenging.

[0003] Fully Convolutional Networks (FCNs) are deep learning-based semantic segmentation networks that can achieve end-to-end automatic image segmentation and are widely used in medical image segmentation. In addition, various variants of U-Net networks have proven effective in segmenting brain structures. However, accurate segmentation of neonatal brain MR images remains a challenging task due to several issues: 1) First, most networks ignore the anisotropic resolution of brain MR images, i.e., inconsistent intra- and inter-slice resolution. For example, high intra-slice resolution and low inter-slice resolution result in different volumetric image sizes for each dimension. Typical 3D FCNs use fixed-size convolutional kernels (e.g., 3x3x3), making it difficult to simultaneously capture sufficient intra- and inter-slice information. 2) Second, there is a lack of capture of long-range dependency information. Besides local and global features, in-plane long-range dependency information can also guide tissue segmentation in brain MR images. 3) Multimodal MR images contain complementary information from different modalities. Existing networks generally use attention mechanisms to guide brain tissue segmentation networks to automatically focus on effective spatial and channel information, but existing attention modules cannot utilize the complementary information of multimodal MR images. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a brain tissue segmentation method based on image semantic information fusion. By extracting and fusing image semantic information from multiple modalities and introducing long-range dependency information of the image, and under the constraint of boundary auxiliary information, accurate segmentation of brain tissue can be achieved.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A brain tissue segmentation method based on image semantic information fusion includes the following steps: S1: Obtain a multimodal brain tissue MR image dataset, and divide the dataset into a training set and a test set; divide the data in the training set and the test set into blocks, that is, crop the 3D data and preprocess it; S2: Process the T1-weighted modal and T2-weighted modal MR images in the training set through two encoding streams respectively, and fuse the features of the two different modalities to obtain fused features; S3: Calculate the similarity between the segments by transforming the fused features; S4: Select feature maps from the first two encoder layers to provide boundary feature maps with fine-grained constraints in a deeply supervised manner to assist in segmentation; S5: After training is complete, test the brain tissue segmentation model constructed in steps S2-S4 using the test set.

[0006] Furthermore, step S1 specifically includes the following steps: S11: Obtain the brain tissue dataset and divide the dataset into training and testing sets; S12: Use the FSL tool to perform skull dissection and intensity non-uniformity correction on the image; S13: Perform z-score normalization on the non-background pixels of the T1 and T2 modality images. This involves subtracting the mean from the non-background image pixels and dividing by the variance to make all data cluster around 0, with a variance of 1. The specific formula is as follows:

[0007] in The mean of the original data. The variance of the original data; S14: Segment the brain tissue image, that is, crop the three-dimensional data into blocks. The size.

[0008] Furthermore, step S2 includes the following steps: S21: Two initial feature tensors are obtained after the first layer encoding module. , The specific formula for the encoding module is as follows:

[0009] in, Indicates that the convolution kernel is A 3D convolution operation with a stride of 1. express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. It is the input feature map. , These are the initial feature tensors of the two outputs and , ; S22: Use the two feature tensors obtained in step S21 as input for cross-modal feature fusion, and concatenate the first-dimensional feature tensor along the channel dimension, as follows:

[0010] Here, [...] represents the concatenation operation, and the tensor... This involves stitching together multimodal image features. Next, regarding the spliced ​​features Perform Max-pooling and Average-pooling operations to obtain global context information along the channel dimension, denoted as... and The tensor representation of global information after pooling is as follows: The spatial information of the entire channel dimension is represented as: ; S23: The spatial information of the channel dimension obtained in step S22 is fed into a pyramid structure embedded with anisotropic convolution. This pyramid structure has three branches, each branch decomposing the three-dimensional convolution into a kernel of size [missing information]. The 3D convolution and the kernel size are 3D convolution, where ; Then merge the tensors generated by the three branches to obtain a tensor of size . Spatial context information, through Convolutions are used to adjust the number of channels, and normalized BN and Leaky are used. Generate feature tensors that incorporate multi-scale spatial information ; Finally, the features are concatenated with the initial multimodal fusion features to obtain fusion features with different levels, represented as follows: .

[0011] Furthermore, step S3 specifically includes the following steps: S31: Input the fused features at different levels obtained in step S23 into the rotation-based Transformer layer and rotate the feature map in multiples of 90°. S32: Each patch undergoes 4 rotations followed by convolution operations to generate 4-channel feature maps; then max pooling is used to obtain the output of the rotated layer. S33: The output of the rotated layer obtained in step S32 is fed into the Transformer to calculate the similarity between sequences; then, a linear mapping is used to integrate spatial information through position encoding; the Transformer block consists of multi-layer self-attention MSA and multi-layer perceptron MLP operations, which include layer normalization operations; after n-1 Transformer layers are stacked, the output of the nth layer is calculated as follows:

[0012]

[0013] in, Normalization operation of the presentation layer Indicates the first The output of each Transformer layer.

[0014] Furthermore, step S4 specifically includes the following steps: S41: Concatenate the two initial feature tensors of the first layer obtained in step S22. and the fusion feature tensor of the second layer ,go through Convolution and The convolution is fed into the boundary auxiliary module, represented as:

[0015]

[0016] in, Indicates that the convolution kernel is A 3D convolution operation with a stride of 1. Indicates that the convolution kernel is , express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. Represents the eigenvector; S42: Obtain the result from step S41 The image resolution is restored by performing residual connections and convolution operations, followed by deconvolution. This process can be represented as follows:

[0017]

[0018]

[0019]

[0020] Where deconv3d represents deconvolution. express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. Represents the eigenvector; S43: Obtain boundary auxiliary information from step S42 The decoded stream is concatenated with the SE-block weighted output, and then deconvolved and 1 1 1. Convolution completes the final segmentation. The specific process is as follows:

[0021]

[0022]

[0023] in, Represents the number of floors. Indicates the SE module, This indicates a splicing operation.

[0024] The beneficial effects of this invention are as follows: by extracting and fusing the semantic information of MR images themselves, and by using the idea of ​​boundary-assisted segmentation, the problems of lack of long-range dependency information and multimodal information extraction and fusion in brain tissue images are solved, thereby improving the accuracy of brain tissue segmentation tasks.

[0025] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a network structure diagram of the brain tissue segmentation model in this invention; Figure 2 This is a network structure diagram of the cross-modal feature fusion network in this invention; Figure 3 This is a schematic diagram of the Transformer based on position transformation in this invention; Figure 4 This is a network structure diagram of the boundary auxiliary constraint in this invention. Detailed Implementation

[0027] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0028] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0029] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0030] Please see Figures 1-4 This invention provides a brain tissue segmentation method based on image semantic information fusion, which includes the following steps: Step 1: Data Preparation Stage: Obtain the brain tissue dataset and divide it into training and testing sets. Then, the data in the training and testing sets is further segmented, meaning the 3D data is cropped and preprocessed, including skull dissection and intensity non-uniformity correction.

[0031] Step 101: Obtain the brain tissue dataset and divide the dataset into a training set and a test set; Step 102: Use the FSL tool to perform skull dissection and intensity non-uniformity correction on the image; Step 103: Perform z-score normalization on the non-background pixels of the T1 and T2 modality images. This involves subtracting the mean from the non-background image pixels and dividing by the variance to make all data cluster around 0, with a variance of 1. The specific formula is as follows:

[0032] in The mean of the original data. The variance of the original data; Step 104: Segment the brain tissue image, that is, crop the three-dimensional data into blocks. The size.

[0033] Step 2: Cross-modal feature fusion stage: The two modal MR images are processed by two encoding streams respectively, and the features of the two different modalities are fused to obtain fused features.

[0034] Step 201: Two initial feature tensors are obtained after passing through the first layer encoding module. , The specific formula for the encoding module is as follows: (1) in Indicates that the convolution kernel is A 3D convolution operation with a stride of 1. express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. It is the input feature map. , These are the initial feature tensors of the two outputs and , .

[0035] Step 202: Use the two feature tensors obtained in Step 201 as input for cross-modal feature fusion, and concatenate the first-dimensional feature tensor (channel dimension), which can be represented as: (2) Here, [...] represents the concatenation operation, and the tensor... This involves stitching together multimodal image features. Next, the stitched features are... Performing Max-pooling and Average-pooling operations not only obtains global context information along the channel dimension but also reduces some parameters. Let's denote this as... and Therefore, the tensor of global information after pooling can be represented as: The spatial information of the entire channel dimension is represented as follows: .

[0036] Step 203: The spatial information of the channel dimension obtained in Step 202 is fed into a pyramid structure embedded with anisotropic convolutions to facilitate the extraction of multi-scale spatial information. This pyramid structure has three branches, each branch decomposing the 3D convolution into convolution kernels with a kernel size of... The 3D convolution and the kernel size are 3D convolution. Wherein Then, the tensors generated by the three branches are merged to obtain a tensor of size . Spatial context information. To avoid increasing computational complexity, by... Convolutions are used to adjust the number of channels, and normalization (BN) and Leaky are used. Generate feature tensors that incorporate multi-scale spatial information Finally, the features are concatenated with the initial multimodal fusion features to obtain fusion features with different levels, which can be represented as: (3) Step 3: Capturing remote dependency information stage: The fusion features obtained in Step 2 are transformed (rotated) to better calculate the similarity between the segments, so as to utilize the image semantic information in the brain structure and integrate the boundary information in the brain structure.

[0037] Step 301: Input the fused features of different levels obtained in step 203 into the rotation-based Transformer layer, first rotating the feature map in multiples of 90°.

[0038] Step 302: After each patch undergoes 4 rotations, a convolution operation is performed to generate a 4-channel feature map. Then, max pooling is used to obtain the output of the rotated layer. Step 303: The output of the rotated layer obtained in step 302 is fed into the Transformer to calculate the similarity between sequences. Spatial information is then integrated through linear mapping and positional encoding. The Transformer block consists of multilayer self-attention (MSA) and multilayer perceptron (MLP) operations, including layer normalization operations. After stacking n-1 Transformer layers, the output of the nth layer can be calculated as: (4) (5) in, Normalization operation of the presentation layer Indicates the first The output of each Transformer layer.

[0039] Step 4: Boundary Assistance Stage: Select feature maps from the first two encoder layers to provide boundary feature maps with fine-grained constraints in a deeply supervised manner to assist segmentation.

[0040] Step 401: Concatenate the two initial feature tensors of the first layer obtained in step 202. and the fusion feature tensor of the second layer ,go through Convolution and The convolutional input to the boundary auxiliary module can be represented as: (6) (7) in, Indicates that the convolution kernel is A 3D convolution operation with a stride of 1. Indicates that the convolution kernel is , express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. This represents the eigenvector.

[0041] Step 402: Take the result obtained in step 401 The process of restoring image resolution involves performing residual connections and convolution operations, followed by deconvolution. This process can be expressed by the following formula: (8) (9) (10) (11) Where deconv3d represents deconvolution. express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. This represents the eigenvector.

[0042] Step 403: Obtain the boundary auxiliary information from step 402 The decoded stream is concatenated with the SE-block weighted output, and then deconvolutioned. Convolution completes the final segmentation. The specific process can be represented as follows: (12) (13) (14) in, The value range for the representative layer number is set in this invention as follows: , Indicates the SE module, This indicates a splicing operation.

[0043] Step 5: After training is complete, test the brain tissue segmentation model constructed in steps 2-4 using the test set.

[0044] To verify the effectiveness of the present invention, the following experiments were conducted: A 3D FCN network was constructed based on this method and tested on the iSeg2017 dataset. Ten samples were selected from the iSeg2017 dataset for experiments. The dataset aims to segment infant brain tissue into cerebrospinal fluid (CSF), gray matter (GM), and white matter (WM). Nine brain samples were used to train the neural network, and the remaining sample was used for testing. The experiment employed 10-fold cross-validation. The method of this invention (Method 5) was compared with 3D UNet (Method 1), VoxResNet (Method 2), 3D-SkipDense-Seg (Method 3), and Non-Local Unet (Method 4), using the Dice coefficient as the evaluation metric. (15) Here, A represents the result of network segmentation, and B represents the gold standard given by the dataset.

[0045] Table 1 presents the test results on the database. It can be seen that, in terms of the Dice coefficient, the neural network based on this invention performs better in each segmentation result.

[0046] Table 1

[0047] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it can implement the steps of the method. The storage medium may be, for example, ROM / RAM, magnetic disk, optical disk, etc.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A brain tissue segmentation method based on image semantic information fusion, characterized in that: Includes the following steps: S1: Obtain a multimodal brain tissue MR image dataset and divide the dataset into a training set and a test set; The training and test sets are divided into blocks, which involves cropping and preprocessing the 3D data. S2: Process the T1-weighted modal and T2-weighted modal MR images in the training set through two encoding streams respectively, and fuse the features of the two different modalities to obtain fused features; S3: Calculate the similarity between the segments by transforming the fused features; S4: Select feature maps from the first two encoder layers to provide boundary feature maps with fine-grained constraints in a deeply supervised manner to assist in segmentation; S5: After training is complete, test the brain tissue segmentation model constructed in steps S2-S4 using the test set; Step S2 includes the following steps: S21: Two initial feature tensors are obtained after the first layer encoding module. , The specific formula for the encoding module is as follows: in, Indicates that the convolution kernel is A 3D convolution operation with a stride of 1. express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. It is the input feature map. , These are the initial feature tensors of the two outputs and , ; S22: Use the two feature tensors obtained in step S21 as input for cross-modal feature fusion, and concatenate the first-dimensional feature tensor along the channel dimension, as follows: Here, [...] represents the concatenation operation, and the tensor... This involves stitching together multimodal image features. Next, regarding the spliced ​​features Perform Max-pooling and Average-pooling operations to obtain global context information at the channel level, denoted as... and The tensor representation of global information after pooling is as follows: The spatial information of the entire channel dimension is represented as: ; S23: The spatial information of the channel dimension obtained in step S22 is fed into a pyramid structure embedded with anisotropic convolution. This pyramid structure has three branches, each branch decomposing the three-dimensional convolution into a kernel of size [missing information]. The 3D convolution and the kernel size are 3D convolution, where ; Then merge the tensors generated by the three branches to obtain a tensor of size . Spatial context information, through Convolutions are used to adjust the number of channels, and normalized BN and Leaky are used. Generate feature tensors that incorporate multi-scale spatial information ; Finally, the features are concatenated with the initial multimodal fusion features to obtain fusion features with different levels, represented as follows: Step S3 specifically includes the following steps: S31: Input the fused features at different levels obtained in step S23 into the rotation-based Transformer layer and rotate the feature map in multiples of 90°. S32: Each patch undergoes 4 rotations followed by convolution operations to generate 4-channel feature maps; then max pooling is used to obtain the output of the rotated layer. S33: The output of the rotated layer obtained in step S32 is fed into the Transformer to calculate the similarity between sequences; then, a linear mapping is used to integrate spatial information through position encoding; the Transformer block consists of multi-layer self-attention MSA and multi-layer perceptron MLP operations, which include layer normalization operations; after n-1 Transformer layers are stacked, the output of the nth layer is calculated as follows: in, Normalization operation of the presentation layer Indicates the first The output of each Transformer layer; Step S4 specifically includes the following steps: S41: Concatenate the two initial feature tensors of the first layer obtained in step S22. and the fusion feature tensor of the second layer ,go through Convolution and The convolution is fed into the boundary auxiliary module, represented as: in, Indicates that the convolution kernel is A 3D convolution operation with a stride of 1. Indicates that the convolution kernel is , express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. Represents the eigenvector; S42: Obtain the result from step S41 The image resolution is restored by performing residual connections and convolution operations, followed by deconvolution. This process can be represented as follows: Where deconv3d represents deconvolution. express Activation function This represents a max-pooling operation with 2 kernels and a step size of 2. Represents the eigenvector; S43: Obtain boundary auxiliary information from step S42 The decoded stream is concatenated with the SE-block weighted output, and then deconvolved and 1 1 1. Convolution completes the final segmentation. The specific process is as follows: in, Represents the number of floors. Indicates the SE module, This indicates a splicing operation.

2. The brain tissue segmentation method based on image semantic information fusion according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Obtain the brain tissue dataset and divide the dataset into training and testing sets; S12: Use the FSL tool to perform skull dissection and intensity non-uniformity correction on the image; S13: Perform z-score normalization on the non-background pixels of the T1 and T2 modality images. This involves subtracting the mean from the non-background image pixels and dividing by the variance to make all data cluster around 0, with a variance of 1. The specific formula is as follows: in The mean of the original data. The variance of the original data; S14: Segment the brain tissue image, that is, crop the three-dimensional data into blocks. The size.

Images