A method for segmenting and extracting fetal brain images from twin MR images

Abstract

The present invention discloses a method for segmenting and extracting fetal brain images from twin MR images. The method includes training a target detector using a twin brain dataset with bounding box labels. The trained target detector is then tested on the twin brain dataset to obtain a twin brain dataset after removing maternal tissue. The single fetal brain dataset is then input into a segmentation network to encode the fetal brain region and extract initial low-level features. Low-resolution features are upsampled to obtain deep high-level features. Low-level features and high-level features of the same resolution are concatenated using skip connections. The output feature map is calculated to obtain a probability output, which is then input into a cross-entropy loss function along with the corresponding labels to calculate the loss value. The network parameters are then updated to obtain a pre-trained model. The pre-trained model is then loaded, and the twin brain dataset, after having been subjected to the target detector and removed maternal tissue, is input into a segmentation network and trained to obtain the final segmentation model. The present invention can quickly and efficiently segment and extract twin brain images.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a method for segmenting and extracting fetal brain images in twin MR images. Background Art

[0002] In recent years, deep learning methods have achieved great success in medical image segmentation and computer-aided diagnosis. Deep learning networks can automatically learn the contextual information of images through convolution operations, thereby obtaining better image feature representations.

[0003] A multi-scale loss function in the prior art is used to train a 2D P-Net to localize and segment the fetal brain. The 2D P-Net consists of six convolutional blocks with dilated convolutions. The dilation parameters of the first five convolutional blocks are 1, 2, 4, 8, and 16, respectively, so they have features at different scales. The features from these five blocks are concatenated and input into the sixth convolutional block, which serves as a classifier. Finally, a Softmax layer is used to obtain a probability output. This prior art localization method is to perform a coarse segmentation of the fetal brain region and perform morphological opening and closing operations. The maximum connected domain is used to find the largest marked area as the fetal brain region, and a bounding box is generated based on this area as the localization result. The localization result still depends heavily on the coarse segmentation result. This solution is for single fetuses, and there is currently no brain extraction solution specifically for twins.

[0004] It can be seen that the existing technology has the following shortcomings: (1) It can only locate one brain region at a time, resulting in the segmentation network being unable to learn the unique structural information between the twin brains. (2) Although 2D P-Net has multi-scale features, it does not fully combine low-level features with high-level features. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for segmenting and extracting fetal brain images in twin MR images.

[0006] The technical solution adopted in the present invention is:

[0007] A method for segmenting and extracting fetal brain images from twin MR images comprises the following steps:

[0008] Step S1: Train an object detector using the twin brain dataset with bounding box labels;

[0009] Step S2: Testing the twin brain dataset with the trained object detector to obtain the twin brain dataset after removing maternal tissue;

[0010] Step S3: Input the single-cell fetal brain dataset into the encoder of the segmentation network to encode the fetal brain region and extract initial low-level features;

[0011] Step S4: After the encoding process is completed, the obtained low-resolution features are input into the decoder of the segmentation network for upsampling to obtain deep high-level features;

[0012] Step S5: The low-level features and high-level features of the same resolution obtained during encoding and decoding are concatenated through skip connections to incorporate contextual information.

[0013] Step S6: Perform a 1×1 convolution operation and a sigmoid function on the output feature map to obtain the probability output, and input it together with the corresponding label into the cross entropy loss function to calculate the loss value, and then update the network parameters to obtain the pre-trained model.

[0014] Step S7: Load the pre-trained model and input the twin brain dataset from which the maternal tissue has been removed by the object detector into the segmentation network. Repeat steps S3 to S6 to obtain the final segmentation model.

[0015] Step S8: Input the brain MR images of the twins in the test set into the segmentation model output to obtain the fetal brain.

[0016] Furthermore, in step S1, YOLOv7 is used as the target detector of the framework, and a bounding box is used to wrap the two brains in the twin MR images. The specific formula of the loss function for constraining the bounding box is as follows:

[0017]

[0018] Where v is a parameter that measures the consistency of aspect ratio, μ is a weight function, IoU(·) represents the calculation of the intersection over union (IoU) of two bounding boxes, A and B are the predicted value and the true value of the bounding box, respectively, c is the diagonal distance of the smallest rectangular area that contains both the predicted bounding box and the corresponding true value, ρ is the Euclidean distance between the center points of A and B, w and h are the width and height of B, and w′ and h′ are the width and height of A.

[0019] Furthermore, each layer of feature extraction block in step S3 includes a layer of NAM, which further suppresses less significant features after capturing the features.

[0020] Furthermore, the backbone structure of the segmentation network in step S3 adopts a classic encoder-decoder architecture; the encoder path consists of four special feature learning blocks; the feature learning block learns the features of the twin brain MR images through two 3×3 convolutional layers with batch normalization (BN) and ReLU functions, and a 2×2 pooling operation is followed by a normalization-based attention module (NAM) to adaptively aggregate the learned feature maps in a task-oriented manner and further suppress insignificant features.

[0021] Furthermore, in step S3, NAM is expressed as:

[0022]

[0023] F out =Sigmoid(ω×BN(F in )) (7)

[0024] Where x is the input of BN(·), k and m represent the mean and standard deviation of each batch, σ is a constant to ensure that the denominator is not 0, α and β are trainable parameters, and α is used as the weight scaling factor, ω is the function weight, and F in and F out Represents the input and output features of NAM. Compared with other attention mechanisms, NAM does not have additional operations such as convolution and full connection, while maintaining highly competitive performance.

[0025] Furthermore, in step S4, bilinear interpolation is used to upsample the feature map in the decoder path.

[0026] Furthermore, the 1×1 convolutional layer and Sigmoid function in step S6 are used to generate the probability map of the twin brain extraction results. p m ∈R represents the probability that the mth pixel belongs to the twin fetal brain region; the cross entropy loss function is applied to train the segmentation network based on the label corresponding to the input image, and the formula is as follows:

[0027]

[0028] Where I is the batch size, g m,n is the label of the mth pixel in the nth input image in each batch; p m,n represents the probability predicted by the network that the mth pixel of each batch of nth input images belongs to the twin brain region.

[0029] The present invention employs the above technical solution and has the following advantages over existing technologies: (1) It locates all brain regions at once, preserving structural information between the twin brains. (2) It addresses the shortcomings of the segmentation network and strengthens the relationship between low-level and high-level features. (3) It addresses the problem of insufficient twin data samples, which prevents the segmentation network from being fully trained. (4) It removes erroneous pixels that may be generated by the segmentation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0031] Figure 1 A schematic diagram of the framework principle of a method for segmenting and extracting fetal brain images from twin MR images of the present invention;

[0032] Figure 2 Schematic diagram of the structure of the segmented network of the present invention. Implementation Method

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0034] like Figure 1 As shown in FIG2 , the present invention discloses a method for segmenting and extracting fetal brain images in twin MR images, which comprises the following steps:

[0035] Step S1: Train an object detector using a twin brain dataset with bounding box labels; introduce an object detector to lock the location of all fetal brains in maternal tissue, which can reduce false positives and preserve information between twin brains.

[0036] Step S2: Testing the twin brain dataset with the trained object detector to obtain the twin brain dataset after removing maternal tissue;

[0037] Step S3: The single-child fetal brain dataset is fed into the encoder of the segmentation network, which encodes the fetal brain regions and extracts initial low-level features. The segmentation network strengthens the relationship between low-level and high-level features and further suppresses insignificant features. It can also learn common features between single and twin fetal brains, improving network model performance.

[0038] Step S4: After the encoding process is completed, the obtained low-resolution features are input into the decoder of the segmentation network for upsampling to obtain deep high-level features.

[0039] Step S5: The low-level features and high-level features of the same resolution obtained during encoding and decoding are concatenated through skip connections and combined with contextual information.

[0040] Step S6: Perform a 1×1 convolution operation and a sigmoid function on the output feature map to obtain the probability output, input it together with the corresponding label into the cross entropy loss function to calculate the loss value, and update the network parameters to obtain the pre-trained model.

[0041] Step S7: Load the pre-trained model, input the twin brain dataset from which maternal tissue has been removed by the object detector into the segmentation network, and repeat steps S3 to S6 to obtain the final segmentation model. Specifically, to address the problem of insufficient twin datasets and the inability to train the segmentation network from scratch, a two-step training strategy based on transfer learning is proposed. This strategy has three advantages: 1) Before the second training step, the initial performance of the model is high. 2) During the training process, the model converges faster. 3) After training, the model performance is stronger. This is because after the first training step, the segmentation network has already learned certain features of the single fetus brain, and the second training step only requires fine-tuning the weights based on this.

[0042] The specific working principle of the present invention is described in detail below:

[0043] The present invention comprises two parts, such as Figure 1 As shown. Specifically, in order to relieve the pressure on the segmentation network, the region of interest is narrowed to the vicinity of the brain in the first part to exclude the interference of maternal tissue. This is achieved based on target detection. In particular, in order to retain the valuable structural information between the two brains, the present invention detects all brains at once instead of locating them one by one. In the second part, a new segmentation network is proposed to extract the twin brains located in the first stage, and a normalization-based attention module (NAM) is used to suppress non-significant features. At the same time, a two-stage training strategy is proposed to improve the performance of the segmentation model, and the results of the segmentation network are corrected according to the anatomical structure of the twin brains.

[0044] Twin brain detection: Real-time target detection has a wide range of applications, including medical image analysis. Following the principles of speed and efficiency, the present invention first uses YOLOv7 as the target detector of the framework. This detector can directly output the positioning result based on the bounding box regression technology. In particular, the present invention only uses one bounding box to wrap the two brains in the twin MR images, instead of assigning a bounding box to each brain. This has two purposes: 1) remove maternal tissue outside the brain area to reduce false positives in the segmentation results; 2) retain the key structural features between the two brains. The loss function used to constrain the bounding box takes into account the overlapping area, center distance, and aspect ratio. The specific formula is as follows:

[0045]

[0046]

[0047] Where v is a parameter that measures the consistency of aspect ratio, μ is a weight function, IoU(·) represents the calculation of the intersection over union (IoU) of two bounding boxes, A and B are the predicted value and the true value of the bounding box, respectively, c is the diagonal distance of the smallest rectangular area that contains both the predicted bounding box and the corresponding true value, ρ is the Euclidean distance between the center points of A and B, w and h are the width and height of B, and w′ and h′ are the width and height of A.

[0048] Segmented network: The network framework proposed in this invention is as follows Figure 2 As shown in the figure, the backbone structure adopts the classic encoder-decoder structure. The encoder path consists of four special feature learning blocks. The present invention first adds two 3×3 convolutional layers together with batch normalization (BN) and ReLU functions to the feature learning block to learn the features of the twin brain MR images. After a layer of 2×2 pooling operation, it is followed by a normalization-based attention module (NAM), which adaptively aggregates the learned feature maps in a task-oriented manner and further suppresses insignificant features. NAM can be summarized by the formula:

[0049]

[0050] F out =Sigmoid(ω×BN(F in )) (7)

[0051] Where x is the input of BN(·), k and m represent the mean and standard deviation of each batch, σ is a constant to ensure that the denominator is not 0, α and β are trainable parameters, and α is used as the weight scaling factor, ω is the function weight, and F in and F out Represents the input and output features of NAM. Compared with other attention mechanisms, NAM does not have additional operations such as convolution and full connection, while maintaining highly competitive performance.

[0052] In the decoder path, the present invention uses bilinear interpolation to upsample the feature map. The upsampled feature map is then connected to the relatively high-resolution feature map in the encoder path through a skip connection operation. Therefore, the segmentation network of the present invention aggregates local spatial features from the encoder and high-level contextual features from the decoder. The last part of the decoder is a 1×1 convolutional layer and a sigmoid function to generate a probability map of the twin brain extraction results. p m∈R represents the probability that the mth pixel belongs to the twin fetal brain region. Based on the labels corresponding to the input images, the present invention applies the cross entropy loss function to train the segmentation network, and the formula is as follows:

[0053]

[0054] Where w and h are the width and height of the input image, I is the batch size, and g m,n is the label of the mth pixel in the nth input image in each batch. m,n represents the probability predicted by the network that the mth pixel of each batch of nth input images belongs to the twin brain region.

[0055] Two-step training strategy: To address the problem of insufficient twin datasets and the inability to train the segmentation network from scratch, a two-step training strategy based on transfer learning is proposed. First, the segmentation network is trained using a large single-fetal brain dataset to generate a pre-trained model. Second, the pre-trained model is loaded and input into the twin dataset for weight fine-tuning to obtain the final model. This training strategy has three advantages: 1) Before the second training step, the model's initial performance is high. 2) During training, the model converges faster. 3) After training, the model performance is stronger. This is because after the first training step, the segmentation network has already learned certain features of the single-fetal brain, and the second training step only requires fine-tuning the weights based on this.

[0056] Correction of Extraction Results: After the framework's first phase of multi-brain detection, the region of interest is reduced, significantly reducing false positive areas. However, the segmentation network's predictions may still contain some small erroneous pixels or regions. Furthermore, the present invention notes that since the two fetuses have nearly identical gestational ages, their brain sizes are also roughly similar. Therefore, this paper labels all connected regions based on a 4-connected component labeling algorithm and removes smaller voxel blocks, thereby eliminating erroneous pixels generated by the segmentation network. As a post-processing step, ERC will effectively improve the segmentation accuracy of twin brain images.

[0057] Experimental verification: The deep environment for implementing the algorithm is NVIDIA GeForce RTX 3060, and the experimental data includes two datasets: 1) contains 1104 axial T2-weighted imaging single slices; 2) contains 208 axial T2-weighted imaging single slices.

[0058] In this experiment, the first-stage object detector training used the twin brain dataset with bounding box labels. The image size was set to 512×512 pixels, and the training, validation, and test sets were split in a 6:2:2 ratio. The second-stage segmentation network training employed a two-step strategy: The first step used the single fetal brain dataset, with the input image size set to 256×256 pixels. The second step used the twin brain dataset, localizing the object detector trained in the first step, cropping the region of interest, and loading the pre-trained model from the first step with the input image size set to 192×192 pixels. The dataset split ratio for this stage remained the same as for the first stage.

[0059] Evaluation indicators: In order to quantitatively analyze the present invention, four common indicators are used to evaluate the automatic extraction framework, including precision, sensitivity, dice coefficient, and Hausdorff distance. The formula is defined as:

[0060]

[0061]

[0062] Where TP, TN, FP, and FN represent the number of true positive, true negative, false positive, and false negative samples, respectively. A and B represent the true value and the corresponding segmentation result, respectively. d(a, b) represents the Euclidean distance between a and b.

[0063] The present invention adopts the above technical solution and proposes an automatic brain extraction framework for twin MR images. It introduces a target detector to detect the brain areas in the twins at one time and retain the unique structural information between the brains. A new segmentation network is proposed, which fully combines low-level features and high-level features. In order to solve the problem that the twin data set is not enough to train the segmentation network from scratch, a two-step training strategy based on the idea of ​​transfer learning is proposed. According to the anatomical structure information of the twin brains, a method for correcting the extraction results is proposed to remove the erroneous pixels that may be generated by the segmentation network. The present invention uses a target detector to locate all brain areas at one time and successfully retains the unique structural information between the brains. The segmentation network of the present invention effectively combines the relationship between low-level features and high-level features, and uses a two-stage training strategy to give full play to the network performance.

[0064] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

Claims

1. A method for segmenting and extracting fetal brain images from twin MR images, characterized by: It includes the following steps: Step S1: Train an object detector using the twin brain dataset with bounding box labels. Use YOLOv7 as the framework for the object detector. Use a bounding box to enclose the two brains in the twin MR images. The specific formula of the loss function for constraining the bounding box is as follows: Where v is a parameter that measures the consistency of aspect ratio, μ is a weight function, IoU(·) represents the intersection over union (IoU) of two bounding boxes, and B are the predicted value and the true value of the bounding box, respectively. c is the diagonal distance of the smallest rectangular area that contains both the predicted bounding box and the corresponding true value. ρ is the Euclidean distance between the center points of A and B. w and h are the width and height of B, and w′ and h′ are the width and height of A. Step S2: Testing the twin brain dataset with the trained object detector to obtain the twin brain dataset after removing maternal tissue; Step S3: Input the single-cell fetal brain dataset into the encoder of the segmentation network to encode the fetal brain region and extract initial low-level features; Step S4: After the encoding process is completed, the low-resolution features are input into the decoder of the segmentation network for upsampling to obtain deep high-level features; Step S5: The low-level features and high-level features of the same resolution obtained during encoding and decoding are concatenated through skip connections to incorporate contextual information. Step S6: Perform a 1×1 convolution operation and a sigmoid function on the output feature map to obtain a probability output, and input the probability output together with the corresponding label into the cross entropy loss function to calculate the loss value, and then update the network parameters to obtain a pre-trained model; Step S7: Load the pre-trained model and input the twin brain dataset from which the maternal tissue has been removed by the object detector into the segmentation network. Repeat steps S3 to S6 to obtain the final segmentation model. Step S8: Input the brain MR images of the twins in the test set into the segmentation model output to obtain a predicted image of the fetal brain.

2. The method for segmenting and extracting fetal brain images from twin MR images according to claim 1, characterized in that: The backbone structure of the segmentation network in step S3 adopts a classic encoder-decoder architecture. The encoder path consists of four special feature learning blocks. The feature learning block learns the features of the twin brain MR images through two 3×3 convolutional layers with batch normalization (BN) and ReLU functions. After a layer of 2×2 pooling operation, a normalized attention module (NAM) is used to adaptively aggregate the learned feature maps in a task-oriented manner and further suppress insignificant features.

3. The method for segmenting and extracting fetal brain images from twin MR images according to claim 1 or 2, characterized in that: In step S3, NAM is expressed as: Among them, k and m represent the mean and standard deviation of each batch, σ is a constant used to ensure that the denominator is not 0; α and β are trainable parameters, ω is the function weight; F in and F out Represents the input and output features of NAM.

4. The method for segmenting and extracting fetal brain images from twin MR images according to claim 1, characterized in that: In step S4, bilinear interpolation is used to upsample the feature map in the decoder path.

5. The method for segmenting and extracting fetal brain images from twin MR images according to claim 1, characterized in that: The 1×1 convolutional layer and Sigmoid function in step S6 are used to generate the probability map of the twin brain extraction results. p m ∈R represents the probability that the mth pixel belongs to the twin fetal brain region; the cross entropy loss function is applied to train the segmentation network based on the label corresponding to the input image, and the formula is as follows: Where w and h are the width and height of the input image, I is the batch size, and g m,n is the label of the mth pixel in the nth input image in each batch, p m,n represents the probability predicted by the network that the mth pixel of each batch of nth input images belongs to the twin brain region.

Images