Fetal brain mri segmentation method based on deep contrastive learning

Abstract

The application discloses a kind of fetal brain MRI segmentation methods, devices, media and terminal based on deep contrast learning, it is related to medical image field, the model training method includes: constructing brain tissue segmentation model;At least one training sample and at least one semantic segmentation true value graph are obtained;The feature extraction is carried out to training sample, and feature map is obtained;Boundary key point graph is generated according to semantic segmentation true value graph;The training process of feature extraction network is guided by contrast learning branch;The feature map is used as the input of segmentation output branch, and segmentation result is obtained;Brain tissue segmentation model is adjusted by back propagation algorithm to obtain the fetal brain MRI segmentation model based on deep contrast learning.The application makes up the defect of low contrast in MRI imaging process, improves the segmentation accuracy of model on brain tissue boundary point, further improves the segmentation effect of fetal brain tissue, which has certain significance for prenatal examination of fetal brain deformity and reduction of neonatal defect rate.

Description

Technical Field

[0001] This invention relates to the field of medical imaging, specifically to a method, device, medium, and terminal for fetal brain MRI segmentation based on deep contrast learning. Background Technology

[0002] The fetal stage is an exceptionally active and critical period for the development of the human brain's structure, function, and connectivity. Accurately and reliably mapping the dynamic trajectory of early brain development helps in understanding normal and abnormal growth. In medicine, fetal brain malformations are not uncommon in clinical practice, with an incidence rate of approximately 0.6%. Furthermore, limb malformations are diverse and have complex causes, leading to frequent missed diagnoses. Related research indicates that chromosomal abnormalities can manifest as various structural malformations during fetal development. Diagnosing brain malformations allows us to identify the abnormal chromosome, which is of significant clinical value for clinical chromosomal abnormality markers. Therefore, prenatal screening for fetal brain malformations is of great clinical significance in improving and reducing the rate of birth defects.

[0003] Magnetic resonance imaging (MRI) offers advantages such as a large field of view, high soft tissue resolution, and multi-plane imaging. It can not only analyze the structural morphology of the fetal brain but also better display its fine anatomical structures, observing the appearance and disappearance of transient structures. MRI utilizes image processing methods based on inter-slice motion correction and super-resolution volume reconstruction to effectively reduce image artifacts, improve image resolution, and better quantitatively assess fetal brain development. Precisely segmenting fetal brain MRI images into different tissue regions, such as white matter, gray matter, and cerebrospinal fluid, is a fundamental step in quantifying early brain development.

[0004] However, segmenting fetal brain MRI images is more challenging than segmenting adult brain MRI images due to reduced tissue contrast, severe partial volume effects, high image noise, strong motion effects, and dynamic white matter myelination. Furthermore, the segmentation process requires pixel-level annotation information, and precise annotation down to each pixel requires not only specialized medical knowledge but also significant time and effort. Therefore, developing automated segmentation algorithms to improve doctors' work efficiency is of great importance.

[0005] Therefore, how to efficiently and accurately segment fetal brain MRI images is a problem that urgently needs to be solved. Summary of the Invention

[0006] In view of this, the present invention provides a fetal brain MRI segmentation method, device, medium, and terminal based on deep contrastive learning. This method combines the SwinUNETR model with contrastive learning incorporating boundary information to increase the contrast of brain tissue boundary points in feature representation, thereby compensating for the low contrast deficiency in MRI imaging and improving the segmentation accuracy of the model at brain tissue boundary points, thus further enhancing the segmentation effect of fetal brain MRI images. It addresses the problem of automatically and accurately extracting brain tissue from unlabeled clinical fetal brain images in existing technologies.

[0007] To achieve the above and other related objectives, in a first aspect, the present invention provides a fetal brain MRI segmentation method based on deep contrastive learning, the method comprising:

[0008] A brain tissue segmentation model is constructed, which includes a feature extraction network, a boundary key point generation unit, a contrast learning branch, and a segmentation output branch.

[0009] Obtain a training sample set, which includes at least one training sample and at least one semantic segmentation ground truth map, wherein the training sample is a fetal brain MRI image;

[0010] The training samples are input into the feature extraction network to extract features and obtain a feature map.

[0011] The semantic segmentation truth map is used as input to the boundary key point generation unit to generate a boundary key point map;

[0012] The feature map is used as input to the contrastive learning branch, and the boundary key point map is referenced to guide the training process of the feature extraction network and calculate the contrastive loss.

[0013] The feature map is used as the input to the segmentation output branch to obtain the segmentation result, and the segmentation loss is calculated.

[0014] The loss function is calculated based on contrast loss and segmentation loss. The parameters of the brain tissue segmentation model are adjusted by the backpropagation algorithm to obtain a fetal brain MRI segmentation model for segmenting the fetal brain MRI.

[0015] In one embodiment of the present invention, after obtaining the training samples and before inputting the training samples into the feature extraction network, the method further includes: performing data augmentation on the training samples, including:

[0016] The portion of the training sample value that is greater than a preset value is cropped out, and then cropped to a fixed size;

[0017] The cropped training samples are randomly flipped along three axes and then randomly flipped vertically to increase the diversity of the training samples.

[0018] The flipped training samples are converted into tensor types required for model training.

[0019] In one embodiment of the present invention, the method for obtaining the semantic segmentation truth graph includes:

[0020] High-level semantic feature maps are extracted from training samples using an encoder;

[0021] The high-level semantic feature map is restored to the same resolution as the training samples by the decoder, and then pixel-by-pixel classification is performed to obtain the semantic segmentation ground truth map.

[0022] In one embodiment of the present invention, the step of using the semantic segmentation truth map as input to the boundary key point generation unit to generate the boundary key point map includes:

[0023] The semantic segmentation truth map is cut into 2D slice images along a certain perspective, and a set of boundary points is obtained on each 2D slice image;

[0024] Based on the size of the boundary point set on each 2D slice image, a first radius and a threshold are set. Each point in the boundary point set is used as the center of a circle, and a circle with the first radius is drawn. The proportion of points with the same category as the center of the circle is calculated.

[0025] Boundary points with a proportion greater than the threshold are retained as boundary key points, a 2D boundary key point map is generated, and the 2D boundary key point map is stitched together along the cross-section to obtain a boundary key point map from a certain perspective.

[0026] The above methods were applied to the axial, coronal, and sagittal planes of fetal brain images, respectively.

[0027] Apply a logical "OR" operation to the boundary key point maps generated from three different perspectives to obtain the final 3D boundary key point map.

[0028] In one embodiment of the present invention, the feature extraction network includes: an encoder-decoder network based on SwinUNETR, wherein the encoder includes several 3D Swin Transformer blocks and downsampling layers, and the decoder includes several upsampling layers based on transposed convolution.

[0029] In one embodiment of the present invention, the step of using the feature map as input to the contrastive learning branch and referring to the boundary keypoint map to guide the training process of the feature extraction network includes:

[0030] Based on the boundary key point map, difficult feature points are sampled from the corresponding positions of the fetal brain feature map, and then easy feature points are randomly sampled from non-boundary key points to form a sampling point set;

[0031] In the set of sampling points, samples whose feature points and feature anchors belong to the same semantic category are considered positive samples, and those whose feature points and feature anchors belong to the same semantic category are considered negative samples.

[0032] By contrastive learning loss, the distance between feature anchors and positive samples in feature embedding is narrowed, while the distance between feature anchors and negative samples in feature embedding is widened, thus guiding the training process of the feature extraction network.

[0033] In one embodiment of the present invention, the step of using the feature map as input to the segmentation output branch to obtain the segmentation result includes:

[0034] Receive the feature map;

[0035] The convolutional layer maps the dimension of each feature point in the feature map to 8 dimensions.

[0036] The segmentation results of the processed feature map are calculated using an activation function to obtain the segmentation results of the fetal brain MRI image.

[0037] Secondly, the present invention also proposes a fetal brain MRI segmentation device based on deep contrastive learning, the device comprising:

[0038] Model building module: used to build a brain tissue segmentation model, which includes a feature extraction network, a boundary key point generation unit, a contrast learning branch, and a segmentation output branch;

[0039] Sample acquisition module: used to acquire a training sample set, the training sample set including: at least one training sample and at least one semantic segmentation ground truth map, the training sample being a fetal brain MRI image;

[0040] Feature extraction module: used to input the training samples into the feature extraction network, perform feature extraction, and obtain feature maps;

[0041] Boundary key point generation module: used to take the semantic segmentation truth map as input to the boundary key point generation unit to generate a boundary key point map;

[0042] Contrastive learning module: used to take the feature map as input to the contrastive learning branch and refer to the boundary key point map to guide the training process of the feature extraction network and calculate the contrastive loss;

[0043] Segmentation output module: used to take the feature map as input to the segmentation output branch, obtain the segmentation result, and calculate the segmentation loss;

[0044] Model adjustment module: used to calculate the loss function based on contrast loss and segmentation loss, and to adjust the parameters of the brain tissue segmentation model through the backpropagation algorithm to obtain a fetal brain MRI segmentation model for segmenting the fetal brain MRI.

[0045] Thirdly, the present invention also proposes a computer-readable storage medium, characterized in that it stores computer instructions, which, when executed, perform a method for training a fetal brain MRI image model segmentation model in any possible embodiment of the first or second aspect.

[0046] Fourthly, the present invention also proposes an electronic terminal, characterized in that the device includes: a memory and a processor; the memory is used to store computer instructions; the processor executes the computer instructions to implement the fetal brain MRI image model segmentation model training method in any possible embodiment of the first or second aspect.

[0047] The beneficial effects of this invention are:

[0048] This paper presents a fetal brain MRI segmentation method based on deep contrastive learning. By combining the SwinUNETR model with contrastive learning that incorporates boundary information, the contrast of brain tissue boundary points in feature representation is increased, thereby compensating for the low contrast in MRI imaging and improving the segmentation accuracy of the model at brain tissue boundary points, thus further improving the segmentation effect of fetal brain MRI images.

[0049] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0051] Figure 1 This is a schematic diagram illustrating the structure of the SwinUNETR encoder-decoder network, as shown in an exemplary embodiment of the present invention.

[0052] Figure 2 This is a schematic flowchart illustrating a fetal brain MRI segmentation method based on deep contrastive learning, as shown in an exemplary embodiment of the present invention.

[0053] Figure 3A schematic diagram of the structure of a fetal brain MRI segmentation model based on deep contrastive learning, as illustrated in an exemplary embodiment of the present invention, is shown.

[0054] Figure 4 A schematic diagram of a boundary key point map generation module illustrating an exemplary embodiment of the present invention is shown;

[0055] Figure 5 A schematic diagram illustrating the code implementation logic of a brain tissue segmentation method for fetal brain MRI images, as shown in an exemplary embodiment of the present invention, is provided.

[0056] Figure 6 This is a schematic diagram of the structure of a fetal brain MRI segmentation device based on deep contrastive learning, as shown in an exemplary embodiment of the present invention.

[0057] Figure 7 This is a schematic diagram of the structure of a fetal brain MRI segmentation terminal based on deep contrastive learning, as shown in an exemplary embodiment of the present invention. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. In addition, the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show components related to the present invention and are not drawn according to the actual number, shape and size of components in the actual implementation. In the actual implementation, the type, quantity and proportion of each component can be arbitrarily changed, and the component layout may also be more complex.

[0059] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0060] The following explains the technical terms used in this invention:

[0061] Magnetic Resonance Imaging (MRI) and Medical Image Analysis: MRI is a biomagnetic spin imaging technique. It utilizes the characteristics of atomic nucleus spin motion to generate signals after being excited by radiofrequency pulses within an external magnetic field. These signals are detected by detectors and input into a computer, where they are processed and displayed as images on a screen. Compared to computed tomography (CT), MRI offers higher soft tissue resolution and has significant advantages in examining the musculoskeletal system and central nervous system. It can directly produce tomographic images in axial, coronal, sagittal, and various oblique planes without artifacts found in CT scans. Furthermore, it eliminates the need for contrast agents and produces no ionizing radiation, thus avoiding adverse effects on the body. Medical imaging plays a crucial role in clinical diagnosis, especially in treatment planning, surgery, and prognostic assessment. By acquiring non-invasive medical images reflecting information about human organs, physicians can make treatment decisions that are most beneficial to the patient's life.

[0062] Deep learning is a new research direction in the field of machine learning. It learns the inherent patterns and representational levels of sample data by simulating the information interaction process of human neurons. The information obtained in this learning process is of great help in interpreting text, images, sound, and even multimodal data. Its ultimate goal is to achieve artificial intelligence that enables machines to think like humans. Currently, there are two main models in the field of deep learning: one is the Convolutional Neural Network (CNN) represented by ResNet and VGG, and the other is the Transformer model based on the self-attention mechanism. The former weights local pixels of an image through convolutional kernels, which can effectively extract local information of the image and has translational equivariance. However, due to the limited receptive field of the convolutional kernel, it cannot capture long-distance dependencies between features very well. The latter calculates attention weights pairwise on the embedded information to obtain information on the relative importance between features, thereby achieving more effective long-distance global semantic modeling.

[0063] Semantic segmentation: Image semantic segmentation is a pixel-level classification problem that assigns a semantic category to each pixel in an image. Early methods for semantic segmentation involved manual feature extraction processes, such as Support Vector Machines (SVM) and Markov Random Fields (MRF). With the rise of deep learning, the classic encoder-decoder structure emerged in this field. The encoder first extracts high-level semantic features from the image, and then the decoder restores the extracted high-level semantic features to match the original image. Figure 1 At the same resolution, the feature maps, which are the same size as the original image, are then classified pixel by pixel.

[0064] Fetal brain image segmentation: In the medical field, fetal brain malformations are not uncommon in clinical practice, with an incidence rate of approximately 0.6%. Furthermore, limb malformations are diverse and have complex causes, leading to frequent missed diagnoses. While brain malformations do not cause fatal harm to the fetus, they can have a severe impact on the family after birth. Related research indicates that chromosomal abnormalities can result in various structural malformations during fetal development. Diagnosing brain malformations allows us to identify the abnormal chromosome, which is of significant clinical value for clinical chromosomal abnormality markers. Therefore, prenatal screening for fetal brain malformations is of great clinical significance in reducing the rate of birth defects. Accurately segmenting fetal brain MRI images into different brain tissue regions provides a quantitative basis for the diagnosis of fetal brain malformations. However, the presence of noise and fetal movement leads to blurred and low-quality fetal brain MRI images, causing existing algorithms to easily misclassify pixels near brain tissue boundaries.

[0065] Edge detection: Early edge detection operators such as Sobel and Canny primarily obtained image edges through image gradients. These methods provided basic low-level cues and were widely used in computer vision applications. Learning-based methods tend to integrate different low-level features and trained classifiers to obtain edges. These methods are mainly based on hand-extracted features and still have certain limitations in detecting meaningful semantic boundaries. Recently, some convolutional neural network-based methods have achieved better performance by fusing multi-level features. Some Transformer-based methods also utilize their ability to model long-distance dependencies between features for image edge detection, achieving good performance when the model can capture better semantic features. However, existing methods capture boundary features by fusing multi-scale features rather than learning the intrinsic relationships between features.

[0066] Contrastive learning aims to narrow the gap in embedded features between similar or class-specific objects while widening the gap between different objects, thus learning the intrinsic relationships between features. Contrastive learning can be divided into instance-level and pixel-level methods. Instance-level contrastive learning is suitable for classification tasks, while pixel-level contrastive learning is suitable for segmentation tasks. The key to contrastive learning lies in the design of the surrogate task and sampling strategy. However, in the area of ​​fetal brain MRI segmentation, due to the different morphological manifestations of different brain tissues, there are different optimal viewing angles on MRI (including three viewing angles: axial, coronal, and sagittal). If images of MRI sliced ​​according to a certain viewing angle are fed into a deep learning network for training, some spatial information will be lost. Therefore, how to fully utilize the spatial information of the input data is an area that needs to be considered, improved, and enhanced.

[0067] Figure 1 This is a schematic diagram illustrating the structure of the SwinUNETR encoder-decoder network, as shown in an exemplary embodiment of the present invention.

[0068] Reference Figure 1 As shown, the SwinUNETR model is mainly divided into two parts: an encoder and a decoder. The encoder consists of several 3D Swin Transformer blocks and downsampling layers, which are used for feature extraction. The decoder consists of several upsampling layers based on transposed convolution, which are used to restore the resolution of the extracted feature map to the input size, and then classify each feature point on the feature map.

[0069] Assuming the size of the input medical image is First, the image is divided into several blocks, each block being of size . Then arrange these blocks into a shape of length [length missing]. Given a sequence, project each element of the sequence linearly onto a plane of dimension 1. The embedding space. For each element in the sequence, it is divided into... There are 1 window, and the feature map size in each window is 1. Then, self-attention operations are performed on the pixels within each window. For the first... Layer and first For the encoder of the first layer, if the first layer... The window size of the layer is Then the first The window size of the layer is , No. The output of the layer and the first The calculation formula for the input is as follows:

[0070]

[0071]

[0072]

[0073]

[0074] Here, and Multi-head self-attention is performed when the feature map is divided into windows of the same size, and when the feature map is divided into windows of different sizes. and They represent Output and The output, and These represent the multilayer perceptron and the layer normalization operation, respectively. To calculate the multi-head self-attention of windows of different sizes, this invention utilizes a combination of 3D cyclic shift operations and masks to divide the windows into different sizes, and calculates the self-attention according to the following formula:

[0075]

[0076] in, Indicates a query. Indicates keywords, Represents value, This indicates the dimensions of the query and keywords. The specific calculation formula is as follows:

[0077]

[0078] In this invention, the embedding dimension of the feature vector The total number of layers in the encoder is 48. Furthermore, the Swin UNETR encoder comprises four stages, each containing two Transformer Blocks. Therefore, the total number of layers in the encoder is... In the first stage, Stage 1, a Linear Embed layer is used to create a size of... The 3D token vector; to maintain the hierarchical structure of the encoder, the first stage employs a block fusion layer (Merging) with a factor of 2 to reduce the feature resolution; furthermore, a block fusion layer... The feature maps are grouped according to their resolution, resulting in feature map sizes of [resolution value missing]. The embedding dimension is 8 times that before downsampling, and then it passes through a linear layer in the block fusion layer, making the feature embedding dimension twice that before downsampling. Stage 2 also uses a block fusion layer with a factor of 2 (Merging) to reduce the feature resolution; a block fusion layer... The feature maps are grouped according to their resolution, resulting in feature map sizes of [resolution value missing]. The embedding dimension is 8 times that before downsampling, and then it passes through a linear layer in the block fusion layer, making the embedding dimension of the feature twice that before downsampling. The network structures of stages three and four are similar to those of stage two, and the resolutions of the feature maps are respectively... and .

[0079] Swin UNETR is a U-shaped network structure where features extracted by the encoder are used by the decoder at each stage through cross-layer connections; at each stage of the encoder... The size of the output feature map is They were then sent to a facility containing two The residual blocks of the convolutional layer (Conv Block) are normalized through instance normalization; then, the feature map is expanded by a factor of 2 through a deconvolution operation and concatenated with the output feature map of the previous stage. The concatenated feature map is then fed into the residual block; the final segmentation result is processed through a... A convolutional layer and a The activation function is used for calculation.

[0080] The UpSample Block module aims to gradually restore the resolution of the feature map to the original image size. Each time the feature map undergoes an upsampling operation, its resolution is increased to twice its original size. Specifically, this involves: increasing the resolution of the feature map to twice its original size using a TransConv Block, concatenating and convolving it with the feature map from the previous layer, and finally completing an upsampling process using residual connections and the ReLU activation function.

[0081] This invention uses the Dice loss function to calculate the segmentation loss during training on a voxel-by-voxel basis:

[0082]

[0083] in This refers to the number of voxels. Indicates the number of brain tissue types. and They refer to the first Individual elements in the first The predicted probabilities for each category are compared with the one-hot vectors of the ground truth labels.

[0084] Therefore, the loss function of the entire method is a linear combination of the segmentation loss and the contrast loss:

[0085]

[0086] In one embodiment of the present invention, the following is provided: To optimize .

[0087] To address the problem that existing technologies cannot efficiently and accurately segment fetal brain MRI images, embodiments of the present invention propose a fetal brain MRI segmentation method, device, medium, and terminal based on deep contrastive learning. These embodiments will be described in detail below.

[0088] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a fetal brain MRI segmentation method based on deep contrastive learning, as shown in an exemplary embodiment of the present invention. Figure 2 As shown, in an exemplary embodiment, the fetal brain MRI segmentation method based on deep contrastive learning includes at least steps S210 to S270, which are described in detail below:

[0089] In step S210, a brain tissue segmentation model is constructed, which includes a feature extraction network, a boundary key point generation unit, a contrast learning branch, and a segmentation output branch.

[0090] In step S220, a training sample set is obtained, which includes at least one training sample and at least one semantic segmentation ground truth map, wherein the training sample is a fetal brain MRI image.

[0091] First, it should be noted that for each fetal brain MRI image acquired in utero, a semantic segmentation ground truth map is generated to label the category of each voxel in the image. This invention divides the fetal brain into seven semantic categories of brain tissue: cerebrospinal fluid, gray matter, white matter, ventricles, cerebellum, dark gray matter, and brainstem. In the semantic segmentation ground truth map, 0 represents the background region (not the fetal brain), 1 represents cerebrospinal fluid, 2 represents gray matter, 3 represents white matter, 4 represents ventricles, 5 represents cerebellum, 6 represents dark gray matter, and 7 represents brainstem.

[0092] It should also be noted that training datasets for medical image segmentation are typically relatively small in size because annotating medical images requires specialized medical personnel, and since most medical images are grayscale, observing the lesion regions is relatively difficult. Therefore, the annotation of each training dataset is very time-consuming and labor-intensive. Taking fetal brain imaging as an example, currently, open-source and annotated datasets are very limited. The dataset used in this invention is FeTA, a fetal brain image segmentation competition organized by the International Association for Medical Image Computing and Computer-Aided Intervention, which contains single-modality (T2-weighted) data of intrauterine MRI images from 80 subjects, with a resolution of [resolution missing]. The fetuses in these MRI images ranged in gestational age from 20 to 33 weeks, some of whom had congenital diseases (spina bifida), and the image quality varied.

[0093] Furthermore, after obtaining the training samples, data augmentation is required on the training samples, i.e., the fetal brain MRI images, including: cropping out the portions of the image with values ​​greater than 0; and cropping the images to [a specific size] due to GPU computing power limitations. ; Randomly flip the image along each of the three axes; Randomly flip the image vertically; Convert the image into the tensor type required for training.

[0094] Finally, the semantic segmentation ground truth map is obtained by: extracting high-level semantic feature maps from the training samples through an encoder; restoring the high-level semantic feature maps to the same resolution as the training samples through a decoder, and then performing pixel-by-pixel classification to obtain the semantic segmentation ground truth map.

[0095] In step S230, the training samples are input into the feature extraction network to extract features and obtain a feature map.

[0096] It should be noted that the feature extraction network includes an encoder-decoder network based on SwinUNETR. The encoder includes several 3D Swin Transformer blocks and downsampling layers, and the decoder includes several upsampling layers based on transposed convolutions. During the training phase, the fetal brain MRI image is input into the feature extraction network for feature extraction to obtain a feature map of the fetal brain MRI. This feature map is the same size as the input fetal brain MRI image.

[0097] In step S240, the semantic segmentation truth map is used as the input to the boundary key point generation unit to generate a boundary key point map.

[0098] It should be noted that the semantic segmentation ground truth map is cut into 2D slice images along a certain viewpoint, and then a conventional edge detection algorithm is applied to each 2D slice image to obtain a set of boundary points. A radius is set according to the size of the set of boundary points on each 2D slice image. With a threshold For each point in the set of boundary points, draw a circle with radius [missing information]. Find the circle and calculate the proportion of points inside the circle that have the same category as the center. ,in The larger the value, the less smooth the boundary of the circular region. This represents the unevenness at the center of the circle. Next, we will use unevenness greater than its... The boundary points of neighboring boundary points are retained as boundary keypoints to generate a 2D boundary keypoint map. These 2D boundary keypoint maps are then stitched together along the cross-section to form a boundary keypoint map generated from a specific viewpoint. However, fetal brain MRI images have three viewpoints: axial, coronal, and sagittal. Image quality and resolution vary depending on the viewpoint, and the optimal viewing viewpoint varies depending on the brain tissue. We aim to fully utilize the information from these three different viewpoints; therefore, we apply this method to each viewpoint. Finally, we apply a logical "OR" operation to the boundary keypoint maps generated from the three different viewpoints to obtain the final 3D boundary keypoint map, achieving complementarity between the information from different viewpoints.

[0099] In step S250, the feature map is used as the input to the contrastive learning branch, and the boundary key point map is referenced to guide the training process of the feature extraction network, and the contrastive loss is calculated.

[0100] It should be noted that existing methods propose that contrastive learning requires combining easy and difficult samples. This can improve the model's performance on difficult samples while avoiding the model getting trapped in local optima. In this invention, easy and difficult samples refer to non-boundary keypoints and boundary keypoints, respectively, with the latter being more difficult to classify correctly than the former. Therefore, this invention samples difficult feature points from the corresponding positions of the fetal brain MRI feature map based on the boundary keypoint map, and then randomly samples non-boundary keypoints as easy feature points to form a sampling point set. In this sampling point set, apart from each feature anchor point, the remaining feature points of the same class are positive samples, and the remaining feature points of different classes are negative samples. Contrastive learning loss is used to narrow the distance between the feature anchor point and positive samples in feature embedding, while simultaneously widening the distance between the feature anchor point and negative samples in feature embedding. The formula is:

[0101]

[0102] in Indicates a positive sample. This represents the set of feature points of a negative sample.

[0103] In step S260, the feature map is used as the input to the segmentation output branch to obtain the segmentation result, and the segmentation loss is calculated.

[0104] The fetal brain MRI feature map is input into the segmentation branch to classify each feature point on the feature map to obtain the predicted result of fetal brain tissue segmentation, and the segmentation loss is calculated. For the first... Given input samples, with segmentation loss:

[0105] ,

[0106] in, ,at this time For the first The categories of each sample, The model predicts the first The sample belongs to the first The probability of each category:

[0107] .

[0108] In step S270, a loss function is calculated based on contrast loss and segmentation loss, and the parameters of the brain tissue segmentation model are adjusted by backpropagation algorithm to obtain a fetal brain MRI segmentation model for segmenting the fetal brain MRI.

[0109] It should be noted that the loss function is a linear combination of the segmentation loss and the contrast loss:

[0110]

[0111] Among them, settings To optimize , For the segmentation loss, The comparison loss is given.

[0112] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the structure of a fetal brain MRI segmentation model based on deep contrastive learning, as shown in an exemplary embodiment of the present invention. Figure 3 As shown, the fetal brain MRI segmentation model based on deep contrastive learning is constructed based on the SwinUNETR model. The fetal brain MRI segmentation model based on deep contrastive learning includes an encoder-decoder feature extraction network based on SwinUNETR, a boundary key point generation module, a contrastive learning module for boundary information-guided sampling, and a segmentation output module.

[0113] It should be noted that the feature extraction network is used to extract feature maps from fetal brain MRI; the boundary key point generation module acts on the semantic segmentation ground truth map to generate boundary key point maps, thereby providing boundary information reference for the sampling process of the contrastive learning module; the contrastive learning sampling process combines relatively easy-to-classify non-boundary key points with relatively difficult-to-classify boundary key points, so that the contrastive learning process can improve performance on difficult samples and avoid getting trapped in local optima, further guiding the feature extraction network to learn more discriminative feature maps; the segmentation output module inputs the feature maps into the classification network, and the classification result of each feature point on the feature map is the segmentation result of the fetal brain tissue.

[0114] Specifically, the encoder-decoder extraction network structure and boundary key point generation module based on SwinUNETR have been described in detail above, so they will not be repeated here; the contrastive learning branch consists of 2... Convolution expands the dimension of each feature point in the feature map output by the feature extraction network to 256 dimensions. Then, it samples the feature points in the expanded feature map and calculates the contrast loss, thereby improving the discriminative power of the features extracted by the feature extraction network. The segmentation output module receives the feature map output by the feature extraction network and processes it through two... The convolutional layer maps the dimension of each feature point in the feature map to 8 dimensions, and then passes it through a... The function predicts the probability of belonging to each category and calculates the segmentation loss.

[0115] Please see Figure 4 , Figure 4 A schematic diagram of a boundary key point map generation module is shown in an exemplary embodiment of the present invention.

[0116] In one specific embodiment, the semantic segmentation ground truth map is sliced ​​into 2D slice images along a certain viewpoint. Then, a conventional edge detection algorithm is applied to each 2D slice image to obtain a set of boundary points. A radius is set according to the size of the set of boundary points on each 2D slice image. With a threshold For each point in the set of boundary points, draw a circle with radius [missing information]. Find the circle and calculate the proportion of points inside the circle that have the same category as the center. ,in The larger the value, the less smooth the boundary of the circular region. This represents the unevenness at the center of the circle. Next, we will use unevenness greater than its... The boundary points of neighboring boundary points are retained as boundary keypoints to generate a 2D boundary keypoint map. These 2D boundary keypoint maps are then stitched together along the cross-section to form a boundary keypoint map generated from a specific viewpoint. However, fetal brain MRI images have three viewpoints: axial, coronal, and sagittal. Image quality and resolution vary depending on the viewpoint, and the optimal viewing viewpoint varies depending on the brain tissue. We aim to fully utilize the information from these three different viewpoints; therefore, we apply this method to each viewpoint. Finally, we apply a logical "OR" operation to the boundary keypoint maps generated from the three different viewpoints to obtain the final 3D boundary keypoint map, achieving complementarity between the information from different viewpoints.

[0117] Please see Figure 5 , Figure 5 This diagram illustrates the code implementation logic of a brain tissue segmentation method for fetal brain MRI images, as shown in an exemplary embodiment of the present invention.

[0118] In one specific embodiment, the data is loaded and enhanced before being input into the training module to obtain the model. Finally, the model parameters are loaded into the testing module to test the model. In addition, since the contrastive learning module is only connected to the feature extraction network during the training phase, and the sampling process of contrastive learning requires information on boundary key points, the semantic ground truth map needs to be generated into a boundary key point map by the boundary key point generation module during the data loading phase, and then loaded into the model training phase for model training.

[0119] like Figure 6 The diagram illustrates a schematic representation of a fetal brain MRI segmentation device based on deep contrastive learning, as shown in this embodiment of the invention. Figure 6 The diagram illustrates the structure of a fetal brain MRI segmentation device based on deep contrastive learning according to an embodiment of the present invention. In this embodiment, the fetal brain MRI segmentation device 600 based on deep contrastive learning includes:

[0120] Model building module 610: used to build a brain tissue segmentation model, the brain tissue segmentation model including a feature extraction network, a boundary key point generation unit, a contrast learning branch and a segmentation output branch;

[0121] Sample acquisition module 620: used to acquire a training sample set, the training sample set including: at least one training sample and at least one semantic segmentation ground truth map, the training sample being a fetal brain MRI image;

[0122] Feature extraction module 630: used to input the training samples into the feature extraction network, perform feature extraction, and obtain a feature map;

[0123] Boundary key point generation module 640: used to take the semantic segmentation truth map as input to the boundary key point generation unit to generate a boundary key point map;

[0124] Contrast learning module 650: used to take the feature map as input to the contrast learning branch and refer to the boundary key point map to guide the training process of the feature extraction network and calculate the contrast loss;

[0125] Segmentation output module 660: used to take the feature map as input to the segmentation output branch, obtain the segmentation result, and calculate the segmentation loss;

[0126] Model adjustment module 670: used to calculate the loss function based on contrast loss and segmentation loss, and to adjust the parameters of the brain tissue segmentation model through the backpropagation algorithm to obtain a fetal brain MRI segmentation model for segmenting the fetal brain MRI.

[0127] The fetal brain MRI segmentation method based on deep contrastive learning provided in this invention can be implemented on the terminal side or the server side. For the hardware structure of the fetal brain MRI segmentation terminal based on deep contrastive learning, please refer to [link to relevant documentation]. Figure 7 This is a schematic diagram of an optional hardware structure of a fetal brain MRI segmentation terminal 700 based on deep contrastive learning provided in an embodiment of the present invention. The terminal 700 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The fetal brain MRI segmentation terminal 700 based on deep contrastive learning includes: at least one processor 710, a memory 720, at least one network interface 750, and a user interface 740. The various components in the device are coupled together through a bus system 730. It is understood that the bus system 730 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 730 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 7Various buses are referred to as bus systems. The user interface 740 may include a display, keyboard, mouse, trackball, clicker, buttons, a touchpad, or a touchscreen. It is understood that the memory 720 may be volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memory described in this embodiment is intended to include, but is not limited to, these and any other suitable categories of memory. The memory 720 in this embodiment is used to store various categories of data to support the operation of the deep contrastive learning-based fetal brain MRI segmentation terminal 700. Examples of this data include: any executable program, such as operating system 721 and application program 722, for operation on the deep contrast learning-based fetal brain MRI segmentation terminal 700; operating system 721 includes various system programs, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. Application program 722 may include various applications, such as media player, browser, etc., for implementing various application services. The deep contrast learning-based fetal brain MRI segmentation method provided in the embodiments of the present invention can be included in application program 722. The method disclosed in the above embodiments of the present invention can be applied to processor 710, or implemented by processor 710. Processor 710 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 710 or by instructions in software form. The processor 710 may be a general-purpose processor, digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 710 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 710 can be a microprocessor or any conventional processor. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as the hardware decoding processor executing the steps, or as a combination of hardware and software modules in the decoding processor executing the steps.The software module can reside in a storage medium, which is located in a memory. The processor reads information from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method. In an exemplary embodiment, the fetal brain MRI segmentation terminal 700 based on deep contrastive learning can be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0128] In summary, the present invention discloses a fetal brain MRI segmentation method, device, medium, and terminal based on deep contrastive learning. By combining the SwinUNETR model with contrastive learning that incorporates boundary information, the contrast of brain tissue boundary points in feature representation is increased, thereby compensating for the low contrast in MRI imaging, improving the segmentation accuracy of the model at brain tissue boundary points, and further enhancing the segmentation effect of fetal brain MRI images.

[0129] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A fetal brain MRI segmentation method based on deep contrastive learning, characterized in that, The method includes: A brain tissue segmentation model is constructed, which includes a feature extraction network, a boundary key point generation unit, a contrast learning branch, and a segmentation output branch. Obtain a training sample set, which includes at least one training sample and at least one semantic segmentation ground truth map, wherein the training sample is a fetal brain MRI image; The training samples are input into the feature extraction network to extract features and obtain a feature map. The semantic segmentation truth map is used as input to the boundary key point generation unit to generate a boundary key point map; The feature map is used as input to the contrastive learning branch, and the boundary key point map is referenced to guide the training process of the feature extraction network and calculate the contrastive loss. The feature map is used as the input to the segmentation output branch to obtain the segmentation result, and the segmentation loss is calculated. The loss function is calculated based on contrast loss and segmentation loss. The parameters of the brain tissue segmentation model are adjusted by backpropagation algorithm to obtain a fetal brain MRI segmentation model for segmenting fetal brain MRI. The step of using the feature map as input to the contrastive learning branch and referring to the boundary key point map to guide the training process of the feature extraction network includes: Based on the boundary key point map, difficult feature points are sampled from the corresponding positions of the fetal brain feature map, and then easy feature points are randomly sampled from non-boundary key points to form a sampling point set; In the set of sampling points, samples whose feature points and feature anchors belong to the same semantic category are considered positive samples, and those whose feature points and feature anchors belong to the same semantic category are considered negative samples. By contrastive learning loss, the distance between feature anchors and positive samples in feature embedding is narrowed, while the distance between feature anchors and negative samples in feature embedding is widened, thus guiding the training process of the feature extraction network.

2. The fetal brain MRI segmentation method based on deep contrastive learning according to claim 1, characterized in that, After acquiring the training samples and before inputting them into the feature extraction network, the method further includes: performing data augmentation on the training samples, including: The portion of the training sample value that is greater than a preset value is cropped out, and then cropped to a fixed size; The cropped training samples are randomly flipped along three axes and then randomly flipped vertically to increase the diversity of the training samples. The flipped training samples are converted into tensor types required for model training.

3. The fetal brain MRI segmentation method based on deep contrastive learning according to claim 1, characterized in that, The methods for obtaining the semantic segmentation truth map include: High-level semantic feature maps are extracted from training samples using an encoder; The high-level semantic feature map is restored to the same resolution as the training samples by the decoder, and then pixel-by-pixel classification is performed to obtain the semantic segmentation ground truth map.

4. The fetal brain MRI segmentation method based on deep contrastive learning according to claim 1, characterized in that, The step of using the semantic segmentation truth map as input to the boundary key point generation unit to generate the boundary key point map includes: The semantic segmentation truth map is cut into 2D slice images along a certain perspective, and a set of boundary points is obtained on each 2D slice image; Based on the size of the boundary point set on each 2D slice image, a first radius and a threshold are set. Each point in the boundary point set is used as the center of a circle, and a circle with the first radius is drawn. The proportion of points with the same category as the center of the circle is calculated. Boundary points with a proportion greater than the threshold are retained as boundary key points, a 2D boundary key point map is generated, and the 2D boundary key point map is stitched together along the cross-section to obtain a boundary key point map from a certain perspective. The above methods were applied to the axial, coronal, and sagittal planes of fetal brain images, respectively. Apply a logical "OR" operation to the boundary key point maps generated from three different perspectives to obtain the final 3D boundary key point map.

5. The fetal brain MRI segmentation method based on deep contrastive learning according to claim 1, characterized in that, The feature extraction network includes an encoder-decoder network based on SwinUNETR, wherein the encoder includes several 3D Swin Transformer blocks and downsampling layers, and the decoder includes several upsampling layers based on transposed convolution.

6. The fetal brain MRI segmentation method based on deep contrastive learning according to claim 1, characterized in that, The step of using the feature map as input to the segmentation output branch to obtain the segmentation result includes: Receive the feature map; The convolutional layer maps the dimension of each feature point in the feature map to 8 dimensions. The segmentation results of the processed feature map are calculated using an activation function to obtain the segmentation results of the fetal brain MRI image.

7. A fetal brain MRI segmentation device based on deep contrastive learning, characterized in that, include: Model building module: used to build a brain tissue segmentation model, which includes a feature extraction network, a boundary key point generation unit, a contrast learning branch, and a segmentation output branch; Sample acquisition module: used to acquire a training sample set, the training sample set including: at least one training sample and at least one semantic segmentation ground truth map, the training sample being a fetal brain MRI image; Feature extraction module: used to input the training samples into the feature extraction network, perform feature extraction, and obtain feature maps; Boundary key point generation module: used to take the semantic segmentation truth map as input to the boundary key point generation unit to generate a boundary key point map; Contrastive learning module: used to take the feature map as input to the contrastive learning branch and refer to the boundary key point map to guide the training process of the feature extraction network and calculate the contrastive loss; Segmentation output module: used to take the feature map as input to the segmentation output branch, obtain the segmentation result, and calculate the segmentation loss; Model adjustment module: used to calculate the loss function based on contrast loss and segmentation loss, and to adjust the parameters of the brain tissue segmentation model through the backpropagation algorithm to obtain a fetal brain MRI segmentation model for segmenting the fetal brain MRI. The step of using the feature map as input to the contrastive learning branch and referring to the boundary key point map to guide the training process of the feature extraction network includes: Based on the boundary key point map, difficult feature points are sampled from the corresponding positions of the fetal brain feature map, and then easy feature points are randomly sampled from non-boundary key points to form a sampling point set; In the set of sampling points, samples whose feature points and feature anchors belong to the same semantic category are considered positive samples, and those whose feature points and feature anchors belong to the same semantic category are considered negative samples. By contrastive learning loss, the distance between feature anchors and positive samples in feature embedding is narrowed, while the distance between feature anchors and negative samples in feature embedding is widened, thus guiding the training process of the feature extraction network.

8. A computer-readable storage medium, characterized in that, The device stores computer instructions that, when executed, perform the fetal brain MRI segmentation method based on deep contrastive learning as described in any one of claims 1 to 6.

9. An electronic terminal, characterized in that, The device includes: a memory and a processor; the memory is used to store computer instructions; the processor executes the computer instructions to implement the fetal brain MRI segmentation method based on deep contrastive learning as described in any one of claims 1 to 6.

Images