A method for segmenting carotid artery unstable plaques based on multi-sequence magnetic resonance images

Abstract

The present invention discloses a method for segmenting carotid artery unstable plaques based on multi-sequence magnetic resonance images, comprising: (1) obtaining multi-sequence magnetic resonance images of the neck; (2) registering the multi-sequence magnetic resonance images to obtain three-dimensional registered images; (3) inputting the registered images into a pre-constructed U-net neural network model to obtain plaque segmentation images corresponding to the neck; when constructing the U-net neural network model, modifying the channel parameters of the U-net neural network model according to the number of multi-sequences; constructing the U-net neural network model with known registered images as inputs and corresponding plaque mask images as outputs. The present invention learns a method for automatically predicting the positions of unstable plaques from multi-sequence magnetic resonance images through a neural network model, so as to predict new neck magnetic resonance image samples, determine whether there are unstable plaques and give specific positions, which can be used as a reference for doctors' diagnosis and improve the efficiency of doctors' diagnosis.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic image recognition, and particularly relates to a method for segmenting carotid artery unstable plaques based on multi-sequence magnetic resonance images. Background Art

[0002] As is well known, the shedding of carotid atherosclerotic plaques can form emboli, block intracranial arteries, and cause ischemia of distal brain tissues, leading to the occurrence of cerebral infarction. High-resolution magnetic resonance imaging (HR-MRI) has been proven to be an effective tool for detecting atherosclerotic vulnerable plaques. High-resolution magnetic resonance imaging can clearly display the external morphological characteristics, internal structural components, and position distribution information of plaques. In addition to traditional T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), and proton density (PDWI), HR-MRI also includes black blood sequences, bright blood sequences, and time-of-flight sequences (TOF-MRA).

[0003] Magnetic resonance imaging is currently the only non-invasive imaging technique that can clearly display atherosclerotic plaques throughout the body. However, due to the huge number of three-dimensional high-resolution magnetic resonance vessel wall images, with each examiner's images reaching 500, even experienced professional doctors need to spend a long time to complete the diagnosis of the examiner, resulting in low work efficiency.

[0004] In order to achieve rapid intelligent plaque image segmentation, researchers have begun to develop segmentation methods based on automatic recognition:

[0005] The patent document with the publication number CN109932720A discloses an intelligent segmentation method for intracranial plaques and carotid artery plaques in magnetic resonance images, including: step S1, obtaining the magnetic resonance image of the vessel wall of the user; step S2, preprocessing the magnetic resonance image of the vessel wall to obtain a preprocessed image; step S3, segmenting a preset plaque area in the preprocessed image through a pre-trained convolutional neural network model; step S4, outputting a segmentation image of the plaque tissue area corresponding to the magnetic resonance image of the vessel wall. When constructing the convolutional neural network model, the entire three-dimensional magnetic resonance image is used as the input, and a large amount of training set data is required.

[0006] The patent document with the publication number CN111598891A, in order to overcome the above problems, adopts a U-convolutional network (U-Net), which also has a good segmentation effect when the amount of image data is very small.

[0007] However, the existing technologies all use a single series of magnetic resonance images for model training, and the accuracy of the obtained neural network model is still not ideal enough. Summary of the Invention

[0008] The present invention provides a method for segmenting carotid artery unstable plaques based on multi-sequence magnetic resonance images. This method constructs multiple neural network models based on multi-sequence magnetic resonance images and then fuses the segmentation results, greatly improving the segmentation accuracy and robustness.

[0009] A method for segmenting carotid artery unstable plaques based on multi-sequence magnetic resonance images, comprising the following steps:

[0010] (1) Obtain multi-sequence magnetic resonance images of the neck;

[0011] (2) Register the multi-sequence magnetic resonance images to obtain registered images;

[0012] (3) Input the registered images into the constructed U-net neural network model to obtain plaque segmentation images corresponding to the neck;

[0013] When constructing the U-net neural network model, modify the channel parameters of the U-net neural network model according to the number of multi-sequences (for example, when the number of sequences is N, change the input_channel parameter of the U-net to N, and the rest remains unchanged); construct the U-net neural network model with the known registered images as the input and the corresponding plaque mask images as the results.

[0014] Preferably, when constructing the U-net neural network model:

[0015] (i) First, collect multi-sequence magnetic resonance images of the neck;

[0016] (ii) Register the multi-sequence magnetic resonance images to obtain three-dimensional registered images;

[0017] (iii) Mark the carotid artery unstable plaque regions in the registered images to obtain plaque mask images;

[0018] (iv) Use the registered images described in step (ii) as the input and the corresponding mask images as the output to form a training sample set, and construct the U-net neural network model.

[0019] Before modeling, slice the obtained multi-sequence three-dimensional registered images, and use the obtained two-dimensional slice images as the input in step (iv), and use the corresponding plaque mask images as the true prediction results to construct the U-net neural network model.

[0020] Since the labeled regions in the data are small, only a small number of two-dimensional slice images contain labeled regions. Preferably, first use the slice images containing the labeled unstable plaque regions to train the model. After the model meets the set requirements (such as accuracy), then input the slice images that do not contain the labeled unstable plaque regions to continue training the model.

[0021] Meanwhile, before modeling, it is necessary to standardize the input two-dimensional slice images so that the mean of their distribution range is 0 and the variance is 1.

[0022] Preferably, during the model training process, data augmentation is performed by one or more operations such as randomly flipping each two-dimensional slice image and its mask image left and right, randomly zooming in or out, adding random Gaussian noise, and randomly cropping regions to the same size.

[0023] The patch mask image can be manually labeled using existing software (such as Slicer software), and the final patch mask image format is a binary image. By defining each pixel value as 0 / 255, the labeled area and the unlabeled area are distinguished. For example, the pixel values of the points corresponding to the labeled area can be set to 0, and the pixel values of the points in the remaining unlabeled area can be set to 255.

[0024] When training the U-net neural network model of the present invention, the Focal loss function is used to monitor the training process. The U-net model uses an "encoding-decoding" structure. The encoding part extracts the feature map of the image through a convolutional neural network, and the decoding part obtains the segmentation result from the feature map through a deconvolution operation.

[0025] Preferably, the test sample set is constructed simultaneously according to steps (i) to (iii), the training sample set is used to optimize the model, and the test sample set is used to verify the model; when the verification result does not meet the requirements, the model training continues; when the requirements are met, the constructed U-net neural network model is output.

[0026] Preferably, during the model training process, the optimizer uses the Adam algorithm, and the learning rate adjustment strategy uses the cosine annealing method. The value of the loss function of the test set is used as the criterion for judging the quality of model training. The Adam algorithm is an adaptive motion estimation algorithm, which is an extension of the stochastic gradient descent algorithm. Its adaptive mechanism makes the model training faster, more robust, reduces the dependence on the hyperparameter learning rate, and reduces the difficulty and time of training the neural network. Cosine annealing can reduce the learning rate through the cosine function. In the cosine function, as x increases, the cosine value first decreases slowly, then accelerates, and then decreases slowly again. This decreasing pattern can cooperate with the learning rate to produce good results in a very effective calculation method.

[0027] Preferably, the multi-sequence magnetic resonance images adopted in the present invention are three-dimensional magnetic resonance image data of the T1W sequence and the TOF sequence.

[0028] Preferably, we can introduce a multi-view fusion method to optimize the model and the detection method, specifically including:

[0029] Model construction stage: The three-dimensional registration image and the corresponding patch mask image are sliced from multiple perspectives respectively to obtain multiple groups of two-dimensional sliced images and mask sliced images, and multiple U-net neural network models corresponding to multiple perspectives are constructed respectively.

[0030] Image segmentation: The three-dimensional registration image to be segmented is sliced from multiple perspectives. The two-dimensional sliced images of each perspective are respectively input into the corresponding U-net neural network model to obtain multiple sub-segmentation images. After all the sub-segmentation images are obtained, all the sub-segmentation images are fused to obtain the final patch segmentation image. Optionally, the sub-segmentation images or the fused segmentation image are binarized before or after fusion to obtain the final patch segmentation image.

[0031] As a further preference, the present invention slices from three perspectives, namely the coronal plane, axial plane, and sagittal plane of the three-dimensional magnetic resonance registration image.

[0032] As a further preference, in the training stage or the actual application stage of the model, the three-dimensional magnetic resonance registration image is sliced from multiple perspectives respectively for training to obtain three models (training stage) or respectively input into the constructed corresponding models to obtain the segmentation results of the corresponding perspectives (application stage); in the training stage, the sliced data of multiple perspectives are used as input, and the mask images of the corresponding perspectives are used as results to realize the construction of the model; in the application stage, the sliced data of multiple perspectives are respectively input into the corresponding models to obtain the segmentation results (segmentation images) of multiple perspective models, and then the multiple segmentation results are fused to obtain a final segmentation result. The advantage of multi-perspective fusion is that it can better utilize the image spatial information and has better performance than the single-perspective model.

[0033] As a preference, the method of weight summation is used for fusion.

[0034] After obtaining the final segmentation result, in order to ensure the clarity of the output result, morphological operations can be performed on the segmentation result, including: a) For the three-dimensional binarized segmentation result, use a cross-shaped convolution kernel for image erosion operation. b) Use a square convolution kernel for image dilation operation on the result of the erosion operation.

[0035] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0036] The present invention learns a method for automatically predicting the position of unstable plaques from multi-sequence magnetic resonance images through a neural network model, so as to predict new neck magnetic resonance image samples, judge whether there are unstable plaques and give the specific positions, as a reference for doctors' diagnosis, and improve the efficiency of doctors' diagnosis. Brief Description of the Drawings

[0037] Figure 1 is the flow chart adopted in the embodiment of the present invention;

[0038] Figure 2 is a two-dimensional slice image of a certain T1W sequence;

[0039] Figure 3 is a plaque mask image marked for the unstable plaque area;

[0040] Figure 4 is a probability map output by the U-net neural network without binarization processing;

[0041] Figure 5 is a probability map after binarization processing. Specific implementation manner

[0042] As Figure 1 shown, a method for segmenting carotid artery unstable plaques based on multi-sequence magnetic resonance images includes:

[0043] (1) Obtain multi-sequence magnetic resonance images of the neck;

[0044] (2) Register images of different sequences to obtain three-dimensional image registration;

[0045] (3) Slice the three-dimensional image registration from the coronal plane, axial plane, and sagittal plane respectively to obtain slice images of three perspectives;

[0046] (4) Input the slice images of the three perspectives into the trained U-net neural network model respectively to obtain segmentation results corresponding to the three perspectives;

[0047] (5) Fuse the three segmentation results to obtain the final segmentation result.

[0048] In the implementation, the U-Net model is used as the neural network model of this method. U-Net was published in 2015 and was proposed by Olaf Ronneberger, Philipp Fischer, and Thomas Brox. It belongs to a variant of FCN. The original intention of U-Net was to solve problems in biomedical images. Due to its good effect, it has also been widely used in various directions of semantic segmentation, such as satellite image segmentation and industrial defect detection.

[0049] The U-Net network structure is symmetric and shaped like the English letter U, so it is called U-Net. The whole figure is composed of blue / white frames and arrows of various colors. Among them, the blue / white frames represent feature maps; the blue arrows represent 3x3 convolutions for feature extraction; the gray arrows represent skip-connections for feature fusion; the red arrows represent pooling for dimensionality reduction; the green arrows represent upsampling for dimensionality restoration; and the cyan arrows represent 1x1 convolutions for output results.

[0050] The U-net model uses an "encoding-decoding" structure. The encoding part extracts the feature map of the image through a convolutional neural network, and the decoding part obtains the segmentation result from the feature map through deconvolution operations. A sigmoid operation is used on each pixel of the segmentation result to obtain the confidence that each pixel is classified as an unstable plaque area.

[0051] The U-Net encoder part uses convolutional layers, batch normalization layers, and max pooling layers. The role of the batch normalization layer is to increase the robustness of the model.

[0052] The U-Net decoder uses convolutional layers and interpolation layers.

[0053] All convolutional layers in U-Net use 3*3 sized convolutional kernels.

[0054] The number of layers and channels of U-net can be adjusted according to specific data. If overfitting occurs, reduce the number of layers and channels; if the features are not obvious enough, increase the number of layers and channels.

[0055] Use Focal loss as the loss function of this method. Set the parameters as alpha = 0.9 and gamma = 0.4.

[0056] The actual modeling process is as follows:

[0057] (i) Obtain multi-sequence magnetic resonance images of the neck;

[0058] (ii) Register different sequence images to obtain three-dimensional registered images;

[0059] The multi-sequence three-dimensional magnetic resonance images that can be used include time-of-flight MR angiography (3D-TOF), T1-weighted image (T1WI), contrast-enhanced T1-weighted image (CE-T1WI), T2-weighted image (T2WI), proton density-weighted image (PWI), etc. In this embodiment, the multi-sequence magnetic resonance images used are T1W sequence and TOF sequence magnetic resonance images, and the three-dimensional magnetic resonance images of the T1W sequence and TOF sequence are registered;

[0060] (iii) Use a mask to label the region of carotid artery unstable plaque to obtain a binary three-dimensional mask image;

[0061] In this embodiment, the Slicer software is used to manually frame the region where the unstable plaque is located in the three-dimensional magnetic resonance image with a cube. Then, the labeled image is binarized, and the pixel values of the labeled region are set to 0, and the pixel values of other regions are set to 255.

[0062] At the same time, according to the different perspectives (or coordinate axes) adopted in step (4), the binary three-dimensional mask image is sliced according to the corresponding perspective; a corresponding two-dimensional slice mask image is obtained; as Figure 3 One of the obtained binary two-dimensional mask slice images is shown.

[0063] (iv) At the same time, slice the three-dimensional registration image from the coronal plane, axial plane, and sagittal plane to obtain two-dimensional slice images of three perspectives respectively; Figure 2 is a certain two-dimensional slice image corresponding to the T1W sequence.

[0064] Perform data normalization processing on each two-dimensional slice image, that is, make the mean of its numerical distribution range 0 and the variance 1 through the following formula.

[0065] x' = (x - mean(x)) / (std(x))

[0066] where x' is the pixel value of a certain point after normalization; x is the pixel value of a certain point before normalization; mean(x) is the average pixel value of the pixel points in the two-dimensional slice image; std(x) is the standard deviation of the pixel values of the pixel points in the two-dimensional slice image.

[0067] (v) Use the sliced images of the three perspectives after normalization and the corresponding two-dimensional slice mask images as input and true prediction results respectively to construct a U-net model;

[0068] During the process of training the model, since the labeled region in the data is small, only a small part of the slice images contain the labeled region. In this embodiment, in the initial training stage, only the slice images containing labeled data are used for training, and after the model has a certain accuracy, the slice images without labeled data are added proportionally.

[0069] The original images have different resolutions, imaging regions, and voxel spacings, so normalization is required. Therefore, before performing the registration operation described above, it can be processed according to the following method:

[0070] First, use the N4 bias field correction algorithm to correct the intensity inhomogeneity of all sequence magnetic resonance image volumes.

[0071] Then, B-spline interpolation is used to resample all the sequence images to obtain 3D images with approximately isotropic voxel sizes.

[0072] Thereafter, based on an adaptive mask with 3D rigid and affine transformations, an automatic registration method is applied to match these volumes.

[0073] The volume of TOF is selected as the reference image, and the T1W, T1, T2 and other applicable sequence images are selected as the registration images. All sequence volumes are unified to the same spatial volume / pixel size and correspond to the same arterial position.

[0074] Meanwhile, in order to enhance the generalization ability of the model, data augmentation needs to be performed on the data during the model training process, that is, new training data is obtained by performing some random operations on the original data, as follows:

[0075] (a) Randomly flip each image and its annotation left and right, that is, take a random number between 0 and 1. If the random number is greater than 0.5, the image is flipped left and right, otherwise it remains unchanged.

[0076] (b) Randomly scale each image and its annotation. Take a random number between 0.5 and 1.5 as the scaling ratio, and scale the length and width of the image and the mask to the random ratio at the same time. Random scaling can enable the neural network to learn targets of different sizes instead of a single size, effectively increasing the model accuracy.

[0077] (c) Add random Gaussian noise to each image, that is, add a random number that conforms to a normal distribution with a mean of 0 and a standard deviation of 0.05 to the value of each pixel of the image.

[0078] (d) Randomly crop regions of each image and its annotation. The purpose is to make the sizes of all images input into the neural network the same for batch input and speed up the calculation. Specifically, a rectangular region of a specified size is randomly cropped from each image and its annotation. If the image size is smaller than the specified size, the image is expanded (for example, two in a row) and then cropped.

[0079] In order to further increase the generalization ability of the model, we can divide the dataset into two parts: a training set and a test set. The training set is used to optimize the model, and then the test set is used to judge the generalization ability of the model. During the model training process, the Adam algorithm is used as the optimizer. The initial learning rate is set to 0.0001. The learning rate adjustment strategy uses the cosine annealing method. The value of the loss function of the test set is used as the criterion for judging the quality of the model training.

[0080] In this embodiment, the loss function used during the training of the training set is the Focal loss function. The Focal loss function is an improvement of the cross-entropy loss function, which improves the effect of small target segmentation.

[0081] The formula of the Focal loss function is as follows, where p is the predicted value and ranges from 0 to 1.

[0082] Focal(p t ) = -α(1 - p t ) γ log p t

[0083]

[0084] y is the mask value of a certain pixel;

[0085] When testing on the test set, the value of the test set loss function is used as the criterion for evaluating the quality of model training. The loss function here is not specifically limited, and general loss functions can be used.

[0086] In step (5), the multi-modal (perspective) three-dimensional image data is sliced into two-dimensional images in three directions and input into the model in sequence, and then a column of two-dimensional segmentation result images obtained is combined into a three-dimensional segmentation result according to the original positions. The sub-segmentation results (sub-segmentation images) of multiple models are weighted and averaged, and a fixed threshold is used to perform binary processing on the value of each pixel to obtain the final plaque segmentation image.

[0087] In step (5), after obtaining the binary segmentation result (segmentation image), morphological operations can be performed on it to improve the accuracy of the result, specifically including:

[0088] For the three-dimensional binary segmentation result, an image erosion operation is performed using a cross-shaped convolution kernel. The purpose is to filter out isolated segmentation regions.

[0089] An image dilation operation is performed on the result of the erosion operation using a square convolution kernel. The purpose is to expand the segmentation range. Figure 4 and Figure 5 are the plaque distribution probability maps (i.e., segmentation images) before and after binary processing respectively. The value of each pixel in the figure represents the probability of being judged as an unstable plaque, and the whiter it is, the higher the probability.

Claims

1. A method for segmenting carotid artery unstable plaques based on multi-sequence magnetic resonance images, characterized in that It includes the following steps: (1) Obtain multi-sequence magnetic resonance images of the neck; (2) Register the multi-sequence magnetic resonance images to obtain a three-dimensional registered image; (3) Input the registered image into the pre-constructed U-net neural network model to obtain a plaque segmentation image corresponding to the neck; When constructing the U-net neural network model, modify the channel parameters of the U-net neural network model according to the number of multi-sequences; construct the U-net neural network model with the known registered image as the input and the corresponding plaque mask image as the result; When constructing the U-net neural network model: (i) First, collect multi-sequence magnetic resonance images of the neck; (ii) Register the multi-sequence magnetic resonance images to obtain a three-dimensional registered image; slice the three-dimensional registered image, and use the obtained two-dimensional slice images as the input in step (iv) to construct the U-net neural network model; (iii) Label the carotid artery unstable plaque area in the registered image to obtain a plaque mask image; (iv) Use the registered image described in step (ii) as the input and the corresponding mask image as the output to form a training sample set, and construct the U-net neural network model; When registering the images: First, use the N4 bias field correction algorithm to correct the intensity inhomogeneity of all sequence magnetic resonance image volumes; Then, use B-spline interpolation to resample all sequence images to obtain 3D images with approximately isotropic voxel sizes; Thereafter, based on an adaptive mask with 3D rigid and affine transformations, apply an automatic registration method to match these 3D images to achieve image registration; During the model construction stage or when using the pre-constructed U-net neural network model for image segmentation: Model construction stage: Slice the three-dimensional registered image and the corresponding plaque mask image from multiple perspectives respectively to obtain multiple groups of two-dimensional slice images and mask slice images, and construct U-net neural network models corresponding to multiple perspectives respectively; Perform image segmentation: Slice the three-dimensional registered image to be segmented from multiple perspectives, input the two-dimensional slice images of each perspective into the corresponding U-net neural network model respectively to obtain a sub-segmentation image; after obtaining all sub-segmentation images, fuse all sub-segmentation images to obtain the final plaque segmentation image; Optionally, perform binarization processing on the sub-segmentation images or the fused segmentation image before or after fusion to obtain the final plaque segmentation image; When training the U-net neural network model using the training sample set, use the Focal loss function.

2. The carotid artery unstable plaque segmentation method based on multi-sequence magnetic resonance images according to claim 1, characterized in that First, use the slice images containing the labeled unstable plaque area to train the model. After the model meets the set requirements, then input the slice images that do not contain the labeled unstable plaque area to continue training the model.

3. The carotid artery unstable plaque segmentation method based on multi-sequence magnetic resonance images according to claim 1, characterized in that, During the model training process, perform data augmentation by performing one or more operations such as randomly flipping left and right, randomly zooming in or out, adding random Gaussian noise, and randomly cropping regions of the same size on each two-dimensional slice image and its mask image.

4. The carotid artery unstable plaque segmentation method based on multi-sequence magnetic resonance images according to claim 1, wherein Construct a test sample set simultaneously according to steps (i) to (iii), optimize the model using the training sample set, and perform model verification using the test sample set; when the verification result does not meet the requirements, continue training the model; when the requirements are met, output the constructed U-net neural network model.

5. The carotid artery unstable plaque segmentation method based on multi-sequence magnetic resonance images according to claim 1, characterized in that The multi-sequence magnetic resonance images are two or more of three-dimensional time-of-flight MR angiography, T1-weighted image, enhanced T1-weighted image scan, T2-weighted image (T2WI), and proton density weighted image.

Images