Medical image segmentation method based on calibration point detection and shape grayscale model matching

Abstract

The present invention discloses a medical image segmentation method based on calibration point detection and shape grayscale model matching, belonging to the field of medical image analysis technology. The method first constructs a parameterized atlas with both shape and grayscale information based on a large number of sample images and their annotation results; creates an anatomical feature point detection model based on a convolutional neural network, and trains the model using sample images and their annotation results; uses the matching of the shape grayscale model and the prediction results of the neural network to achieve high-precision registration and organ mapping between the atlas and the target image. The present invention utilizes the deformation field of the image area to characterize the shape change of the model, introduces the modeling of grayscale information and the guidance of anatomical feature points, and improves the robustness of the algorithm while ensuring the segmentation accuracy, so that it can be applied to medical images with different noise conditions and contrasts.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a medical image segmentation method based on calibration point detection and shape grayscale model matching, which is mainly applicable to the automated analysis of different types of medical images. Background Art

[0002] Medical images are the main basis for current clinical diagnosis and treatment, and are also an important means of medical research. With the continuous development of computer technology and medical imaging technology, the amount of medical image data has increased dramatically; therefore, a fast, accurate, highly automated and multi-scenario medical image segmentation method is needed to help doctors and researchers quickly locate the target's area of ​​interest, thereby assisting clinical diagnosis and medical research. In the field of clinical diagnosis, differences in patient body shape, the choice of imaging modality, and the difference in imaging instruments will affect the final imaging results; and for micro-computed tomography (micro-CT), which is the most commonly used in medical research, it will also be affected by the posture changes and radiation dose of small animals. The above reasons make the final image data have significant differences in organ distribution, image contrast, and noise level, which also makes accurate segmentation of various types of medical images a huge challenge. The segmentation method based on statistical shape model matching uses shape prior knowledge to overcome individual differences, but due to its lack of grayscale information, it cannot be generalized to different image contrasts and noise levels. Although the segmentation method based entirely on deep learning can achieve good segmentation results, its network model is still trained using specific image contrast or noise levels, which has certain limitations. If we can improve on these two, build a statistical map with both shape and grayscale information, accurately match it to the target image, and use convolutional neural network-based calibration point detection to guide and correct this process, we can achieve accurate segmentation of the target image and give the algorithm a certain generalization ability.

[0003] In order to achieve medical image segmentation based on calibration point detection and shape grayscale model matching, it is first necessary to use a large number of individual images and their anatomical calibration points to model the shape and grayscale. This process uses the deformation field of the image area to describe the change of shape, and the pixel grayscale value of the image introduces grayscale information to the atlas. Then, the above images and their annotation results are used to train the convolutional neural network, so that the grid can predict the coordinates of each anatomical feature point of the target image. The shape parameters and grayscale parameters used to express the characteristics of the target image can be solved by aligning the statistical atlas with the target image and the prediction results of the neural network, so that the atlas is as close to the target image as possible in the model space. Finally, the segmentation of the target image is achieved using nonlinear deformation registration and organ mapping. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a medical image segmentation method based on calibration point detection and shape grayscale model matching. A large number of individual images and their anatomical feature point calibration results are used as training sets to construct a parameterized anatomical atlas with both shape and grayscale information, and train a convolutional neural network that can predict the anatomical feature points corresponding to the image. The model parameters of the target image are fitted by using the prediction of the neural network and the registration of the atlas, and the segmentation of the target image is completed through nonlinear correction and organ mapping.

[0005] To achieve the above objectives, the technical solution of the present invention is:

[0006] Step A: Construct a parameterized atlas with both shape and grayscale information.

[0007] Step A1: anatomical feature point calibration

[0008] Anatomical feature points refer to points in medical images that have specific anatomical significance. They usually have obvious texture features and can significantly reflect the changes between individuals. For different modeling subjects, matching anatomical feature points should be used, such as the joints of bones and the boundary points of internal organs. For accuracy reasons, anatomical feature points should be manually calibrated by doctors or imaging experts; there is no requirement for the number of calibration points, which can be added or deleted according to the actual effect of the user. In this step, a large number of individual images need to be annotated. These individual images and their annotation results serve as training sets for subsequent parameterized atlas construction and feature point detection model training.

[0009] Step A2, average image calculation

[0010] An image is randomly selected from the training set as the initial reference image, and the remaining images are transformed to the reference image through nonlinear deformation registration, and the corresponding spatial transformation is obtained to calculate the average inverse transformation. The average inverse transformation is applied to the entire training set and averaged to obtain a preliminary average image; the preliminary average image is set as the new reference image and the above process is repeated until convergence. The average image obtained after the above process converges needs to be calibrated with anatomical feature points; at the same time, it also needs to be accurately segmented. This process is also obtained by manual segmentation by imaging experts.

[0011] Step A3, Shape Modeling

[0012] The main task of shape modeling is to learn the deformation pattern of the average image between the images in the training set. This deformation pattern is characterized by the deformation field of the image area. The specific process is as follows:

[0013] First, the average image obtained in step A2 is registered to each training sample using nonlinear deformation, and a shape vector is constructed for each registered training sample. Specifically,

[0014] Assume i =[p i , l i ]∈R (N+K)×3 is the shape vector corresponding to the training sample i; where p i ∈R N×3 is the 3D coordinate set of all pixels after the average image is registered to the training sample i, and the number of pixels is N; l i ∈R K×3 is the three-dimensional coordinate set of all anatomical feature points in the sample image, and there are K anatomical feature points in total.

[0015] Then, the shape vector s is analyzed using the principal components analysis (PCA) method. i The feature extraction is performed on the set of , and the obtained shape model is expressed as follows:

[0016]

[0017] in, represents the average shape of the training set; is the shape feature vector matrix obtained by PCA, M represents the M deformation field change modes learned from the training set; b s ∈R M is the shape parameter; s∈R (N+K)×3 Indicates the current model shape, affected by b s control, so it can be adjusted by b s Control the deformation of the model.

[0018] Step A4: Grayscale modeling

[0019] The main purpose of grayscale modeling is to learn the variation pattern of grayscale values ​​of the average image among the images in the training set. The specific process is as follows:

[0020] Assume g i ∈R N For p i The corresponding grayscale vector represents the grayscale value of each pixel in the average image at the corresponding position in the training sample i. The principal component analysis method is used to analyze the grayscale vector g i The grayscale model is expressed as follows:

[0021]

[0022] in, Represents the gray value of each pixel in the average image; is the grayscale eigenvector matrix obtained by PCA, in which the M eigenvectors represent the M grayscale change patterns learned from the training set; b g ∈R M is the grayscale parameter; g∈R N Indicates the current model grayscale value, affected by b g control, so it can be adjusted by b g The value controls the change of the model grayscale.

[0023] Step B: Create an anatomical feature point detection model based on a convolutional neural network, and train the anatomical feature point detection model using image data and its annotation results.

[0024] Step B1, create a network model

[0025] The anatomical feature point detection model uses the 3D ResNet-18 network as the basic model for feature extraction; the features of each channel are then gradually fused through the convolutional layer, so that the network outputs the heat map of K channels (corresponding to K anatomical feature points), and the size of the output heat map is 1 / 4 of the input.

[0026] Step B2, model training

[0027] Using the training set described in step A1, a Gaussian heat map is generated for each anatomical feature point in each image for model training. The size of the heat map is 1 / 4 of the original image. During the model training process, the loss function is the Focal Loss function shown below:

[0028]

[0029] Where Y is the label value, is the result of network prediction, K is the number of anatomical feature points, and α and β are both hyperparameters.

[0030] Step C: Use the prediction results of the anatomical feature point detection model and the fitting of the model parameters to achieve the registration of the atlas and the target image and organ mapping.

[0031] Step C1: anatomical feature point detection

[0032] The anatomical feature point detection model trained in step B is used to predict the feature point heat map of the target image; the heat map of each output channel is decoded, and the extreme value of the heat value is taken as the coordinate of the corresponding anatomical feature point; finally, K anatomical feature points corresponding to the target image are obtained through prediction.

[0033] Step C2, shape model fitting

[0034] First, the target image is registered to the average image using a nonlinear deformation registration algorithm to obtain the corresponding spatial transformation, while incorporating the constraints and guidance of anatomical feature points. The loss function of nonlinear registration is as follows:

[0035] L=ω s E+ω l D

[0036] Among them, E is the grayscale similarity measure between the target image and the average image, which can guide the matching between the average image and the target image based on the pixel grayscale value, and the corresponding weight is ω s ; D is the distance measure of the anatomical feature points, which can align the anatomical feature points corresponding to the average image and the target image as much as possible, and its corresponding weight is ω l .

[0037] Then, the spatial transformation obtained by image registration is applied to the average shape of the atlas Get s′, and use different deformation field change modes in the shape model to match the difference between the two; this process is expressed as:

[0038]

[0039] in, is the shape parameter fitting result of the target image, is the eigenvector matrix of shape Therefore, the shape model fitting problem is transformed into the problem of solving a system of linear equations, and the corresponding system of equations can be solved by singular value decomposition.

[0040] Step C3, grayscale model fitting

[0041] Similar to the fitting method of the shape model, the grayscale change pattern in the model is also used to fit the grayscale value of the target image; this process is expressed as:

[0042]

[0043] in, is the grayscale parameter fitting result of the target image, is the grayscale eigenvector matrix The pseudo-inverse of , g′ is obtained by interpolating the target image at s′.

[0044] Step C4, non-standard grid interpolation

[0045] After fitting the shape model and grayscale model, the parameter-adjusted atlas needs to be converted into a rasterized pixel image. This process involves inserting the pixel grayscale values ​​on the irregular grid into the voxel grid of the target image, which can be achieved through Gaussian kernel interpolation.

[0046] Step C5, nonlinear shape refinement

[0047] Although the image obtained in step C4 is very close to the target image, there are still slight shape differences due to the constraints of the PCA model. Therefore, the image obtained in step C4 is registered with the target image using differential homeomorphism transformation; and similar to step C2, the distance measure of the anatomical feature points predicted by step C1 is added to the loss function of image registration. Since differential homeomorphism transformation is differentiable, reversible and differentiable, it can maintain the detailed structure of the image unchanged during the deformation process, and with the constraints and guidance of the anatomical feature points, shape refinement can be achieved.

[0048] Step C6, organ segmentation mapping

[0049] Since the shape model fitting result obtained in step C2 and the spatial transformation of the nonlinear shape refinement in step C5 can both be represented in the form of a displacement field, the combined spatial transformation of the two is applied to the average image organ segmentation result obtained in step A2 in the form of spatial point displacement, thereby mapping it to the target image to achieve organ segmentation.

[0050] Beneficial effects of the present invention: The present invention innovatively introduces the modeling of grayscale information in the process of atlas construction, enhances the robustness of the algorithm, and enables the atlas to adapt to images with different contrasts and different noise levels during parameter regulation; and uses rich shape prior information in the form of image region deformation field to predict the organ distribution of the target image. Even in the face of extremely low-quality medical images, relatively accurate segmentation results can be obtained. In addition, by adding anatomical feature point detection based on convolutional neural networks, a high degree of automation is guaranteed while further improving the detailed performance of organ segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method of the present invention.

[0052] Figure 2 It is a flowchart for constructing a graph with both shape and grayscale information.

[0053] Figure 3 It is the network structure diagram of the anatomical feature point detection model.

[0054] Figure 4 It is a flow chart of parametric atlas registration and organ mapping. DETAILED DESCRIPTION

[0055] The present invention is further described below by taking the trunk organ segmentation of mouse CT images as an example in combination with specific embodiments. The medical image segmentation method based on calibration point detection and shape grayscale model matching is as follows: Figure 1 As shown. It mainly includes three parts: constructing a parameterized atlas with both shape and grayscale information; creating an anatomical feature point detection model based on a convolutional neural network and training it; using the prediction results of the anatomical feature point detection model and the model parameters for fitting and organ mapping. The specific steps are as follows:

[0056] Step A: Construct a parameterized atlas with both shape and grayscale information, such as Figure 2 shown.

[0057] Step A1: Data preprocessing and anatomical feature point calibration

[0058] A large number of mouse CT images are resampled to the same pixel size (such as 0.2mm*0.2mm*0.2mm), and the trunk of the mouse is retained by image cropping, while keeping the image size after cropping consistent (such as 256*256*480). The image that completes the pretreatment is calibrated with anatomical feature points, including 6 anatomical feature points, including the tail of the cervical vertebra, the center of the pelvis, the lower edge of the left kidney, the upper edge of the left kidney, the lower edge of the right kidney, and the upper edge of the right kidney. The number of feature points selected by the present invention is not limited to 6, and can be added or deleted according to demand and algorithm performance. All individual images and their anatomical feature point calibration results in this step are used as training sets for subsequent parameterized atlas construction and feature point detection models.

[0059] Step A2, average image calculation

[0060] In the above training set, an image is randomly selected as the initial reference image, and the remaining images are transformed to the reference image through nonlinear deformation registration, and the corresponding spatial transformation is obtained to calculate the average inverse transformation. The average inverse transformation is applied to the entire training set and averaged to obtain a preliminary average image; the preliminary average image is set as the new reference image and the above process is repeated until convergence. The anatomical feature points of the average image obtained after convergence are annotated; at the same time, the organ segmentation of the average image is performed, including manual segmentation and automatic segmentation.

[0061] Step A3, Shape Modeling

[0062] First, the average image obtained in step A2 is registered to each training sample using nonlinear transformation, and a shape vector is constructed for each registered training sample. Then, the PCA method is used to extract features from the set of shape vectors. The resulting shape model is expressed as follows:

[0063]

[0064] in, represents the average shape of the training set; is the shape feature vector matrix obtained by PCA, M represents the M deformation field change modes learned from the training set; b s ∈R M is the shape parameter; s∈R (N+K)×3 Indicates the current model shape.

[0065] Step A4: Grayscale modeling

[0066] Assume g i ∈R N For p i The corresponding grayscale vector is analyzed using the principal component analysis method. i The grayscale model is expressed as follows:

[0067]

[0068] in, Represents the gray value of each pixel in the average image; is the grayscale eigenvector matrix obtained by PCA, in which the M eigenvectors represent the M grayscale change patterns learned from the training set; b g ∈R M is the grayscale parameter; g∈R N Indicates the current model grayscale value.

[0069] Step B: Create an anatomical feature point detection model based on a convolutional neural network, and train the anatomical feature point detection model using image data and its annotation results.

[0070] Step B1, create a network model

[0071] The network structure of the anatomical feature point detection model is as follows Figure 3 As shown in the figure, the model uses a 3D ResNet-18 network as the basic model for feature extraction; the features of each channel are gradually fused through the convolution layer, and finally the network outputs a heat map of 6 channels (corresponding to 6 anatomical feature points), and the size of the output heat map is 1 / 4 of the input.

[0072] Step B2, model training

[0073] Using the training set in step A1, for each anatomical feature point in each image, a Gaussian heat map with the anatomical feature point as the center and a radius of 4 pixels is generated. The size of the heat map is 1 / 4 of the original image. During the model training process, the loss function is the Focal Loss function shown below:

[0074]

[0075] Where Y is the label value, is the result of network prediction, K=6 is the number of anatomical feature points, and the hyperparameters α and β are set to 3 and 4 respectively.

[0076] Step C: Use the prediction results of the neural network and the fitting of the model parameters to achieve the registration of the atlas and the target image and organ mapping; the specific registration process is as follows Figure 4 shown.

[0077] Step C1: anatomical feature point detection

[0078] The anatomical feature point detection model trained in step B is used to predict the feature point heat map of the target image; the heat map is decoded, and the extreme points of the heat value are taken as the coordinates of the corresponding anatomical feature points; finally, the six anatomical feature points corresponding to the target image are obtained through prediction.

[0079] Step C2, shape model fitting

[0080] Firstly, the target image is registered to the mean image using B-spline transform to obtain the corresponding spatial transformation, while incorporating the constraints and guidance of anatomical feature points.

[0081] Then, the spatial transformation obtained by image registration is applied to the average shape of the atlas Get s′, and use different deformation field change modes in the shape model to match the difference between the two; the shape parameter fitting result of the target image is expressed as:

[0082]

[0083] in, is the eigenvector matrix of shape The pseudo-inverse of can be solved by singular value decomposition to solve the corresponding system of equations.

[0084] Step C3, grayscale model fitting

[0085] The grayscale change pattern in the model is used to fit the grayscale value of the target image; the grayscale parameter fitting result of the target image is expressed as:

[0086]

[0087] in, is the grayscale eigenvector matrix The pseudo-inverse of , g′ is obtained by interpolating the target image at s′, and the corresponding system of equations can be solved by singular value decomposition.

[0088] Step C4, non-standard grid interpolation

[0089] The pixel grayscale values ​​on the irregular grid are interpolated to the voxel grid of the target image through the Gaussian interpolation kernel to obtain a rasterized image. The interpolation radius is set to 0.1 mm and the sharpness is 10.

[0090] Step C5, nonlinear shape refinement

[0091] The fitting result is finely registered with the target image using a diffeomorphic transformation, while the anatomical feature points predicted by the neural network are again used to guide the registration.

[0092] Step C6, organ segmentation mapping

[0093] The combined spatial transformation of shape fitting results and nonlinear morphological refinement is applied to the organ segmentation of the average image, which is mapped to the target image to achieve organ segmentation.

[0094] Here, only the trunk organ segmentation of mouse CT images is used as an example for illustration. Although different imaging subjects have different change patterns in shape and grayscale, the idea and method of the present invention are also applicable to other imaging subjects, such as human PET / CT images, etc. Therefore, several modifications made without departing from the principles and basic ideas of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A medical image segmentation method based on calibration point detection and shape grayscale model matching, characterized in that: The method comprises the following steps: Step A: Construct a parametric atlas with both shape and grayscale information: Step B: Create an anatomical feature point detection model based on a convolutional neural network, and train the anatomical feature point detection model using image data and its annotation results; Step C: Using the prediction results of the anatomical feature point detection model and the fitting of the model parameters, the atlas and the target image are aligned and the organ is mapped; The specific steps of step A are as follows: Step A1: anatomical feature point calibration Obtain a large number of sample images and annotate the anatomical feature points of the sample images. The sample images and their annotation results serve as training sets for subsequent parameterized atlas construction and feature point detection model training. Step A2, average image calculation An image is randomly selected from the training set as the initial reference image, and the remaining images are transformed to the reference image through nonlinear deformation registration, and the corresponding spatial transformation is obtained to calculate the average inverse transformation; the average inverse transformation is applied to the entire training set and averaged to obtain a preliminary average image; the preliminary average image is set as the new reference image and the above process is repeated until convergence; the average image obtained after convergence needs to be calibrated with anatomical feature points; at the same time, it also needs to be accurately segmented; Step A3, Shape Modeling First, the average image obtained in step A2 is registered to each training sample using nonlinear deformation, and a shape vector is constructed for each registered training sample; specifically, Assume i =[p i ,l i ]∈R (N+K)×3 is the shape vector corresponding to the training sample i; where p i ∈R N×3 is the 3D coordinate set of all pixels after the average image is registered to the training sample i, and the number of pixels is N; l i ∈R K×3 is the 3D coordinate set of all anatomical feature points in the sample image, and there are K anatomical feature points in total; Then, the principal component analysis method is used to extract features from the set of shape vectors, and the resulting shape model is expressed as follows: in, represents the average shape of the training set; is the shape feature vector matrix obtained by principal component analysis, where M represents the M deformation field change modes learned from the training set; b s ∈R M is the shape parameter; s∈R (N+K)×3 Indicates the current model shape; Step A4: Grayscale modeling Assume g i ∈R N For p i The corresponding grayscale vector represents the grayscale value of each pixel in the average image at the corresponding position in the training sample i. The principal component analysis method is used to extract features from the set of grayscale vectors, and the obtained grayscale model is expressed as follows: in, Represents the gray value of each pixel in the average image; is the grayscale feature vector matrix obtained by principal component analysis, where the M feature vectors represent the M grayscale change patterns learned from the training set; b g ∈R M is the grayscale parameter; g∈R N Indicates the current model grayscale value.

2. The medical image segmentation method based on calibration point detection and shape grayscale model matching according to claim 1, characterized in that: The specific steps of step B are as follows: Step B1, create a network model The anatomical feature point detection model uses the 3D ResNet-18 network as the basic model for feature extraction. The features of each channel are then gradually fused through the convolution layer, so that the network outputs K channels of heat maps corresponding to K anatomical feature points. The size of the output heat map is 1 / 4 of the input. Step B2, model training During model training, the heat map labels are generated using the annotation results in step A, and the loss function is the FocalLoss function shown below: Among them, Y is the label value, is the result of network prediction, K is the number of anatomical feature points, and α and β are both hyperparameters.

3. A medical image segmentation method based on calibration point detection and shape grayscale model matching according to claim 1 or 2, characterized in that: The specific steps of step C are as follows: Step C1: anatomical feature point detection The anatomical feature point detection model trained in step B is used to predict the feature point heat map of the target image; the heat map of each output channel is decoded, and the extreme value of the heat value is taken as the coordinate of the corresponding anatomical feature point; finally, K anatomical feature points corresponding to the target image are obtained through prediction; Step C2, shape model fitting First, the target image is registered to the average image using a nonlinear deformation registration algorithm to obtain the corresponding spatial transformation, while incorporating the constraints and guidance of anatomical feature points; the loss function of the nonlinear registration is as follows: L=ω s E+ω l D Among them, E is the grayscale similarity measure between the target image and the average image, which can guide the matching between the average image and the target image based on the pixel grayscale value, and the corresponding weight is ω s ; D is the distance measure of the anatomical feature points, which can align the anatomical feature points corresponding to the average image and the target image, and its corresponding weight is ω l ; Then, the spatial transformation obtained by image registration is applied to the average shape of the atlas Get ′ , and use different deformation field change modes in the shape model to match the difference between the two; this process is expressed as: in, is the shape parameter fitting result of the target image, is the eigenvector matrix of shape The pseudo-inverse of ; therefore, the shape model fitting problem is transformed into the problem of solving a linear equation system, and the corresponding equation system is solved by singular value decomposition to obtain the shape parameters of the target image; Step C3, grayscale model fitting The grayscale change mode in the model is used to fit the grayscale value of the target image; the grayscale parameter fitting result of the target image is: in, is the grayscale parameter fitting result of the target image, is the grayscale eigenvector matrix The pseudo-inverse of g ′ The target image is in s ′ The interpolation value is obtained; Step C4, non-standard grid interpolation Gaussian kernel interpolation is used to insert the pixel grayscale values ​​on the irregular grid into the voxel grid of the target image, thereby obtaining a rasterized image after parameter fitting; Step C5, nonlinear shape refinement The image obtained in step C4 is registered with the target image using a differential homeomorphic transformation, and the constraints and guidance of the anatomical feature points predicted by the anatomical feature point detection model are used again to achieve shape refinement; Step C6, organ segmentation mapping Since the shape model fitting result obtained in step C2 and the spatial transformation of the nonlinear shape refinement in step C5 can both be represented in the form of a displacement field, the combined spatial transformation of the two is applied to the average image organ segmentation result obtained in step A2 in the form of spatial point displacement, thereby mapping it to the target image to achieve organ segmentation.

Images