A semi-supervised medical image landmark positioning method based on topological structure constraint

Abstract

The application provides a kind of semi-supervised medical image landmark positioning method based on topological structure constraint, it is characterized in that, including: step S1, pre-process image, obtain initial training set.Step S2, initial training set is input to deep neural network and is trained, according to whether there is annotation information, sample is separated and is supervised training and self-supervised training.Step S3, to be measured image is input to the deep neural network of training completion, and the coordinate offset map and predicted hot spot map are predicted.Step S4, the probability that the landmark point represented by predicted hot spot map appears in the graph is used to vote method, and the pixel point higher than the set threshold value is voted, and the final landmark point coordinate on the image to be measured is marked according to the voting result.Deep neural network includes multi-task U-Net network, multi-resolution attention module and multi-branch hollow convolution module.

Description

Technical Field

[0001] This invention relates to a semi-supervised method for locating medical image landmarks based on topological constraints. Background Technology

[0002] Medical imaging is an important auxiliary tool for doctors in clinical diagnosis and surgical planning. In clinical practice, it is usually necessary to mark key landmarks in medical images according to anatomical structures, and then calculate the relevant line segment lengths or measure angles through these landmarks to complete the disease diagnosis [1]. It can be seen that accurate anatomical landmark localization is the foundation of clinical image analysis and is of great significance in assisting doctors to quickly diagnose diseases.

[0003] The continuous development of artificial intelligence technology has opened up a new perspective for the task of locating medical image landmarks: computers learn a large amount of medical image data and automatically summarize features from them, and then complete further landmark prediction based on these features. Reference [2] proposes a random forest-based method to automatically detect important landmarks in dental data of 3D cone-beam computed tomography (CBCT); Reference [3] proposes a cephalometric landmark detection algorithm that can be used to determine the type and severity of skeletal deformities in patients; Reference [4] proposes a two-stage deep learning method to realize real-time joint detection of large-scale anatomical landmarks on limited training data.

[0004] Therefore, the use of machine learning technology can help improve the efficiency of automated medical image interpretation. However, most of the mature machine learning algorithms mentioned above are based on supervised learning, requiring large-scale labeled data for model training. Acquiring medical images faces numerous difficulties: firstly, acquiring image data requires expensive specialized acquisition equipment; secondly, data labeling requires the participation of specially trained physicians, and medical experts often lack the time for extensive data labeling work. Therefore, the field of medical image analysis faces the real problem of a shortage of high-quality labeled data, which poses a significant challenge to the comprehensive utilization of artificial intelligence technology in medical image analysis.

[0005] [1]Leonardi R,Giordano D,Maiorana F,et al.Automatic cephalometric analysis:a systematic review[J].The Angle Orthodontist,2008,78(1):145-151.

[0006] [2]Cheng E,Chen J,Yang J,et al.Automatic Dent-landmark detection in 3-DCBCT dental volumes[C] / / 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.IEEE,2011:6204-6207.

[0007] [3]Lindner C,Wang C W,Huang C T,et al.Fully automatic system for accurate localisation and analysis of cephalometric landmarks in lateral cephalograms[J].Scientific reports,2016,6:33581.

[0008] [4]Zhang J,Liu M,Shen D.Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks[J].IEEE Transactions on Image Processing,2017,26(10):4753-4764. SUMMARY

[0009] To solve the above problems, a semi-supervised medical image landmark positioning method based on topological structure constraint is provided, and the technical scheme adopted by the present application is as follows:

[0010] The application provides a semi-supervised medical image landmark positioning method based on topological structure constraints, and has the characteristics that the method comprises the following steps: S1, collecting images and performing preprocessing to obtain an initial training set, wherein the initial training set comprises sample images with landmark coordinate annotation information and sample images without annotation information; S2, inputting the initial training set into a deep neural network for training, performing supervised training on the sample images with landmark coordinate annotation information by using the landmark coordinate annotation information, performing self-supervised training on the sample images without annotation information by using equivalent landmark changes, and obtaining a trained deep neural network when the number of training reaches a set number; S3, inputting a to-be-tested image into the trained deep neural network to obtain a predicted coordinate offset map and a predicted hotspot map corresponding to the to-be-tested image, wherein the hotspot map represents the probability of the appearance of a landmark at each pixel position in the image, and the offset map represents the coordinate offset between a predicted landmark and an actual landmark; and S4, according to the coordinate offset between the predicted landmark and the actual landmark represented by the predicted coordinate offset map, using a voting method to vote for the pixel points with a probability of the appearance of a landmark in the to-be-tested image being higher than a set threshold, and marking the final landmark coordinate on the to-be-tested image according to the voting result; wherein the deep neural network comprises a multi-task U-Net network as a backbone network, a multi-resolution attention module for generating attention coefficient maps of different resolutions according to feature maps of different resolutions generated by the multi-task U-Net network, and a multi-branch hollow convolution module for extracting context information of images of different scales by using different branches in parallel.

[0011] The semi-supervised medical image landmark positioning method based on topological structure constraints provided by the application can also have the following technical features: step S2 comprises the following steps: S2-1, inputting the initial training set into the deep neural network for training; S2-2, judging whether the input image has annotation information, if the judgment result is yes, performing step S2-3, and if the judgment result is no, performing step S2-4; S2-3, performing supervised training on the sample images with landmark coordinate annotation information; S2-4, performing self-supervised training on the sample images without annotation information; and S2-5, obtaining the trained deep neural network when the number of training reaches a set number.

[0012] The semi-supervised medical image landmark positioning method based on topological structure constraint provided by the application can also have the following technical features: step S2-3 includes the following steps: step S2-3-1, calculating a real hot spot map and a real offset map according to the annotation information; step S2-3-2, inputting a sample image with annotation information of the landmark coordinates into the deep neural network to obtain a predicted hot spot map and a predicted offset map; and step S2-3-3, constructing a loss function according to the real hot spot map, the real offset map, and the predicted hot spot map and the predicted offset map, and updating the parameters of the deep neural network through back propagation.

[0013] The semi-supervised medical image landmark positioning method based on topological structure constraint provided by the application can also have the following technical features: step S2-4 includes the following steps: step S2-4-1, inputting a sample image without annotation information into the deep neural network to obtain a predicted hot spot map and a predicted offset map; step S2-4-2, performing random angle rotation transformation on the sample image without annotation information; step S2-4-3, inputting the sample image after the rotation transformation into the deep neural network to obtain a new predicted hot spot map and a new predicted offset map; and step S2-4-4, using the self-similarity of the topological structure formed by the landmarks in the image after the rotation transformation, comparing the similarity between the hot spot maps and offset maps output by the deep neural network before and after the rotation transformation, constructing a loss function, and updating the parameters of the deep neural network through back propagation.

[0014] Invention action and effect

[0015] According to the semi-supervised medical image landmark positioning method based on topological structure constraint, first, due to the self-supervised module, the topological structure between the landmarks on the image has the transformation invariance feature, the hot spot map and the offset map are predicted for the original image and the image after rotation transformation, and the self-supervised loss is constructed by comparing the similarity of the hot spot maps and the offset maps before and after the transformation to constrain the network, so that when the medical image landmark positioning is performed, the unannotated data can be effectively used, and the accuracy of the landmark positioning of the neural network is improved.

[0016] Secondly, in the training of the deep neural network, the multi-resolution attention module and the multi-branch hollow convolution module are added, so that the neural network can pay more attention to the significant features in the image and consider more context information during training, thereby further improving the feature extraction capability of the neural network, so that the model can accurately locate the anatomical landmarks even in the case of less annotation data, and has the advantages of good positioning effect, strong robustness, good generalization capability and the like, and has practical application value. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1is a flow chart of a semi-supervised medical image landmark positioning method based on topological structure constraints in an embodiment of the present application.

[0018] Figure 2 is a structural schematic diagram of a deep neural network in an embodiment of the present application.

[0019] Figure 3 is a specific implementation flow chart of a supervised module training process in an embodiment of the present application.

[0020] Figure 4 is a specific implementation flow chart of a self-supervised module training process in an embodiment of the present application.

[0021] Figure 5 is a voting result map of landmark positioning in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the following will specifically describe a semi-supervised medical image landmark positioning method based on topological structure constraints in combination with embodiments and drawings.

[0023] <EMBODIMENT>

[0024] Figure 1 is a flow chart of a semi-supervised medical image landmark positioning method based on topological structure constraints in an embodiment of the present application.

[0025] As shown in Figure 1 , a semi-supervised medical image landmark positioning method based on topological structure constraints includes the following steps:

[0026] Step S1, collect images and perform preprocessing to obtain an initial training set.

[0027] Step S2, input the initial training set to a deep neural network for training, use the annotation information of the landmark point coordinates for supervised training on sample images containing the annotation information of the landmark point coordinates, use equivalent landmark point changes for self-supervised training on sample images without annotation information, and train a certain number of times to obtain a trained deep neural network.

[0028] Figure 2 is a structural schematic diagram of a deep neural network in an embodiment of the present application.

[0029] As shown in Figure 2 , the deep neural network 10 includes a multi-task U-Net network 11, a multi-resolution attention module 12, and a multi-branch hollow convolution module 13.

[0030] The multi-task U-Net network 11 is used to predict a hotspot map and an offset map for an input picture.

[0031] In the embodiment, the multi-task U-Net network 11 simultaneously predicts a heat map and an offset map for an input picture, wherein the heat map represents the probability of the key point appearing at each pixel position in the picture, and the offset map represents the coordinate offset between the key point and the real landmark point, which is divided into two directions of x-axis and y-axis.

[0032] The multi-resolution attention module 12 is used to strengthen the attention of the network to important features.

[0033] In the embodiment, the multi-resolution attention module 12 generates different resolution attention coefficient maps for different resolution feature maps obtained by the network, so that the low-level features obtained by the network are weighted by the attention coefficient and then spliced with the high-level features, the attention of the network to the significant features is strengthened, and the feature extraction capability of the model is improved.

[0034] The multi-branch dilated convolution module 13 is used to increase the receptive field of the network so that the model can capture more context information.

[0035] In the embodiment, the multi-branch dilated convolution module 13 extracts different scale context information in parallel using different branches, and uses dilated convolution to avoid the increase of network parameter quantity, so that the network can better capture the correlation between pixels and enhance the utilization of multi-scale context features.

[0036] Step S2 further includes the following steps:

[0037] Step S2-1, input the initial training set to the deep neural network for training.

[0038] Step S2-2, determine whether the input image has annotation information, if the determination is yes, execute step S2-3, and if the determination is no, execute step S2-4.

[0039] Step S2-3, perform supervised training on the sample image with the landmark point coordinates of the annotation information.

[0040] Figure 3 is a specific implementation flowchart of the supervision module training process in the embodiment.

[0041] As shown in Figure 3 , step S2-3-1 includes the following sub-steps:

[0042] Step S2-3-1, calculate the real heat map and the real offset map according to the annotation information.

[0043] In the embodiment, the heat map is calculated using a Gaussian function, and the offset map calculates the relative offset of the landmark point in the x-axis and y-axis.

[0044] Step S2-3-2, inputting the sample image with the annotation information of the landmark point coordinates to the deep neural network to obtain a predicted hotspot map and a predicted offset map;

[0045] Step S2-3-3, constructing a loss function of the supervision module according to the real hotspot map, the real offset map, the predicted hotspot map and the predicted offset map, and updating the parameters of the deep neural network through back propagation.

[0046] In this embodiment, the binary cross-entropy loss function is used to punish the difference between the predicted hotspot map and the real hotspot map, and the L1 loss function is used to punish the coordinate offset.

[0047] Step S2-4, judging whether the input image has annotation information, and entering this step when the judgment is no, using the self-supervision module for network training.

[0048] Figure 4 is a specific implementation flowchart of the self-supervision module training process in the embodiment of the application.

[0049] As shown in Figure 4 , step S2-4 includes the following sub-steps:

[0050] Step S2-4-1, inputting the sample image without annotation information to the deep neural network to obtain a predicted hotspot map and a predicted offset map;

[0051] Step S2-4-2, performing random angle rotation transformation on the sample image without annotation information;

[0052] Step S2-4-3, inputting the sample image after the rotation transformation to the deep neural network to obtain a new predicted hotspot map and a new predicted offset map;

[0053] Step S2-4-4, using the self-similarity of the topological structure formed by the landmark points after the rotation transformation, comparing the similarity between the hotspot maps and the offset maps output by the deep neural network before and after the rotation transformation, constructing a loss function of the self-supervision module, and updating the parameters of the deep neural network through back propagation.

[0054] In this embodiment, the MSE loss function is used to punish the prediction difference of the network before and after the image transformation for both the hotspot map and the offset map.

[0055] Step S3, inputting the to-be-tested image to the trained deep neural network to obtain a predicted coordinate offset map and a predicted hotspot map corresponding to the to-be-tested image, wherein the predicted hotspot map represents the probability of the appearance of the landmark point at each pixel position in the image, and the predicted offset map represents the coordinate offset between the predicted landmark point and the real landmark point.

[0056] Step S4, according to the network predicted coordinate offset map, voting is performed on all pixel points with hot spot map values higher than the threshold, and a prediction result of the final landmark point coordinate is obtained according to the voting result.

[0057] Figure 5 is a voting result map of landmark point positioning in the embodiment of the application.

[0058] As shown in Figure 5 , the first column of pictures is the original image, the second column of pictures is the real landmark point map, the black landmark points are real landmark points, and the third column of pictures is the predicted landmark point map, wherein the white landmark points are predicted landmark points. Figure 5 It can be seen that the embodiment has good accuracy in the medical image landmark point positioning task.

[0059] Effects of the embodiment

[0060] According to the semi-supervised medical image landmark point positioning method based on topological structure constraint, first, the self-supervised module in the embodiment can utilize the transformation invariance of the topological structure between the landmark points on the image, predict the hot spot map and the offset map through the original image and the image after rotation transformation respectively, and then construct the self-supervised loss by the similarity of the hot spot map and the offset map before and after the transformation to constrain the network. Therefore, when performing medical image landmark point positioning, the unlabeled data can be effectively utilized, so that the model can accurately locate the anatomical landmark points even in the case of less labeled data, and has the advantages of good positioning effect, strong robustness, good generalization ability and the like, and has practical application value.

[0061] Secondly, the deep neural network in the embodiment generates feature maps of different resolutions during the training stage, and the multi-resolution attention module can enhance the attention of the neural network to the significant features in the image, avoiding the interference of redundant features. And the attention coefficient map will be updated continuously with the training of the model, so that the network can gradually improve the feature extraction capability of the model.

[0062] Finally, the multi-branch dilated convolution module is added to the deep neural network in the embodiment to increase the receptive field of the neural network. The multiple branches enable the network to consider multi-scale context information in parallel during training, and the use of dilated convolution also avoids the increase of network computation and parameter amount, thereby further improving the attention of the neural network to multi-scale features.

[0063] The above embodiments are only used to illustrate the specific implementation of the application, and the application is not limited to the description range of the above embodiments.

Claims

1. A topology constraint based semi-supervised medical image landmark localization method, characterized in that, The method comprises the following steps: Step S1, collecting images and pre-processing to obtain an initial training set, wherein the initial training set comprises sample images containing landmark point coordinate annotation information and sample images without annotation information; Step S2, inputting the initial training set into a deep neural network for training, using landmark point coordinate annotation information for supervised training on the sample images containing landmark point coordinate annotation information, using equivalent landmark point changes for self-supervised training on the sample images without annotation information, and obtaining a trained deep neural network when the number of training reaches a set number; Step S3, inputting a to-be-tested image into the trained deep neural network to obtain a predicted coordinate offset map and a predicted hot spot map corresponding to the to-be-tested image, wherein the predicted hot spot map represents the probability of the appearance of a landmark point at each pixel position in the image, and the predicted coordinate offset map represents the coordinate offset between the predicted landmark point and the real landmark point; Step S4, according to the coordinate offset between the predicted landmark point and the real landmark point represented by the predicted coordinate offset map, using a voting method to vote for pixel points higher than a set threshold according to the probability of the appearance of a landmark point in the image represented by the predicted hot spot map, and marking the final landmark point coordinates on the to-be-tested image according to the voting result; Wherein, the deep neural network comprises a multi-task U-Net network as a backbone network, a multi-resolution attention module for generating attention coefficient maps of different resolutions according to feature maps of different resolutions generated by the multi-task U-Net network, and a multi-branch hollow convolution module for extracting context information of images of different scales using different branches in parallel; Wherein, the step S2 comprises the following steps: Step S2-1, inputting the initial training set into the deep neural network for training; Step S2-2, judging whether the input image has annotation information, if yes, executing step S2-3, if not, executing step S2-4; Step S2-3, performing the supervised training on the sample images containing landmark point coordinate annotation information; Step S2-4, performing the self-supervised training on the sample images without annotation information; Step S2-5, obtaining the trained deep neural network when the number of training reaches a set number; Wherein, the step S2-3 comprises the following steps: Step S2-3-1, calculating a real hot spot map and a real offset map according to the annotation information; Step S2-3-2, inputting the sample images with landmark point coordinate annotation information into the deep neural network to obtain the predicted hot spot map and the predicted coordinate offset map; Step S2-3-3, constructing a loss function according to the real hot spot map, the real offset map, the predicted hot spot map and the predicted coordinate offset map, and updating the parameters of the deep neural network through back propagation; Wherein, the step S2-4 comprises the following steps: Step S2-4-1, inputting the sample images without annotation information into the deep neural network to obtain the predicted hot spot map and the predicted coordinate offset map; Step S2-4-2, performing random angle rotation transformation on the sample image without annotation information; Step S2-4-3, inputting the sample image after the rotation transformation into the deep neural network to obtain a new predicted hotspot map and a new predicted coordinate offset map; Step S2-4-4, comparing the similarity between the hotspot map and the offset map output by the deep neural network before and after the rotation transformation by using the self-similarity of the topological structure formed by the landmark points in the image, constructing a loss function, and updating the parameters of the deep neural network through back propagation.

Images