Tooth image segmentation method and device and computer readable storage medium

Abstract

The invention provides a tooth image segmentation method, which comprises the following steps of: acquiring an initial medical image data set, screening out part of most valuable image data for manual annotation on the premise of ensuring the uncertainty and diversity of acquired samples, and carrying out domain incremental learning on a segmentation network model to obtain a segmentation result. And newly annotated samples are added in the next iteration for further training and fine tuning so as to continuously improve the segmentation performance of the segmentation network model. The segmentation network model is incrementally updated by continuously and actively screening valuable samples, so that the segmentation network model with a certain evolutionary ability can be obtained through training under the condition of a small amount of annotated image data.

Description

Technical Field

[0001] The present application relates to the field of digital oral technology, and specifically provides a tooth image segmentation method, device and computer-readable storage medium. Background Art

[0002] In recent years, the rapid development of digital information collection methods such as cone beam CT (CBCT) and intraoral scanning in the oral diagnosis and treatment industry has assisted dentists in diagnosis, surgery and designing treatment plans, making digital dentistry the direction and mainstream of future development.

[0003] Dental medical image segmentation is a basic task in digital dentistry. Quite a number of deep learning-based methods have achieved good performance in this task. However, most of them are based on fixed data sets with the same data distribution, assuming that the data distribution is fixed or stationary, the training samples are independent and identically distributed, and all data need to be provided to the model in a single training step. Once the model is deployed, the parameters are fixed, and only images with the same data distribution can be recognized subsequently, which often ignores the potential need for model evolution and updating.

[0004] In real clinical scenarios, new patients appear every day, inducing new data samples to increase over time across institutions. The traditional approach is to mix new data with historical data and retrain, but repeated learning of large-scale historical data results in a huge waste of computing resources.

[0005] At the same time, although deep learning technology has achieved great success in medical image diagnosis, there is a contradiction between deep learning-based segmentation methods and data annotation, that is, the contradiction between the high dependence on a large amount of high-quality labeled data and the expensive pixel-level annotation cost of medical images.

[0006] Active learning is considered to be one of the feasible solutions to alleviate this contradiction. It can alleviate this contradiction by incrementally selecting the most valuable or informative samples for manual annotation during the training process, thereby achieving the best model performance with as little labeled data as possible. However, the main segmentation methods based on active learning still have some problems, such as unreliable uncertainty assessment, inability to learn the boundaries of data samples well, etc., and often ignore the amount of information in the acquired samples.

[0007] Accordingly, the art needs a new technical solution to solve the above problems. Summary of the invention

[0008] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of how to improve image segmentation performance in the case of a small amount of annotated image data.

[0009] In a first aspect, the present application provides a method for tooth image segmentation, which includes the following steps S101-S108: Step S101: Acquire an initial medical image data set, select a first image data set and a second image data set; Step S102: Annotate the first image data set, and obtain a first pre-trained segmentation network model through fully supervised training, wherein the first pre-trained segmentation network model includes a first basic segmentation module and a first accuracy prediction module; S103: The first basic segmentation module filters out a part of the information-rich and diverse image data set from the annotated first image data set to form an annotated replay image data set; Step S104: Use an active learning strategy, combined with a hybrid strategy of accuracy and diversity, to sample the second image data set to filter out a certain number of high-value image data set; step S105: manually annotate the high-value image dataset to obtain a first annotated incremental image dataset; step S106: merge the annotated first incremental image dataset into the annotated replay image dataset to obtain a second annotated updated image dataset; step S107: use the annotated second updated image dataset to perform domain incremental learning on the first pre-trained segmentation network model, and adopt the knowledge distillation strategy to train to obtain an optimized second pre-trained segmentation network model, wherein the second pre-trained segmentation network model includes a second basic segmentation module and a second precision prediction module; step S108: for other unlabeled datasets, repeat steps S104-S107 until the preset conditions are met; step S109: use the finally updated segmentation network model to obtain an image segmentation result.

[0010] In a second aspect, the present application provides a method for tooth image segmentation, the method comprising the following steps S201-S209: Step S201: Acquire an initial medical image data set, construct a training data queue Q as {D1, D2, D3, ..., D N}, and then train them one by one, select D1 as the first image data set, and D2 as the second image data set; step S202: label the first image data set D1, and obtain the first pre-trained segmentation network model M1 through full supervised training, the first pre-trained segmentation network model M1 includes a first basic segmentation module and a first precision prediction module; S203: the first basic segmentation module selects a part of the image data set with rich information and diversity from the labeled first image data set D1 to form a labeled first replay image data set R1, and stores it in the replay image data queue P1 to form P1{R1}; step S204: adopt an active learning strategy, and sample the second image data set D2 in combination with a hybrid strategy of accuracy and diversity, A certain number of high-value image data sets are screened out; step S205: manually annotating the high-value image data set to obtain an annotated first incremental image data set I1; step S206: merging the annotated first incremental image data set I1 into the annotated replay image data queue P1{R1} to obtain an annotated second updated image data set; step S207: using the annotated second updated image data set to perform domain incremental learning on the first pre-trained segmentation network model M1, and adopting the knowledge distillation strategy to train to obtain an optimized second pre-trained segmentation network model M2, wherein the second pre-trained segmentation network model M2 includes a second basic segmentation module and a second precision prediction module; step S208: for the unlabeled data set {D3,…,D N} and the labeled replay image data queue that needs to be iteratively replayed, repeatedly execute steps S204-S207 until the preset conditions are met; step S209: use the final updated segmentation network model to obtain the image segmentation result.

[0011] In a third aspect, a computer device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the dental image segmentation method described in any one of the above technical solutions.

[0012] In a fourth aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, wherein the program codes are suitable for being loaded and run by a processor to execute the tooth image segmentation method described in any one of the above technical solutions.

[0013] This application uses an efficient active learning strategy to screen out some of the most valuable image data for manual annotation while ensuring the uncertainty and diversity of the samples obtained. After that, the segmentation network model is further trained and fine-tuned by adding newly annotated samples in the next iteration through domain incremental learning to continuously improve the segmentation performance of the segmentation network model. By continuously and actively screening valuable samples to incrementally update the segmentation network model, it is possible to train a segmentation model with certain evolutionary capabilities in the case of a small amount of annotated image data. Since the data sets provided by different institutions are usually heterogeneous and there are domain differences, the updated model needs to avoid catastrophic forgetting while promoting data streams with changes in data distribution. Therefore, the role of domain incremental learning is not only to maintain the stability of old data, but also to improve the plasticity of new data. BRIEF DESCRIPTION OF THE DRAWINGS The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0014] Figure 1 is a flow chart of the main steps of a tooth image segmentation method according to an embodiment of the present application;

[0015] Figure 2 is a schematic diagram of a tooth image segmentation method framework based on domain incremental learning according to an embodiment of the present application;

[0016] Figure 3 is a schematic diagram of a framework for model training according to an embodiment of the present application;

[0017] Figure 4 is a schematic diagram of a framework of domain incremental learning according to an embodiment of the present application;

[0018] Figure 5 is a flow chart of the main steps of a tooth image segmentation method according to another embodiment of the present application;

[0019] Figure 6 is a schematic diagram of a domain incremental learning framework based on data stream according to another embodiment of the present application;

[0020] Figure 7 It is a schematic diagram of the computer device structure provided by this application. DETAILED DESCRIPTION

[0021] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0022] Some embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0023] In the description of the present invention, "processor" may include hardware, software or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc.

[0024] Reference Figure 1 and Figure 2 , Figure 1 is a flowchart of the main steps of a tooth image segmentation method according to an embodiment of the present application, Figure 2 FIG. 1 is a schematic diagram of a tooth image segmentation method framework based on domain incremental learning according to an embodiment of the present application. Figure 1 As shown, the tooth image segmentation method in the embodiment 1 of the present application mainly includes the following steps S101 to S109. The above steps S101 to S109 are further explained below.

[0025] Step S101: Obtain an initial medical image dataset. The initial medical image dataset refers to an unlabeled image dataset, which is divided into image datasets with different data distributions according to information such as different institutions or CBCT machines, and the image dataset with the largest number of samples is selected as the first image dataset, and the image dataset with the second largest number of samples is selected as the second image dataset for subsequent steps.

[0026] Specifically, the initial medical image dataset is usually collected by multiple medical sites or hospitals. Since the data of different sites with different imaging equipment or protocols have obvious data heterogeneity, in order to simulate the actual clinical scenario, the data of different sites are made to arrive in sequence. The image datasets of different sites can be divided according to information such as institution, manufacturer, equipment field strength and resolution, and the one with the largest number of samples is selected as the first image dataset, and the one with the second largest number of samples is selected as the second image dataset.

[0027] Step S102: annotating the first image data set, and obtaining a first pre-trained segmentation network model through fully supervised training, wherein the first pre-trained segmentation network model includes a first basic segmentation module and a first accuracy prediction module.

[0028] In this step, the first image data set is manually annotated to obtain corresponding labels, and the initial neural network model is trained using the annotated first image data set through fully supervised learning to obtain a first pre-trained segmentation network model with a certain segmentation capability. The model includes a first basic segmentation module and a first precision prediction module. The first basic segmentation module predicts segmentation of subsequent unlabeled image data, and the first precision prediction module predicts segmentation accuracy of the unlabeled image data.

[0029] Specifically, refer to Figure 3 , the model training in the above step S102 can be implemented through the following steps S1021 to S1024.

[0030] Step S1021: The first pre-trained segmentation network model is constructed by the first basic segmentation module and the first precision prediction module. During the training process, the two modules can be trained simultaneously. At the same time, their optimization processes are decoupled and they are embedded in an end-to-end unified training framework in a cascade manner.

[0031] Step S1022: For the first basic segmentation module, the first image data set is used as input, and the optimization process uses cross entropy loss to minimize the difference between the predicted segmentation result and the true label of the first image data set. The cross entropy loss calculation formula is as follows:

[0032]

[0033] Among them, CE represents the cross entropy loss, p h,w and h,w They represent the predicted label and true label corresponding to the (h,w)th pixel, respectively. H and W represent the height and width of the image, respectively.

[0034] Step S1023: For the first accuracy prediction module, the images in the first image data set and the corresponding first basic segmentation module prediction posterior probabilities are concatenated as input, and the segmentation prediction results obtained by the first basic segmentation module and the multi-classification cross entropy loss of the one-hot encoded (one-hot) labels in the first image data set are taken as targets, aiming to use the posterior probability of the basic segmentation module to guide the accuracy prediction, and the optimization process uses the mean square error loss to minimize the difference between the prediction accuracy and the actual accuracy. The mean square error loss L msr The calculation formula is as follows:

[0035]

[0036] Where n is the number of samples, y i represents the true value of the i-th sample, is the predicted value of the accuracy prediction module for the i-th sample.

[0037] Step S1024: In the early stage of training, a short silent period is set for the precision prediction module, which can be 5 epochs. This means that only the basic segmentation module is trained during this period. The purpose is to avoid the posterior probability of the basic segmentation module in the early stage bringing more noise to the precision prediction module.

[0038] Step S103: the first basic segmentation module selects a portion of image data sets with rich and diverse information from the first labeled image data set to form a labeled replay image data set.

[0039] During the process of inference and prediction of the second image data set by the first basic segmentation module, if it is found that the segmentation effect is poor and the accuracy is not high on the data distribution, the active learning strategy can be used to sample the data set for subsequent domain incremental learning. The data replay strategy of domain incremental learning is used to select some information-rich and diverse image data sets from the labeled first image data set to form a labeled replay image data set.

[0040] Specifically, in order to ensure the diversity of the replay data, the KMeans algorithm is used to perform unsupervised clustering on the labeled first image dataset samples based on the hidden features of the segmentation model to obtain K clusters, and adaptive global average pooling is used to compress the original representation. Then, the iterative query selects the same number of image data from each cluster to form a certain number of replay image datasets, which can be set to 128.

[0041] Step S104: adopting an active learning strategy and combining a hybrid strategy of accuracy and diversity to sample the second image data set, and screen out a certain number of high-value image data sets.

[0042] The active learning strategy combines the accuracy and diversity of samples, that is, while utilizing the accuracy of the precision prediction module, it also utilizes the feature representation of the samples to balance the uncertainty and diversity of the acquired samples.

[0043] For the diversity strategy, when active learning iteratively selects N samples, N can be set to 64, and the feature maps of the labeled first image data set and the unlabeled second image data set in the penultimate layer space of the segmentation model decoder are obtained through the basic segmentation module.

[0044] Faced with the hidden space of the annotated feature map of the first image dataset, the nearest neighbor distance between the feature map of the second image dataset and the annotated feature map of the first image dataset is calculated and recorded by using the Euclidean distance formula, and a sample with the largest distance to its nearest neighbor is selected, and the sampling is repeated until the first N*P unlabeled image data are selected as the partial unlabeled image dataset based on diversity with a number P times the number of samples selected by the specified iteration, where P can be set to 4. Specifically, it is necessary to ensure that the amount of data screened based on diversity is greater than the value of N. The formula is as follows:

[0045]

[0046] Among them, h(x) represents the feature map of the penultimate layer space of the model decoder, D l Denotes the first labeled image dataset, D u represents the second unlabeled image dataset.

[0047] At the same time, the second image data set can obtain the corresponding prediction accuracy through the first pre-trained segmentation network model, and all image data in the partially unlabeled second image data set based on diversity are sorted in the order of the prediction accuracy from small to large. According to the sorting, N image data with smaller prediction accuracy are selected to form the high-value image data set.

[0048] Step S105: manually annotating the high-value image dataset to obtain an annotated first incremental image dataset, and the second image dataset is formed by removing the first incremental image dataset to form an updated second unannotated image dataset.

[0049] Among them, after manually annotating the high-value image dataset to obtain a labeled first incremental image dataset and obtaining the corresponding labels, the second image dataset needs to remove the first incremental image dataset and update it to other datasets, which are unlabeled to ensure that there are no labels in the unlabeled image dataset.

[0050] Step S106: merging the labeled first incremental image data set into the labeled replay image data set to obtain a labeled second updated image data set;

[0051] The labeled first incremental image dataset is used as new data for the next iterative training, and the labeled replay image dataset is used as old data for the next iterative training. They need to be merged and updated into a labeled second updated image dataset, and then transmitted to the model for training together.

[0052] Step S107: Use the labeled second updated image data set to perform domain incremental learning on the first pre-trained segmentation network model, and use the knowledge distillation strategy to train to obtain an optimized second pre-trained segmentation network model, wherein the second pre-trained model includes a second basic segmentation module and a second precision prediction module.

[0053] Among them, the labeled second image dataset, i.e., 128 old data and 64 new data, is used to train the optimized second basic segmentation module and the second precision prediction module through domain incremental learning of data replay and knowledge distillation strategy, aiming to overcome catastrophic forgetting.

[0054] See attached Figure 4 , the framework of domain incremental learning in the above step S107 is specifically implemented as follows.

[0055] In the knowledge distillation (KD) setting, the basic segmentation module of the pre-trained segmentation network model is used as the original segmentation model (also called the teacher network), and the structure of the new segmentation model (also called the student network) is consistent with the former, and the relevant weights are obtained through the exponentially weighted average EMA.

[0056] During the training phase of domain incremental learning, only the new segmentation network model is updated with parameters, where the replayed image dataset needs to pass through the original segmentation network model to obtain the logit feature soft label of the last layer and the feature maps of each layer of the decoder.

[0057] The labeled second updated image data set is passed to the new segmentation model for training to obtain the logit feature soft labels of the replayed image data set in the labeled second updated image data set and the feature maps of each layer of the decoder, as well as the hard labels of the last segmentation of the labeled second updated image data set.

[0058] Then, the original segmentation network model and the new segmentation network model are constrained by pixel-by-pixel distillation, and the soft label output based on logit features of the representative replayed image data (i.e., the distillation loss L kd1 ), transfer prior knowledge to protect the previous knowledge, and the pixel-by-pixel distillation loss calculation formula is as follows:

[0059]

[0060] in, and They represent the logit feature soft targets corresponding to the (h,w)th pixel generated by the new segmentation network model and the original segmentation network model, respectively. KL represents the Kullback Leibler divergence. represents the L2 distance, T is the temperature, and H and W represent the height and width of the image respectively.

[0061] According to the feature maps of each layer of the decoder obtained by the original segmentation network model and the new segmentation network model of the replayed image dataset, paired graph structure data is constructed for each image at each layer, including node-level features and graph-level features, wherein the graph feature interaction matrix is ​​constructed through the graph-level features, and then the graph similarity of the node feature interaction matrix of the paired graph structure data of each image in the replayed image dataset is solved to obtain the distillation loss of paired pixel similarity, wherein the square difference is used to calculate the distillation loss of the paired graph structure data of a certain layer, and the formula is as follows:

[0062]

[0063] Where s = norm(f i T f i ) represents the graph feature interaction matrix, norm represents the normalization process, superscripts s and t represent the student model and the teacher model respectively, and W′ and H′ represent the width and height of the feature map respectively.

[0064] The pairwise graph structure data distillation loss for the feature maps of each layer of the decoder is as follows: Where N is the number of decoder layers.

[0065] Under the guidance of prior knowledge, the new segmentation network model is committed to narrowing the difference between the hard label output and the true label of the annotated second updated image dataset (i.e., the segmentation loss L ce ) to correct and optimize the parameters of the new segmentation network model to adapt to the new data and further consolidate the previous knowledge. The segmentation loss formula is as follows:

[0066]

[0067] Among them, CE represents the cross entropy loss, p h,w and h,w They represent the predicted label and true label corresponding to the (h,w)th pixel, respectively. H and W represent the height and width of the image, respectively.

[0068] According to the above principle, the total loss function of knowledge distillation can be expressed as L ce and L fea_kd and L p_kd The weighted

[0069] L=L_ce+γ(L fe_kd +L p_kd )

[0070] Among them, γ is used to balance the plasticity and stability of the model. When γ = 0, it is equivalent to a fine-tuning operation.

[0071] Step S108: Repeat steps S104-S107 using other unlabeled data sets until the preset conditions are met.

[0072] The preset condition is that the amount of labeled image data reaches the labeling budget, or the performance of the evolved segmentation network model reaches the standard. The labeling budget is determined according to the actual amount of unlabeled data. Specifically, the labeling budget is less than the total number of samples of unlabeled image data.

[0073] Step S109: using the finally updated segmentation network model to obtain the image segmentation result.

[0074] In the above embodiment, the initial medical image data set obtained only selected image data sets with two different data distributions, and the segmentation model finally updated can maintain the segmentation accuracy of the first image data set while gradually learning and adapting to the second image data set with a small amount of labeled image data and reduced training costs.

[0075] Furthermore, real clinical scenarios will encounter continuous data flows, far more than limited data sets. Embodiment 2 is described below, in the process of obtaining the initial medical image data set partition, a data column is constructed.

[0076] Reference Figure 5 and Figure 6 , Figure 5 is a flowchart of the main steps of a tooth image segmentation method according to another embodiment of the present application, Figure 6 2 is a schematic diagram of a domain incremental learning framework based on data stream according to another embodiment of the present application. The tooth image segmentation method in Embodiment 2 of the present application mainly includes the following steps S201 to S209. The above steps S201 to S209 are further described below.

[0077] Step S201: Obtain an initial medical image dataset, divide it into multiple domain image datasets, and then construct a training data queue including N datasets from large to small according to the number of samples in each domain dataset. The training data queue Q is {D1, D2, D3, ..., D N}, and then train them one by one, selecting D1 as the first image data set and D2 as the second image data set for subsequent steps.

[0078] Among them, dental images in the medical field are used as an example to illustrate that in real clinical scenarios, new patients appear every day and induce new data samples to increase over time across institutions. In order to simulate this situation, the initial medical image dataset is obtained, divided into image datasets of multiple domains according to information such as different institutions or CBCT machines, and then a training data queue is constructed from large to small according to the number of samples in the dataset of each domain to form a data stream.

[0079] Step S202: annotate the first image data set D1, and obtain a first pre-trained segmentation network model M1 through full supervised training, wherein the first pre-trained segmentation network model M1 includes a first basic segmentation module and a first accuracy prediction module.

[0080] The training of the model is the same as step S102 described in Example 1, and finally a first pre-trained segmentation network model M1 with excellent performance is obtained based on the data set D1, including a first basic segmentation module and a first accuracy prediction module.

[0081] Step S203: the first basic segmentation module selects part of the image data sets with rich and diverse information from the first labeled image data set D1 to form a first labeled replay image data set R1, and stores it in the replay image data queue P1 to form P1{R1};

[0082] Among them, before the first pre-trained segmentation network model M1 learns the second image data set D2 in the training queue Q, it needs to use the data replay strategy of domain incremental learning, that is, to screen out some information-rich and diverse image data sets from the labeled data set D1 to form a labeled first replay image data set R1, and store it in the replay image data queue P1 to form P1{R1}.

[0083] In order to ensure the diversity of the replay data, the KMeans algorithm is used to perform unsupervised clustering on the labeled dataset D1 based on the hidden features obtained by the basic segmentation module to obtain K clusters, and adaptive global average pooling is used to compress the original representation. Then, the iterative query selects the same number of image data from each cluster to form the first replay image dataset R1, and stores them in the replay image data queue P1 to form P1{R1}, where the sample size of the first replay image dataset R1 can be set to 128.

[0084] Step S204: adopting an active learning strategy and combining a hybrid strategy of accuracy and diversity to sample the second image data set D2, and screening out a certain number of high-value image data sets.

[0085] The active learning strategy is the same as step S104 described in Example 1, and query sampling is performed on the second image data set D2 through a hybrid strategy based on accuracy and diversity, wherein the selection of image data samples can be divided into two stages.

[0086] Phase 1: Using a diversity-based query strategy, the first image dataset D1 and the second image dataset D2 are obtained through the first pre-trained segmentation network model to obtain the feature map of the corresponding decoder's penultimate layer space, and then a certain proportion of sample data of the second image dataset D2 is screened out through Euclidean distance iterative comparison, and the proportion can be set to 50%.

[0087] Stage 2: The 50% second image data set D2 can obtain the segmentation prediction accuracy value of the unlabeled image through the first accuracy prediction module, and then the segmentation prediction accuracy values ​​are sorted in ascending order, and the first 25% of the data samples are selected as the high-value image data set.

[0088] Step S205: manually annotating the high-value image dataset to obtain an annotated first incremental image dataset I1.

[0089] Step S206: merging the labeled first incremental image data set I1 into the labeled replay image data queue P1{R1} to obtain a labeled second updated image data set.

[0090] Step S207: Use the labeled second updated image data set to perform domain incremental learning on the first pre-trained segmentation network model M1, and adopt a knowledge distillation strategy to obtain an optimized second pre-trained segmentation network model M2, wherein the second pre-trained model M2 includes a second basic segmentation module and a second precision prediction module.

[0091] The domain incremental learning is the same as the specific operation in step S107 described in Example 1, see Figure 6 , using data replay and knowledge distillation, the labeled replay data queue P1{R1} is used as old data, and the labeled first incremental image data set I1 is used as new data, and domain incremental learning is performed on the first pre-trained segmentation network model. While maintaining the memory of the old data, it learns to adapt to the new data, thereby enabling the new model to perform well on both old and new data while reducing training time and annotation costs.

[0092] Step S208: Repeat steps S204-S207 for other unlabeled data sets and labeled replay image data queues that need to be iteratively replayed until the preset conditions are met. The other unlabeled data sets are {D3, ..., D N}, the preset condition is that the training queue Q is empty.

[0093] For {D3,…,D N} data set, the steps S204-S207 are repeatedly executed in a loop until the training queue Q is empty, specifically the following steps:.

[0094] An active learning strategy is adopted to combine the hybrid strategy of accuracy and diversity to the tth image dataset D t Sampling is performed to screen out a certain number of high-value image data sets. Specifically, 25% of the high-value image data sets may be screened.

[0095] Manually annotate the high-value image dataset to obtain an annotated t-1th incremental image dataset I t-1 ; The labeled t-1th incremental image dataset I t-1 Merge the labeled replay image data queue P t-1 {R1,R2,…,R t-1}, get the labeled tth updated image dataset;

[0096] The t-1th pre-trained segmentation network model M is trained using the tth updated image dataset that has been labeled t-1 Perform domain incremental learning and use the knowledge distillation strategy to obtain the optimized t-th pre-trained segmentation network model M t , the tth pre-trained model M t It includes the tth basic segmentation module and the tth precision prediction module.

[0097] Specifically, the well-trained model M at the t-1th iteration t-1 As the teacher model, the structure of the student model to be learned is the same as that of the teacher model, and the parameters are obtained through EMA. t-1 And the labeled image dataset D t The student model is trained with 25% of the high-value image data to obtain the model M at the t-th iteration. t , where the replay image data queue P t-1 {R1,R2,…,R t-1} contains information about {D1,D2,…,D t-1 The purpose of this paper is to use a partial set of labeled samples from the dataset to avoid catastrophic forgetting when the new model learns new data.

[0098] When the model M t When convergence occurs, the KMeans algorithm is used to select a certain number of image datasets D based on diversity t, and stored in the replay data queue P for subsequent data replay of domain incremental learning.

[0099] Among them, the marked replay image data queue P t-1 {R1,R2,…,R t-1}, using the t-1th pre-trained segmentation network model M t-1 The t-1th basic segmentation module is used to label the t-2th incremental image dataset I t-2 Filter out some information-rich and diverse image datasets to form the labeled t-1th replay image dataset R t-1 , and stored in the replay data queue P t-1 In, update to P t-1 {R1,R2,…,R t-1}.

[0100] For example, for the third image dataset D3, an active learning strategy is adopted, and a hybrid strategy combining precision and diversity is used to sample the third image dataset D3, and a certain number of high-value image datasets are screened out. Then, the high-value image dataset is manually labeled to obtain a labeled second incremental image dataset I2, and the labeled second incremental image dataset I2 is merged with the labeled replay image data queue P2{R1, R2} to obtain a labeled third updated image dataset. The second pre-trained model M2 is subjected to domain incremental learning using the labeled third updated image dataset, and a knowledge distillation strategy is used to obtain an optimized third pre-trained segmentation network model, wherein the third pre-trained model includes a third basic segmentation module and a third precision prediction module. Among them, in order to ensure the diversity of the replay data, the specific operation is the same as that in the step S203. Based on the hidden features obtained by the second basic segmentation module, the labeled replay image data queue P2{R1, R2} uses the second basic segmentation module of the second pre-trained segmentation network model M2 to filter out some information-rich and diverse image data sets from the labeled first incremental image data set I1 to form a labeled second replay image data set R2, and store it in the replay data queue P2, and update it to P2{R1, R2}.

[0101] The sample size of the second replayed image dataset R2 may be set to 5% of the first incremental image dataset O1.

[0102] Step S209: using the finally updated segmentation network model to obtain the image segmentation result.

[0103] This application uses an efficient active learning strategy to screen out some of the most valuable image data for manual annotation while ensuring the uncertainty and diversity of the samples obtained. After that, the segmentation network model is further trained and fine-tuned by adding newly annotated samples in the next iteration through domain incremental learning to continuously improve the segmentation performance of the segmentation network model. By continuously and actively screening valuable samples to incrementally update the segmentation network model, it is possible to train a segmentation model with certain evolutionary capabilities in the case of a small amount of annotated image data. Since the data sets provided by different institutions are usually heterogeneous and there are domain differences, the updated model needs to avoid catastrophic forgetting while promoting data streams with changes in data distribution. Therefore, the role of domain incremental learning is not only to maintain the stability of old data, but also to improve the plasticity of new data.

[0104] It is understood by those skilled in the art that all or part of the processes in the method for implementing any of the above embodiments of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0105] Another aspect of the present application also provides a computer device. In an embodiment of a computer device according to the present application, the computer device mainly includes a storage device and a processor. The storage device can be configured to store a program for executing the tooth image segmentation method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the tooth image segmentation method of the above method embodiment. Figure 7 , Figure 7 In the example, the storage device 11 and the processor 12 are connected to each other through a bus. For the convenience of explanation, only the parts related to this embodiment are shown. For the specific technical details not disclosed, please refer to the method part of the embodiment of this application.

[0106] In some possible implementations, the computer device may include multiple storage devices and multiple processors. The multiple processors may be processors deployed on the same device. For example, the computer device may be a high-performance device composed of multiple processors, and the multiple processors may be processors configured on the high-performance device. In addition, the multiple processors may also be processors deployed on different devices. For example, the computer device may be a server cluster, and the multiple processors may be processors on different servers in the server cluster.

[0107] Another aspect of the present application also provides a computer-readable storage medium. In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium can be configured to store a program for executing the tooth image segmentation method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned tooth image segmentation method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

Claims

1. A tooth image segmentation method, characterized in that: The method comprises the following steps S101-S108: Step S101: Acquire an initial medical image dataset, select a first image dataset and a second image dataset; Step S102: Annotate the first image dataset, and obtain a first pre-trained segmentation network model through fully supervised training, wherein the first pre-trained segmentation network model comprises a first basic segmentation module and a first precision prediction module; Step S103: The first basic segmentation module selects a part of information-rich and diverse image datasets from the annotated first image dataset to form an annotated replay image dataset; Step S104: Adopt an active learning strategy, combine a hybrid strategy of precision and diversity to sample the second image dataset, and select a certain number of high-value image datasets; Step S105: The high-value image dataset is manually annotated to obtain a first annotated incremental image dataset; step S106: merging the annotated first incremental image dataset into the annotated replay image dataset to obtain a second annotated updated image dataset; step S107: using the annotated second updated image dataset to perform domain incremental learning on the first pre-trained segmentation network model, and adopting the knowledge distillation strategy to train to obtain an optimized second pre-trained segmentation network model, wherein the second pre-trained segmentation network model includes a second basic segmentation module and a second precision prediction module; step S108: for other unlabeled datasets, repeating steps S104-S107 in a loop until the preset conditions are met; step S109: using the finally updated segmentation network model to obtain an image segmentation result.

2. The tooth image segmentation method according to claim 1, characterized in that: The initial medical image datasets are all unlabeled image datasets. The one with the largest number of samples is selected as the first image dataset, and the one with the second largest number of samples is selected as the second image dataset for subsequent steps.

3. The tooth image segmentation method according to claim 1, characterized in that: The other data set is the second image data set minus the first incremental image data set to form an updated second unlabeled image data set, and steps S104-S107 are repeatedly executed in a loop using the data set.

4. The tooth image segmentation method according to claim 1, characterized in that: The preset condition is that the amount of labeled image data reaches the labeling budget, or the performance effect of the updated segmentation network model has reached the standard.

5. A tooth image segmentation method, characterized in that: The method comprises the following steps S201 to S209: Step S201: Acquire an initial medical image data set, construct a training data queue Q as {D1, D2, D3, ..., D N }, and then train one by one, select D1 as the first image data set, and D2 as the second image data set; step S202: annotate the first image data set D1, and obtain a first pre-trained segmentation network model M1 through full supervised training, wherein the first pre-trained segmentation network model M1 includes a first basic segmentation module and a first accuracy prediction module; S203: The first basic segmentation module selects some information-rich and diverse image datasets from the labeled first image dataset D1 to form a labeled first replay image dataset R1, and stores it in the replay image data queue P1 to form P1{R1}; Step S204: Adopt an active learning strategy, combine the precision and diversity hybrid strategy to sample the second image dataset D2, and select a certain number of high-value image datasets; Step S205: Manually annotate the high-value image dataset to obtain a labeled first incremental image dataset I1; Step S206: Merge the labeled first incremental image dataset I1 into the labeled replay image data queue P1{R1} to obtain a labeled second updated image dataset; Step S207: Use the labeled second updated image dataset to perform domain incremental learning on the first pre-trained segmentation network model M1, and adopt the knowledge distillation strategy to train to obtain an optimized second pre-trained segmentation network model M2, the second pre-trained segmentation network model model M2 includes a second basic segmentation module and a second precision prediction module; Step S208: For the unlabeled dataset {D3,…,D N } and the labeled replay image data queue that needs to be iteratively replayed, repeatedly execute steps S204-S207 until the preset conditions are met; step S209: use the final updated segmentation network model to obtain the image segmentation result.

6. The tooth image segmentation method according to claim 5, characterized in that: In step S204, a certain proportion of sample data of the second image data set D2 is screened out through iterative comparison of Euclidean distance for the second image data set D2, and then the segmentation prediction accuracy value of the unlabeled image can be obtained through the first accuracy prediction module, and then the segmentation prediction accuracy values ​​are sorted in ascending order, and a certain proportion of data samples are selected as the high-value image data set.

7. The tooth image segmentation method according to claim 5, characterized in that: In step S208, the replayed image data array is used to segment the network model M using the t-1th pre-trained segmentation network model M. t-1 The t-1th basic segmentation module is used to label the t-2th incremental image dataset I t-2 Filter out some information-rich and diverse image datasets to form the labeled t-1th replay image dataset R t-1 , and stored in the replay data queue P t-1 In, update to P t-1 {R1,R2,…,R t-1 }.

8. The tooth image segmentation method according to claim 5, characterized in that: The steps S204-S207 are repeatedly executed in a loop and are as follows: an active learning strategy is adopted to combine the hybrid strategy of accuracy and diversity to the t image dataset D t Sampling is performed to screen out a certain number of high-value image data sets; the high-value image data sets are manually annotated to obtain the annotated t-1th incremental image data set I t-1 ; The labeled t-1th incremental image dataset I t-1 Merge to the marked replay image data queue P t-1 {R1,R2,…,R t-1 }, obtain the labeled tth updated image dataset; use the labeled tth updated image dataset to train the t-1th pre-trained segmentation network model M t-1 Perform domain incremental learning and use the knowledge distillation strategy to obtain the optimized t-th pre-trained segmentation network model M t , the tth pre-trained model M t It includes the tth basic segmentation module and the tth precision prediction module.

9. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the tooth image segmentation method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the tooth image segmentation method according to any one of claims 1 to 8.

Images