A labeling method for multi-objective medical imaging data

Abstract

The present invention provides a labeling method for multi-target medical image data. After collecting medical image samples to be labeled, a coarse labeling task is sent to the labeling subject with the highest professional level. The coarsely labeled lesion labels are then divided into corresponding professional levels. After that, a fine labeling task is sent to the labeling subject with the corresponding professional level. When a labeling difference occurs, a conflict resolution task is sent to the labeling subject with the highest professional level. Subsequently, the finely labeled samples after consensus and conflict resolution are aggregated to form a data set, and the differences in fine labeling between the labeling subjects are counted. Finally, a more robust segmentation model is trained by constraining the consistency between different labeling results of different models, and the differences between the model and multiple labeling subjects are evaluated. While ensuring the accuracy of labeling, the cost is greatly reduced and the labeling speed is accelerated. The key information of the labeling process is recorded, providing an important reference for building a robust medical image analysis model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology in machine learning, and in particular to the labeling and segmentation technology of multi-target medical image data. Background Art

[0002] Deep learning methods have achieved remarkable results in various medical image segmentation and detection tasks. However, most cutting-edge lesion segmentation and detection models typically utilize deep and wide networks, and their robust generalization capabilities rely heavily on large-scale, high-quality, pixel-level annotated data. Furthermore, manual pixel-level annotation of medical images is an extremely expensive and time-consuming process, requiring extensive clinical knowledge and image interpretation skills.

[0003] There are three main ways to annotate traditional medical image segmentation datasets:

[0004] 1. Large-scale data is manually annotated at the pixel level by clinical experts only. This method can effectively ensure the quality of data annotation.

[0005] Second, clinical experts perform rough or partial annotation on large-scale data. While this approach reduces some annotation costs, it significantly increases the cost of developing medical image processing algorithms. The effectiveness and generalization capabilities of medical image analysis models built using only partially annotated data still need to be improved.

[0006] Third, ordinary annotators perform pixel-level manual annotation based on expert consensus. Although this method speeds up the annotation process, the annotation quality is often not high and there is a lack of annotation checking and correction mechanisms.

[0007] Most current public medical imaging datasets only provide the image data and labeling results, lacking detailed information about the annotation process, such as the annotator's qualifications and the differences between different annotators. Furthermore, developing more robust segmentation models and comprehensively evaluating them on datasets constructed with the participation of multiple annotators is a pressing challenge in the field of automated medical image analysis. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a multi-target medical image data annotation method that can more effectively mobilize the capabilities of the annotation subjects, improve the annotation efficiency, and reflect the differences between different annotation subjects, as well as to establish a more robust and comprehensive dataset evaluation index by utilizing the differences between different annotation subjects on a co-constructed dataset during segmentation training.

[0009] The technical solution adopted by the present invention to solve the above technical problems is a method for labeling multi-target medical image data, comprising the steps of:

[0010] Data preprocessing step: preprocess the collected medical imaging data;

[0011] Coarse labeling task distribution step: Send coarse labeling tasks to the labeling subject with the highest professional level, and receive the coarsely labeled medical image samples after the labeling subject with the highest professional level has annotated all lesions in the medical image samples at the video level or image level;

[0012] The steps for classifying the difficulty of annotation are as follows: the corresponding professional levels are divided according to the lesion annotations on the roughly annotated medical image samples; the number of annotation subjects for each professional level is more than two;

[0013] Fine-labeling task distribution step: Send fine-labeling tasks to the annotation subjects who assign each coarse-labeled medical image sample to a professional level, and receive fine-labeled medical image samples from the annotation subjects of each professional level after performing pixel-level or instance-level annotation on the lesions or tissues they are responsible for on the coarse-labeled medical image samples;

[0014] Consensus and conflict resolution steps: The determination of the fine annotation of the same lesion or tissue in the finely annotated medical image samples requires the consensus of no less than two annotators. When different annotators have different fine annotations for the same lesion or tissue in the finely annotated medical image samples, a conflict resolution task is sent to the annotator with the highest professional level, and then the annotator with the highest professional level confirms the fine annotation of the same lesion or tissue with differences.

[0015] Aggregation step: Aggregate the finely labeled medical image samples after consensus and conflict resolution to form a dataset.

[0016] Furthermore, a difference statistics step is included: in the annotation overlapping area of ​​each lesion or tissue of the finely annotated medical image sample after consensus and conflict processing, the difference in fine annotations between the annotated subjects is counted.

[0017] Furthermore, the summary step also organizes and records the professional levels of all annotation subjects, the lesions they participated in annotating, and the data scale, and uses them as the attributes of the annotation subjects in the medical imaging sample dataset; organizes and records the difference indicators between all annotation subjects of all lesions, and uses the difference indicators as the annotation attributes of the medical imaging sample dataset for the lesions.

[0018] Furthermore, when segmentation training is performed based on a dataset, the specific implementation method for establishing a more robust and comprehensive dataset evaluation index by utilizing the differences between different annotation subjects on the co-constructed dataset is as follows:

[0019] The medical image sample dataset is divided into training set, validation set and test set;

[0020] After the aggregation step, in the task model training process based on the medical image sample dataset, the training set is first input into two segmentation models with different structures, and the multi-objective loss of different lesion prediction results and fine annotation is calculated: the difference between the prediction results and fine annotation results output by the two segmentation models is calculated to obtain the segmentation loss Loss seg , the consistency loss Loss is obtained by the difference between the prediction results output by the two segmentation models con ; Then, the segmentation model weights are updated based on the iterative training of the multi-objective loss, and the best segmentation model is selected on the validation set. Finally, the test set is input into the best segmentation model, and the segmentation performance of the fine-labeling results of different labeled subjects corresponding to each lesion is calculated; when a lesion corresponds to more than two fine-labeling results, the differences between the fine-labeling results are compared.

[0021] Specifically, the multi-objective loss Loss is the weighted average sum of M different lesions: w i is the weight of the loss corresponding to the i-th different lesion, which is related to the professional level corresponding to the lesion annotation; i is the lesion number, ranging from 1 to M; λ e Is a variable that changes with the number of iterations, namely the consistency loss Loss con The weight of Pred changes over time; 1i Represents the prediction result of lesion i output by the first segmentation model, Pred 2i Represents the prediction result of lesion i output by the second segmentation model; when lesion i is finely annotated by only one annotation subject, GT 1i With GT 2i are the same fine annotation results; when the lesion i is finely annotated by two annotation subjects, GT 1i With GT 2i They are respectively the fine annotation results of two different annotation subjects; when there are three or more annotation subjects finely annotating together, Pred is calculated one by one 1i Segmentation loss with all fine annotation results, Loss Seg (Pred 1i ,GT 1i ) is the average of these segmentation losses, and accordingly, Loss Seg (Pred 2i ,GT 2i ) is Pred 2i Average segmentation loss over all fine-grained results.

[0022] The beneficial effect of the present invention is that, compared with traditional medical image annotation and data set construction methods, the present invention adopts the method of using the highest professional level annotation subject to perform video-level / image-level rough annotation and different professional levels annotation subjects to perform graded and fine annotation of lesions, which can greatly reduce costs and speed up the annotation speed while ensuring professionalism. The strategy of using two or more annotation subjects’ consensus to confirm annotations and the highest professional level annotation subject to resolve annotation conflicts helps to improve the annotation quality and ensure its accuracy. Furthermore, the present invention also counts the professional levels of all annotators, the organizations and scales involved in the annotation, and the annotation differences between different lesion annotators, providing an important reference for constructing a medical lesion / tissue analysis model. In addition, the present invention uses the prediction consistency of two different models and the consistency of multiple annotations to construct a more robust segmentation model, and evaluates its differences with multiple annotation subjects, providing a more comprehensive reference for its clinical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of a labeling process for multi-target medical imaging data according to an embodiment.

[0024] Figure 2 This is a schematic diagram of the attribute composition of a multi-target dataset, overlapping annotations of the same lesion, and difference evaluation between annotators.

[0025] Figure 3 (a) is a schematic diagram of segmentation of multi-target medical imaging data for multiple labeled subjects in an embodiment; (b) is a schematic diagram of a loss function for calculating a certain lesion in an embodiment. DETAILED DESCRIPTION

[0026] The annotation subject in the present invention can be a human or a machine with learning capabilities. In this embodiment, clinicians are assigned to the highest level of annotation. Ordinary personnel who have undergone minimal training are assigned to different levels of annotation based on their experience reading images and annotation skills at the time of training. The professional level corresponding to lesion annotation is achieved by assigning different levels of annotation difficulty to lesions.

[0027] like Figure 1 As shown, a method for multi-objective medical image data annotation, verification, and evaluation includes the following steps:

[0028] S1. Collect and organize large-scale medical imaging data;

[0029] S2, rough screening and preprocessing of medical imaging data;

[0030] S3. Send clinical experts to perform rough video-level or image-level annotation of all lesions in all samples;

[0031] S4. After clinical experts roughly annotate all lesions in all samples at the video or image level, they classify the lesions into different annotation difficulty levels. Based on their image reading experience and annotation level, annotators of different levels are assigned lesions or tissues of different difficulty levels.

[0032] S5. Send the task of performing pixel-level or instance-level fine annotation on the lesions or tissues they are responsible for to annotators of different levels;

[0033] S6. The fine annotation of the same lesion or tissue is determined by the consensus of no less than two annotators. When different annotators have different fine annotations for the same lesion or tissue, a task is sent to clinical experts to resolve the annotation conflict.

[0034] S7. Summarize the finely annotated medical image samples after consensus and conflict resolution to form a dataset, and count the differences in fine annotations of the same lesion or tissue among multiple annotators.

[0035] Step S1 specifically includes:

[0036] S101. Based on the task requirements and under the guidance of a clinician, determine the required data modality, the number of lesions / tissues M to be annotated, and the number of valid samples N to be collected;

[0037] S102. Collect medical imaging data containing multiple lesions / tissues under relevant regulations. The number of samples must be greater than N.

[0038] S103. The collected medical imaging data are sorted based on patients as basic units and grouped according to disease types.

[0039] Step S2 specifically includes:

[0040] S201. Quickly and roughly screen the collected medical imaging data to exclude data with poor imaging quality or that does not meet task requirements;

[0041] S202. Taking into account factors such as clinician usage habits and task requirements, the data is pre-processed by performing coordinate system conversion, contrast stretching, and size transformation.

[0042] Step S3 specifically includes:

[0043] S301. Experienced clinicians conduct detailed screening of the data after rough screening, and the total number of samples after screening is no less than N;

[0044] S302: While screening the data, the clinician performs image-level annotation on the data. For 3D medical imaging data, the presence of M lesions / tissues in the video / image sequence, along with the start and end frames of their appearance, is recorded one by one. For 2D imaging data, the presence of M lesions / tissues in a single frame is recorded.

[0045] S303. After completing the screening and image-level annotation, the clinician divides the M lesions / tissues into L annotation difficulty levels based on experience and data set conditions, providing guidance for subsequent graded and fine annotation.

[0046] Step S4 specifically includes:

[0047] S401: Randomly select a small sample of data and have H intermediate and junior annotators who participated in the annotation of this dataset perform trial annotation, and then have the clinicians check and evaluate it;

[0048] S402. Based on image reading ability, sensitivity to different lesions, and labeling experience, the H different levels of labelers are divided into L levels from high to low (the same number as the difficulty levels of lesion / tissue labeling);

[0049] S403. Assign lesion / tissue labeling tasks of different difficulty levels to annotators of different levels. Generally speaking, annotators with strong abilities will label lesions / tissues with high difficulty levels, and so on.

[0050] S404. For the same lesion / tissue, ensure that there are no less than two annotators involved in the annotation, and ensure that part of the data they annotate is overlapping.

[0051] like Figure 2 As shown in the figure, a portion of the data of a certain lesion / tissue is randomly selected to ensure that all the labelers need to label it.

[0052] Step S5 specifically includes:

[0053] S501, referring to the lesion / tissue distribution in S403, divide the data into L subsets;

[0054] S502. Labelers of different levels cyclically complete the labeling of lesions / tissues of different subsets and different difficulty levels.

[0055] For example, an annotator with a labeling level of 1 labels the lesions with a labeling difficulty level of 1 in data subset 1. An annotator with a labeling level of 2 labels the lesions with a labeling difficulty level of 2 in data subset 2. After the labeling is completed, an annotator with a labeling level of 1 labels the lesions with a labeling difficulty level of 1 in data subset L. Annotators with a labeling level of 2 label the lesions with a labeling difficulty level of 2 in data subset 1. After repeating this process L times, all lesions / tissues in all subsets are labeled.

[0056] Step S6 specifically includes:

[0057] S601: For the fine annotation of the same lesion / tissue, the annotators check and revise each other, and the final annotation result is determined by the consensus of the annotators;

[0058] S602. Clinical experts resolve labeling conflicts.

[0059] Step S7 specifically includes:

[0060] S701, such as Figure 2 As shown in Figure 3, the differences in pixel-level fine annotations between the annotators are counted in the overlapping areas of annotations for each lesion / tissue.

[0061] The annotation difference calculation method includes pixel-level classification and semantic segmentation and other indicators and their variants.

[0062] For example, pixel-level classification metrics:

[0063] Accuracy = (TP + TN) / (TP + TN + FP + FN),

[0064] Precision = TP / (TP+FP),

[0065] Recall = TP / (TP+FN),

[0066] F1-Score,F1-Score=2TP / (2TP+FP+FN),

[0067] Among them, TP, FP, TN and FN are true positive examples, false positive examples, true negative examples and false negative examples respectively. When calculating, the labeling result of one labeler is assumed to be the true value, and the labeling result of the other labeler is assumed to be the predicted value.

[0068] And, semantic segmentation indicators:

[0069] dice coefficient:

[0070] Intersection-over-Union (IOU):

[0071] Among them, A and B are the labeled areas of the two annotators respectively.

[0072] S702. The difference indicators between all the labeled subjects of all the lesions are sorted and recorded, and the difference indicators are used as the labeling attributes of the dataset regarding the lesions. Specifically, the difference indicators of all the annotators of all the lesions are sorted into the form of mean ± labeled difference, and used as the attributes of the dataset regarding the lesion / tissue to provide a reference for training and evaluating the lesion / tissue segmentation and analysis model.

[0073] S703. Organize and record the professional level of all annotators, the lesions / tissues they participated in annotating, and the data size, and use them as attributes of the dataset regarding the annotator.

[0074] In addition to having better accuracy, the data set formed after adopting the annotation method of the embodiment also contains more information that is beneficial to subsequent training. The same lesion has two different levels of annotations, coarse and fine, and the two can serve as auxiliary information for each other. When the data set is used to train different lesion analysis models, it can be decided whether to use coarse or fine annotations of the data set according to the actual analysis model task. The use of coarse annotation is suitable for models whose training purpose is to complete video classification or image classification, and the use of fine annotation is suitable for models whose training purpose is to complete image segmentation. To make fuller use of the two levels of annotation, an image classification branch can be added to the training of models such as medical image segmentation, and a loss function that combines segmentation and classification losses can be designed so that the coarse annotation at the image level helps the image segmentation task, and vice versa.

[0075] like Figure 3 As shown in (a) and (b), a method for multi-target medical image data segmentation includes two segmentation models with different structures, consistency loss, and segmentation loss. First, the medical image data in the training set are input into segmentation model 1 and segmentation model 2 one by one, respectively, to obtain prediction results 1 and 2 for multiple lesions. Then, the segmentation loss between prediction results 1 and 2 and the annotation results, as well as the consistency loss between prediction results 1 and 2, are calculated respectively. The model weights are then updated and iterative training is performed. Finally, the best model is selected on the validation set, and then the segmentation performance of the best model on the test set is calculated for each lesion based on the annotation results of different annotators, and the differences between the data annotated by multiple annotators and the prediction results are compared.

[0076] Segmentation model 1 and segmentation model 2 are two segmentation models with different structures. For example, segmentation model 1 is a classic U-Net, and segmentation model 2 is a novel ConvNeXt. For another example, segmentation model 1 is a CNN-based model, and segmentation model 2 is a Transformer-based model.

[0077] Among them, the overall loss is the weighted average sum of M different lesions:

[0078]

[0079] w i is the weight of the loss corresponding to M different lesions, which is related to the difficulty level of lesion / tissue annotation;

[0080] Among them, the calculation of segmentation loss Loss Seg (Pred 1i ,GT 1i ) and Loss Seg (Pred 2i ,GT 2i ), when only one person marks a certain lesion i, GT 2i With GT 1i Same; when there are two annotators annotating together, GT 2i With GT 1i are the annotation results of two different annotators respectively; when there are three or more annotators annotating together, Loss Seg (Pred 1i ,GT 1i ) is the average segmentation loss of the predicted result 1Pred1 and all the labeled results, Loss Seg (Pred 2i ,GT 2i ) is the average segmentation loss of the predicted result 2Pred2 and all the labeled results.

[0081] Among them, λ e is a variable that changes with the number of iterations, that is, the weight of each consistency loss changes over time. For example, before the number of iterations E, the weight of the consistency loss becomes higher and higher, and after E, the weight remains unchanged.

[0082] The embodiments described above are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

Claims

1. A method for labeling multi-target medical image data, characterized in that: Including steps: Data preprocessing step: preprocess the collected medical imaging data; Coarse labeling task distribution step: Send coarse labeling tasks to the labeling subject with the highest professional level, and receive the coarsely labeled medical image samples after the labeling subject with the highest professional level has performed video-level or image-level labeling on all lesions in the medical image samples to be labeled; The difficulty of labeling is divided into the following steps: according to the professional level corresponding to the lesion labeling on the roughly labeled medical image samples; the number of labeling subjects for each professional level is more than two; Fine-labeling task distribution step: Send fine-labeling tasks to the annotation subjects who assign each coarse-labeled medical image sample to a professional level, and receive fine-labeled medical image samples from the annotation subjects of each professional level after performing pixel-level or instance-level annotation on the lesions or tissues they are responsible for on the coarse-labeled medical image samples; Consensus and conflict resolution steps: The determination of the fine annotation of the same lesion or tissue in the finely annotated medical image samples requires the consensus of no less than two annotators. When different annotators have different fine annotations for the same lesion or tissue in the finely annotated medical image samples, a conflict resolution task is sent to the annotator with the highest professional level, and then the annotator with the highest professional level confirms the fine annotation of the same lesion or tissue with differences. Aggregation step: Aggregate the finely annotated medical image samples after consensus and conflict resolution to form a medical image sample dataset. The aggregation step also organizes and records the professional level of all annotation subjects, the lesions they participated in annotating, and the data size, and uses them as the attributes of the annotation subjects in the medical image sample dataset. The difference index between all annotation subjects of all lesions is organized and recorded, and this difference index is used as the annotation attribute of the medical image sample dataset for the lesion. Difference statistics step: After consensus and conflict resolution, the differences in fine annotations between the annotation subjects are counted in the overlapping areas of annotations of each lesion or tissue in the finely annotated medical image samples; The differences in fine annotations between statistically annotated subjects include relevant indicators and their variants of pixel-level classification and semantic segmentation of fine annotations between two annotated subjects. When calculating the relevant indicators of annotation differences, the fine annotation results of one annotated subject are taken as the true value, and the fine annotation results of the other annotated subject are taken as the predicted value.

2. The method according to claim 1, wherein: The medical image sample dataset is divided into training set, validation set and test set; After the aggregation step, in the task model training process based on the medical image sample data set, the training set is first input into two segmentation models with different structures, and the multi-objective loss of different lesion prediction results and fine annotation is calculated: the segmentation loss Loss is obtained by calculating the difference between the prediction results and fine annotation results output by the two segmentation models respectively. seg , the consistency loss Loss is obtained by the difference between the prediction results output by the two segmentation models con ; Then, the segmentation model weights are updated based on the iterative training of the multi-objective loss, and the best segmentation model is selected on the validation set. Finally, the test set is input into the best segmentation model, and the segmentation performance of the fine-labeling results of different labeled subjects corresponding to each lesion is calculated; when a lesion corresponds to more than two fine-labeling results, the differences between the fine-labeling results are compared.

3. The method according to claim 2, wherein: The multi-target loss Loss is the weighted average sum of M different lesions: w i is the weight of the loss corresponding to the i-th different lesion, which is related to the professional level corresponding to the lesion annotation; i is the lesion number, ranging from 1 to M; λ e Is a variable that changes with the number of iterations, namely the consistency loss Loss con The weight of Pred changes over time; 1i Represents the prediction result of lesion i output by the first segmentation model, Pred 2i Represents the prediction result of lesion i output by the second segmentation model; when lesion i is finely annotated by only one annotation subject, GT 1i With GT 2i The same fine annotation results; When the lesion i is finely annotated by two annotators, GT 1i With GT 2i They are respectively the fine annotation results of two different annotation subjects; when there are three or more annotation subjects finely annotating together, Pred is calculated one by one 1i Segmentation loss with all fine annotation results, Loss Seg (Pred 1i ,GT 1i ) is the average of these segmentation losses, Loss Seg (Pred 2i ,GT 2i ) is Pred 2i Average segmentation loss over all fine-grained results.

Images