Medical image evaluation index selection method based on attribute matching

Abstract

The invention relates to a medical image evaluation index selection method based on attribute matching, and provides a more reasonable evaluation index selection mode by deeply analyzing the inherent attributes of evaluation indexes and combining the attributes of a current task, an algorithm and a data set. Different from a traditional task directional selection method, the method determines applicable evaluation indexes only according to task targets, and the method pays attention to the mutual relation among the three attributes, so that the index matching process is optimized. The innovative method aims to improve the applicability of medical image evaluation, so that the evaluation result is more reliable and credible in clinical practice.

Description

Technical Field

[0001] The invention relates to a method for selecting medical image evaluation indicators based on attribute matching, and belongs to the technical field of medical image processing. Background Art

[0002] With the continuous development of the field of medical image processing, various evaluation indicators have been proposed and applied to different tasks. These evaluation indicators are crucial to the effectiveness of model evaluation and its reliability in clinical applications. Appropriate evaluation indicators can not only accurately reflect the performance of the model, but also provide valuable information for clinicians to help them make more accurate decisions. However, due to the increasing complexity of the tasks, the scope of application of some indicators has become blurred, leading to confusion between different tasks.

[0003] Medical image processing tasks include medical image segmentation, lesion detection, image reconstruction, image registration, 3D reconstruction, and imaging diagnosis. Each task has different requirements for evaluation indicators. In image segmentation, it is crucial to accurately distinguish the target structure; while in lesion detection, it is necessary to effectively identify and locate abnormal areas. Image reconstruction tasks focus on improving image quality and eliminating artifacts. Since the goals of different tasks are inconsistent, using the same evaluation indicator may lead to misleading evaluation results and affect the effectiveness of the model in clinical applications. Therefore, it is particularly important to choose appropriate evaluation indicators for each specific task.

[0004] At present, most studies still adopt a task-oriented approach, focusing on the applicability of indicators under specific tasks. This approach may improve the evaluation effect of specific tasks in the short term, but researchers often ignore the intrinsic properties of the indicators. Take the Dice similarity coefficient as an example. This indicator is widely used in medical image segmentation tasks, but in some cases it may not accurately reflect the segmentation effect. If you only focus on the direct goal of the task and ignore the intrinsic properties of the indicator, it will be difficult to fully evaluate the actual performance of the model.

[0005] New evaluation indicators are constantly being introduced, but existing research often lacks systematic analysis of new indicators and cannot determine whether they are applicable to other tasks. Some emerging deep learning algorithms have achieved good results in medical image processing, but there are not enough specialized evaluation indicators for the characteristics of these algorithms. Therefore, there is an urgent need for a systematic method to deeply analyze the intrinsic properties of evaluation indicators and combine them with task requirements, algorithm characteristics, and data set properties. Through this method, a more reasonable evaluation indicator selection strategy can be provided for the field of medical image processing, and the credibility and clinical practicality of the evaluation results can be improved. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a more reasonable evaluation index selection strategy. The method analyzes the intrinsic properties of the evaluation index and combines it with the current task, algorithm characteristics and data set properties to systematically optimize the matching process of the evaluation index.

[0007] A method for selecting medical image evaluation indicators based on attribute matching, comprising the following steps:

[0008] Step 1: Indicator attribute analysis: By standardizing the definition formula of the evaluation indicators, they are divided into seven categories according to their characteristics, including: overlap-based, volume-based, pair count-based, information theory-based, probability-based, spatial distance-based, and pixel error-based, which is convenient for subsequent analysis and comparison;

[0009] Step 2: Assignment of indicator attributes: Assign attributes to each indicator, including computational complexity, sensitivity to different types of errors, and relevance to clinical needs, etc., to lay the foundation for the rational selection of evaluation indicators. By assigning different attributes to the evaluation indicators, it is convenient to match them with the task algorithm and data set in the future. Computational complexity determines the efficiency and resource consumption of the indicator in practical applications; sensitivity attributes, such as sensitivity to outliers, true negative areas, and point locations, affect the performance of the indicator when dealing with errors; relevance to clinical needs ensures that the selected indicator can be connected with the actual needs of the task, thereby providing more clinically valuable evaluation results.

[0010] Step 3: Task and algorithm attribute definition: Task and algorithm attribute definition, by analyzing the goals of the current medical image processing tasks, algorithm mechanisms and their adaptability to the characteristics of the data set, clarify the core requirements of the task and identify the performance characteristics of the algorithm. According to the specific requirements of the task (such as sensitivity to outliers, small areas, and complex boundaries), the tasks and algorithms are given relevant attributes to facilitate the matching selection of subsequent evaluation indicators. This process ensures that the selected evaluation indicators can accurately reflect the actual performance of the task goals and algorithms.

[0011] Step 4: In this step, the distribution, quality, and structural properties of the dataset used need to be analyzed to ensure that the selected evaluation indicators match the characteristics of the dataset. First, analyze the distribution of the target area of ​​the dataset and select evaluation indicators that are suitable for dealing with imbalance or morphological changes. Next, evaluate the quality factors of the dataset, such as image resolution and noise level, to determine whether indicators that are more robust to noise are needed. Then, consider the shape and structure of the target area and select indicators that are sensitive to shape or can better match the volume. Finally, combine the task requirements and algorithm characteristics to ensure that the selected evaluation indicators can effectively match the characteristics of the dataset, thereby improving the applicability and reliability of the evaluation results.

[0012] Step 5: Attribute matching selection suggestions: Construct an attribute matching table to compare the attributes of the evaluation indicators with the attributes of the tasks, algorithms, and datasets to obtain evaluation indicator selection suggestions that meet the current application needs. First, list all evaluation indicators and their key attributes, including sensitivity to error types, computational complexity, and clinical relevance. Next, analyze the task requirements, algorithm characteristics, and dataset characteristics, clarify the specific requirements of each task, and understand the characteristics of the algorithm and dataset. Then, construct an attribute matching table to compare the attributes of the evaluation indicators with the attributes of the tasks, algorithms, and datasets to obtain the applicability score of each indicator. Finally, based on the matching results, make recommendations for the selection of evaluation indicators for specific tasks to ensure that the selected indicators can most effectively reflect the task requirements and optimize the evaluation process.

[0013] The step 2 specifically includes:

[0014] Potential factors affecting the attribute assignment process in the assignment task include: outlier sensitivity, true negative consideration, chance adjustment, sensitivity to point location, ignoring alignment errors, recall bonus, overall shape and alignment, overlap-based, distance-based, pair count-based, information-theoretic-based, probability-based, spatial distance-based, and pixel error-based.

[0015] The core of the present invention is to build an attribute matching framework that can effectively identify and utilize the relationship between evaluation indicators, tasks, algorithms and datasets.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention relates to a method for selecting medical image evaluation indicators based on attribute matching to achieve a more reasonable evaluation indicator selection strategy. The method systematically optimizes the matching process of evaluation indicators by deeply analyzing the intrinsic attributes of the evaluation indicators and combining them with the current task, algorithm characteristics and data set attributes, aiming to improve the applicability of medical image evaluation and make the evaluation results more valuable for reference in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A classification diagram of evaluation indicators of a method for selecting medical image evaluation indicators based on attribute matching according to the present invention;

[0020] Figure 2An evaluation index attribute allocation diagram of a medical image evaluation index selection method based on attribute matching according to the present invention;

[0021] Figure 3 The present invention discloses an attribute matching table of evaluation indexes for a method for selecting medical image evaluation indexes based on attribute matching. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The specific implementation method will be demonstrated with a specific example. In this example, 20 different types of indicators are used. The segmentation task is used as an example to demonstrate the attribute matching process. Before officially starting to introduce the method, the 20 indicators used in the example will be briefly introduced to help understand the subsequent step allocation process.

[0024] ACC (Accuracy): The ratio of the number of correctly classified samples to the total number of samples, which is used to measure the accuracy of the overall classification of the model.

[0025] F1 (F1 Score): The harmonic mean of precision and recall, used to comprehensively evaluate the accuracy and recall ability of the model.

[0026] MCC (Matthews Correlation Coefficient): used to measure the balance and correlation of prediction results in binary classification problems.

[0027] BACC (Balanced Accuracy): The average of sensitivity and specificity, used to evaluate model performance when the categories are unbalanced.

[0028] NPV (Negative Predictive Value): The ratio of the number of true negative samples to the number of predicted negative samples, which is used to measure the accuracy of the model's prediction of negative samples.

[0029] ER (Error Rate, misclassification rate): measures the proportion of incorrect predictions among all predictions, which is a supplement to the accuracy rate.

[0030] FDR (False Discovery Rate): The ratio of the number of false positive samples to the number of predicted positive samples, which is used to measure the error proportion of the model predicted positive samples.

[0031] G-Mean (Geometric Mean): The geometric mean of precision and recall, used to evaluate model performance in unbalanced data sets.

[0032] VOE (Volumetric Overlap Error): It is used to measure the overlap error between the predicted volume and the actual volume.

[0033] RVD (Relative Volume Difference): The relative difference between the predicted volume and the actual volume, used to evaluate the volume accuracy of the segmentation model.

[0034] CC (Correlation Coefficient): used to measure the linear correlation between two variables.

[0035] Log-Loss (Logarithmic Loss): It is used to measure the difference between the predicted probability of the classification model and the actual classification result. The smaller the value, the better the model performance.

[0036] NMI (Normalized Mutual Information): used to measure the correlation between two variables, normalized to eliminate scale differences.

[0037] SSIM (Structural Similarity Index Measure): used to measure the similarity between two images in terms of brightness, contrast, and structure.

[0038] NCC (Normalized Cross-Correlation): used to measure the correlation between two images, and normalized to eliminate scale differences.

[0039] FID (Frechet Inception Distance): It is used to measure the distance between the generated image and the real image in the feature space, and is usually used to evaluate the quality of images generated by generative adversarial networks (GANs).

[0040] 95HD (95th Percentile Hausdorff Distance): The 95th percentile of the Hausdorff distance, used to reduce the impact of extreme values ​​on the evaluation

[0041] PSNR (Peak Signal-to-Noise Ratio): used to measure the quality of image compression or restoration. The higher the value, the better the quality.

[0042] MAE (Mean Absolute Error): The average value of the absolute error between the predicted value and the actual value, which is used to measure the error size of the prediction model.

[0043] MSE (Mean Squared Error): The average of the sum of squares of the errors between the predicted values ​​and the actual values, used to measure the error of the regression model.

[0044] A method for selecting medical image evaluation indicators based on attribute matching, comprising the following steps:

[0045] Step 1: Analysis of indicator attributes. We first divide each indicator into seven categories by normalizing the formula of each indicator, namely: Spatial overlap based, Volume based, Pair counting based, Information theoretic based, Probabilistic based, Spatial distance based and Pixel-wise error based. The basis for the division of each category is given. Overlap-based indicators are derived from the four basic elements in the confusion matrix: true positive (TP), false positive (FP), true negative (TN) and false negative (FN). They evaluate the performance of the model by calculating the ratio of the overlapping part (such as TP) to other parts, emphasizing the similarity between the predicted results and the actual results. Volume-based indicators evaluate the segmentation effect by calculating the difference between the model predicted volume and the true volume. Pair counting-based indicators measure the correlation by comparing the pair counts between the predicted results and the actual results, and are often used to evaluate the prediction accuracy of continuous variables. Information theory-based indicators evaluate model performance through the framework of information theory. Probability-based indicators evaluate the quality of the generated model by measuring the difference in distribution between the generated image and the real image in the feature space, emphasizing the similarity between the image generated by the model and the real image. Spatial distance-based indicators evaluate spatial accuracy by calculating the maximum distance (or other distance metrics) between the predicted segmentation boundary and the real boundary, paying special attention to the accuracy of boundaries and shapes, and emphasizing the importance of spatial position. Pixel error-based indicators evaluate model performance by directly comparing the difference between the predicted and real pixel values, emphasizing the accuracy of each pixel. These indicators are suitable for pixel-level classification and regression tasks, providing quantitative assessments of the overall prediction quality. For some indicators, they may belong to multiple categories at the same time (for example, the Dice coefficient can be classified as an overlapping class or the volume correlation calculated can be considered). In this case, they need to be classified according to their main characteristics. The classification results are shown in Figure 1 .

[0046] Step 2: Assignment of indicator attributes. Before selecting evaluation indicators reasonably, we need to assign attributes to each indicator, including computational complexity, sensitivity to different types of errors, and relevance to clinical needs, to lay the foundation for the reasonable selection of indicators. The following are various potential influencing factors considered in the indicator attribute assignment process in the segmentation task:

[0047] Outlier Sensitivity: A metric has this property if its calculation method is particularly sensitive to outliers in the segmentation result (e.g., a few mis-segmented pixels). For example, metrics based on pixel errors, such as MSE, are often very sensitive to outliers because they square or take the absolute value of the error for each pixel, which magnifies the impact of a few errors. Metrics that calculate errors by weighted average or squared difference often have this property. When there are a small number of pixels in the automatic segmentation result that are outside the target area, metrics with this property will severely penalize these outliers.

[0048] True Negatives Consideration: This property applies to situations where the evaluation metric considers the consistency of the background region (true negative voxels) with the true label. In a binary classification problem, if a metric can fully consider the consistency between the prediction of the background region and the true value, it should have this property. For example, when calculating accuracy or a specific overlap metric (such as the Dice coefficient), the correct identification of background pixels will directly affect the evaluation result, so such metrics have this property.

[0049] Chance Adjustment: If an evaluation metric adjusts for chance consistency (e.g., overlapping results of random splits) to remove the influence of random factors, this metric has this property. For example, the adjusted RandIndex (ARI) or the Jaccard coefficient after background noise correction can take random factors into account, thereby eliminating high scores caused by pure chance.

[0050] Sensitivity to Point Positions: Some metrics are more sensitive to positional changes of false positive voxels, while others may ignore positional differences and focus more on the accuracy of the overall segmentation result. Metrics with the "Sensitivity to Point Positions" property usually adjust their scores as the spatial position of the false positive voxels changes. For example, some distance-based metrics, such as the Hausdorff distance, will have a greater impact on the alignment accuracy of the segmentation result with the true value due to positional offsets.

[0051] Ignoring Alignment Errors: Some metrics focus on the overall shape or volume without considering the alignment accuracy between the segmentation result and the true value. Metrics with this property usually do not affect the results due to alignment errors when matching shapes or volumes. For example, some overlap metrics (such as Dice coefficient and Jaccard coefficient) do not consider the exact position of the segmented region when calculating, but only focus on the coverage of the region, thus ignoring the alignment error.

[0052] Recall Rewarding: Metrics with this attribute focus more on recall than precision, and usually penalize errors caused by insufficient segmentation volume or imprecise boundaries. Suitable for metrics that emphasize recall, such as the F1 score or some variants of the Dice coefficient, which are more tolerant of increasing the number of detected positive samples (such as tumors or organs) and do not attach importance to the accuracy of boundaries or volumes.

[0053] General Shape and Alignment: Metrics with this property ignore details and only focus on the overall shape and alignment of the segmented region. For example, VOE and RVD usually focus on evaluating the overall shape of the segmented region without caring about specific details or alignment errors.

[0054] Overlap-based: This property describes the metric based on four types of overlap (TP, TN, FP, FN) between the automatic segmentation and the true value.

[0055] Distance-based: This property describes the metric defined based on the Euclidean distance of voxels between segments in the automatic segmentation and segments in the ground truth.

[0056] Information Theoretical-based: This property describes metrics that are based on information theoretic factors.

[0057] Probabilistic-based: This property describes a metric defined based on the statistics of voxels in the overlapping regions of the segmented regions.

[0058] Pair-counting-based: This property describes a metric that is computed based on the grouping of all possible pairs of objects in an image partition.

[0059] Volume-based: This property describes the metric defined based on the volume of the segmented region.

[0060] Pixel-wise Error-based: This property describes pixel-wise error-based metrics such as mean square error (MSE), mean absolute error (MAE), and peak signal-to-noise ratio (PSNR).

[0061] By assigning different attributes to the evaluation indicators, it is easier to match them with the task algorithm and data set. Figure 2 .

[0062] Step 3: Task and algorithm attribute definition. Before selecting indicators, considering that segmentation tasks of different organs may have different sensitivities, we need to clarify the specific requirements of the task and assign relevant attributes of the task and algorithm according to specific situations. For example, in the segmentation task, for image segmentation tasks with outliers, when the outliers have little effect on the overall results, evaluation indicators that are sensitive to outliers, such as Hausdorff Distance (HD), should be avoided. When processing Small Segments, if at least one dimension is less than 5% of the image grid dimension, indicators based on overlap number and volume may not be applicable. In this case, it is recommended to use distance-based evaluation indicators. For segmentation areas with ComplexBoundary, evaluation indicators that are sensitive to point positions (such as HD and mean volume difference) should be used, rather than volume-based evaluation indicators. In addition, for segmentation areas with Low Densities, indicators based on four overlap numbers may be over-penalized, so it is recommended to use distance-related evaluation methods (such as HD, mean volume difference, and Modified Hausdorff Distance, MHD). Finally, for the segmentation area with Low SegmentationQuality, the distance-based evaluation index is more effective than the volume-based index in distinguishing the segmentation quality, and the volume similarity (VS) index should be avoided. By analyzing the special situations that may occur in the target task, the task and algorithm-related attributes are assigned to prepare for the selection of appropriate indicators.

[0063] Step 4: Target dataset attribute analysis: Analyze the unique distribution, quality, and structural attributes of the dataset used to ensure that the selected indicators match the characteristics of the dataset, thereby improving the applicability of the evaluation results. When analyzing the dataset, the segmentation algorithm must also have corresponding requirements for different segmentation targets. Many different requirements can be defined, which may vary from case to case. The following are some common requirements for segmentation algorithms:

[0064] Contour is Important: In some tasks, the accuracy of the segmentation boundary is critical. In this case, evaluation metrics that are sensitive to point positions (such as Hausdorff distance and average voxel deviation) should be selected because they are better at evaluating the accuracy of boundaries and are not suitable for volume-based metrics.

[0065] Alignment is Important: When the overall position and alignment of the segmentation result are more important than the boundary accuracy, volume-based metrics may not be applicable. In this case, the overall alignment of the segmented region needs to be evaluated to ensure that the segmented region is consistent with the shape of the true target region.

[0066] Recall is Important: In some cases, the segmentation result needs to include all the real target areas as much as possible, even if this will introduce some false areas. In this case, the accuracy of the boundary is not so important, and maximizing the recall rate should be prioritized. Mutual information and true positive rate are indicators that reward recall and can effectively reflect the comprehensiveness of the segmentation.

[0067] Volume is Important: Sometimes the size of the segmented region is more important than the accuracy of its boundaries and position. In this case, the algorithm should ensure that the volume of the segmented region is as close as possible to the volume of the true object. It is recommended to use a volume similarity metric that accurately reflects the overall size of the segmented region.

[0068] Only General Shape and Alignment: In some applications, precise boundaries and high overlap are not required, only rough shapes and positions are needed. For example, when identifying lesions and estimating their sizes, the Mahalanobis distance is a suitable evaluation metric that meets the requirements for rough shapes and alignment.

[0069] Step 5: Attribute matching selection recommendations. Based on the above analysis results, an attribute matching table is constructed to compare the attributes of the evaluation indicators with the attributes of the tasks, algorithms, and datasets. This comparison process aims to obtain evaluation indicator selection recommendations that meet the current application needs.

[0070] When constructing the attribute matching table, it is first necessary to list all relevant evaluation indicators and their specific attributes, including sensitivity to different types of errors, computational complexity, and clinical relevance. Next, these attributes are compared with the specific requirements of each task, such as contour accuracy, overall alignment, recall, and volume requirements. In addition, it is necessary to analyze the applicability of each indicator in these contexts for different dataset characteristics, such as data distribution, quality, and structure. This approach not only helps to confirm which indicators can meet the needs of the current task, but also reveals possible limitations or defects. After completing the matching table, score each evaluation indicator according to its degree of match with the task requirements and dataset characteristics (in this case, a check mark indicates recommended use, a cross indicates not recommended use, and a blank indicates neutral). Finally, based on the analysis results of the matching table, suggestions for indicator selection for specific tasks are put forward to ensure that the selected indicators can fully reflect the actual needs of the task, optimize the evaluation process, and improve the reliability of the model evaluation results. See the matching table for details. Figure 3 .

[0071] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A method for selecting medical image evaluation indicators based on attribute matching, characterized in that: The following steps are involved: Step 1: Indicator attribute analysis: By standardizing the definition formula of the evaluation indicators, they are divided into seven categories according to their characteristics, including: overlap-based, volume-based, pair count-based, information theory-based, probability-based, spatial distance-based, and pixel error-based, which is convenient for subsequent analysis and comparison; Step 2: Indicator attribute assignment: Assign attributes to each indicator, including computational complexity, sensitivity to different types of errors, and relevance to clinical needs, etc., to lay the foundation for the rational selection of evaluation indicators. By assigning different attributes to the evaluation indicators, it is convenient to match them with the task algorithm and data set in the future. Step 3: Task and algorithm attribute definition: Based on the current medical image processing task and the algorithm used, define its unique attributes, which involves a detailed evaluation of the task objectives, algorithm mechanisms and their adaptability to logarithmic characteristics; Step 4: Target dataset attribute analysis: Analyze the unique distribution, quality and structural attributes of the dataset used to ensure that the selected indicators match the characteristics of the dataset and improve the applicability of the evaluation results; Step 5: Attribute matching selection recommendations: Build an attribute matching table, compare the attributes of the evaluation indicators with the attributes of the task, algorithm, and dataset, and obtain evaluation indicator selection recommendations that meet the current application requirements.

2. The method for selecting medical image evaluation indicators based on attribute matching according to claim 1, characterized in that: The step 2 specifically includes: Potential factors affecting the attribute assignment process in the assignment task include: outlier sensitivity, true negative consideration, chance adjustment, sensitivity to point location, ignoring alignment errors, recall bonus, overall shape and alignment, overlap-based, distance-based, pair count-based, information-theoretic-based, probability-based, spatial distance-based, and pixel error-based.

3. The method for selecting medical image evaluation indicators based on attribute matching according to claim 1, characterized in that: The step four specifically includes: analyzing the target area distribution of the data set, selecting evaluation indicators suitable for processing imbalance or morphological changes, evaluating the quality factors of the data set to determine whether it is necessary to adopt indicators that are more robust to noise, considering the shape and structure of the target area, selecting indicators that are sensitive to shape or can better match the volume, combining task requirements and algorithm characteristics, ensuring that the selected evaluation indicators can effectively match the characteristics of the data set, and improving the applicability and reliability of the evaluation results.

4. The method for selecting medical image evaluation indicators based on attribute matching according to claim 1, characterized in that: The step four specifically includes: first listing all evaluation indicators and their key attributes, including sensitivity to error types, computational complexity, and clinical relevance, analyzing task requirements, algorithm characteristics, and data set characteristics, clarifying the specific requirements of each task and understanding the characteristics of the algorithm and data set, constructing an attribute matching table, comparing the attributes of the evaluation indicators with the attributes of the task, algorithm, and data set, and deriving a suitability score for each indicator. Based on the matching results, recommendations for selecting evaluation indicators for specific tasks are proposed to ensure that the selected indicators can most effectively reflect task requirements and optimize the evaluation process.

Images