A region-based multi-index multi-organ medical image segmentation model evaluation system

Abstract

This invention discloses a region-based, multi-index, multi-organ medical image segmentation model evaluation system, comprising a data acquisition module, an organ region delineation module, a data preprocessing module, a medical image segmentation model training module, a medical image segmentation model testing module, and a medical image segmentation model evaluation module. This system proposes a multi-index, multi-organ medical image segmentation model evaluation method that simultaneously quantifies multi-organ segmentation results, multiple accuracy indices, and confidence estimates in a concise and unified metric. It also provides information on the model's overall performance and its clinical usability, providing a novel and intuitive standard for clinically evaluating multi-index and multi-organ medical image segmentation models.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a region-based multi-index multi-organ medical image segmentation model evaluation system. Background Art

[0002] In recent years, U-Net and its derivatives have achieved state-of-the-art results in various medical image segmentation tasks in clinical practice. However, compared to the development of these models, research on model evaluation techniques has been relatively lacking. Existing model evaluation techniques face the complexity of guiding clinical practice, and model evaluation techniques themselves are still incomplete. Evaluation techniques tailored to specific clinical application scenarios, particularly multi-organ segmentation tasks, remain elusive.

[0003] In clinical applications, the most primitive model evaluation technique involves clinicians subjectively evaluating the model's segmentation results to assess model accuracy. While this method is highly reliable and professional, it is time-consuming, increases staff workload, and is subject to subjective variability. Traditional model evaluation techniques introduce objective metrics and focus on the accuracy of model segmentation, calculating the correlation coefficient between the segmentation results and the true value labels. This method is relatively objective and can accurately reflect clinical diagnostic results. However, due to the different focuses and clinical significance of various accuracy metrics, the ranking of model evaluation results can be inconsistent, increasing the complexity of clinical evaluation models and causing confusion among clinicians when selecting models. However, solely considering the accuracy of model segmentation cannot guarantee the reliability of the model's predictions in clinical applications. Model reliability is crucial for clinical deployment. Assessing model reliability is another important technique. However, the introduction of reliability metrics inevitably leads to ambiguity between accuracy and reliability. It is common for one model to be more accurate while another model is more reliable. When deploying models clinically, doctors must consider and even make trade-offs between the accuracy and reliability of the models. Existing evaluation technologies have not yet effectively solved the problem of comprehensively measuring multiple accuracy indicators and reliability estimates in multi-organ segmentation clinical application scenarios, and there is uncertainty and lack of uniformity in the model evaluation results.

[0004] In general, the model evaluation techniques currently used in clinical practice perform well in specific medical image segmentation tasks, but they still have the following shortcomings:

[0005] 1. The method based on the subjective evaluation of doctors requires more time and energy, and there is the possibility of subjective differences, which will increase the workload of doctors and the uncertainty of the deployment model.

[0006] 2. Evaluation methods based on objective accuracy metrics can lead to inconsistent rankings across multiple accuracy metrics. This is due to the different emphases and clinical significance of each metric. This increases the complexity of the clinical evaluation model and can cause difficulties for physicians in selecting models.

[0007] 3. Evaluation methods based on reliability metrics can lead to ambiguity in model evaluation results. In clinical practice, evaluating model accuracy and reliability separately is not comprehensive and cannot provide a unified perspective for model evaluation. Physicians must make a trade-off between accuracy and reliability.

[0008] In summary, in the medical image segmentation model evaluation system, how to simultaneously consider multiple accuracy indicators and reliability metrics, reduce the uncertainty and complexity of model evaluation, and provide clinicians with a more reliable and effective basis for model selection is a key issue that needs to be solved urgently. Summary of the Invention

[0009] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology, and proposes a region-based multi-index multi-organ medical image segmentation model evaluation system. By adopting statistical methods, a new clinical model evaluation method is proposed, which uses a unified metric to simultaneously quantify multi-organ segmentation results, multiple accuracy indicators and reliability estimates, thereby providing a comprehensive and concise perspective for clinical evaluation models.

[0010] To achieve the above objectives, the present invention provides a technical solution: a region-based multi-index multi-organ medical image segmentation model evaluation system, comprising:

[0011] A data acquisition module is used to acquire an image dataset, including CT image data of the same multiple organ parts in multiple samples; and randomly divide the image dataset into a training set, a validation set, and a test set;

[0012] The organ region delineation module is used to delineate the organ region of interest in the acquired CT image data as the true value label;

[0013] The data preprocessing module is used to preprocess the collected CT image data and the outlined true value labels by cropping, format conversion and normalization to meet the input requirements of the medical image segmentation model, thereby obtaining a new data format;

[0014] The medical image segmentation model training module is used to iteratively train the medical image segmentation model using the divided training set. During the training process, the model parameters are adjusted according to the value of the loss function so that they gradually converge to the optimal value. At the same time, the model parameters are adjusted using the validation set to prevent overfitting of the model training, and finally a trained medical image segmentation model is obtained.

[0015] The medical image segmentation model testing module is used to input the test set into the trained models obtained in the medical image segmentation model training module to generate the corresponding organ segmentation results. Based on the organ segmentation results, the confidence estimate value of each organ under different models is calculated as the reliability indicator of the model prediction. The confidence estimate value is then combined with the true value label obtained by the organ region delineation module to evaluate the segmentation quality of each organ under different models and obtain the corresponding model prediction accuracy indicator.

[0016] The medical image segmentation model evaluation module is used to evaluate the advantages and disadvantages of different medical image segmentation models. It summarizes the accuracy indicators of all models generated in the medical image segmentation model test module, and uses statistical methods to autonomously generate thresholds within the model to screen whether the segmentation results of each sample organ are clinically applicable. Then, for each model to be evaluated, a ranking correlation between accuracy and confidence is established, and a clinically acceptable confidence interval is statistically calculated. The generated threshold is then used for screening. Finally, a regional value-based metric is used to uniformly quantify multiple accuracy indicators, confidence estimates, and segmentation results of multiple organs to generate a specific and concise measurement indicator, and provide information on the comprehensive performance of the model and the degree of clinical applicability.

[0017] Furthermore, the organ region delineation module manually determines the organ region of interest for each sample and delineates it according to the tissue structure characteristics in the reference image.

[0018] Furthermore, the data preprocessing module pre-processes the data according to the requirements and characteristics of the medical image segmentation model, so that the performance and generalization ability of the medical image segmentation model can be improved, thereby better solving practical problems;

[0019] For medical image segmentation models that can only process two-dimensional data, the model can only input two-dimensional CT images. The data preprocessing module performs the following operations: split the three-dimensional CT image data into two-dimensional CT images, convert them into Numpy format, and crop the images within the numerical range of -125 to 275. Then, each CT image is normalized and the processed two-dimensional CT images are randomly divided into training set, validation set and test set in a ratio of 8:1:1. The test set is saved in h5 format.

[0020] For medical image segmentation models that can directly process three-dimensional data, the model can directly input three-dimensional images. The data preprocessing module performs the following operations: normalize the CT values ​​of the entire three-dimensional CT image data from the numerical range of -1000 to 1000 to the numerical range of 0 to 1, and resample each slice to an isotropic voxel spacing of 1.0 mm during the preprocessing process, and then randomly divide the processed data into training set, validation set and test set according to the ratio of 8:1:1.

[0021] Furthermore, the medical image segmentation model training module divides the training set obtained in the data preprocessing module into n small batches and trains the medical image segmentation model batch by batch; and adopts data enhancement strategies during the training phase, including random rotation of 90°, 180° and 270°, random axial, sagittal and coronal flips and random scaling; using I CT The current batch of CT image data is represented by g, and the true value label corresponding to the multi-organ segmentation is represented by g. For the medical image segmentation model training module, the training process includes the following steps:

[0022] 1) For medical image segmentation models that require pre-training, pre-train them in the large database ImageNet and use the generated weights to initialize the model;

[0023] 2) Put I CT Input into the medical image segmentation model S for forward propagation. During the forward propagation process, the input data enters the encoder to obtain a series of feature images, and then passes through the decoder to obtain a segmentation result p with the same size as the input image; the segmentation result p is determined by the following formula:

[0024] p=S(I CT )

[0025] 3) Compare the segmentation result p output by the model with the true value label g and calculate the loss function L; where F represents the function that calculates the correlation coefficient between the segmentation result and the true value label, and L is determined by the following formula:

[0026] L=F(p,g)

[0027] 4) Based on the gradient of the loss function, the back-propagation algorithm is used to update the model parameters. During the back-propagation process, the medical image segmentation model automatically calculates the gradient of the loss function for each parameter and uses the Adam optimization algorithm to adaptively adjust the learning rate and update the parameters.

[0028] 5) Using the Adam optimization algorithm, the model parameters are updated using the calculated gradients. Each parameter update reduces the model's loss function L. After a preset number of iterations, the medical image segmentation model is tested using the validation set and the model's accuracy is calculated. If the model's test results do not meet the requirements, the model structure, loss function, and hyperparameters need to be adjusted and the model retrained.

[0029] 6) Repeat steps 2)-5) above until the model's loss function L stabilizes or reaches the preset number of iterations.

[0030] Furthermore, the medical image segmentation model testing module tests the medical image segmentation model obtained by the medical image segmentation model training module, inputs the test set into each trained model, obtains the segmentation result of each organ, and calculates the accuracy index and reliability index according to the segmentation result;

[0031] The accuracy index calculation is performed by measuring the difference between the model segmentation results and the true value labels. It is divided into region-based calculation methods and boundary-based calculation methods. Among them, the region-based metric compares the similarity between the segmentation results and the true value labels, and the indicators used are Dice coefficient and intersection-over-union (IOU).

[0032] The Dice coefficient is an indicator that compares the degree of overlap between the segmentation result and the true value label. It is calculated as follows: the segmentation result and the true value label are converted into binary images respectively, and then the two are multiplied to obtain the intersection. The number of pixels in the intersection is divided by the sum of the total number of pixels in the two binary images. The result of the division is the Dice coefficient. The Dice coefficient ranges from 0 to 1. The closer the value is to 1, the better the quality of the segmentation result.

[0033] Let p represent the segmentation result, g represent the true value label, i represent the i-th voxel value, I represent the total number of voxels, and the Dice coefficient is determined by the following expression:

[0034]

[0035] The intersection-over-union (IOU) is used to measure the precision and accuracy of the segmentation results. It is calculated as follows: the segmentation result and the true value label are converted into binary images respectively, and then the two are multiplied to obtain the intersection, the two are added and then subtracted to obtain the union, and finally the number of pixels in the intersection is divided by the number of pixels in the union to obtain the IOU. The value range of the IOU is 0 to 1. The closer the value is to 1, the better the quality of the segmentation result.

[0036] The intersection-over-union (IOU) is determined by the following expression:

[0037]

[0038] The boundary-based metric compares the boundary difference between the segmentation result and the true value label. The indicator used is the Hausdorff distance, which is calculated as follows: the shortest distance from each pixel in the true value label to the segmentation result and the shortest distance from each pixel in the segmentation result to the true value label are calculated respectively, and then the larger value of the two maximum values ​​is taken as the Hausdorff distance. The smaller the Hausdorff distance value, the smaller the boundary difference between the segmentation result and the true value label.

[0039] G' and P' represent the set of points on the ground truth label and the segmentation result surface, respectively. g' and p' represent the points in the point set, respectively. HD represents the Hausdorff distance, which is determined by the following expression:

[0040] HD(G',P')=max{max g'∈G' min p'∈P' ||g'-p'||,max p'∈P' min g'∈G' ||p'-g'||}

[0041] The medical image segmentation model testing module needs to select two or more accuracy indicators;

[0042] The reliability index of the medical image segmentation model can be obtained by direct or indirect calculation;

[0043] The direct method is to obtain the confidence index. Confidence is a measure of the reliability of the prediction results of the medical image segmentation model, that is, the degree of certainty of the model's classification results for each pixel. It is calculated by taking the maximum value of the predicted logits for each pixel, specifically the value after the Sigmoid or Softmax activation function, and then calculating the average value of all pixels in each channel of each test sample as the confidence value of each organ. Conf represents the confidence value, N represents the total number of samples, and r represents the confidence value. a Represents the probability value of the a-th sample prediction, and the confidence value is determined by the following expression:

[0044]

[0045] The indirect method is calculated by combining the accuracy index. The two indicators used are the predicted calibration error and the maximum calibration error. Both of them measure the stability of the medical image segmentation model. The calculation method is the absolute difference between the statistical accuracy index and the confidence level. The predicted calibration error is obtained by averaging the absolute differences, and the maximum calibration error is the maximum value of the absolute differences.

[0046] ECE is used to represent the prediction calibration error, N is the total number of samples, and s a Indicates the accuracy index of the a-th sample, confa represents the confidence index of the a-th sample, and the prediction calibration error is determined by the following expression:

[0047]

[0048] MCE represents the maximum calibration error, which is determined by the following expression:

[0049]

[0050] In the medical image segmentation model testing module, the confidence value output by the model itself, that is, the direct calculation method, is preferred as the reliability indicator.

[0051] Furthermore, the medical image segmentation model evaluation module evaluates the performance of the model generated by the medical image segmentation model testing module from a comprehensive perspective, as follows:

[0052] Summarize the accuracy indicators of all models generated in the medical image segmentation model testing module. For each organ segmentation result under each sample, a bootstrapping statistical method is adopted to generate thresholds for each accuracy indicator under each organ within the model, which are used to screen the sample organ segmentation results that can be used clinically.

[0053] Bootstrapping is a computer-based statistical inference method that does not rely on data being in a specific distribution. It can extract a preset number of samples from existing data and then infer sample characteristics that are more consistent with the actual distribution through statistical analysis of these samples. The core idea of ​​the bootstrapping method is to resample the existing finite samples, continuously randomly select smaller sample sets, and perform data processing on each small sample set to construct a sample distribution that is more consistent with the actual distribution for inference.

[0054] The specific implementation method of obtaining the accuracy index threshold by Bootstrapping in the medical image segmentation model evaluation module is as follows: under each accuracy index, randomly select o samples from the total test set samples according to the organ, where o ≤ the total number of samples, to form a new sample set, and repeat B times to generate B new sets; use the statistic θ to evaluate each set, and generate a total of B estimators of θ; the medical image segmentation model evaluation module sets the mean as the default value of the statistic θ; use the B estimators to construct a new distribution, sort the distribution in descending order, and select the value corresponding to a certain percentile as the threshold under the accuracy index, wherein the percentile is set to 50% by default in the medical image segmentation model evaluation module, which is the median of the distribution;

[0055] For each model, we summarize the multi-organ segmentation results, multiple accuracy indicators, confidence estimates, and thresholds for each accuracy indicator across all test set samples. We then perform ranking calculations, calculate confidence intervals, and filter and compare by threshold to generate a set of sample organs that simultaneously meet all threshold conditions. We then calculate the model's usability / comprehensiveness score, which will be used to compare the overall performance of multiple models or their clinical usability.

[0056] According to the results generated by the medical image segmentation model test module, each test set sample in a model will have a corresponding accuracy index and confidence estimate under each organ; the specific implementation of calculating the model's usability / comprehensive score in the medical image segmentation model evaluation module is: for each organ segmentation result of the sample, a one-to-one association is established between different accuracy indicators and confidence estimates, and they are sorted in descending order according to the size of the confidence value to obtain a new sorted set, and the sorted results are traversed until the end. When the jth element in the set is traversed, the confidence interval of each accuracy indicator is re-calculated for the first j elements in the set using the Bootstrapping technology, among which the clinically acceptable segmentation accuracy threshold should include a 95% confidence interval. Therefore, in the percentile selection process, the accuracy index value corresponding to the 95% percentile is selected and compared with the generated threshold. Since there are multiple different accuracy indicators, in a sample, only the organ segmentation results whose accuracy indicators simultaneously meet the threshold conditions are considered to meet the standards. The same statistical calculation is performed on all organ segmentation results to generate a set of sample organs that meet the threshold conditions in all samples. The larger the area of ​​the set, the better the overall performance of a model. The final usability / comprehensive score result is obtained by dividing the set that meets the conditions by the cumulative value of all organs to be segmented in the total sample in the form of area, and generates a value between 0 and 1. The larger the value, the better the overall performance of the model or the higher the clinical usability.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] 1. The system of the present invention targets specific clinical application scenarios of multi-index and multi-organ segmentation tasks. It adopts a method of generating thresholds within the model to be evaluated, combines multi-organ segmentation results, multiple accuracy indicators and confidence estimates to perform calculations, and screens and measures the results of each sample organ, generates a set of sample organs that meet the threshold conditions, and finally concisely and intuitively generates a usability and comprehensive score value representing the quality of the model, bringing more comprehensive, unified and novel insights to clinical model evaluation.

[0059] 2. Compared with the method based on the subjective evaluation of doctors, the system of the present invention is an objective evaluation standard that does not require human intervention and is more objective without subjective differences.

[0060] 3. Compared with the evaluation method based on objective accuracy indicators, the system of the present invention measures and unifies different accuracy indicators at the same time, eliminating the problem of inconsistent ranking results of the clinical evaluation model, reducing the complexity of the clinical evaluation model and making it more comprehensive.

[0061] 4. Compared with the evaluation method based on reliability indicators, the system of the present invention considers the accuracy and reliability of the medical image segmentation model at the same time and measures both uniformly, eliminating the ambiguity between accuracy and reliability faced in clinical evaluation models and reducing the uncertainty and complexity of the evaluation.

[0062] 5. Compared to existing medical image segmentation model evaluation methods, the proposed system is more comprehensive, objective, and quantitative, and has a wider scope of applicability, including being not limited to the neighborhood of medical image segmentation model evaluation. It also offers greater comparability, allowing the comprehensive performance of different models to be compared, thereby determining the pros and cons of each model, and more conveniently helping clinicians understand the characteristics of the models and make decisions about them. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is an architectural diagram of the system of the present invention.

[0064] Figure 2 This is the schematic diagram of the medical image segmentation model testing module.

[0065] Figure 3 This is a schematic diagram of the medical image segmentation model evaluation module.

[0066] Figure 4 This is a specific implementation diagram of the usability / comprehensiveness score calculation. DETAILED DESCRIPTION

[0067] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0068] like Figure 1 As shown, this embodiment discloses a region-based multi-index multi-organ medical image segmentation model evaluation system, including: a data acquisition module, an organ region delineation module, a data preprocessing module, a medical image segmentation model training module, a medical image segmentation model testing module and a medical image segmentation model evaluation module.

[0069] The data acquisition module is used to collect image datasets, which should include multi-organ CT image data of multiple samples obtained at the same location at the same time. In this embodiment, the samples are all abdominal multi-organ samples, and the organs selected are all abdominal organs, including the stomach, liver, kidneys, gallbladder, left and right adrenal glands, spleen, esophagus, pancreas, and portal vein. Two public datasets of different sizes, BTCV and AMOS 2022, are used for implementation. Patients are randomly divided into training sets, validation sets, and test sets with a ratio of 8:1:1.

[0070] The organ region delineation module is used to delineate the organ region of interest in the reference image. Since the evaluation system is applied to a multi-organ segmentation scenario, radiologists delineate various organ regions of the human abdomen on CT images, and the results of the organ delineation regions are used as the true value labels in this embodiment.

[0071] The data preprocessing module is used to preprocess the CT images acquired by the data acquisition module so that they meet the input requirements of the medical image segmentation model;

[0072] In this embodiment, for medical image segmentation models such as Transunet and Swin-unet that can only process two-dimensional data, the data preprocessing module performs the following operations: dividing the three-dimensional CT image data into two-dimensional CT images, converting them into "Numpy" format, and cropping the images within the numerical range of -125 to 275. Each CT image is then normalized and the processed two-dimensional CT images are randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The test set is saved in the "h5" format.

[0073] In this embodiment, for medical image segmentation models such as 3D Unet, nnU-Net, V-Net, Attention U-Net, Unetr, and Swinunetr that can directly process 3D data, the data preprocessing module normalizes the CT values ​​of the entire 3D CT image data from a range of -1000 to 1000 to a range of 0 to 1, resamples each slice to an isotropic voxel spacing of 1.0 mm during preprocessing, and then randomly partitions the processed data into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0074] The medical image segmentation model training module divides the training set into n small batches (mini batch) of data to train the medical image segmentation model batch by batch, wherein the mini batch size is set to m, and the specific value can be adjusted according to the size of the GPU available video memory. In this embodiment, m is selected as 1; during the training phase, a data enhancement strategy is adopted, including random rotation of 90°, 180° and 270°, random axial, sagittal and coronal flipping and random scaling. Using I CT The current batch of CT image data is represented by g, and the true value label corresponding to the multi-organ segmentation is represented by g. For the medical image segmentation model training module, the training process includes the following steps:

[0075] 1) For medical image segmentation models that require pre-training for Transunet and Swin-unet, pre-training is performed on the large-scale ImageNet database and the generated weights are used to initialize the model;

[0076] 2) For 3D Unet, nnU-Net, V-Net, Attention u-net, Unetr and Swin unetr models, I CT Input into the medical image segmentation model S for forward propagation. During the forward propagation process, the input data enters the encoder to obtain a series of feature images, and then passes through the decoder to obtain a segmentation result p of the same size as the input image; the segmentation result p is determined by the following formula:

[0077] p=S(I CT )

[0078] 3) Compare the segmentation result p output by the model with the true value label g and calculate the loss function L. Here, F represents the function that calculates the correlation coefficient between the segmentation result and the true value label, and L is determined by the following formula:

[0079] L=F(p,g)

[0080] 4) Based on the gradient of the loss function, the model parameters are updated using the backpropagation algorithm. During the backpropagation process, the medical image segmentation model automatically calculates the gradient of the loss function with respect to each parameter and, in this embodiment, uses the Adam optimization algorithm to adaptively adjust the learning rate and update the parameters.

[0081] 5) Using the Adam optimization algorithm, update the model parameters using the calculated gradients. Each parameter update will reduce the model's loss function L to a certain extent. After a certain number of iterations, test the medical image segmentation model using the validation set and calculate the model's accuracy. If the model's test results do not meet the requirements, adjust the model structure, loss function, and hyperparameters and retrain the model.

[0082] 6) Repeat steps 2)-5) above until the model's loss function L stabilizes or reaches the preset number of iterations.

[0083] The medical image segmentation model testing module inputs the CT image data in the test set into each trained model obtained in the medical image segmentation model training module, generates corresponding organ segmentation results, and then calculates the accuracy index and reliability index respectively according to the segmentation results;

[0084] like Figure 2 As shown in Figure 1, the CT image data in the test set is input into the image segmentation model S to obtain the segmentation result p. Combined with the true value labels g manually annotated by doctors obtained by the organ region delineation module, the accuracy index is calculated voxel by voxel, with i representing the i-th voxel value and I representing the total number of voxels.

[0085] The medical image segmentation model test module needs to select two or more accuracy indicators. In this embodiment, the Dice coefficient and Hausdorff distance are mainly used for the accuracy indicators. Among them, the Dice coefficient is an accuracy indicator that measures the degree of overlap and similarity between the segmentation result and the true value label. It is calculated as follows: the segmentation result and the true value label are converted into binary images respectively, and then the two are multiplied to obtain the intersection, and then the number of pixels in the intersection is divided by the sum of the total number of pixels in the two binary images. The result of the division is used as the Dice coefficient. The value range of the Dice coefficient is 0 to 1. The closer the value is to 1, the better the quality of the segmentation result. The Dice coefficient is determined by the following expression:

[0086]

[0087] The Hausdorff distance compares the boundary differences between the segmentation result and the true value label. It is calculated by calculating the shortest distance from each pixel in the true value label to the segmentation result and the shortest distance from each pixel in the segmentation result to the true value label. The larger of the two maximum values ​​is taken as the Hausdorff distance. A smaller Hausdorff distance value indicates a smaller boundary difference between the segmentation result and the true value label.

[0088] G' and P' represent the set of points on the ground truth label and the segmentation result surface, respectively. g' and p' represent the points in the point set, respectively. HD represents the Hausdorff distance, which is determined by the following expression:

[0089] HD(G',P')=max{max g'∈G' min p'∈P' ||g'-p'||,max p'∈P' min g'∈G' ||p'-g'||}

[0090] In this embodiment, different medical image segmentation models are tested on two abdominal multi-organ public datasets, BTCV and AMOS 2022, and the results of calculating the Dice coefficient and Hausdorff distance are as follows:

[0091] Table 1 Test results of average Dice coefficient and Hausdorff distance of each model in two datasets

[0092]

[0093] As shown in the table above, based on the average Dice coefficient (↑), the Attention U-Net and Swin U-Net models achieved average Dice coefficients of 0.832 and 0.838, respectively, on the BTCV dataset. In the AMOS 2022 dataset, the average Dice coefficients for the nn U-Net and Swin U-Net were 0.87 and 0.876, respectively. While Swin U-Net outperformed other models in both datasets, the gap in scores was smaller compared to the next-best model. Notably, the worst-performing models in terms of the Dice coefficient were Swin U-Net and U-Net, with the rankings varying across datasets. Regarding the average Hausdorff distance (↓), Transunet achieved the lowest score in the BTCV dataset, representing the best performance, with a significant gap compared to other models. In the AMOS 2022 dataset, Swin U-Net ranked first, slightly lower than Transunet by 0.03. However, in both datasets, U-Net demonstrated the worst boundary similarity. However, the different rankings between Hausdorff distance and Dice coefficient can conflict with each other. This ambiguity in the results of multi-metric evaluations can lead to different final rankings of models, which makes it more complicated for doctors and experts to deploy models in clinical practice.

[0094] As for the reliability index, in this embodiment, the confidence value is directly obtained. The confidence mainly measures the reliability of the prediction results of the medical image segmentation model, that is, the degree of certainty of the model's classification results for each pixel. It is calculated by taking the maximum value of the predicted logits for each pixel, usually the value after the Sigmoid or Softmax activation function, and then calculating the average value of all pixels in each channel of each test sample as the confidence value of each organ. Conf represents the confidence value, N represents the total number of samples, and r represents the confidence value of each organ. a Represents the probability value of the a-th sample prediction, and the confidence value is determined by the following expression:

[0095]

[0096] In addition, the reliability index can also be obtained indirectly. The indirect method is calculated by combining the accuracy index. The two indicators used are the predicted calibration error and the maximum calibration error. The two indicators balance the stability of the medical image segmentation model. The calculation method is the absolute difference between the statistical accuracy index and the confidence level. The predicted calibration error is obtained by averaging the absolute differences, and the maximum calibration error is the maximum value of the absolute differences.

[0097] ECE is used to represent the prediction calibration error, N is the total number of samples, and s a Indicates the accuracy index of the a-th sample, conf a represents the confidence index of the a-th sample, and the prediction calibration error is determined by the following expression:

[0098]

[0099] MCE represents the maximum calibration error, which is determined by the following expression:

[0100]

[0101] In this embodiment, different medical image segmentation models are tested on two abdominal multi-organ public datasets, BTCV and AMOS 2022, and the results of calculating the predicted calibration error ECE and the maximum calibration error MCE are as follows:

[0102] Table 2 Test results of prediction calibration error ECE and maximum calibration error MCE of each model in two datasets

[0103]

[0104]

[0105] As shown in the table above, from the perspectives of ECE (↓) and MCE (↓), Swin unetr demonstrated the best calibration in both abdominal multi-organ datasets. Conversely, the U-Net model achieved the highest scores in both ECE and MCE, indicating that this model performed the worst in terms of reliability. However, it is important to note that the ranking of other models will vary depending on whether ECE or MCE is considered, and will also be affected by different datasets. Furthermore, there is a phenomenon where, for example, the V-Net model outperforms Unetr in predictive accuracy but performs worse than Unetr in terms of reliability. This creates difficulties in selecting models for clinical deployment, as a choice must be made between model accuracy and reliability, making it difficult to directly compare models.

[0106] The medical image segmentation model evaluation module is used to evaluate the performance of different medical image segmentation models. It aggregates the accuracy metrics of all models in the model testing module and uses statistical methods to autonomously generate thresholds within the models to screen whether each sample organ segmentation result is clinically applicable. For each model to be evaluated, a ranking correlation between accuracy and confidence is established, and clinically acceptable confidence intervals are calculated. The generated thresholds are then used for screening. A unified metric based on regional values ​​is used to quantify multiple accuracy metrics, confidence estimates, and segmentation results for multiple organs, generating a specific and concise measurement indicator that provides information on the overall performance of the model and its degree of clinical applicability.

[0107] like Figure 3 As shown, in this embodiment, the Dice coefficient and Hausdorff distance metrics of all models generated in the medical image segmentation model testing module are summarized. For each organ segmentation result in each sample, a bootstrapping statistical method is used to generate thresholds for each accuracy metric within the model for each organ. This is used to screen clinically applicable sample organ segmentation results. Specifically, using the Dice coefficient or Hausdorff distance metric, a new sample set (o), where o ≤ the total number of samples, is randomly selected from the total test set, forming a new sample set. This is repeated B times to generate B new sets. Each set is evaluated using the statistic θ, generating a total of B estimates of θ. For general purposes, this embodiment sets the "mean" as the default value for the statistic θ. A new distribution is constructed using these B estimates, which are sorted in descending order and the value corresponding to a percentile is selected as the threshold for the accuracy metric. In this embodiment, the percentile is set to 50% by default, which is the median of the distribution.

[0108] For each model trained in the medical image segmentation model training module of this embodiment, the multi-organ segmentation results, Dice coefficients, Hausdorff distances, confidence estimates, and generated thresholds for all test set samples are aggregated. Sorting calculations are performed based on the confidence values, confidence intervals are calculated, and the samples are screened and compared based on the thresholds. A set of sample organs that simultaneously meet all threshold conditions is generated, and a usability / comprehensiveness score is further calculated for each model. The results of this score will be used as a basis for comparing the comprehensive performance of multiple models or their clinical usability.

[0109] like Figure 4As shown, in this embodiment, the specific implementation of calculating the usability / comprehensiveness score of each model is as follows: for each organ segmentation result of the sample, a one-to-one association is established between different Dice coefficients and Hausdorff distance indicators and confidence estimates. The confidence values ​​are sorted in descending order to obtain a new sorted set. The sorted results are also traversed according to different organs until the end. When the jth element in the set is traversed, the confidence intervals of the Dice coefficient and Hausdorff distance are recalculated for the first j elements in the set using the Bootstrapping method. The clinically acceptable segmentation accuracy threshold should include a 95% confidence interval. Therefore, in the percentile selection process after descending sorting, the accuracy indicator corresponding to the 95% percentile is selected and compared with the generated threshold. Since there are multiple different accuracy indicators, in a sample, only the organ segmentation results whose accuracy indicators meet the threshold conditions at the same time are considered to be up to standard. The same statistical calculation is performed on all organ segmentation results to generate a set of sample organs that meet the threshold conditions in all samples, as shown in FIG. Figure 4 As shown in the dark area, the larger the area of ​​the set, the better the overall performance of the model. The final model usability / comprehensiveness score is obtained by dividing the set that meets the conditions by the cumulative value of all organs to be segmented in the total sample in the form of "area domain" and generating a value between 0 and 1. The larger the value, the better the overall performance of the model or the higher its clinical usability.

[0110] In this example, different medical image segmentation models were tested on two public abdominal multi-organ datasets, BTCV and AMOS 2022, and the final usability / comprehensiveness scores of the models were calculated. The results are as follows:

[0111] Table 3 Usability / comprehensiveness score results of each model in two datasets

[0112]

[0113] As shown in the table above, the medical image segmentation model evaluation module ultimately generates a very intuitive number. The Swin unetr model has the best usability and comprehensiveness, while U-Net has the lowest score, making it the worst-performing model. The rankings of the remaining models may vary across datasets, but duplicate scores are rare. It effectively resolves ambiguity between multiple metrics and between accuracy and reliability, successfully using a unified metric to simultaneously quantify multiple accuracy metrics, confidence estimates, and multi-organ segmentation results. This provides comprehensive performance insights that accuracy, confidence estimates, and calibration error metrics cannot fully capture. Finally, a concise score is presented, reducing the complexity and uncertainty of model comparison and deployment in the clinic.

[0114] The above embodiments are preferred implementations of the present invention, but the implementations of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A region-based multi-index multi-organ medical image segmentation model evaluation system, characterized by: include: A data acquisition module is used to acquire an image dataset, including CT image data of the same multiple organ parts in multiple samples; and randomly divide the image dataset into a training set, a validation set, and a test set; The organ region delineation module is used to delineate the organ region of interest in the acquired CT image data as the true value label; The data preprocessing module is used to preprocess the collected CT image data and the outlined true value labels by cropping, format conversion and normalization to meet the input requirements of the medical image segmentation model, thereby obtaining a new data format; The medical image segmentation model training module is used to iteratively train the medical image segmentation model using the divided training set. During the training process, the model parameters are adjusted according to the value of the loss function so that they gradually converge to the optimal value. At the same time, the model parameters are adjusted using the validation set to prevent overfitting of the model training, and finally a trained medical image segmentation model is obtained. The medical image segmentation model testing module is used to input the test set into the trained models obtained in the medical image segmentation model training module to generate the corresponding organ segmentation results. Based on the organ segmentation results, the confidence estimate value of each organ under different models is calculated as the reliability indicator of the model prediction. The confidence estimate value is then combined with the true value label obtained by the organ region delineation module to evaluate the segmentation quality of each organ under different models and obtain the corresponding model prediction accuracy indicator. The medical image segmentation model evaluation module is used to evaluate the advantages and disadvantages of different medical image segmentation models. It summarizes the accuracy indicators of all models generated in the medical image segmentation model test module, and uses statistical methods to autonomously generate thresholds within the model to screen whether the segmentation results of each sample organ are clinically applicable. Then, for each model to be evaluated, a ranking correlation between accuracy and confidence is established, and a clinically acceptable confidence interval is statistically calculated. The generated threshold is then used for screening. Finally, a regional value-based metric is used to uniformly quantify multiple accuracy indicators, confidence estimates, and segmentation results of multiple organs to generate a specific and concise measurement indicator, and provide information on the comprehensive performance of the model and the degree of clinical applicability.

2. The region-based multi-index multi-organ medical image segmentation model evaluation system according to claim 1, characterized in that: The organ region delineation module manually determines the organ region of interest for each sample and delineates it according to the tissue structure characteristics in the reference image.

3. The region-based multi-index multi-organ medical image segmentation model evaluation system according to claim 2, characterized in that: The data preprocessing module pre-processes the data according to the requirements and characteristics of the medical image segmentation model, so that the performance and generalization ability of the medical image segmentation model can be improved, thereby better solving practical problems; For medical image segmentation models that can only process two-dimensional data, the model can only input two-dimensional CT images. The data preprocessing module performs the following operations: split the three-dimensional CT image data into two-dimensional CT images, convert them into Numpy format, and crop the images within the numerical range of -125 to 275. Then, each CT image is normalized and the processed two-dimensional CT images are randomly divided into training set, validation set and test set in a ratio of 8:1:

1. The test set is saved in h5 format. For medical image segmentation models that can directly process three-dimensional data, the model can directly input three-dimensional images. The data preprocessing module performs the following operations: normalize the CT values ​​of the entire three-dimensional CT image data from the numerical range of -1000 to 1000 to the numerical range of 0 to 1, and resample each slice to an isotropic voxel spacing of 1.0 mm during the preprocessing process, and then randomly divide the processed data into training set, validation set and test set according to the ratio of 8:1:

1.

4. The region-based multi-index multi-organ medical image segmentation model evaluation system according to claim 3, characterized in that: The medical image segmentation model training module divides the training set obtained in the data preprocessing module into n small batches and trains the medical image segmentation model batch by batch; and adopts data enhancement strategies during the training phase, including random rotation of 90°, 180° and 270°, random axial, sagittal and coronal flips and random scaling; using I CT The current batch of CT image data is represented by g, and the true value label corresponding to the multi-organ segmentation is represented by g. For the medical image segmentation model training module, the training process includes the following steps: 1) For medical image segmentation models that require pre-training, pre-train them in the large database ImageNet and use the generated weights to initialize the model; 2) Will I CT Input into the medical image segmentation model S for forward propagation. During the forward propagation process, the input data enters the encoder to obtain a series of feature images, and then passes through the decoder to obtain a segmentation result p with the same size as the input image; the segmentation result p is determined by the following formula: p=S(I CT ) 3) Compare the segmentation result p output by the model with the true value label g and calculate the loss function L; where F represents the function that calculates the correlation coefficient between the segmentation result and the true value label, and L is determined by the following formula: L=F(p,g) 4) Based on the gradient of the loss function, the back-propagation algorithm is used to update the model parameters. During the back-propagation process, the medical image segmentation model automatically calculates the gradient of the loss function for each parameter and uses the Adam optimization algorithm to adaptively adjust the learning rate and update the parameters. 5) Using the Adam optimization algorithm, the model parameters are updated using the calculated gradients. Each parameter update reduces the model's loss function L. After a preset number of iterations, the medical image segmentation model is tested using the validation set and the model's accuracy is calculated. If the model's test results do not meet the requirements, the model structure, loss function, and hyperparameters need to be adjusted and the model retrained. 6) Repeat steps 2)-5) above until the model's loss function L stabilizes or reaches the preset number of iterations.

5. The region-based multi-index multi-organ medical image segmentation model evaluation system according to claim 4, characterized in that: The medical image segmentation model testing module tests the medical image segmentation model obtained by the medical image segmentation model training module, inputs the test set into each trained model, obtains the segmentation result of each organ, and calculates the accuracy index and reliability index according to the segmentation result; The accuracy index calculation is performed by measuring the difference between the model segmentation results and the true value labels. It is divided into region-based calculation methods and boundary-based calculation methods. Among them, the region-based metric compares the similarity between the segmentation results and the true value labels, and the indicators used are Dice coefficient and intersection-over-union (IOU). The Dice coefficient is an indicator that compares the degree of overlap between the segmentation result and the true value label. It is calculated as follows: the segmentation result and the true value label are converted into binary images respectively, and then the two are multiplied to obtain the intersection. The number of pixels in the intersection is divided by the sum of the total number of pixels in the two binary images. The result of the division is the Dice coefficient. The Dice coefficient ranges from 0 to 1. The closer the value is to 1, the better the quality of the segmentation result. Let p represent the segmentation result, g represent the true value label, i represent the i-th voxel value, I represent the total number of voxels, and the Dice coefficient is determined by the following expression: The intersection-over-union (IOU) is used to measure the precision and accuracy of the segmentation results. It is calculated as follows: the segmentation result and the true value label are converted into binary images respectively, and then the two are multiplied to obtain the intersection, the two are added and then subtracted to obtain the union, and finally the number of pixels in the intersection is divided by the number of pixels in the union to obtain the IOU. The value range of the IOU is 0 to 1. The closer the value is to 1, the better the quality of the segmentation result. The intersection-over-union (IOU) is determined by the following expression: The boundary-based metric compares the boundary difference between the segmentation result and the true value label. The indicator used is the Hausdorff distance, which is calculated as follows: the shortest distance from each pixel in the true value label to the segmentation result and the shortest distance from each pixel in the segmentation result to the true value label are calculated respectively, and then the larger value of the two maximum values ​​is taken as the Hausdorff distance. The smaller the Hausdorff distance value, the smaller the boundary difference between the segmentation result and the true value label. G' and P' represent the set of points on the ground truth label and the segmentation result surface, respectively. g' and p' represent the points in the point set, respectively. HD represents the Hausdorff distance, which is determined by the following expression: HD(G',P')=max{max g'∈G' min p'∈P' ||g'-p'||,max p'∈P' min g'∈G' ||p'-g'||} The medical image segmentation model testing module needs to select two or more accuracy indicators; The reliability index of the medical image segmentation model can be obtained by direct or indirect calculation; The direct method is to obtain the confidence index. Confidence is a measure of the reliability of the prediction results of the medical image segmentation model, that is, the degree of certainty of the model's classification results for each pixel. It is calculated by taking the maximum value of the predicted logits for each pixel, specifically the value after the Sigmoid or Softmax activation function, and then calculating the average value of all pixels in each channel of each test sample as the confidence value of each organ. Conf represents the confidence value, N represents the total number of samples, and r represents the confidence value. a Represents the probability value of the a-th sample prediction, and the confidence value is determined by the following expression: The indirect method is calculated by combining the accuracy index. The two indicators used are the predicted calibration error and the maximum calibration error. Both of them measure the stability of the medical image segmentation model. The calculation method is the absolute difference between the statistical accuracy index and the confidence level. The predicted calibration error is obtained by averaging the absolute differences, and the maximum calibration error is the maximum value of the absolute differences. ECE is used to represent the prediction calibration error, N is the total number of samples, and s a Indicates the accuracy index of the a-th sample, conf a represents the confidence index of the a-th sample, and the prediction calibration error is determined by the following expression: MCE represents the maximum calibration error, which is determined by the following expression: In the medical image segmentation model testing module, the confidence value output by the model itself, that is, the direct calculation method, is preferred as the reliability indicator.

6. The region-based multi-index multi-organ medical image segmentation model evaluation system according to claim 5, characterized in that: The medical image segmentation model evaluation module evaluates the performance of the model generated by the medical image segmentation model testing module from a comprehensive perspective, as follows: Summarize the accuracy indicators of all models generated in the medical image segmentation model testing module. For each organ segmentation result under each sample, a bootstrapping statistical method is adopted to generate thresholds for each accuracy indicator under each organ within the model, which are used to screen the sample organ segmentation results that can be used clinically. Bootstrapping is a computer-based statistical inference method that does not rely on data being in a specific distribution. It can extract a preset number of samples from existing data and then infer sample characteristics that are more consistent with the actual distribution through statistical analysis of these samples. The core idea of ​​the bootstrapping method is to resample the existing finite samples, continuously randomly select smaller sample sets, and perform data processing on each small sample set to construct a sample distribution that is more consistent with the actual distribution for inference. The specific implementation method of obtaining the accuracy index threshold by Bootstrapping in the medical image segmentation model evaluation module is as follows: under each accuracy index, randomly select o samples from the total test set samples according to the organ, where o ≤ the total number of samples, to form a new sample set, and repeat B times to generate B new sets; use the statistic θ to evaluate each set, and generate a total of B estimators of θ; the medical image segmentation model evaluation module sets the mean as the default value of the statistic θ; use the B estimators to construct a new distribution, sort the distribution in descending order, and select the value corresponding to a certain percentile as the threshold under the accuracy index, wherein the percentile is set to 50% by default in the medical image segmentation model evaluation module, which is the median of the distribution; For each model, we summarize the multi-organ segmentation results, multiple accuracy indicators, confidence estimates, and thresholds for each accuracy indicator across all test set samples. We then perform ranking calculations, calculate confidence intervals, and filter and compare by threshold to generate a set of sample organs that simultaneously meet all threshold conditions. We then calculate the model's usability / comprehensiveness score, which will be used to compare the overall performance of multiple models or their clinical usability. According to the results generated by the medical image segmentation model test module, each test set sample in a model will have a corresponding accuracy index and confidence estimate under each organ; the specific implementation of calculating the model's usability / comprehensive score in the medical image segmentation model evaluation module is: for each organ segmentation result of the sample, a one-to-one association is established between different accuracy indicators and confidence estimates, and they are sorted in descending order according to the size of the confidence value to obtain a new sorted set, and the sorted results are traversed until the end. When the jth element in the set is traversed, the confidence interval of each accuracy indicator is re-calculated for the first j elements in the set using the Bootstrapping technology, among which the clinically acceptable segmentation accuracy threshold should include a 95% confidence interval. Therefore, in the percentile selection process, the accuracy index value corresponding to the 95% percentile is selected and compared with the generated threshold. Since there are multiple different accuracy indicators, in a sample, only the organ segmentation results whose accuracy indicators simultaneously meet the threshold conditions are considered to meet the standards. The same statistical calculation is performed on all organ segmentation results to generate a set of sample organs that meet the threshold conditions in all samples. The larger the area of ​​the set, the better the overall performance of a model. The final usability / comprehensive score result is obtained by dividing the set that meets the conditions by the cumulative value of all organs to be segmented in the total sample in the form of area, and generates a value between 0 and 1. The larger the value, the better the overall performance of the model or the higher the clinical usability.

Images