Segmentation model evaluation method and device, electronic equipment, storage medium and program

Abstract

The invention provides a segmentation model evaluation method and device, electronic equipment, a storage medium and a program. The method comprises the steps that multiple groups of test images and corresponding actual segmentation results are acquired, each test image comprises multiple objects, and the actual segmentation results comprise respective actual object segmentation results of the multiple objects; predicting test segmentation results of a plurality of objects in each test image by using the segmentation model; on the basis of each test object segmentation result and the corresponding actual object segmentation result, determining a performance index of the segmentation model, evaluating the morphological consistency of the target object and evaluating the accuracy degree of the state category of the target object; and the prediction capability of the segmentation model is evaluated based on the performance index, the morphological consistency evaluation of the target object and the accuracy degree. According to the embodiment, the segmentation model can be accurately evaluated, so that the proper segmentation model can be selected in different scenes.

Description

Technical Field

[0001] The embodiments of this application relate to the field of information processing technology, and in particular to a method, apparatus, electronic device, storage medium and program for evaluating a segmentation model. Background Technology

[0002] For patients with aneurysms, it is necessary to detect and monitor the patient's condition by using angiography images to show the aneurysm and the parent artery. In this process, it is necessary to segment the aneurysm and the parent artery from the angiography images. Whether the aneurysm and the parent artery can be accurately segmented will affect the detection and monitoring of the condition.

[0003] Aneurysms and tumor-bearing arteries can usually be segmented from angiographic images using segmentation models (e.g., deep learning models). However, different segmentation models have different characteristics, resulting in varying segmentation capabilities for angiographic images.

[0004] Therefore, for different segmentation models, the segmentation performance can usually be evaluated using a single indicator such as intersection-over-union ratio or similarity coefficient. However, evaluating a segmentation model using a single indicator is difficult to comprehensively and accurately assess. Even if multiple indicators are used for evaluation, it still only assesses the segmentation model from the perspective of its own parameters, ignoring the clinical relevance of the segmentation results, and thus cannot guarantee that the evaluation results can comprehensively evaluate the segmentation model. Summary of the Invention

[0005] In view of this, embodiments of the present disclosure provide a method, apparatus, electronic device, storage medium, and program for evaluating a segmentation model.

[0006] In a first aspect, embodiments of this disclosure provide a method for evaluating a segmentation model, the method comprising:

[0007] Obtain multiple sets of test images and the actual segmentation results corresponding to each set of test images. Each test image includes multiple objects, and the corresponding actual segmentation results include the actual object segmentation results of each of the multiple objects.

[0008] The trained segmentation model is used to predict the test segmentation results of multiple objects in each test image. Each test segmentation result includes the test object segmentation result for each object.

[0009] The performance metrics of the segmentation model are determined based on the differences between the segmentation results of each test object and the corresponding segmentation results of the actual object.

[0010] Determine the morphological consistency evaluation between the test object segmentation result of the target object in each object and the corresponding actual object segmentation result;

[0011] Predict the state category of the target object using the test object segmentation results, and evaluate the accuracy of the state category using the corresponding actual object segmentation results;

[0012] The predictive ability of the segmentation model is evaluated based on performance indicators, the morphological consistency of the target object, and the accuracy.

[0013] In some optional implementations, multiple sets of test images and the actual segmentation results corresponding to each set of test images are obtained, including:

[0014] Accordingly, at least two angiographic image sequences are acquired, each angiographic image sequence being acquired from at least two aneurysm patients before and / or after surgery, and each aneurysm patient undergoing at least two surgical procedures;

[0015] Select the image to be labeled from each angiography image sequence to label the actual segmentation results;

[0016] In each angiography image sequence, the remaining multiple angiography images other than the image to be labeled are determined as a group of test images. The actual segmentation result corresponding to each group of test images is the actual segmentation result of the image to be labeled selected from the same angiography image sequence.

[0017] In some alternative implementations, the multiple objects include an aneurysm, a tumor-bearing artery, and a background; the actual object segmentation results for each of the multiple objects include an aneurysm mask, a tumor-bearing artery mask, and a background mask;

[0018] Accordingly, the trained segmentation model is used to predict the test segmentation results for multiple objects in each test image, including:

[0019] The trained segmentation model was used to predict the aneurysm mask, the aneurysm-bearing artery mask, and the background mask in each angiography image.

[0020] In some alternative implementations, performance metrics include at least one of the following: intersection-union ratio, similarity coefficient, accuracy, and harmonic mean of the segmentation model;

[0021] Accordingly, the performance metrics of the segmentation model are determined based on the differences between the segmentation results of each test object and the corresponding segmentation results of the actual objects, including:

[0022] The intersection-union ratio of the segmentation model is determined by the degree of overlap between the segmentation result of each test object and the corresponding segmentation result of the actual object.

[0023] The similarity coefficient of the segmentation model is determined by the similarity between the segmentation result of each test object and the segmentation result of the corresponding actual object.

[0024] The accuracy of the segmentation model is determined by the probability that the segmentation result of each test object matches the segmentation result of the corresponding actual object.

[0025] The harmonic mean of the segmentation model is determined using precision and the corresponding recall.

[0026] In some optional implementations, the target object includes an aneurysm; the test object segmentation result of the target object includes the aneurysm mask of the aneurysm in each test image; the corresponding actual object segmentation result includes the aneurysm-annotated region in the corresponding actual segmentation result;

[0027] Accordingly, the morphological consistency evaluation between the test object segmentation results and the corresponding actual object segmentation results of the target objects in each object is determined, including:

[0028] Determine the various morphological data of each aneurysm mask and determine the various morphological parameters of the corresponding aneurysm annotation area;

[0029] Each type of morphological data in each aneurysm mask and corresponding aneurysm annotation area is used to construct a corresponding deviation distribution feature map, and the corresponding correlation coefficient is determined.

[0030] The morphological consistency evaluation is determined by using the deviation distribution characteristic map and correlation coefficient of each type of morphological data.

[0031] In some alternative implementations, the status categories include complete occlusion and incomplete occlusion of the aneurysm.

[0032] Accordingly, the target object's state category is predicted using the test object segmentation results, and the accuracy of the state category is evaluated using the corresponding actual object segmentation results, including:

[0033] The aneurysm mask of each angiography image in each angiography image sequence is used to generate the corresponding time density curve, which represents the intensity change of the contrast agent in the corresponding angiography image sequence.

[0034] Multiple angiography parameters and imaging indices were extracted from each time-density curve.

[0035] Using multiple angiography parameters corresponding to each time density curve, the aneurysm in the corresponding angiography image sequence is predicted to be either completely occluded or partially occluded.

[0036] The accuracy of assessing the complete or incomplete occlusion status of aneurysms in the corresponding angiographic image sequence is evaluated based on the complete or incomplete occlusion status of the aneurysm in each aneurysm annotation region.

[0037] Secondly, embodiments of this disclosure provide an evaluation device for a segmentation model, the device comprising: an acquisition module, a prediction module, a performance index determination module, a morphological consistency module, a state category evaluation module, and a model evaluation module;

[0038] The acquisition module is configured to acquire multiple sets of test images and the actual segmentation results corresponding to each set of test images. Each test image includes multiple objects, and the corresponding actual segmentation results include the actual object segmentation results of each of the multiple objects.

[0039] The prediction module is configured to use the trained segmentation model to predict the test segmentation results of multiple objects in each test image, and each test segmentation result includes the test object segmentation result for each object.

[0040] The performance metric determination module is configured to determine the performance metrics of the segmentation model based on the difference between the segmentation results of each test object and the corresponding actual object segmentation results.

[0041] The morphological consistency determination module is configured to determine the morphological consistency evaluation between the test object segmentation result of the target object and the corresponding actual object segmentation result in each object.

[0042] The state category evaluation module is configured to predict the state category of the target object using the test object segmentation results of the target object, and evaluate the accuracy of the state category using the corresponding actual object segmentation results.

[0043] The model evaluation module is configured to evaluate the predictive ability of the segmentation model based on performance metrics, morphological consistency of the target object, and accuracy.

[0044] In some alternative implementations, the acquisition module is further configured to:

[0045] Acquire at least two angiographic image sequences, each angiographic image sequence being acquired from at least two aneurysm patients before and / or after surgery, and each aneurysm patient having at least two surgical procedures;

[0046] Select the image to be labeled from each angiography image sequence to label the actual segmentation results;

[0047] In each angiography image sequence, the remaining multiple angiography images other than the image to be labeled are determined as a group of test images. The actual segmentation result corresponding to each group of test images is the actual segmentation result of the image to be labeled selected from the same angiography image sequence.

[0048] In some alternative implementations, the multiple objects include an aneurysm, a tumor-bearing artery, and a background; the actual object segmentation results for each of the multiple objects include an aneurysm mask, a tumor-bearing artery mask, and a background mask;

[0049] Accordingly, the prediction module is further configured as follows:

[0050] The trained segmentation model was used to predict the aneurysm mask, the aneurysm-bearing artery mask, and the background mask in each angiography image.

[0051] In some alternative implementations, performance metrics include at least one of the following: intersection-union ratio, similarity coefficient, accuracy, and harmonic mean of the segmentation model;

[0052] Accordingly, the performance indicator determination module is further configured as follows:

[0053] The intersection-union ratio of the segmentation model is determined by the degree of overlap between the segmentation result of each test object and the corresponding segmentation result of the actual object.

[0054] The similarity coefficient of the segmentation model is determined by the similarity between the segmentation result of each test object and the segmentation result of the corresponding actual object.

[0055] The accuracy of the segmentation model is determined by the probability that the segmentation result of each test object matches the segmentation result of the corresponding actual object.

[0056] The harmonic mean of the segmentation model is determined using precision and the corresponding recall.

[0057] In some optional implementations, the target object includes an aneurysm; the test object segmentation result of the target object includes the aneurysm mask of the aneurysm in each test image; the corresponding actual object segmentation result includes the aneurysm-annotated region in the corresponding actual segmentation result;

[0058] Accordingly, the morphological consistency determination module is further configured as follows:

[0059] Determine the various morphological data of each aneurysm mask and determine the various morphological parameters of the corresponding aneurysm annotation area;

[0060] Each type of morphological data in each aneurysm mask and corresponding aneurysm annotation area is used to construct a corresponding deviation distribution feature map, and the corresponding correlation coefficient is determined.

[0061] The morphological consistency evaluation is determined by using the deviation distribution characteristic map and correlation coefficient of each type of morphological data.

[0062] In some alternative implementations, the status categories include complete occlusion and incomplete occlusion of the aneurysm.

[0063] Accordingly, the state category evaluation module is further configured as follows:

[0064] The aneurysm mask of each angiography image in each angiography image sequence is used to generate the corresponding time density curve, which represents the intensity change of the contrast agent in the corresponding angiography image sequence.

[0065] Multiple angiography parameters and imaging indices were extracted from each time-density curve.

[0066] Using multiple angiography parameters corresponding to each time density curve, the aneurysm in the corresponding angiography image sequence is predicted to be either completely occluded or partially occluded.

[0067] The accuracy of assessing the complete or incomplete occlusion status of aneurysms in the corresponding angiographic image sequence is evaluated based on the complete or incomplete occlusion status of the aneurysm in each aneurysm annotation region.

[0068] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0069] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any implementation of the first aspect.

[0070] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in any of the implementations of the first aspect.

[0071] To address the problem of the inability to comprehensively evaluate segmentation models, the embodiments of this disclosure provide a segmentation model evaluation method, apparatus, electronic device, storage medium, and program. These methods determine performance indicators based on the difference between the segmentation results of the test object and the segmentation results of the actual object, quantifying the basic capabilities of the segmentation model from the perspective of segmentation accuracy. Furthermore, based on the morphological consistency of the target object, the morphological matching degree between the segmentation results of the test object and the actual object is verified. Finally, through state category prediction and accuracy evaluation, the segmentation results of the test object are correlated with practical application value. This achieves a comprehensive evaluation of the segmentation model from performance-related technical parameters to its clinical relevance, avoiding the problem of relying solely on performance parameters while ignoring practical application effectiveness. This ensures that the evaluation results truly reflect the comprehensive performance of the segmentation model.

[0072] Therefore, by fusing multi-dimensional evaluation results, the applicability of segmentation models in different application scenarios can be determined based on a comprehensive consideration of performance indicators, morphological consistency evaluation, and the accuracy of state categories. For example, in scenarios where priority must be given to ensuring the matching of segmentation accuracy with clinical morphology, or in scenarios where the effectiveness of segmentation results in predicting the state of the target object must also be considered, this method can be used to clarify the advantages and disadvantages of the model, thereby accurately selecting a segmentation model that suits the needs of the scenario and improving the relevance and reliability of the segmentation model application. Attached Figure Description

[0073] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0074] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure may be applied;

[0075] Figure 2A A flowchart illustrating an embodiment of the evaluation method for the segmentation model according to this disclosure;

[0076] Figure 2B This is a flowchart illustrating an embodiment of the decomposition process 2010 according to one embodiment of step 201 of this disclosure;

[0077] Figure 2C This is a breakdown flowchart of an embodiment of the breakdown process 2030 according to step 203 of this disclosure;

[0078] Figure 2D This is a flowchart illustrating an embodiment of the decomposition process 2040 according to one embodiment of step 204 of this disclosure;

[0079] Figure 2E This is a flowchart illustrating an embodiment of the decomposition process 2050 according to step 205 of this disclosure;

[0080] Figure 3 This is a schematic diagram of the structure of an evaluation apparatus according to an embodiment of the segmentation model of this disclosure;

[0081] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0082] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0083] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0084] Figure 1 An exemplary system architecture 100 is shown, illustrating embodiments of evaluation methods, apparatuses, electronic devices, storage media, and programs to which the segmentation model of this disclosure can be applied.

[0085] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0086] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as deep learning models, neural network applications, natural language processing applications, large model applications, speech recognition applications, short video social applications, audio and video conferencing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0087] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with sound acquisition devices (e.g., microphones), video acquisition devices or angiography image acquisition devices (e.g., cameras) and displays, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed on the terminal devices listed above. They can be implemented as multiple software programs or software modules (e.g., used to provide services related to angiography image segmentation) or as a single software program or software module. No specific limitations are made here.

[0088] In some cases, the segmentation model evaluation method provided in this disclosure can be executed by terminal devices 101, 102, and 103, and correspondingly, the segmentation model evaluation device can be set in terminal devices 101, 102, and 103. In this case, system architecture 100 may not include server 105.

[0089] In some cases, the segmentation model evaluation method provided in this disclosure can be jointly executed by terminal devices 101, 102, 103 and server 105. For example, the step of "obtaining multiple sets of test images and the actual segmentation results corresponding to each set of test images" can be executed by terminal devices 101, 102, 103, and the step of "evaluating the predictive ability of the segmentation model based on the performance indicators, the morphological consistency evaluation of the target object, and the accuracy" can be executed by server 105. This disclosure does not limit this. Accordingly, the segmentation model evaluation device can also be respectively set in terminal devices 101, 102, 103 and server 105.

[0090] In some cases, the evaluation method of the segmentation model provided in this disclosure can be executed by the server 105. Accordingly, the evaluation device of the segmentation model can also be set in the server 105. In this case, the system architecture 100 may not include the terminal devices 101, 102, and 103.

[0091] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide services related to angiography image segmentation), or as a single software program or software module. No specific limitations are made here.

[0092] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0093] In some alternative implementations, refer to Figure 2A The diagram illustrates a flow 200 of an embodiment of the segmentation model evaluation method of this disclosure. This segmentation model evaluation method can be used to evaluate a single segmentation model or to evaluate multiple segmentation models simultaneously. The method includes the following steps 201 to 206:

[0094] Step 201: Obtain multiple sets of test images and the actual segmentation results corresponding to each set of test images.

[0095] Before evaluating the segmentation model to be evaluated, test images and corresponding actual segmentation results can be obtained. The segmentation model can perform image segmentation on the test images, while the actual segmentation results are used to detect the image segmentation effect of the segmentation model.

[0096] In some cases, the test image and the corresponding actual segmentation result can be a medical angiography image, such as an angiography image of an aneurysm patient, i.e., a DSA image (Digital Subtraction Angiography image). Correspondingly, the actual segmentation result can be multiple ROIs (Regions of Interest) marked on the selected DSA image. Each ROI can correspond to the region of an object, which serves as the target that the segmentation model needs to segment when performing image segmentation on the test image, i.e., the region that needs to be segmented.

[0097] Based on this, when the test image is a DSA image of an aneurysm patient, a set of DSA images of the same aneurysm patient at the same time can be obtained as a set of test images. Then, one image is selected from each set of test images and multiple ROIs are labeled for it as the corresponding actual segmentation result.

[0098] Step 202: Use the trained segmentation model to predict the test segmentation results for multiple objects in each test image.

[0099] The segmentation model can be a deep learning model or a neural network model, such as U-Net (U-Shaped Networking), VNet (V-Shaped Networking), DeepLabV3 (Deep Labeling Version 3), TransUNet (Transformer-U-Net), and / or SwinUNet (Swin Transformer U-Net). Both the test image and the actual segmentation result contain multiple Regions of Interest (ROIs), meaning regions corresponding to multiple objects. The region of each object in the test image is the region that the segmentation model needs to segment from the test image. Therefore, after the segmentation model segments the regions of each object, the result obtained is the corresponding test object segmentation result; that is, each test object segmentation result defines the region of its corresponding object.

[0100] Based on the test image and the corresponding actual segmentation result determined in step 201 above, the test image can be segmented using the segmentation model to be evaluated. In this process, when the test image and the corresponding actual segmentation result are DSA images of a patient with an aneurysm, the multiple objects contained in the test image and the corresponding actual segmentation result can be the aneurysm, the tumor-bearing artery, and the background in the DSA image.

[0101] Therefore, after the segmentation model to be evaluated performs image segmentation on the training image, for each test object segmentation result in the obtained test segmentation result, if the output segmentation result of each test object is accurate, it should be able to accurately define the aneurysm mask of the aneurysm region, the aneurysm carrier artery mask of the aneurysm carrier artery region, and the background mask of the background region.

[0102] The segmentation model to be evaluated can be a pre-trained segmentation model. Since the segmentation model can be one or more deep learning models or neural network models, in this step, the segmentation model that predicts the segmentation result of the test object for the test image can be one of the five pre-trained models: U-Net, VNet, DeepLabV3, TransUNet, and SwinUNet.

[0103] Based on this, the acquired test images, i.e., each DSA image, can be input into the U-Net model, VNet model, DeepLabV3 model, TransUNet model, and SwinUNet model mentioned above, and each model can be used to predict the aneurysm mask, the aneurysm-bearing artery mask, and the background mask in each DSA image.

[0104] Based on the test object segmentation results predicted by the segmentation model in step 202, the segmentation model can be comprehensively evaluated from both the performance perspective and the clinical perspective of the segmentation results during the subsequent evaluation process. This will yield a more comprehensive evaluation result than one that is evaluated from a single perspective.

[0105] Step 203: Determine the performance index of the segmentation model based on the difference between the segmentation results of each test object and the corresponding actual object segmentation results.

[0106] The actual object segmentation result is the ROI marked in the previous steps. In other words, each actual object segmentation result represents the region of a corresponding object.

[0107] In evaluating the segmentation model from a performance perspective, one or more performance metrics can be used to represent the difference between each test object segmentation result and the corresponding actual object segmentation result in the actual segmentation result, based on the test object segmentation results determined in step 202 above.

[0108] The difference between the segmentation results of each test object and the corresponding segmentation results of the actual object can be a difference in one aspect or multiple aspects. Specifically, it can include, for example, differences in area, differences in pixel values, and differences in the objects they correspond to.

[0109] Therefore, for each aspect of the difference, an index related to that difference can be calculated and used as a performance metric to represent the performance of the segmentation model.

[0110] In other words, when evaluating segmentation models using performance metrics that represent unilateral differences, the evaluation results are usually rather one-sided, especially when there are multiple segmentation models to be evaluated. Since different segmentation models have different structures and strengths, it is easier to obtain inaccurate evaluations when evaluating each segmentation model separately using performance metrics that represent unilateral differences. However, when evaluating segmentation models separately using multiple performance metrics that describe differences in different aspects, the capabilities of each segmentation model from multiple perspectives can be taken into account, thus obtaining a more comprehensive evaluation.

[0111] Step 204: Determine the morphological consistency evaluation between the test object segmentation result of the target object and the corresponding actual object segmentation result in each object.

[0112] In this context, the target object is one of multiple objects in the test image, the test object segmentation result of the target object is the test object segmentation result corresponding to the target object among the multiple test object segmentation results predicted by the segmentation model, and the corresponding actual object segmentation result is the ROI corresponding to the target object among the multiple ROIs of the actual segmentation result.

[0113] Based on the segmentation results of multiple test objects determined in step 202 above, in the process of evaluating the segmentation model from a clinical perspective, one of the multiple objects can be used as the target object. Thus, the performance of the segmentation model can be evaluated from a clinical perspective when segmenting the specified target object. In some cases, the performance of the segmentation model can also be evaluated in multiple rounds by taking turns using each object as the target object.

[0114] The clinical perspective can include the actual clinical morphology of the target object.

[0115] In the segmentation results predicted by the segmentation model for each test object, the segmentation result corresponding to the target object should, as expected, completely and accurately reflect the actual morphology of the target object in clinical practice; that is, the two should be consistent. However, due to the different capabilities of segmentation models, the morphology of the target object represented by the segmentation result may not perfectly match the actual morphology in clinical practice. Based on this, the process of evaluating the segmentation model from a clinical perspective can include: using the morphology of the target object in clinical practice as a benchmark, measuring whether the morphology of the target object represented by the segmentation result is consistent with this benchmark. Here, the morphology of the target object in clinical practice can be the morphology of the corresponding actual object segmentation result as labeled.

[0116] Specifically, the morphological consistency evaluation of the target object in the segmentation result of the test object and the corresponding actual object segmentation result can be calculated, and a scoring threshold for measuring the morphological consistency evaluation can be set.

[0117] The morphological consistency evaluation may include one or more morphological indicators.

[0118] When morphological consistency evaluation includes a morphological index, when measuring morphological consistency, if the morphological index between the test object segmentation result and the corresponding actual object segmentation result is greater than or equal to the scoring threshold, it can be determined that the morphology of the target object represented by the test object segmentation result is consistent with the morphology of the corresponding actual object segmentation result used as the benchmark. If the morphological index between the test object segmentation result and the corresponding actual object segmentation result is less than the scoring threshold, it can be determined that the morphology of the target object represented by the test object segmentation result is inconsistent with the morphology of the corresponding actual object segmentation result used as the benchmark.

[0119] When morphological consistency evaluation includes multiple morphological indicators, a corresponding scoring threshold can be set for each morphological indicator when measuring morphological consistency. If all morphological indicators between the test object segmentation result and the actual object segmentation result are greater than or equal to the corresponding scoring threshold, it can be determined that the morphology of the target object represented by the test object segmentation result is consistent with the morphology of the corresponding actual object segmentation result used as the benchmark. If any morphological indicator between the test object segmentation result and the actual object segmentation result is less than the corresponding scoring threshold, it can be determined that the morphology of the target object represented by the test object segmentation result is inconsistent with the morphology of the corresponding actual object segmentation result used as the benchmark.

[0120] In a specific example using DSA images of an aneurysm patient, when the target object is an aneurysm, the shape of the target object can be the geometric shape of the aneurysm, while the reference shape can be the geometric shape of the aneurysm marked in the actual segmentation result.

[0121] Step 205: Predict the state category of the target object using the test object segmentation results of the target object, and evaluate the accuracy of the state category using the corresponding actual object segmentation results.

[0122] In this step, the clinical perspective can also include the perspective of the target object's clinical status category. The status category of the target object can be the clinical characteristics of the target object. In a specific example using DSA images of an aneurysm patient, when the target object is an aneurysm, its status category can be, for example, the occlusion status of the aneurysm.

[0123] Based on the test object segmentation results determined in step 202 above, during the evaluation of the segmentation model from the perspective of state category, the predicted test object segmentation results of the target object can be used to predict the state category of the target object. If the test object segmentation results of the target object are accurate, an accurate prediction result should be obtained when predicting the state category. However, due to the different capabilities of segmentation models, there may be cases where the test object segmentation results of the target object predicted by the segmentation model are inaccurate. Therefore, when using inaccurate test object segmentation results to predict the state category of the target object, an accurate prediction result cannot be obtained. Accordingly, the ability of the segmentation model to generate test object segmentation results can be evaluated based on the accuracy of the state category prediction results.

[0124] In a specific example using DSA images of an aneurysm patient, the occlusion status of the aneurysm can be predicted by using a pre-set feature scoring model to analyze the various features exhibited by the aneurysm mask as the target test object segmentation result. The occlusion status can include complete occlusion and incomplete occlusion.

[0125] Furthermore, based on the DSA image which corresponds to the actual segmentation result, the actual occlusion status of the aneurysm can be determined according to the various features of the aneurysm annotation region which corresponds to the actual object segmentation result.

[0126] Based on this, the accuracy of the occlusion state predicted by the aneurysm mask compared to the actual occlusion state of the aneurysm can be determined.

[0127] Step 206: Evaluate the predictive ability of the segmentation model based on performance indicators, morphological consistency evaluation of the target object, and accuracy assessment.

[0128] Based on the performance indicators, morphological consistency evaluation, and accuracy determined in steps 203, 204, and 205 above, the predictive ability of the segmentation model can be evaluated from multiple perspectives. At the same time, when each test image covers DSA images of different aneurysm patients at different times before and after surgery, the predictive ability of different segmentation models for DSA images at different times can be evaluated.

[0129] As can be seen, performance indicators are determined based on the difference between the segmentation results of the test objects and the segmentation results of the actual objects, quantifying the basic capabilities of the segmentation model from the perspective of segmentation accuracy. Then, based on the morphological consistency of the target objects, the matching degree between the segmentation results of the test objects and the actual object segmentation results in terms of morphology is verified. Finally, through state category prediction and accuracy evaluation, the segmentation results of the test objects are associated with practical application value, realizing a comprehensive evaluation of the segmentation model from the technical parameters of performance to the correlation value from a clinical perspective. This avoids the problem of relying solely on performance parameters and ignoring the effectiveness of practical applications, ensuring that the evaluation results can truly reflect the comprehensive performance of the segmentation model.

[0130] Therefore, by fusing multi-dimensional evaluation results, the applicability of segmentation models in different application scenarios can be determined based on a comprehensive consideration of performance indicators, morphological consistency evaluation, and the accuracy of state categories. For example, in scenarios where priority must be given to ensuring the matching of segmentation accuracy with clinical morphology, or in scenarios where the effectiveness of segmentation results in predicting the state of the target object must also be considered, this method can be used to clarify the advantages and disadvantages of the model, thereby accurately selecting a segmentation model that suits the needs of the scenario and improving the relevance and reliability of the segmentation model application.

[0131] In some alternative implementations, refer to Figure 2B This illustrates a decomposition process 2010 of one embodiment of step 201 of this disclosure. The decomposition process 2010 includes the following steps 2011 to 2013:

[0132] Step 2011: Obtain at least two angiography image sequences.

[0133] Each DSA sequence (Digital Subtraction Angiography sequence) is a video image or continuous image acquired by a DSA acquisition device within a very short time, such as within 1 second or more seconds. It includes multiple video frames, and each video frame is a DSA image. That is, a single DSA sequence includes multiple DSA images.

[0134] In scenarios where segmentation models are used to segment DSA images, DSA sequences can be obtained from at least two aneurysm patients during the acquisition process.

[0135] The obtained DSA sequences can be DSA sequences from different periods before and / or after surgery for each aneurysm patient. In this application, no limitation is made on the surgical method. The surgical method for each aneurysm patient can be the same or include at least two surgical methods.

[0136] In the process of obtaining DSA sequences, the selection criteria for patients with aneurysms from which DSA sequences are obtained may include, for example: age 18 to 80 years; having a single saccular intracranial aneurysm (IA); being eligible for or having received endovascular treatment, which may include simple coil embolization, stent-assisted coil embolization, or flow diverter implantation; having at least one DSA examination with a follow-up period of more than six months; and having complete clinical and imaging records.

[0137] Furthermore, at least one preoperative DSA sequence and / or at least one postoperative DSA sequence are obtained from each aneurysm patient, thereby obtaining multiple DSA images.

[0138] Step 2012: Select the image to be labeled from each angiography image sequence for labeling the actual segmentation results.

[0139] Based on the DSA sequence determined in step 2011 above, for DSA images belonging to the same DSA sequence, one DSA image can be selected as the image to be labeled corresponding to that DSA sequence.

[0140] Since the image to be labeled will be used as a benchmark to test the segmentation effect of the segmentation model, for each DSA sequence, an optimal projection can be selected as the image representing the DSA sequence based on the maximum visibility of the aneurysm and the parent artery during the arterial phase. The regions of the aneurysm, the parent artery, and the background are then labeled in the DSA image, and each labeled region is used as the actual object segmentation result.

[0141] Step 2013: Determine a set of test images from each angiography image sequence, excluding the image to be labeled.

[0142] Based on the multiple DSA images obtained in the aforementioned step 2011, since there are often DSA images that are not suitable for testing, the multiple DSA images can be screened to exclude DSA images that are not suitable for testing the segmentation model, and the remaining DSA images are determined as test images.

[0143] Specifically, exclusion criteria for DSA images may include: images containing aneurysms associated with arteriovenous malformations or other blood flow-related lesions; images containing dissecting, fusiform, or thrombotic aneurysms; images containing multiple or tandem aneurysms; DSA images that are not follow-up images of patients and cannot assess the occlusion status of the aneurysms; and images of poor quality that cannot be used for reliable training or testing, such as severe artifacts or improper projection angles that prevent adequate visualization of the aneurysm and the parent artery.

[0144] Based on the above screening of DSA sequence sources and the exclusion of obtained DSA images, multiple test images can be obtained.

[0145] In some cases, based on the multiple test images obtained after the above exclusion operations, each test image can be further amplified.

[0146] Specifically, each test image can be rotated at different angles, such as 90, 180, and 270, and / or its contrast can be enhanced, thereby obtaining multiple new images based on each test image, which can also be used as test images for subsequent test segmentation models.

[0147] Based on this, targeted DSA sequence acquisition, test image selection, and the construction of actual object segmentation results provide a high-quality and clinically relevant assessment data foundation for segmentation model evaluation. Specifically, by collecting DSA sequences from aneurysm patients before surgery and after different surgical procedures, the acquired image data ensures that it covers key stages of patient treatment and reflects the application needs of the segmentation model in different clinical scenarios. By identifying test images, suitable and effective images for model evaluation can be selected, eliminating the interference of invalid data on the evaluation results. By selecting representative images from each DSA sequence and labeling the regions of each object to generate segmentation results for each actual object, a precise reference standard is provided for subsequent steps such as comparing segmentation results of test objects, calculating performance indicators, and evaluating state categories, avoiding model evaluation bias caused by inaccurate reference data.

[0148] In some alternative implementations, based on the segmentation results of each test object predicted by the segmentation model in step 202 above, the various performance indicators of the segmentation model can be determined using the test object segmentation results.

[0149] The performance metrics that represent the difference between the test object segmentation results and the actual object segmentation results can include, for example, intersection-union ratio, Dice coefficient (similarity coefficient), recall rate, and harmonic mean.

[0150] Further reference Figure 2C This illustrates a breakdown flow 2030 of one embodiment of step 203 of this disclosure. Specifically, the flow 2030 includes the following steps 2031 to 2034:

[0151] Step 2031: Determine the intersection-union ratio of the segmentation model by using the degree of overlap between the segmentation result of each test object and the corresponding segmentation result of the actual object.

[0152] In this step, for each object, the image segmentation effect can be represented by the degree of overlap between the corresponding test object segmentation result and the corresponding actual object segmentation result. The higher the degree of overlap, the better the segmentation effect of the segmentation model is considered.

[0153] Based on this, the degree of overlap between the two can be represented by the cross-union ratio in terms of area or pixels. Specifically, in some specific cases, for each object, the corresponding area cross-union ratio can be determined based on the area of ​​the object in the test object segmentation result and the area of ​​the corresponding actual object segmentation result, or the corresponding pixel number cross-union ratio can be determined based on each pixel in the test object segmentation result and each pixel in the corresponding actual object segmentation result.

[0154] As can be seen, the intersection-union ratio specifically represents the degree of spatial matching between the segmentation result of the test object and the corresponding segmentation result of the actual object, that is, the degree of overlap.

[0155] Step 2032: Determine the similarity coefficient of the segmentation model by comparing the segmentation result of each test object with the segmentation result of the corresponding actual object.

[0156] Among them, the Dice coefficient represents the similarity between the segmentation result of the test object and the corresponding segmentation result of the actual object.

[0157] Based on this, the overlapping area between the test object segmentation result and the corresponding actual object segmentation result can be determined first. Furthermore, twice the overlapping area between the test object segmentation result and the corresponding actual object segmentation result, and twice the total number of pixels in both results, can be determined. Therefore, the ratio of twice the overlapping area to twice the total number of pixels can be defined as the Dice coefficient.

[0158] Step 2033: Determine the accuracy of the segmentation model by the probability that the object represented by the segmentation result of each test object matches the object represented by the segmentation result of the corresponding actual object.

[0159] Among them, accuracy represents the ability of the segmentation model to accurately characterize the corresponding object by predicting the segmentation results of the test object.

[0160] Based on this, in determining the accuracy, for each test object segmentation result predicted by the segmentation model, the number of objects represented by the test object segmentation result that accurately match the objects represented by the corresponding test object segmentation result is determined, that is, the number of objects correctly represented in each test object segmentation result.

[0161] Furthermore, based on the number of correct test object segmentation results, the proportion of test object segmentation results that accurately represent the correct object among all test object segmentation results representing the same object is determined, and this proportion is defined as the accuracy.

[0162] Therefore, the higher the ratio, the higher the accuracy of the corresponding segmentation model.

[0163] Step 2034: Determine the harmonic mean of the segmentation model using precision and the corresponding recall.

[0164] The recall rate specifically represents the probability of missing a test object from the segmentation results of the actual object segmentation results.

[0165] Since precision and recall, as determined in step 2033 above, characterize different aspects of the segmentation model's capabilities, the harmonic mean between precision and recall can be used to represent the overall balanced capability of the segmentation model in order to balance these two aspects.

[0166] In determining the harmonic and summation numbers, we can first determine the product of precision and recall, and then determine the sum of precision and recall.

[0167] Therefore, twice the ratio of the product to the sum can be determined as the corresponding harmonic mean.

[0168] In some cases, in addition to using the above-mentioned metrics that characterize the difference between the segmentation results of the test object and the corresponding actual object segmentation results as performance metrics, other metrics can be added as performance metrics to characterize the segmentation model performance. For example, recall can be used as one of the performance metrics listed above, and the area under the ROC curve (Receiver Operating Characteristic Curve) can also be used as one of the performance metrics listed above.

[0169] Based on the definition of recall mentioned above, in the process of determining recall, we can first determine each pixel in the actual object segmentation result. Since the actual object segmentation result represents an object, each pixel in it also corresponds to the object.

[0170] Furthermore, it is determined whether each pixel accurately corresponds to the correct object in the corresponding test object segmentation result.

[0171] Furthermore, the ratio of the number of pixels that accurately correspond to the correct object to the total number of pixels in the actual object segmentation result is determined as the corresponding recall rate.

[0172] The area under the ROC curve represents the relationship between two categories of proportions: the proportion of objects in each test object segmentation result that are misclassified as incorrect objects, and the proportion of objects in each test object segmentation result that are correctly predicted as correct objects.

[0173] Therefore, the proportion of objects misjudged as incorrect can be used as the horizontal axis, and the proportion of objects predicted as correct can be used as the vertical axis, thus generating an ROC curve, and the area under the ROC curve can be determined accordingly.

[0174] As can be seen, the area under the ROC curve specifically represents the segmentation model's ability to distinguish between different objects.

[0175] Based on this, a comprehensive quantitative evaluation of the segmentation model's performance is achieved through the construction and calculation of multi-dimensional performance indicators, overcoming the limitations of single-indicator evaluation. Specifically, the intersection-union ratio (IUGR) is determined based on the degree of overlap between the test object segmentation results and the actual object segmentation results, intuitively reflecting the spatial matching degree between the two; the similarity coefficient quantifies the similarity between the segmentation results and the real regions; precision measures the accuracy of the test object segmentation results in representing the objects; and the harmonic mean is calculated by combining precision and recall, balancing the segmentation model's ability to accurately represent objects and avoid missing objects. It is evident that these four indicators cover the core performance of the segmentation model from different dimensions, collectively forming a complete performance evaluation system. This system can comprehensively capture the differences between the test object segmentation results and the actual object segmentation results, avoiding the problem that a single indicator cannot reflect the model's comprehensive capabilities. Furthermore, this process, through clear indicator calculation logic, provides a unified and quantifiable standard for comparing the performance of segmentation models.

[0176] In some alternative implementations, refer to Figure 2D The diagram illustrates a decomposition process 2040 of one embodiment of step 204 of this disclosure. This decomposition process 2040 includes the following steps 2041 to 2043:

[0177] Step 2041: Determine the multiple morphological data of each aneurysm mask and determine the various morphological parameters of the corresponding aneurysm annotation area.

[0178] In scenarios where segmentation models perform image segmentation on DSA images, aneurysms can be pre-defined as target objects, and the aneurysm mask predicted by the segmentation model can be used as the segmentation result of the test object corresponding to the target object. In this example, from a medical clinical perspective, the multi-class morphological data of the aneurysm shown in each aneurysm mask can be determined. Specifically, the multi-class morphological data may include, for example, the diameter, perimeter, width, diameter-to-width ratio, surface area, total pixel area of ​​the aneurysm mask, and boundary length of the aneurysm mask.

[0179] Furthermore, the aneurysm annotation region, which serves as the actual verification standard, can be determined from the actual object segmentation results corresponding to the target object, thereby allowing the aforementioned morphological data to be determined within the aneurysm annotation region.

[0180] Step 2042: Construct corresponding deviation distribution feature maps using the morphological data of each aneurysm mask and the corresponding aneurysm annotation area, and determine the corresponding correlation coefficients.

[0181] Among them, the deviation distribution feature map represents the deviation between various morphological data in the segmentation result of the test object corresponding to the target object and various morphological data in the segmentation result of the actual object corresponding to the target object, thereby quantifying the consistency between the two.

[0182] Based on the various types of morphological data determined in step 2041 above, a deviation feature distribution map of each type of morphological data can be constructed.

[0183] Specifically, taking aneurysm masks and aneurysm labeled areas as examples, in some cases, for each type of morphological data, the mean of this type of morphological data in the aneurysm mask and the mean of this type of morphological data in the aneurysm labeled area can be determined, and the difference between this type of morphological data in the aneurysm mask and the mean of this type of morphological data in the aneurysm labeled area can be determined. The mean and the difference are then used as the horizontal axis and the vertical axis, respectively, to form a deviation distribution feature map of each aneurysm mask with respect to this type of morphological data.

[0184] Therefore, the area covered by the curve in the deviation distribution feature map, i.e., the area under the curve, can be used to characterize the consistency between the aneurysm mask and the aneurysm labeled area.

[0185] On the other hand, the correlation coefficient for each type of morphological data specifically represents the consistency between the segmentation results of the test object corresponding to the target object and the segmentation results of the corresponding actual object in clinical morphology from the overall perspective of various types of morphological data.

[0186] Since the deviation between the aneurysm mask and the aneurysm labeled area includes both the actual deviation caused by the image segmentation capability of the segmentation model and the inevitable error caused by worker measurement or measurement method, the correlation coefficient can quantify the consistency between the two from the perspective of including the actual deviation.

[0187] Specifically, taking aneurysm mask and aneurysm labeled area as examples, in the process of determining the correlation coefficient, we can first determine the errors that occur when measuring aneurysm morphology data. For example, when labeling aneurysm labeled area, there are errors due to manual labeling, and when measuring aneurysm mask in the model, there are errors due to the measurement method.

[0188] Based on this, the proportion of the actual deviation between the aneurysm mask and the aneurysm labeled area, excluding the aforementioned errors, can be determined out of all deviations between the aneurysm mask and the aneurysm labeled area. This proportion is then defined as the correlation coefficient.

[0189] Step 2043: Determine the morphological consistency evaluation using the deviation distribution characteristic map and correlation coefficient of each type of morphological data.

[0190] Based on the area under the ROC curve and the correlation coefficient determined in step 2042 above, a first scoring threshold can be set for the area under the ROC curve and a second scoring threshold can be set for the correlation coefficient.

[0191] If the area under the ROC curve is greater than or equal to the first scoring threshold, and the correlation coefficient is greater than or equal to the second scoring threshold, then it can be determined that the morphological consistency between the segmentation result of the test object corresponding to the target object and the segmentation result of the corresponding actual object is good.

[0192] Based on this, by focusing on the morphological data comparison and analysis of the target object, namely aneurysm, we achieved in-depth validation of the segmentation model's performance from a clinical morphological matching perspective, overcoming the shortcomings of relying solely on general performance indicators to measure the clinical applicability of segmentation results. By identifying the aneurysm mask and various types of morphological data for the aneurysm-annotated region, we provided specific and quantifiable analytical dimensions for subsequent morphological consistency evaluation. By constructing a deviation distribution feature map and calculating correlation coefficients, we quantified the difference between the segmentation results and the actual morphology from the perspectives of the deviation distribution pattern and overall correlation of the morphological data. By combining the results of these two types of analysis to determine the morphological consistency evaluation, we can accurately determine whether the segmentation results of the test object corresponding to the target object output by the segmentation model conform to the actual clinical morphology, avoiding the clinical application risks caused by morphological distortion of the test object segmentation results, and ensuring that the segmentation effect of the segmentation model meets the clinical needs for the morphological representation of the target object.

[0193] In some alternative implementations, refer to Figure 2E The diagram illustrates a decomposition process 2050 of one embodiment of step 205 of this disclosure. This decomposition process 2050 includes the following steps 2051 to 2054:

[0194] Step 2051: Generate the corresponding time density curve using the aneurysm mask of each angiography image in each angiography image sequence.

[0195] In the process of evaluating the segmentation model, based on the segmentation results of each test object corresponding to the identified target object, from a medical clinical perspective, it can be determined whether the segmentation results of each test object corresponding to the target object can be used to accurately predict the state category of the target object. If the segmentation results of the test objects corresponding to the target object can be used to accurately predict the state category of the target object, then the segmentation results of the test objects corresponding to the target object segmented by the segmentation model can be considered accurate, and thus the segmentation ability of the segmentation model can be determined to be good.

[0196] In the process of predicting the state category of the target object using the segmentation results of the test object corresponding to the target object, the aneurysm is used as the target object, and the aneurysm mask is used as the segmentation result of the test object corresponding to the target object. Each DSA image in the DSA sequence should cover the complete hemodynamic cycle, and the vascular signal should be reflected by contrast agent.

[0197] Based on this, for each DSA sequence, the density of the contrast agent in each DSA image can be used to construct a time density curve corresponding to that DSA sequence, where the contrast agent is a substance used to enhance the contrast between blood vessels and tissues in the DSA image.

[0198] As can be seen, the time density curve represents the changes in features such as contrast intensity in the DSA sequence.

[0199] Step 2052: Extract multiple angiography parameters from each time-density curve.

[0200] Based on the time-density curve determined in step 2051 above, DSA imaging indicators related to the aneurysm can be determined.

[0201] Among them, DSA imaging parameters may include, for example, the time to peak intensity of the contrast agent, i.e., the time from the start of the process to the peak intensity; the mean transit time of the contrast agent, i.e. the duration for which the intensity is above half maximum; cerebral blood volume, i.e., the area under the time density curve; and cerebral blood flow, i.e., the ratio of cerebral blood volume to mean transit time.

[0202] Since contrast agents are substances used to enhance the contrast between blood vessels and tissues in DSA images, the aforementioned DSA imaging indicators related to or manifested by contrast agents in each DSA image determine whether the corresponding aneurysm mask can be used to accurately predict the occlusion status of the aneurysm.

[0203] Step 2053: Use multiple angiography parameters corresponding to each time density curve to predict whether the aneurysm in the corresponding angiography image sequence is in a state of complete occlusion or incomplete occlusion.

[0204] Based on the DSA imaging indicators determined in step 2052 above, corresponding DSA imaging indicator coefficients can be set in advance for each DSA imaging indicator, and the corresponding QDSA score (Quantitative Digital Subtraction Angiography score) can be obtained by weighting.

[0205] Specifically, the QDSA score can be determined according to the following formula (1):

[0206]

[0207] Among them, X i Let β represent the i-th DSA imaging index. i represents the DSA imaging index coefficient corresponding to the i-th DSA imaging index, and n represents the total number of DSA imaging indexes.

[0208] Furthermore, since QDSA scores may be inaccurate, based on the aforementioned determined ROC curves, the accuracy of each QDSA score can be determined by the area under the ROC curve corresponding to that QDSA score, i.e., whether the occlusion status of the corresponding aneurysm can be assessed based on the QDSA score.

[0209] In some specific examples, the area under the ROC curve can range from, for example, 0.5 to 1. The closer the area under the ROC curve is to 1, the higher the accuracy of judging the occlusion status of the aneurysm based on the corresponding QDSA score; while the closer the area under the ROC curve is to 0.5, the lower the accuracy of judging the occlusion status of the aneurysm based on the corresponding QDSA score.

[0210] Therefore, the QDSA score that can be used to determine the occlusion status of aneurysm can be determined by combining each QDSA score with the corresponding RCO curve area, and the occlusion status of the aneurysm can be determined based on the QDSA score used to determine the occlusion status of the aneurysm.

[0211] In determining the occlusion status of an aneurysm using the QDSA score, an occlusion threshold can be preset. When the QDSA score is greater than or equal to the occlusion threshold, the aneurysm in the DSA sequence corresponding to that QDSA score is considered to be incompletely occluded. When the QDSA score is less than the occlusion threshold, the aneurysm in the DSA sequence corresponding to that QDSA score is considered to be completely occluded.

[0212] Step 2054: Evaluate the accuracy of the complete or incomplete occlusion status of the aneurysm in the corresponding angiographic image sequence based on the complete or incomplete occlusion status of the aneurysm in each aneurysm annotation region.

[0213] Based on the aneurysm occlusion status predicted using DSA images from each DSA sequence, the accuracy of the occlusion status predicted in step 2053 can be judged based on the actual occlusion status of the aneurysm corresponding to each DSA image.

[0214] Specifically, based on the occlusion status of aneurysms in the segmentation results of each actual object, the number of correct occlusion statuses that are consistent with the occlusion status of aneurysms in the segmentation results can be determined from the occlusion statuses predicted by each DSA image in each DSA sequence, and the proportion of this correct number in all DSA images is determined as the accuracy.

[0215] Based on this, by correlating the segmentation results of the test object corresponding to the target object with the clinical state category prediction of the target object, i.e., the aneurysm, the evaluation extends from the effectiveness of the segmentation results to their clinical application value, thus overcoming the shortcomings of focusing only on segmentation accuracy while ignoring the actual clinical significance of the segmentation results. By generating time-density curves using multiple aneurysm masks, a correlation between the segmentation results and contrast agent intensity changes is established; by extracting angiographic imaging parameters, quantitative evidence is provided for state category prediction; by predicting the complete or incomplete occlusion state of the aneurysm based on these parameters, the segmentation results are transformed into clinically interpretable state judgments; and by evaluating the prediction accuracy against the actual object segmentation results corresponding to the target object, the support capability of the segmentation model's output segmentation results for key clinical state judgments is directly verified, ensuring that the segmentation model not only achieves accurate segmentation but also effectively serves the clinical needs for assessing the state of the target object.

[0216] In some alternative implementations, the segmentation model to be tested can be trained before step 202, so that the trained segmentation model can be tested.

[0217] Specifically, after excluding DSA images that are not suitable for testing in step 2012, a portion of the DSA images can be determined as training images and another portion of the DSA images can be determined as test images according to a certain ratio. For example, 60% of the DSA images in each DSA sequence can be determined as training images and the remaining 40% of the DSA images in the DSA sequence can be determined as test images according to a 6:4 ratio.

[0218] Furthermore, the segmentation model to be trained can be trained using the training images that account for 60% of the DSA images, thereby obtaining a trained segmentation model. When testing the trained segmentation model, the test images that account for 40% of the DSA images are used to test the trained segmentation model.

[0219] Further reference Figure 3 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a segmentation model evaluation device, which corresponds to the method embodiment shown in Figure 2, and the device can be specifically applied to various electronic devices.

[0220] like Figure 3 As shown, the segmentation model evaluation device 300 of this embodiment includes: an acquisition module 301, a prediction module 302, a performance index determination module 303, a morphological consistency module 304, a state category evaluation module 305, and a model evaluation module 306.

[0221] The acquisition module 301 is configured to acquire multiple sets of test images and the actual segmentation results corresponding to each set of test images. Each test image includes multiple objects, and the corresponding actual segmentation results include the actual object segmentation results of each of the multiple objects.

[0222] The prediction module 302 is configured to use the trained segmentation model to predict the test segmentation results of multiple objects in each test image, and each test segmentation result includes the test object segmentation result of each object;

[0223] The performance index determination module 303 is configured to determine the performance index of the segmentation model based on the difference between the segmentation results of each test object and the corresponding actual object segmentation results.

[0224] The morphological consistency determination module 304 is configured to determine the morphological consistency evaluation between the test object segmentation result of the target object and the corresponding actual object segmentation result in each object.

[0225] The state category evaluation module 305 is configured to predict the state category of the target object using the test object segmentation results of the target object, and evaluate the accuracy of the state category using the corresponding actual object segmentation results.

[0226] The model evaluation module 306 is configured to evaluate the predictive ability of the segmentation model based on performance metrics, morphological consistency evaluation of the target object, and accuracy.

[0227] In this embodiment, the specific processing of the acquisition module 301, prediction module 302, performance index determination module 303, morphological consistency module 304, state category evaluation module 305, and model evaluation module 306 of the segmentation model evaluation device 300, and the resulting technical effects, can be referred to the relevant descriptions of steps 201, 202, 203, 204, 205, and 206 in the corresponding embodiment of Figure 2, and will not be repeated here.

[0228] In some alternative implementations, the acquisition module 301 is further configured to:

[0229] Acquire at least two angiographic image sequences, each angiographic image sequence being acquired from at least two aneurysm patients before and / or after surgery, with each aneurysm patient undergoing at least two surgical procedures;

[0230] Select the image to be labeled from each angiography image sequence to label the actual segmentation results;

[0231] In each angiography image sequence, the remaining multiple angiography images other than the image to be labeled are determined as a group of test images. The actual segmentation result corresponding to each group of test images is the actual segmentation result of the image to be labeled selected from the same angiography image sequence.

[0232] In some alternative implementations, the multiple objects include an aneurysm, a tumor-bearing artery, and a background; the actual object segmentation results for each of the multiple objects include an aneurysm mask, a tumor-bearing artery mask, and a background mask;

[0233] Accordingly, the prediction module 302 is further configured as follows:

[0234] The trained segmentation model was used to predict the aneurysm mask, the aneurysm-bearing artery mask, and the background mask in each angiography image.

[0235] In some alternative implementations, performance metrics include at least one of the following: intersection-union ratio, similarity coefficient, accuracy, and harmonic mean of the segmentation model;

[0236] Accordingly, the performance indicator determination module 303 is further configured as follows:

[0237] The intersection-union ratio of the segmentation model is determined by the degree of overlap between the segmentation result of each test object and the corresponding segmentation result of the actual object.

[0238] The similarity coefficient of the segmentation model is determined by the similarity between the segmentation result of each test object and the segmentation result of the corresponding actual object.

[0239] The accuracy of the segmentation model is determined by the probability that the segmentation result of each test object matches the segmentation result of the corresponding actual object.

[0240] The harmonic mean of the segmentation model is determined using precision and the corresponding recall.

[0241] In some optional implementations, the target object includes an aneurysm; the test object segmentation result of the target object includes the aneurysm mask of the aneurysm in each test image; the corresponding actual object segmentation result includes the aneurysm-annotated region in the corresponding actual segmentation result;

[0242] Accordingly, the morphological consistency determination module 304 is further configured as follows:

[0243] Determine the various morphological data of each aneurysm mask and determine the various morphological parameters of the corresponding aneurysm annotation area;

[0244] Each type of morphological data in each aneurysm mask and corresponding aneurysm annotation area is used to construct a corresponding deviation distribution feature map, and the corresponding correlation coefficient is determined.

[0245] The morphological consistency evaluation is determined by using the deviation distribution characteristic map and correlation coefficient of each type of morphological data.

[0246] In some alternative implementations, the status categories include complete occlusion and incomplete occlusion of the aneurysm.

[0247] Accordingly, the state category evaluation module 305 is further configured as follows:

[0248] The aneurysm mask of each angiography image in each angiography image sequence is used to generate the corresponding time density curve, which represents the intensity change of the contrast agent in the corresponding angiography image sequence.

[0249] Multiple angiography parameters and imaging indices were extracted from each time-density curve.

[0250] Using multiple angiography parameters corresponding to each time density curve, the aneurysm in the corresponding angiography image sequence is predicted to be either completely occluded or partially occluded.

[0251] The accuracy of assessing the complete or incomplete occlusion status of aneurysms in the corresponding angiographic image sequence is evaluated based on the complete or incomplete occlusion status of the aneurysm in each aneurysm annotation region.

[0252] It should be noted that the implementation details and technical effects of each module in the segmentation model evaluation device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0253] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing the electronic device of the present disclosure. Figure 4 The computer system 400 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0254] like Figure 4As shown, the computer system 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the computer system 400. The processing device 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0255] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows computer system 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 A computer system 400 with various electronic devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0256] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0257] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0258] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0259] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following functions: Figure 2A The embodiments shown and their alternative implementations illustrate the evaluation method of the segmentation model.

[0260] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0261] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0262] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the unit itself; for example, an acquisition module can also be described as "a module that acquires multiple sets of test images and the actual segmentation results corresponding to each set of test images".

[0263] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. An evaluation method for a segmentation model, comprising: Obtain multiple sets of test images and the actual segmentation results corresponding to each set of test images. Each test image includes multiple objects, and the corresponding actual segmentation results include the actual object segmentation results of each of the multiple objects. The trained segmentation model is used to predict the test segmentation results of multiple objects in each test image. Each test segmentation result includes the test object segmentation result for each object. The performance metrics of the segmentation model are determined based on the differences between the segmentation results of each test object and the corresponding segmentation results of the actual object. Determine the morphological consistency evaluation between the test object segmentation result of the target object in each object and the corresponding actual object segmentation result; The state category of the target object is predicted using the test object segmentation results, and the accuracy of the state category is evaluated using the corresponding actual object segmentation results. The predictive ability of the segmentation model is evaluated based on the performance metrics, the morphological consistency evaluation of the target object, and the accuracy.

2. The method according to claim 1, wherein, The process of obtaining multiple sets of test images and the actual segmentation results corresponding to each set of test images includes: Acquire at least two angiographic image sequences, each angiographic image sequence being acquired from at least two aneurysm patients before and / or after surgery, and each aneurysm patient having at least two surgical procedures; Select the image to be labeled from each angiography image sequence to label the actual segmentation results; In each angiography image sequence, the remaining multiple angiography images other than the image to be labeled are determined as a group of test images. The actual segmentation result corresponding to each group of test images is the actual segmentation result of the image to be labeled selected from the same angiography image sequence.

3. The method according to claim 2, wherein, The multiple objects include aneurysm, tumor-bearing artery, and background; the actual object segmentation results for each of the multiple objects include aneurysm mask, tumor-bearing artery mask, and background mask; as well as The step of using the trained segmentation model to predict the test segmentation results of multiple objects in each test image includes: The trained segmentation model is used to predict the aneurysm mask, the aneurysm-bearing artery mask, and the background mask in each angiography image.

4. The method according to claim 1, wherein, The performance metrics include at least one of the following: intersection-union ratio, similarity coefficient, accuracy, and harmonic mean of the segmentation model; as well as The process of determining the performance metrics of the segmentation model based on the difference between the segmentation results of each test object and the corresponding segmentation results of the actual object includes: The intersection-union ratio (IUU) of the segmentation model is determined by the degree of overlap between the segmentation result of each test object and the corresponding segmentation result of the actual object. The similarity coefficient of the segmentation model is determined by comparing the segmentation result of each test object with the segmentation result of the corresponding actual object. The accuracy of the segmentation model is determined by the probability that the object represented by the segmentation result of each test object matches the object represented by the segmentation result of the corresponding actual object. The harmonic mean of the segmentation model is determined using precision and the corresponding recall.

5. The method according to claim 3, wherein, The target object includes the aneurysm; the test object segmentation result of the target object includes the aneurysm mask in each test image; The corresponding actual object segmentation result includes the aneurysm-annotated region in the corresponding actual segmentation result; as well as The evaluation of morphological consistency between the test object segmentation result and the corresponding actual object segmentation result of the target object in each object includes: Determine the various morphological data of each aneurysm mask and determine the various morphological parameters of the corresponding aneurysm annotation area; Each type of morphological data in each aneurysm mask and corresponding aneurysm annotation area is used to construct a corresponding deviation distribution feature map, and the corresponding correlation coefficient is determined. The morphological consistency evaluation is determined by using the deviation distribution characteristic map and correlation coefficient of each type of morphological data.

6. The method according to claim 5, wherein, The status categories include complete occlusion and incomplete occlusion of the aneurysm; and The step of predicting the state category of the target object using the test object segmentation result of the target object, and evaluating the accuracy of the state category using the corresponding actual object segmentation result, includes: The aneurysm mask of each angiography image in each angiography image sequence is used to generate a corresponding time density curve, which represents the intensity change of the contrast agent in the corresponding angiography image sequence. Multiple angiography parameters and imaging indices were extracted from each time-density curve. Using multiple angiography parameters corresponding to each time density curve, the aneurysm in the corresponding angiography image sequence is predicted to be either completely occluded or partially occluded. The accuracy of assessing the complete or incomplete occlusion status of aneurysms in the corresponding angiographic image sequence is evaluated based on the complete or incomplete occlusion status of the aneurysm in each aneurysm annotation region.

7. An evaluation device for a segmentation model, comprising: The acquisition module is configured to acquire multiple sets of test images and the actual segmentation results corresponding to each set of test images. Each test image includes multiple objects, and the corresponding actual segmentation results include the actual object segmentation results of each of the multiple objects. The prediction module is configured to use the trained segmentation model to predict the test segmentation results of multiple objects in each test image, and each test segmentation result includes the test object segmentation result for each object. The performance metric determination module is configured to determine the performance metric of the segmentation model based on the difference between the segmentation results of each test object and the corresponding actual object segmentation results. The morphological consistency determination module is configured to determine the morphological consistency evaluation between the test object segmentation result of the target object and the corresponding actual object segmentation result in each object. The state category evaluation module is configured to predict the state category of the target object using the test object segmentation results of the target object, and evaluate the accuracy of the state category using the corresponding actual object segmentation results; The model evaluation module is configured to evaluate the predictive ability of the segmentation model based on the performance metrics, the morphological consistency evaluation of the target object, and the accuracy.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by one or more processors, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising computer program instructions, wherein, When the computer program instructions are executed on the computer, the computer causes the computer to perform the method as described in any one of claims 1-6.

Images