Pulmonary nodule algorithm performance test method, apparatus and device, and storage medium

Abstract

The invention discloses a pulmonary nodule algorithm performance test method, device and equipment and a storage medium, which are applied to the technical field of artificial intelligence, and comprise the following steps: determining a test method of a target detection scene and a test method of a segmentation and measurement scene as test methods; determining at least one of a selection matching mode, a first recall rate, first accuracy, F1 measurement, average accuracy, an average accuracy mean value and a free response subject operation characteristic curve as a test value; determining and selecting at least one of a second recall rate, a second accuracy, a target area overlap ratio, tree length detection and a surface distance as a test value; and testing the pulmonary nodule algorithm to be detected according to the test data set, and determining a performance test result according to the test result and the test index threshold. The algorithm is tested according to the test method of the target detection scene and the test method of the segmentation and measurement scene, and the algorithm performance is determined according to the test result and the threshold thereof, so that the pulmonary nodule algorithm performance can be detected.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for testing the performance of a pulmonary nodule algorithm. Background Art

[0002] In recent years, various AI algorithms, both domestically and internationally, have been dedicated to addressing hot issues such as lung nodule detection, classification, and segmentation, assisting physicians in quickly and accurately identifying and diagnosing lung nodules, thereby providing auxiliary solutions for early detection, diagnosis, and treatment of lung nodules. Because current AI-assisted diagnosis and testing primarily utilize supervised learning, product development, testing, and quality control rely on a large number of standardized examination cases. However, the accuracy and performance of various AI (artificial intelligence) algorithms cannot be guaranteed.

[0003] Therefore, how to test the performance of the pulmonary nodule clinical algorithm according to different clinical scenarios is a technical problem that technicians in this field urgently need to solve. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for testing the performance of a lung nodule algorithm, which solves the technical problem in the prior art that the performance of a lung nodule clinical algorithm cannot be tested.

[0005] To solve the above technical problems, the present invention provides a method for testing the performance of a pulmonary nodule algorithm, comprising:

[0006] Determine the target detection scenario test method and the segmentation and measurement scenario test method as the test method;

[0007] When testing using the test method of the target detection scenario, determining at least one of a matching method, a first recall rate, a first precision, an F1 measure, an average precision, a mean of average precisions, and a free-response receiver operating characteristic curve as a test value; determining the first recall rate by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first precision represents the proportion of true positives; and the F1 measure is determined based on the first precision and the first recall rate;

[0008] When testing using the test method for the segmentation and measurement scenario, determining and selecting at least one of a second recall rate, a second precision, target area overlap, tree length detection, and surface distance as a test value; determining the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard;

[0009] Determine the test data sampling method corresponding to the algorithm for detecting lung nodules, as well as the test index thresholds corresponding to each test value;

[0010] Determine a test data set according to the test data sampling method;

[0011] According to the test data set, the target detection scenario test method and the segmentation and measurement scenario test method are used as test methods to test the algorithm for detecting lung nodules, determine the test results, and determine the performance test results based on the test results and the test index threshold.

[0012] Optionally, before testing the algorithm for detecting pulmonary nodules using the target detection scenario testing method and the segmentation and measurement scenario testing method according to the test data set, the method further includes:

[0013] Determine the test running environment corresponding to the algorithm for detecting lung nodules; the test running environment includes CPU size, GPU size and memory size.

[0014] Optionally, after testing the algorithm for detecting pulmonary nodules using the target detection scenario test method and the segmentation and measurement scenario test method according to the test data set and determining a test result, and determining a performance test result according to the test result and the test index threshold, the method further includes:

[0015] Generate a test report based on the performance test results and upload the test report.

[0016] Optionally, determining the test data set according to the test data sampling method includes:

[0017] Determining the test data sampling method as a positive data extraction strategy;

[0018] Set the weights of each indicator in the positive data extraction strategy.

[0019] Optionally, before determining the test data sampling method corresponding to the algorithm for detecting pulmonary nodules and the test index thresholds corresponding to the test values, the method further includes:

[0020] Use self-built mirror warehouses to store various algorithm models;

[0021] Configure corresponding test plans for various algorithm models; the test plans include test data sampling strategies and test indicator threshold strategies.

[0022] Optionally, the threshold corresponding to the first recall rate is 0.75, and the threshold corresponding to the precision is 0.8.

[0023] Optionally, determining to use a test method for a target detection scenario and a test method for a segmentation and measurement scenario as the test method includes:

[0024] Determine the test method using the target detection scenario, the test method using the segmentation and measurement scenario as the test method and the test method using the image classification scenario; wherein the parameters used in the test method affecting the classification scenario include sensitivity, specificity, missed detection rate, positive predictive value, negative predictive value and Youden index.

[0025] This application also provides a test device for the performance of a pulmonary nodule algorithm, comprising:

[0026] A test method determination module, used to determine a test method for a target detection scenario and a test method for a segmentation and measurement scenario as a test method;

[0027] a first test value determination module, configured to, when performing a test using the test method for the target detection scenario, determine and select at least one of a matching method, a first recall rate, a first precision, an F1 measure, an average precision, a mean of average precisions, and a free-response receiver operating characteristic curve as a test value; determine the first recall rate by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first precision represents a proportion of true positives; and determine the F1 measure based on the first precision and the first recall rate;

[0028] a second test value determination module, configured to, when performing a test using the test method for segmenting and measuring the scene, determine and select at least one of a second recall rate, a second precision, target area overlap, tree length detection, and surface distance as a test value; and determine the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard;

[0029] The sampling method and threshold determination module is used to determine the test data sampling method corresponding to the algorithm for detecting lung nodules, as well as the test index threshold corresponding to each test value;

[0030] A test data set determination module, configured to determine a test data set according to the test data sampling method;

[0031] A test result determination module is used to test the algorithm for detecting lung nodules based on the test data set using the test method for the target detection scenario and the test method for the segmentation and measurement scenario as test methods, determine the test results, and determine the performance test results based on the test results and the test index threshold.

[0032] This application also provides a test device for the performance of a pulmonary nodule algorithm, including:

[0033] memory for storing computer programs;

[0034] A processor is used to implement the above-mentioned method for testing the performance of the pulmonary nodule algorithm when executing the computer program.

[0035] A storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for testing the performance of a pulmonary nodule algorithm.

[0036] It can be seen that the present invention determines to use the test method of the target detection scenario and the test method of the segmentation and measurement scenario as the test method; when using the test method of the target detection scenario for testing, determines to select at least one of the matching method, the first recall rate, the first accuracy, the F1 measure, the average accuracy, the mean of the average accuracy and the free response receiver operating characteristic curve as the test value; the first recall rate is determined by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first accuracy represents the proportion of true positives; the F1 measure is determined based on the first accuracy and the first recall rate; when using the test method of the segmentation and measurement scenario for testing, determines to select the first At least one of the second recall rate, the second precision, the target area overlap, the tree length detection and the surface distance is used as a test value; the second recall rate and the second precision are determined according to the target area segmented by the algorithm and the target area segmented by the reference standard; the test data sampling method corresponding to the algorithm for the pulmonary nodule to be detected, and the test index threshold corresponding to each test value are determined; the test data set is determined according to the test data sampling method; the test method of the target detection scene and the test method of the segmentation and measurement scene are used as test methods according to the test data set to test the algorithm for the pulmonary nodule to be detected, determine the test results, and determine the performance test results according to the test results and the test index threshold. Compared with the current inability to detect the performance of the pulmonary nodule algorithm, the present application detects the algorithm of the pulmonary nodule algorithm according to the test method of the target detection scene and the test method of the segmentation and measurement scene as test methods, uses multiple parameters for testing during the detection process, and determines the algorithm performance by comparing the test results with the test index threshold, thereby realizing the detectability of the performance of the pulmonary nodule algorithm.

[0037] In addition, the present invention also provides a testing device, equipment and storage medium for the performance of a lung nodule algorithm, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1A flowchart of a method for testing the performance of a pulmonary nodule algorithm provided in an embodiment of the present invention;

[0040] Figure 2 An overall flow chart of the performance of a lung nodule algorithm provided by an embodiment of the present invention;

[0041] Figure 3 A flowchart illustrating a method for testing the performance of a pulmonary nodule algorithm provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the structure of a device for testing the performance of a pulmonary nodule algorithm provided in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the structure of a test device for the performance of a lung nodule algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] Please refer to Figure 1 , Figure 1 A flowchart of a method for testing the performance of a pulmonary nodule algorithm provided in an embodiment of the present invention. The method may include:

[0046] S101: Determine to use a target detection scenario test method and a segmentation and measurement scenario test method as test methods.

[0047] The executor of this embodiment is an electronic device. The steps of testing the target detection scenario test method in this embodiment are: a) inputting the test set into the pulmonary nodule algorithm to be tested, and outputting the result detected by the algorithm; the format of the result should be compatible with the reference standard, and the content should at least include the location of the target and the probability of detection. b) The output result is matched with the reference standard according to the prescribed principles to obtain TP (True Positive), FP (False Positive) and FN (False Negative). c) Calculating each performance index according to the formula. The suspicious lesion is taken as a unit, and the calculation result of the entire set is directly used as the final result; if a case contains multiple lesions, the case can also be used as a unit to calculate the calculation result of the lesion set within each case, and then the calculation result of the case set is averaged. The test method of the segmentation and measurement scene in this embodiment is to test the pulmonary nodule algorithm to be tested in the following steps: a) input the test set to the algorithm to be tested and output the result of the algorithm segmentation; the format of the result should be compatible with the reference standard, and the content should at least include the coordinates of the boundary endpoints of the segmented area; b) the performance indicators of the target area segmented by the algorithm and the target area segmented by the reference standard are calculated according to the formula; c) the average value of the calculated results is used as the final result; d) if the algorithm incorporates the detection function, only the targets with the detection result of TP (true positive) are calculated. It can be understood that this embodiment provides corresponding data verification sets for algorithms of different modalities to further increase the scientific nature of the algorithm detection. The data verification set is equivalent to a question with an answer. After the algorithm is tested, it is judged whether the detection result of the algorithm is consistent with the actual result.

[0048] It should be further explained that in order to improve the efficiency of the test, before testing the algorithm for detecting lung nodules based on the test data set using the test method for the target detection scenario and the test method for the segmentation and measurement scenario, it may also include: determining the test running environment corresponding to the algorithm for detecting lung nodules; the test running environment includes the CPU size, GPU size, and memory size. This embodiment does not limit the specific CPU size, GPU size, and memory size. For example, the CPU size, GPU size, and memory size in this embodiment may be 16 cores, 8 blocks, and 64G, respectively; or the CPU size, GPU size, and memory size in this embodiment may be 32 cores, 16 blocks, and 128G, respectively. The reason why this embodiment determines the test running environment is to ensure that the performance of the algorithm for detecting lung nodules can be successfully tested.

[0049] It should be further explained that in order to improve the accuracy of the performance test of the pulmonary nodule algorithm, the above-mentioned determination of using the test method of the target detection scene and the test method of the segmentation and measurement scene as the test method can include: determining to use the test method of the target detection scene, the test method of the segmentation and measurement scene as the test method and the test method of the image classification scene; wherein, the parameters used in the test method affecting the classification scene include sensitivity, specificity, missed detection rate, positive predictive value, negative predictive value and Youden index. The steps for testing the test method of the image classification scene in this embodiment are: a) inputting a test set to the algorithm to be tested and outputting the result of the algorithm classification; the format of the result should be compatible with the reference standard, and the content includes the classification result and classification probability (if applicable); b) comparing the algorithm classification with the reference standard classification, calculating the number of true positive, false positive, true negative and false negative samples, and constructing a confusion matrix. The sensitivity in this embodiment refers to the number of true positive samples divided by the sum of the number of true positive samples and the number of false negative samples; the specificity in this embodiment refers to the number of true negative samples divided by the sum of the number of true negative samples and the number of false positive samples; the missed detection rate in this embodiment is equal to one minus the sensitivity; the positive predictive value in this embodiment refers to the number of true positive samples divided by the sum of the number of true positive samples and the number of false positive samples; the negative predictive value in this embodiment refers to the number of true negative samples divided by the sum of the number of true negative samples and the number of false negative samples; the Youden index in this embodiment refers to the sum of the sensitivity and specificity minus one. This embodiment improves the accuracy of algorithm performance testing by using three testing methods to test the pulmonary nodule algorithm to be tested.

[0050] S102, when testing using a test method for a target detection scenario, determining to select at least one of a matching method, a first recall rate, a first precision, an F1 measure, an average precision, a mean of average precisions, and a free-response receiver operating characteristic curve as a test value; dividing the number of true positives by the sum of the number of true positives and the number of false positives to determine the first recall rate; the first precision represents the proportion of true positives; and the F1 measure is determined based on the first precision and the first recall rate.

[0051] The specific matching method is not limited in this embodiment. For example, the matching method in this embodiment can be area overlap: the matching result is determined by calculating the degree of overlap between the algorithm-marked target and the reference standard area (such as the Dice (set similarity metric function) coefficient, the Jaccard (similarity between individuals of symbolic metric or Boolean metric) coefficient) and setting a matching threshold. Or the matching method in this embodiment can also be center point distance: the matching result is determined by calculating the distance between the center of the algorithm-marked target and the center of the reference standard area and setting a matching threshold. Or the matching method in this embodiment can also be center point fall-in: the matching result is determined by judging whether the center of the target area marked by the algorithm falls within the reference standard target. In this embodiment, the first recall rate refers to the proportion of true positives, and the recall rate is equal to the number of true positives divided by the sum of the number of true positives and the number of false positives. In this embodiment, a tolerance recall rate can also be set, and the tolerance recall rate refers to the value of the recall rate running deviation, for example, the tolerance recall rate is 0.6. In this embodiment, a positive ratio can also be set, and the positive ratio refers to the ratio of all positives (true positives, false negatives) to all negatives (true negatives, false positives). The first precision in this embodiment refers to the proportion of true positives in all results judged as positive. The F1 metric in this embodiment is equal to the first precision multiplied by the recall rate multiplied by 2, divided by the sum of the precision and recall rate. The average precision in this embodiment changes the algorithm threshold setting and calculates the precision and recall rate corresponding to each threshold. With the recall rate as the horizontal axis and the precision as the vertical axis, a precision-recall curve is generated, and the integral area under the curve is calculated, i.e., the average precision. The test record should describe whether the curve is smoothed and the method used; the free response receiver operating characteristic curve in this embodiment refers to calculating the recall rate and non-lesion rate corresponding to each threshold, with the recall rate as the vertical axis and the non-lesion localization rate as the horizontal axis. The constructed curve is the fRoc curve (receiver operating characteristic curve), and the area under the curve is calculated as the evaluation indicator.

[0052] S103, when testing using the test method of the segmentation and measurement scene, determine and select at least one of the second recall rate, the second precision, the target area overlap, the tree length detection and the surface distance as the test value; determine the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard.

[0053] In this embodiment, the second recall is the intersection of the target area segmented by the algorithm and the target area segmented by the reference standard divided by the target area segmented by the reference standard. The second precision in this embodiment is the intersection of the target area segmented by the algorithm and the target area segmented by the reference standard divided by the target area segmented by the algorithm. The target area overlap (intersection-over-union ratio) in this embodiment indicates that when the target area is a general entity (such as a lung nodule), the Dice coefficient or Jaccard coefficient should be used to evaluate the target area overlap. The Dice coefficient is twice the intersection of the target area segmented by the algorithm and the target area segmented by the reference standard divided by the sum of the two (the harmonic mean of recall and precision). The Jaccard coefficient is the intersection of the target area segmented by the algorithm and the target area segmented by the reference standard divided by the union of the two. The tree length detection in this embodiment refers to the use of tree length detection (TLD) to evaluate the ratio of the correctly segmented trachea length to the reference standard trachea length when the target area is a trachea or other tree-like structure. The surface distance in this embodiment refers to the surface distance between the target areas given by the algorithm and the reference standard, which can be used to evaluate the effectiveness of contour segmentation.

[0054] S104, determining a test data sampling method corresponding to the algorithm for detecting lung nodules, and a test index threshold corresponding to each test value.

[0055] This embodiment does not limit the specific test data sampling method. For example, the test data sampling method in this embodiment may be a positive data extraction strategy, such as: a male-to-female ratio of 4:6, an adult-to-minor ratio of 1:9, etc.; or the test data sampling method in this embodiment may be a negative data extraction strategy. The test index threshold corresponding to the test value in this embodiment can be determined according to different types of algorithms for pulmonary nodules to be detected, and different algorithms for pulmonary nodules to be detected correspond to different test index thresholds. This embodiment does not limit the specific test threshold. For example, in this embodiment, the threshold corresponding to the first recall rate in this embodiment is 0.75, and the threshold corresponding to the first precision is 0.8.

[0056] It should be further explained that, in order to improve the accuracy of performance testing, before determining the test data sampling method corresponding to the pulmonary nodule algorithm to be detected and the test indicator thresholds corresponding to each test value, the following steps may also be included: utilizing a self-built mirror repository to store various algorithm models; configuring corresponding test plans for each algorithm model; and test plans including test data sampling strategies and test indicator threshold strategies. This embodiment provides corresponding test plans for various types of pulmonary nodule algorithms, thereby improving the accuracy of pulmonary nodule algorithm performance testing.

[0057] S105: Determine a test data set according to a test data sampling method.

[0058] This embodiment can select data from a data set according to a test data sampling method to obtain a test data set.

[0059] It should be further explained that, in order to improve the accuracy of sampling, the above-mentioned determination of the test data set according to the test data sampling method may include: determining the test data sampling method as a positive data extraction strategy; setting the weights of various indicators in the positive data extraction strategy. This embodiment can set the weights of various indicators in the positive data extraction strategy, such as the male-female ratio of each indicator; or the male-female ratio; or the proportion of each region. It is understandable that there will be a total number when extracting data, for example, 100 data are extracted this time to form a data set; then, according to the weights and the data tags within the system, the amount of data with a given configuration ratio is dynamically extracted. This embodiment sets the weights of various indicators to ensure that all types of data exist in the extracted data, thereby improving the accuracy of sampling and thus improving the accuracy of performance testing.

[0060] S106, using the target detection scenario test method and the segmentation and measurement scenario test method as the test method according to the test data set, the algorithm for detecting lung nodules is tested, the test results are determined, and the performance test results are determined according to the test results and the test index threshold.

[0061] This embodiment tests the detection algorithm and obtains test results corresponding to the test index threshold, thereby determining whether each parameter in the test result exceeds the corresponding test index threshold. If it exceeds, it is determined that the performance is normal; if it does not exceed, it is determined that it is abnormal and the performance needs to be optimized.

[0062] It should be further explained that, in order to improve the convenience of performance improvement, after the above-mentioned test method of the target detection scenario and the test method of the segmentation and measurement scenario are used as the test method based on the test data set, the algorithm for detecting lung nodules is tested, the test results are determined, and the performance test results are determined based on the test results and the test index threshold, it can also include: generating a test report based on the performance test results, and uploading the test report. This embodiment will generate a test report based on the performance test results to specifically analyze the reasons for good or bad performance. The test report of this embodiment can have a graph to improve the intuitiveness of the report.

[0063] A method for testing the performance of a lung nodule algorithm provided by an embodiment of the present invention may include: S101, determining to use a test method for a target detection scenario and a test method for a segmentation and measurement scenario as test methods; S102, when using the test method for the target detection scenario for testing, determining to select at least one of a matching method, a first recall rate, a first accuracy, an F1 metric, an average accuracy, a mean of average accuracy, and a free response receiver operating characteristic curve as a test value; determining the first recall rate by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first accuracy represents the proportion of true positives; the F1 metric is determined based on the first accuracy and the first recall rate; S103, when using the test method for the segmentation and measurement scenario for testing, determining the first recall rate by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first accuracy represents the proportion of true positives; the F1 metric is determined based on the first accuracy and the first recall rate; When conducting the test, determine and select at least one of the second recall rate, the second precision, the target area overlap, the tree length detection, and the surface distance as the test value; determine the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard; S104, determine the test data sampling method corresponding to the algorithm for the pulmonary nodule to be detected, and the test index threshold corresponding to each test value; S105, determine the test data set based on the test data sampling method; S106, test the algorithm for the pulmonary nodule to be detected using the test method of the target detection scene and the test method of the segmentation and measurement scene as the test method according to the test data set, determine the test results, and determine the performance test results based on the test results and the test index threshold. Compared with the current inability to detect the performance of the pulmonary nodule algorithm, the present application detects the algorithm of the pulmonary nodule algorithm based on the test method of the target detection scene and the test method of the segmentation and measurement scene as the test method, uses multiple parameters for testing during the detection process, and determines the algorithm performance based on the comparison of the test results with the test index threshold, thereby achieving the detectability of the performance of the pulmonary nodule algorithm. Furthermore, the embodiment determines the test operating environment to ensure that the performance of the algorithm to be detected for pulmonary nodules can be successfully tested. Furthermore, the embodiment improves the accuracy of algorithm performance testing by using three test methods to test the algorithm to be tested for pulmonary nodules. Furthermore, the embodiment provides corresponding test plans for various types of pulmonary nodule algorithms, thereby improving the accuracy of pulmonary nodule algorithm performance testing. Furthermore, the embodiment generates a test report based on the performance test results to specifically analyze the reasons for good or bad performance. The test report of this embodiment may include a graph to improve the intuitiveness of the report.

[0064] In order to make the present invention easier to understand, please refer to Figure 2 , Figure 2 The overall flow chart of the performance of a lung nodule algorithm provided by an embodiment of the present invention may specifically include:

[0065] 1. Determine the testing environment and performance testing method corresponding to the algorithm to be tested.

[0066] 2. Select the test data set according to the test data sampling method corresponding to the algorithm to be tested.

[0067] 3. Determine the test threshold corresponding to the algorithm to be tested and the performance detection method.

[0068] 4. Use the test data set to run the detection algorithm and determine the test results.

[0069] 5. Determine whether the test results are as expected.

[0070] 6. Compare the test results with the test threshold and generate a report based on the comparison results.

[0071] In order to make the present invention easier to understand, please refer to Figure 3 , Figure 3 A flowchart of a method for testing the performance of a pulmonary nodule algorithm provided by an embodiment of the present invention is provided. Figure 2 The overall flow chart of the lung nodule algorithm performance provided corresponds to the following:

[0072] S301, determining an operating environment corresponding to a lung nodule image segmentation algorithm; wherein the operating environment includes a 16-core CPU, 8 GPUs, and 64G memory.

[0073] S302: Determine to use a target detection scenario test method and a segmentation and measurement scenario test method as the test method.

[0074] S303, when testing using the test method of the target detection scenario, determine and select at least one of the matching method, first recall rate, tolerance recall rate, positive ratio, confidence and significance level parameters, precision and FROC curve (receiver operating characteristic curve) as the test value.

[0075] The accuracy in this embodiment corresponds to the first accuracy mentioned above. The thresholds corresponding to the confidence and significance level parameters in this embodiment can be configured as 99% & 1%.

[0076] S304 , when performing a test using the test method of the segmentation and measurement scenario, determining to select at least one of the second recall rate, precision, intersection-over-union, tree length detection, measurement value difference, and surface distance as a test value.

[0077] S305: Use the self-built image repository to store the uploaded lung nodule image segmentation algorithm.

[0078] S306, determining a test data sampling method and a test index threshold corresponding to the lung nodule image segmentation algorithm.

[0079] S307: Determine a test data set according to a test data sampling method.

[0080] S308, start an instance of the lung nodule image segmentation algorithm image in the Kubernetes cluster (container cluster management system), and import the test dataset into the instance.

[0081] S309: After the execution of the determination algorithm is completed, the test result is determined according to the test value of each test method and the corresponding test index threshold.

[0082] The method for testing the performance of a pulmonary nodule algorithm provided by an embodiment of the present invention has the following advantages:

[0083] 1. The clinical algorithm detection method for pulmonary nodules described in the present invention is also applicable to the detection of algorithms for some other diseases.

[0084] 2. The algorithm detection described in this invention supports customized detection environment. Whether it is CPU, memory or GPU, it supports specified models and configurations to meet the training requirements of specific algorithms.

[0085] 3. The detection method described in the present invention is the simplest detection method currently available and can be completed by only providing the original algorithm and detection indicators.

[0086] 4. The detection method described in the present invention is completely completed automatically by the server. Whether it is loading from the data set or the test set, calculating indicators, or generating reports, there is no excessive human interference, which greatly ensures the compliance of the detection report.

[0087] The following introduces a test device for the performance of a lung nodule algorithm provided by an embodiment of the present invention. The test device for the performance of a lung nodule algorithm described below and the test method for the performance of a lung nodule algorithm described above can refer to each other.

[0088] Please refer to Figure 4 , Figure 4 A schematic diagram of a device for testing the performance of a pulmonary nodule algorithm provided in an embodiment of the present invention may include:

[0089] A test method determination module 100 is used to determine the test method of the target detection scenario and the test method of the segmentation and measurement scenario as the test method;

[0090] A first test value determination module 200 is configured to, when performing a test using the test method for the target detection scenario, determine and select at least one of a matching method, a first recall rate, a first precision, an F1 measure, an average precision, a mean of average precisions, and a free-response receiver operating characteristic curve as a test value; determine the first recall rate by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first precision represents a proportion of true positives; and determine the F1 measure based on the first precision and the first recall rate;

[0091] A second test value determination module 300 is configured to, when performing a test using the test method for the segmentation and measurement scenario, determine and select at least one of a second recall rate, a second precision, target area overlap, tree length detection, and surface distance as a test value; and determine the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard;

[0092] The sampling method and threshold determination module 400 is used to determine the test data sampling method corresponding to the algorithm for detecting lung nodules, and the test index threshold corresponding to each test value;

[0093] A test data set determination module 500 is configured to determine a test data set according to the test data sampling method;

[0094] The test result determination module 600 is used to test the algorithm for detecting lung nodules according to the test data set using the test method of the target detection scenario and the test method of the segmentation and measurement scenario as the test method, determine the test results, and determine the performance test results based on the test results and the test index threshold.

[0095] Furthermore, based on the above embodiment, the device for testing the performance of the pulmonary nodule algorithm may further include:

[0096] The operating environment determination module is used to determine the test operating environment corresponding to the algorithm for detecting lung nodules; the test operating environment includes CPU size, GPU size and memory size.

[0097] Furthermore, based on any of the above embodiments, the device for testing the performance of the pulmonary nodule algorithm may further include:

[0098] The test report generation module is used to generate a test report based on the performance test results and upload the test report.

[0099] Further, based on any of the above embodiments, the test data set determination module 500 may include:

[0100] a positive data extraction strategy determining unit, configured to determine the test data sampling method as a positive data extraction strategy;

[0101] The weight setting unit is used to set the weight of each indicator in the positive data extraction strategy.

[0102] Furthermore, based on any of the above embodiments, the device for testing the performance of the pulmonary nodule algorithm may further include:

[0103] Algorithm model storage module, used to store various algorithm models using self-built image warehouse;

[0104] The test plan determination module is used to configure the corresponding test plans for various algorithm models; the test plans include test data sampling strategies and test indicator threshold strategies.

[0105] Further, based on any of the above embodiments, the threshold corresponding to the first recall rate is 0.75, and the threshold corresponding to the precision is 0.8.

[0106] Further, based on any of the above embodiments, the test method determination module 100 may include:

[0107] A test method determination unit is used to determine the test method of the target detection scenario, the test method of the segmentation and measurement scenario as the test method and the test method of the image classification scenario; wherein the parameters used in the test method affecting the classification scenario include sensitivity, specificity, missed detection rate, positive predictive value, negative predictive value and Youden index.

[0108] It should be noted that the order of the modules and units in the above-mentioned test device for the performance of a pulmonary nodule algorithm can be changed without affecting the logic.

[0109] A testing device for the performance of a lung nodule algorithm provided by an embodiment of the present invention may include: a test method determination module 100, used to determine the test method of the target detection scenario and the test method of the segmentation and measurement scenario as the test method; a first test value determination module 200, used to determine the selection of at least one of a matching method, a first recall rate, a first accuracy, an F1 measure, an average accuracy, a mean of average accuracy and a free response receiver operating characteristic curve as a test value when testing using the test method of the target detection scenario; the first recall rate is determined by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first accuracy represents the proportion of true positives; the F1 measure is determined based on the first accuracy and the first recall rate; a second test value determination module 300, used to determine the selection of a matching method, a first recall rate, a first accuracy, an F1 measure, an average accuracy, a mean of average accuracy and a free response receiver operating characteristic curve when testing using the test method of the segmentation and measurement scenario During the test, at least one of the second recall rate, the second precision, the target area overlap, the tree length detection and the surface distance is selected as the test value; the second recall rate and the second precision are determined according to the target area segmented by the algorithm and the target area segmented by the reference standard; the sampling method and threshold determination module 400 is used to determine the test data sampling method corresponding to the algorithm for the pulmonary nodule to be detected, and the test index threshold corresponding to each test value; the test data set determination module 500 is used to determine the test data set according to the test data sampling method; the test result determination module 600 is used to test the algorithm for the pulmonary nodule to be detected according to the test data set using the test method of the target detection scenario and the test method of the segmentation and measurement scenario as the test method, determine the test result, and determine the performance test result according to the test result and the test index threshold. Compared with the current inability to test the performance of the lung nodule algorithm, the present application uses the test method of the target detection scenario and the test method of the segmentation and measurement scenario as the test method to test the lung nodule algorithm. During the detection process, multiple parameters are used for testing, and the algorithm performance is determined by comparing the test results with the test index threshold, thereby achieving the detectability of the lung nodule algorithm performance. In addition, the reason why the embodiment determines the test running environment is to ensure that the performance of the lung nodule algorithm to be detected can be successfully tested; and, this embodiment improves the accuracy of the algorithm performance detection by using three test methods to detect the lung nodule algorithm to be tested; and, this embodiment provides corresponding test plans for various types of lung nodule algorithms, thereby improving the accuracy of the lung nodule algorithm performance test; and, this embodiment will generate a test report based on the performance test results to specifically analyze the reasons for good or bad performance. The test report of this embodiment can have a graph to improve the intuitiveness of the report.

[0110] The following introduces a test device for the performance of a lung nodule algorithm provided by an embodiment of the present invention. The test device for the performance of a lung nodule algorithm described below and the test method for the performance of a lung nodule algorithm described above can refer to each other.

[0111] Please refer to Figure 5 , Figure 5 A schematic diagram of a structure of a pulmonary nodule algorithm performance testing device provided in an embodiment of the present invention may include:

[0112] Memory 10, for storing computer programs;

[0113] The processor 20 is configured to execute a computer program to implement the steps of the above-mentioned method for testing the performance of the pulmonary nodule algorithm.

[0114] The memory 10 , the processor 20 , and the communication interface 30 all communicate with each other via a communication bus 40 .

[0115] In the embodiment of the present invention, the memory 10 is used to store one or more programs. The program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0116] Determine the target detection scenario test method and the segmentation and measurement scenario test method as the test method;

[0117] When testing using a test method for a target detection scenario, determining at least one of a matching method, a first recall rate, a first precision, an F1 measure, an average precision, a mean of average precisions, and a free-response receiver operating characteristic curve as a test value; determining the first recall rate by dividing the number of true positives by the sum of the number of true positives and the number of false positives; the first precision represents the proportion of true positives; and the F1 measure is determined based on the first precision and the first recall rate.

[0118] When testing using the test method of the segmentation and measurement scene, determining and selecting at least one of a second recall rate, a second precision, target area overlap, tree length detection, and surface distance as a test value; determining the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard;

[0119] Determine the test data sampling method corresponding to the algorithm for detecting lung nodules, as well as the test index thresholds corresponding to each test value;

[0120] Determine the test data set based on the test data sampling method;

[0121] According to the test data set, the target detection scenario test method and the segmentation and measurement scenario test method are used as the test method to test the algorithm for detecting lung nodules, determine the test results, and determine the performance test results based on the test results and the test indicator threshold.

[0122] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function, etc.; the data storage area may store data created during use.

[0123] In addition, the memory 10 may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset or an extended set thereof. The operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0124] The processor 20 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic device. The processor 20 may be a microprocessor or any conventional processor. The processor 20 may call a program stored in the memory 10.

[0125] The communication interface 30 may be an interface of a communication module, and is used to connect to other devices or systems.

[0126] Of course, it needs to be explained that Figure 5 The structure shown does not constitute a limitation on the test equipment for the performance of the pulmonary nodule algorithm in the embodiment of the present invention. In actual applications, the test equipment for the performance of the pulmonary nodule algorithm may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0127] The storage medium provided in an embodiment of the present invention is introduced below. The storage medium described below and the method for testing the performance of a pulmonary nodule algorithm described above can be referenced to each other.

[0128] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for testing the performance of the pulmonary nodule algorithm.

[0129] The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0131] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0132] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0133] The above is a detailed introduction to the testing method, device, equipment and storage medium for the performance of a lung nodule algorithm provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for testing the performance of a pulmonary nodule algorithm, characterized in that: include: Determine the target detection scenario test method and the segmentation and measurement scenario test method as the test method; When testing using the test method of the target detection scenario, determining at least one of a matching method, a first recall rate, a first precision, an F1 measure, an average precision, a mean of average precisions, and a free-response receiver operating characteristic curve as a test value; dividing the number of true positives by the sum of the number of true positives and the number of false positives to determine the first recall rate; and the first precision represents a proportion of true positives. The F1 measure is determined based on the first precision and the first recall; When performing a test using the test method for the segmentation and measurement scenario, determining to select at least one of the second recall rate, the second precision, the target area overlap, the tree length detection, and the surface distance as a test value; Determining the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard; Determine the test data sampling method corresponding to the algorithm for detecting lung nodules, as well as the test index thresholds corresponding to each test value; Determine a test data set according to the test data sampling method; According to the test data set, the target detection scenario test method and the segmentation and measurement scenario test method are used as test methods to test the algorithm for detecting lung nodules, determine the test results, and determine the performance test results based on the test results and the test index threshold.

2. The method for testing the performance of the pulmonary nodule algorithm according to claim 1, characterized in that: Before testing the algorithm for detecting pulmonary nodules using the target detection scenario testing method and the segmentation and measurement scenario testing method according to the test data set, the method further includes: Determine the test running environment corresponding to the algorithm for detecting lung nodules; the test running environment includes CPU size, GPU size and memory size.

3. The method for testing the performance of the pulmonary nodule algorithm according to claim 1, characterized in that: After testing the algorithm for detecting pulmonary nodules using the target detection scenario testing method and the segmentation and measurement scenario testing method according to the test data set, determining a test result, and determining a performance test result according to the test result and the test index threshold, the method further includes: Generate a test report based on the performance test results and upload the test report.

4. The method for testing the performance of a pulmonary nodule algorithm according to any one of claims 1 to 3, characterized in that: The determining of the test data set according to the test data sampling method includes: Determining the test data sampling method as a positive data extraction strategy; Set the weights of each indicator in the positive data extraction strategy.

5. The method for testing the performance of the pulmonary nodule algorithm according to claim 1, characterized in that: Before determining the test data sampling method corresponding to the algorithm for detecting pulmonary nodules and the test index thresholds corresponding to the test values, the method further includes: Use self-built mirror warehouses to store various algorithm models; Configure corresponding test plans for various algorithm models; the test plans include test data sampling strategies and test indicator threshold strategies.

6. The method for testing the performance of the pulmonary nodule algorithm according to claim 1, characterized in that: The threshold corresponding to the first recall rate is 0.75, the threshold corresponding to the F1 score is 0.75, the threshold corresponding to the first precision is 0.8, the threshold corresponding to the average precision is 0.7, and the threshold corresponding to the mean of the average precision is 0.

75.

7. The method for testing the performance of a pulmonary nodule algorithm according to claim 1, wherein: The determining of using the target detection scenario test method and the segmentation and measurement scenario test method as the test method includes: Determine to use the test method of the target detection scenario, the test method of the segmentation and measurement scenario as the test method and the test method of the image classification scenario; wherein the parameters used in the test method affecting the classification scenario include sensitivity, specificity, missed detection rate, positive predictive value, negative predictive value and Youden index.

8. A device for testing the performance of a pulmonary nodule algorithm, characterized in that: include: A test method determination module, used to determine a test method for a target detection scenario and a test method for a segmentation and measurement scenario as a test method; a first test value determination module, configured to, when performing a test using the test method for the target detection scenario, determine at least one of a matching method, a first recall rate, a first precision, an F1 measure, an average precision, a mean of average precisions, and a free-response receiver operating characteristic curve as a test value; determine the first recall rate by dividing the number of true positives by the sum of the number of true positives and the number of false positives; and the first precision represents a proportion of true positives; The F1 measure is determined based on the first precision and the first recall; A second test value determination module is configured to determine and select at least one of a second recall rate, a second precision, target area overlap, tree length detection, and surface distance as a test value when performing a test using the test method of the segmentation and measurement scenario; Determining the second recall rate and the second precision based on the target area segmented by the algorithm and the target area segmented by the reference standard; The sampling method and threshold determination module is used to determine the test data sampling method corresponding to the algorithm for detecting lung nodules, as well as the test index threshold corresponding to each test value; A test data set determination module, configured to determine a test data set according to the test data sampling method; A test result determination module is used to test the algorithm for detecting lung nodules based on the test data set using the test method for the target detection scenario and the test method for the segmentation and measurement scenario as test methods, determine the test results, and determine the performance test results based on the test results and the test index threshold.

9. A device for testing the performance of a pulmonary nodule algorithm, characterized in that: include: memory for storing computer programs; A processor, configured to implement the method for testing the performance of a pulmonary nodule algorithm as claimed in any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for testing the performance of a pulmonary nodule algorithm according to any one of claims 1 to 7.

Images