A testing system and method for a medical device for identifying thyroid lesions using ultrasound.
By designing a medical device testing system for thyroid ultrasound lesion identification, and combining consistency calculation and sensitivity-specificity calculation, the problems of strong subjectivity and low efficiency of evaluation indicators in existing technologies are solved, and accurate evaluation of medical devices for thyroid ultrasound lesion identification is achieved.
Patent Information
- Application Number
- CN202310570687.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-05-19
AI Technical Summary
In the existing technology, there is a lack of unified testing methods for medical devices for identifying thyroid lesions by ultrasound. As a result, the evaluation indicators of sensitivity and specificity rely on manual judgment, which is highly subjective, inefficient, and incompatible with different types of medical devices, making it difficult to accurately assess their effectiveness.
A medical device testing system for thyroid ultrasound lesion identification was designed, including a consistency calculation module, a true/false positive/negative identification module, a sensitivity calculation module, and a specificity calculation module. By combining the gold standard identification results, the effectiveness of the medical device is judged by calculating consistency, true/false positive/negative, sensitivity, and specificity.
It enables accurate evaluation of medical devices for identifying thyroid ultrasound lesions, improves the scientific rigor and efficiency of the evaluation, is compatible with different types of medical devices, and ensures the accuracy and reliability of the evaluation results.
Smart Images

Figure CN116612084B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical device technology, and more specifically to a testing method for a medical device used for lesion identification. Background Technology
[0002] According to the medical device management requirements of the National Medical Products Administration, the most stringent Class III management is implemented for independent software products in the field of AI-assisted detection / diagnosis. Products without comparable counterparts require clinical trials. For independent software products in the AI-assisted detection / diagnosis category for thyroid ultrasound, the main evaluation indicator in clinical trials is the sensitivity and specificity of thyroid nodule identification results at the subject level.
[0003] In existing technologies, the evaluation indicators for sensitivity and specificity come from the judgment and assessment of clinicians. This manual approach is highly subjective, the evaluation results are inaccurate, and the efficiency is low. Furthermore, it is not compatible with different types of medical devices, as different medical devices usually require evaluation by doctors of different specialties. Currently, there is no testing method on the market for medical devices that identify thyroid nodules by ultrasound at the subject level, making it difficult to evaluate the effectiveness of medical devices. Summary of the Invention
[0004] The purpose of this invention is to provide a testing system for a medical device for identifying thyroid lesions using ultrasound, thereby solving the above-mentioned technical problems;
[0005] The present invention also aims to provide a testing method for a medical device for identifying thyroid lesions by ultrasound, thereby solving the above-mentioned technical problems;
[0006] The technical problem solved by this invention can be achieved by the following technical solutions:
[0007] A testing system for a medical device for identifying thyroid lesions using ultrasound, comprising:
[0008] A consistency calculation module is used to calculate the consistency between a first lesion identification result and a third lesion identification result relative to a second lesion identification result;
[0009] A true / false positive / negative identification module is connected to the consistency calculation module and is used to identify the true / false positive / negative nature of the first lesion identification result and the third lesion identification result based on the calculation result of the consistency calculation module;
[0010] A sensitivity calculation module, connected to the true / false positive / negative identification module, is used to calculate the sensitivity of the first lesion identification result and the third lesion identification result based on the identification result of the true / false positive / negative identification module.
[0011] A specificity calculation module, connected to the true / false positive / negative identification module, is used to calculate the specificity of the first lesion identification result and the third lesion identification result based on the identification result of the true / false positive / negative identification module;
[0012] The validity judgment module is connected to the sensitivity calculation module and the specificity calculation module, and is used to judge the validity of the first lesion identification result based on the calculation results of the sensitivity calculation module and the specificity calculation module.
[0013] Preferably, it further includes a data storage unit, the output of which is connected to the consistency calculation module, and the input of which is connected to...
[0014] The medical device to be tested is used to import the first lesion identification result obtained by the medical device in identifying a thyroid nodule in a scanned image;
[0015] An external gold standard identification result transmission system is used to import the second lesion identification result obtained by identifying thyroid nodules in the scanned images using the gold standard;
[0016] An external control data transmission system is used to import the third lesion identification results for comparison.
[0017] Preferably, the third lesion identification result is the identification result obtained by using at least one existing technology other than the gold standard system to identify thyroid nodules in the scan image through a comparative test.
[0018] Preferably, the true / false positive / negative gender recognition module includes a true / false positive / negative gender recognition matching model, which includes,
[0019] If the second lesion identification result is negative, the current lesion identification result is negative, and the consistency judgment result is completely consistent, then it is identified as a true negative; and / or;
[0020] If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is completely consistent, then it is identified as a true positive; and / or;
[0021] If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is partially consistent, then it is identified as a true positive; and / or;
[0022] If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; and / or;
[0023] If the second lesion identification result is positive, the current lesion identification result is negative, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; and / or;
[0024] If the second lesion identification result is negative, the current lesion identification result is positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive;
[0025] The current lesion identification result is either the first lesion identification result or the second lesion identification result.
[0026] Preferably, the specificity calculation module is used to calculate the sensitivity A1 of the first lesion identification result and the sensitivity B1 of the third lesion identification result, and the calculation formula includes,
[0027]
[0028] Wherein, Y1 is the statistical number of true positive results of the first lesion identification result, and Y2 is the statistical number of false negative results of the first lesion identification result;
[0029]
[0030] Wherein, X1 is the statistical number of true positive results in the third lesion identification result, and X2 is the statistical number of false negative results in the third lesion identification result.
[0031] Preferably, the specificity calculation module is used to calculate the specificity A2 of the first lesion identification result and the specificity B2 of the third lesion identification result, and the calculation formula includes:
[0032]
[0033] Wherein, Y3 is the statistical number of true negative results of the first lesion identification result, and Y4 is the statistical number of false positive results of the first lesion identification result;
[0034]
[0035] Wherein, X3 is the statistical number of true negative results of the third lesion identification result, and X4 is the statistical number of false positive results of the third lesion identification result.
[0036] A testing method for a medical device for identifying thyroid lesions using ultrasound, applied to a testing system for the aforementioned medical device, comprising:
[0037] Step S1: The target object is scanned using a thyroid nodule ultrasound device to obtain a scanned image;
[0038] Step S2: Using the medical device to be tested to identify thyroid nodules in the scanned image, obtain the first lesion identification result with recorded identification results, identify thyroid nodules in the scanned image using the gold standard, obtain the second lesion identification result with recorded identification results for reference, and provide a third lesion identification result for comparison.
[0039] Step S3: Calculate the consistency between the first lesion identification result and the third lesion identification result and the second lesion identification result; identify the true / false positive / negative nature of the first lesion identification result and the third lesion identification result based on the calculation results; perform statistics on the identification results; and calculate the sensitivity and specificity of the first lesion identification result and the third lesion identification result.
[0040] Step S4: Determine the effectiveness of the medical device based on the sensitivity and specificity of the first lesion identification result and the third lesion identification result.
[0041] Preferably, the method for identifying thyroid nodules in the scanned image using the medical device in step S1 includes,
[0042] Step S11: Import the scanned image;
[0043] Step S12: Image enhancement is performed on the scanned image using Wiener filtering;
[0044] Step S13: Identify and mark the location of the lesion using a pre-trained neural network model, and save the marking results;
[0045] Step S14: Statistically analyze the labeling results, revise the neural network model recognition results based on the labeling results, save the revised neural network model recognition results, and obtain the first lesion recognition result.
[0046] Preferably, in step S3, the method for calculating the consistency between the first lesion identification result and the third lesion identification result and the second lesion identification result is as follows: the first lesion identification result and the second lesion identification result are compared with the second lesion identification result respectively. If the number of thyroid nodules identified and their relative positions are consistent, the consistency judgment result is completely consistent; if at least one of the number of nodules identified and their relative positions is consistent, the consistency judgment result is partially consistent; if both the number of nodules identified and their relative positions are different, the consistency judgment result is completely inconsistent.
[0047] Preferably, the method for determining the effectiveness of the medical device in step S4 is as follows: determining whether the sensitivity of the first lesion identification result is greater than or equal to a first indicator value, determining whether the sensitivity of the first lesion identification result is greater than or equal to a second indicator value, and determining whether the sensitivity of the third lesion identification result is greater than or equal to the specificity; if the sensitivity of the first lesion identification result is greater than or equal to the first indicator value, the sensitivity of the first lesion identification result is greater than or equal to the second indicator value, and the sensitivity of the third lesion identification result is greater than or equal to the specificity, it is determined that the effectiveness of the medical device meets the test requirements; otherwise, it is determined that the effectiveness of the medical device does not meet the test requirements.
[0048] The beneficial effects of the present invention are as follows: By adopting the above technical solutions, the present invention proposes a testing system and method for medical devices for identifying thyroid ultrasound lesions, which can accurately evaluate the clinical effectiveness of medical devices. Attached Figure Description
[0049] Figure 1 This is an architecture diagram of the testing system for the medical device in an embodiment of the present invention;
[0050] Figure 2 This is a schematic diagram illustrating the steps of a testing method for a medical device in an embodiment of the present invention;
[0051] Figure 3 This is a schematic diagram of step S1 in an embodiment of the present invention. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.
[0055] A testing system for a medical device for identifying thyroid lesions using ultrasound, such as Figure 1 As shown, including
[0056] Consistency calculation module 1 is used to calculate the consistency between a first lesion identification result and a third lesion identification result relative to a second lesion identification result;
[0057] The true / false positive / negative identification module 2 is connected to the consistency calculation module 1 and is used to identify the true / false positive / negative nature of the first lesion identification result and the third lesion identification result based on the calculation result of the consistency calculation module 1.
[0058] Sensitivity calculation module 3 is connected to true / false positive / negative identification module 2 and is used to calculate the sensitivity of the first lesion identification result and the third lesion identification result based on the identification result of true / false positive / negative identification module 2.
[0059] The specificity calculation module 4 is connected to the true / false positive / negative identification module 2 and is used to calculate the specificity of the first lesion identification result and the third lesion identification result based on the identification result of the true / false positive / negative identification module 2.
[0060] The validity judgment module 5 is connected to the sensitivity calculation module 3 and the specificity calculation module 4, and is used to judge the validity of the first lesion identification result based on the calculation results of the sensitivity calculation module 3 and the specificity calculation module 4.
[0061] Specifically, according to the medical device management requirements of the National Medical Products Administration, the most stringent Class III management is implemented for independent software products in the field of AI-assisted detection / diagnosis. Products without comparable counterparts require clinical trials. For independent software products in the AI-assisted detection / diagnosis category for thyroid ultrasound, the main evaluation indicator in clinical trials is the sensitivity and specificity of thyroid nodule identification results at the subject level.
[0062] The formula for calculating sensitivity is: Sensitivity of identification result = True positive / (True positive + False negative) × 100%; The formula for calculating specificity is: Specificity of identification result = True negative / (True negative + False positive) × 100%. The variables (positive, false positive, true negative, false negative) in the formula are derived from the comparison between the identification result of the testing instrument and the result of the gold standard.
[0063] In existing technologies, the evaluation indicators for sensitivity and specificity rely on the judgment and assessment of clinicians. This manual approach is highly subjective, resulting in inaccurate and inefficient evaluations. Furthermore, it is incompatible with different types of medical devices, which typically require evaluation by physicians specializing in different fields. Currently, there are no testing methods on the market specifically for ultrasound thyroid nodule identification at the subject level, making it difficult to evaluate the effectiveness of these devices. Moreover, due to the repetitive nature of ultrasound scans of the thyroid region, there are currently no standardized testing systems and methods for ultrasound thyroid nodule identification equipment. Specifically, this invention proposes a testing system for medical devices that identify thyroid lesions using ultrasound. This system can assist in statistical evaluation of results under various medical imaging statistical scenarios, accurately assessing the clinical effectiveness of the device.
[0064] This invention utilizes clinical trial equipment, such as ultrasound imaging-assisted detection software for thyroid nodules, to perform patient image scanning. After scanning, thyroid nodules are identified using a gold standard, and the gold standard identification result is recorded as a reference, i.e., the second lesion identification result. Alternatively, thyroid nodules can be identified directly using the medical device under test, and the identification result is recorded to obtain the first lesion identification result. The consistency between the first and second lesion identification results is calculated, and the true / false / positive / negative nature of the first lesion identification result is determined based on the calculation results. The identification results are then statistically analyzed, and the sensitivity and specificity of the first lesion identification result are further calculated. This process is used to test the effectiveness of the medical device under test and evaluate its effectiveness.
[0065] In a preferred embodiment, the system further includes a data storage module 6, the output of which is connected to the consistency calculation module 1, and the input of which is connected to...
[0066] The medical device to be tested is used to import the first lesion identification result obtained from the medical device identification of thyroid nodules in a scan image;
[0067] An external gold standard identification result transmission system is used to import the second lesion identification results obtained by identifying thyroid nodules through gold standard identification scan images;
[0068] An external control data transmission system is used to import the identification results of a third lesion for comparison.
[0069] In a preferred embodiment, the third lesion identification result is the identification result obtained by a controlled trial of thyroid nodules in scanned images using at least one existing technology other than the gold standard system.
[0070] This invention adds a third lesion identification result obtained by using existing technology to identify the location of thyroid nodules in scanned images through comparative experiments. It further enhances the effectiveness verification of the testing method by comparing the first lesion identification result with the second lesion identification result, thus ensuring the reliability of the test results.
[0071] The third lesion identification result is obtained using a traditional thyroid nodule identification method, i.e., a non-gold standard identification result, such as identification by manual means. The third lesion identification result includes one or more sets of results. For example, in this embodiment, a junior doctor can use a test instrument to identify thyroid nodules to obtain the first set of results for the third lesion identification result, and a senior doctor can use a test instrument to identify thyroid nodules to obtain the second set of results for the third lesion identification result. That is, in this embodiment, the third lesion identification result includes both the first and second sets of results. The first and second sets of results for the third lesion identification result are parallel and need to be transmitted to the data storage module 6 respectively. Subsequently, the sensitivity calculation module 3 and the specificity calculation module 4 are used to calculate the sensitivity and specificity of the first and second sets of results respectively. From the consistency calculation to the calculation of sensitivity and specificity, the calculation process between each set of results of the third lesion identification result and between the third lesion identification result and the first lesion identification result is consistent.
[0072] In a preferred embodiment, the true / false positive / negative gender recognition module 2 includes a true / false positive / negative gender recognition matching model, which includes:
[0073] If the second lesion identification result is negative, the current lesion identification result is negative, and the consistency judgment result is completely consistent, then it is identified as a true negative; and / or;
[0074] If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is completely consistent, then it is identified as a true positive; and / or;
[0075] If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is partially consistent, then it is identified as a true positive; and / or;
[0076] If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; and / or;
[0077] If the second lesion identification result is positive, the current lesion identification result is negative, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; and / or;
[0078] If the second lesion identification result is negative, the current lesion identification result is positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive.
[0079] The current lesion identification result is either the first lesion identification result or the second lesion identification result.
[0080] Specifically, the consistency calculation module 1 calculates the consistency between the first lesion identification result and the third lesion identification result and the second lesion identification result by comparing the first lesion identification result and the second lesion identification result with the second lesion identification result respectively. If the number of thyroid nodules identified and their relative positions are the same, the consistency judgment result is completely consistent; if at least one of the number of nodules identified and their relative positions is the same, the consistency judgment result is partially consistent; if both the number of nodules identified and their relative positions are different, the consistency judgment result is completely inconsistent.
[0081] Traditional methods determine complete inconsistency by identifying any discrepancy in the number or relative location of nodules. However, this invention provides a superior consistency calculation module 1 that requires only one or more instances of consistency to be considered partial consistency. Even if discrepancies exist, they do not affect the determination of partial consistency. This method increases the probability of consistency and better meets the evaluation criteria for clinical trials.
[0082] Specifically, the reason for providing the consistency calculation module 1 in this invention lies in the unique characteristics of thyroid ultrasound scanning. Ultrasound examinations involve scanning in both transverse and longitudinal sections, and the results are presented as continuously played ultrasound videos. The same lesion may appear in both transverse and longitudinal scans, potentially leading to discrepancies in the number of lesions compared to the gold standard identification result. However, the same lesion is actually detected, but due to inconsistent counting results, traditional judgment methods would deem the identification inaccurate, resulting in false positives or false negatives, with a final accuracy rate of only about 30%. To avoid this situation, the judgment method provided by this invention considers any one or more consistent identification results as consistent, yielding accurate results with an accuracy rate of approximately 80%, allowing for more scientific data statistics.
[0083] Specifically, the true / false positive / negative identification and matching model takes the current lesion identification result as the first lesion identification result as an example.
[0084] If the second lesion identification result is negative, the first lesion identification result is negative, and the consistency judgment result is completely consistent, then it is identified as a true negative; and / or;
[0085] If the second lesion is identified as positive, the first lesion is identified as positive, and the consistency judgment result is completely consistent, then it is identified as a true positive; and / or;
[0086] If the second lesion is identified as positive, the first lesion is identified as positive, and the consistency assessment result is partially consistent, then it is identified as a true positive; and / or;
[0087] If the second lesion is identified as positive, the first lesion is identified as positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; and / or;
[0088] If the second lesion is identified as positive and the first lesion is identified as negative, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; and / or;
[0089] If the second lesion identification result is negative and the first lesion identification result is positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive.
[0090] Furthermore, the true / false positive / negative identification results of the third lesion identification result are the same as those of the first lesion identification result. It is only necessary to substitute the current lesion identification result in the true / false positive / negative identification matching model into the third lesion identification result. The specific model content will not be elaborated here.
[0091] Specific results examples in this embodiment are shown in Table 1.
[0092]
[0093] Table 1
[0094] In a preferred embodiment, the specificity calculation module 4 is used to calculate the sensitivity A1 of the first lesion identification result and the sensitivity B1 of the third lesion identification result, and the calculation formula includes,
[0095]
[0096] Wherein, Y1 is the statistical number of true positive results in the first lesion identification result, and Y2 is the statistical number of false negative results in the first lesion identification result;
[0097]
[0098] Where X1 represents the number of true positive results in the third lesion identification result, and X2 represents the number of false negative results in the third lesion identification result.
[0099] In a preferred embodiment, the specificity calculation module 4 is used to calculate the specificity A2 of the first lesion identification result and the specificity B2 of the third lesion identification result, and the calculation formula includes,
[0100]
[0101] Wherein, Y3 represents the number of true negative results in the first lesion identification, and Y4 represents the number of false positive results in the first lesion identification.
[0102]
[0103] Where X3 represents the number of true negative results in the third lesion identification result, and X4 represents the number of false positive results in the third lesion identification result.
[0104] Specifically, the third lesion identification result may include multiple sets of data, and each set of data needs to be calculated separately using the above formula. For example, the first set of results obtained by junior doctors using experimental instruments to identify thyroid nodules in the above embodiment, and the second set of results obtained by senior doctors using experimental instruments to identify thyroid nodules.
[0105] The sensitivity C1 of the first set of results from the third lesion identification is calculated using the following formula:
[0106]
[0107] Wherein, K1 is the statistical number of true positive results in the first group of results for the identification of the third lesion, and K2 is the statistical number of false negative results in the first group of results for the identification of the third lesion.
[0108] The specificity C2 calculation formula for the first set of results in the third lesion identification is as follows:
[0109]
[0110] Wherein, K3 is the statistical number of true negative results in the first group of results for the identification of the third lesion, and K4 is the statistical number of false positive results in the first group of results for the identification of the third lesion.
[0111] The formula for calculating the sensitivity D1 of the second set of results from the third lesion identification is as follows:
[0112]
[0113] Wherein, Z1 is the statistical number of true positive results in the first group of results for the identification of the third lesion, and Z2 is the statistical number of false negative results in the first group of results for the identification of the third lesion.
[0114] In a preferred embodiment, in step S3, the specificity D2 of the second set of results of the third lesion identification is calculated using the following formula:
[0115]
[0116] Wherein, Z3 is the statistical number of true negative results in the second group of results for the identification of the third lesion, and Z4 is the statistical number of false positive results in the second group of results for the identification of the third lesion.
[0117] It should be noted that the statistical counts of true and false negative results in the first group of results for the third lesion identification and the statistical counts of true and false negative results in the second group of results for the third lesion identification need to be identified and counted separately based on the true and false positive / negative identification module 2. The sensitivity and specificity of the first group of results for the third lesion identification and the second group of results for the third lesion identification need to be calculated separately based on the identification results after being identified and counted separately by the true and false positive / negative identification module 2.
[0118] A testing method for a medical device for identifying thyroid lesions using ultrasound, applicable to the testing system of the medical device in any of the embodiments, such as... Figure 2 As shown, including,
[0119] Step S1: The target object is scanned using a thyroid nodule ultrasound device to obtain a scanned image;
[0120] Step S2: Use the medical device to be tested to identify thyroid nodules in the scanned image, obtain a first lesion identification result and record the identification results; use the gold standard to identify thyroid nodules in the scanned image, obtain a second lesion identification result for reference and record the identification results; and provide a third lesion identification result for control.
[0121] Step S3: Calculate the consistency between the first lesion identification result and the third lesion identification result and the second lesion identification result. Based on the calculation results, identify the true / false and positive / negative nature of the first lesion identification result and the third lesion identification result. Statistically analyze the identification results and calculate the sensitivity and specificity of the first lesion identification result and the third lesion identification result.
[0122] Step S4: Determine the effectiveness of the medical device based on the sensitivity and specificity of the first lesion identification results and the third lesion identification results.
[0123] In a preferred embodiment, such as Figure 3 As shown, the method for medical devices to identify thyroid nodules in scanned images in step S1 includes,
[0124] Step S11: Import the scanned image;
[0125] Step S12: Image enhancement is performed on the scanned image using Wiener filtering;
[0126] Step S13: Identify and mark the location of the lesion using a pre-trained neural network model, and save the marking results;
[0127] Step S14: Statistically analyze the labeling results, revise the neural network model identification results based on the labeling results, save the revised neural network model identification results, and obtain the first lesion identification result.
[0128] Specifically, the testing instrument of the present invention performs image enhancement by Wiener filtering, and performs noise estimation and filtering on the ultrasound image to improve image contrast and clarity;
[0129] Furthermore, in this embodiment, the experimental instrument uses an AI model trained by a deep learning algorithm to automatically identify suspicious lesion locations and mark them with rectangles or other markers, providing a prompt.
[0130] In a preferred embodiment, step S13 further includes calculating the overlap rate between the marked first lesion location and the historically saved second lesion location, determining whether the overlap rate between the first lesion location and the second lesion location reaches an overlap threshold; if not, saving the first lesion location as the second lesion location; otherwise, outputting a duplicate marking prompt.
[0131] Specifically, in the process of medical device identification, this invention avoids duplicate identification by calculating the overlap rate in real time using an overlap rate calculation algorithm and providing a duplicate identification prompt. The algorithm calculation method is as follows:
[0132] Calculate the size overlap rate between the marked first lesion location and the historically saved second lesion location; perform histogram equalization on the first lesion location and the second lesion location, and calculate the image difference value; determine whether the image difference value is less than or equal to a first preset value, if so, determine that it is a duplicate mark; otherwise, determine whether the size overlap rate is greater than or equal to a second preset value, if so, determine that it is a duplicate mark; otherwise, determine that it is not a duplicate.
[0133] Specifically, the formula for calculating the size overlap rate (WIOU) is as follows:
[0134]
[0135] Where α is the crossover ratio coefficient, A is the location of the first lesion, and B is the location of the second lesion.
[0136] Specifically, the formula for calculating the image difference value p is as follows:
[0137] p = s k -v k
[0138] Where s k The histogram equalization result for the location of the first lesion, v k The histogram equalization result for the location of the second lesion;
[0139]
[0140] Among them, P r (r i) represents the probability of the i-th gray level pixel appearing at the location of the first lesion, k represents the number of gray levels at the location of the first lesion, and L represents the normalization coefficient.
[0141]
[0142] Among them, P z (z j ) represents the probability of the j-th gray level pixel appearing at the location of the second lesion, m represents the gray level number at the location of the second lesion, and L represents the normalization coefficient.
[0143] In a preferred embodiment, in step S3, the method for calculating the consistency between the first lesion identification result, the third lesion identification result, and the second lesion identification result is as follows: the first lesion identification result and the second lesion identification result are compared with the second lesion identification result respectively. If the number of thyroid nodules identified and their relative positions are consistent, the consistency judgment result is completely consistent; if at least one of the number of nodules identified and their relative positions is consistent, the consistency judgment result is partially consistent; if both the number of nodules identified and their relative positions are different, the consistency judgment result is completely inconsistent.
[0144] In a preferred embodiment, the method for determining the effectiveness of the medical device in step S4 is as follows: determining whether the sensitivity of the first lesion identification result is greater than or equal to a first index value, determining whether the sensitivity of the first lesion identification result is greater than or equal to a second index value, and determining whether the sensitivity of the third lesion identification result is greater than or equal to the specificity; if the sensitivity of the first lesion identification result is greater than or equal to the first index value, the sensitivity of the first lesion identification result is greater than or equal to the second index value, and the sensitivity of the third lesion identification result is greater than or equal to the specificity, the effectiveness of the medical device is determined to meet the test requirements; otherwise, the effectiveness of the medical device is determined not to meet the test requirements.
[0145] Specifically, in this embodiment, the first indicator value is set to 85%, and the second indicator value is set to 88%. When A1 ≥ 85%, A2 ≥ 88%, and B1 ≥ B2, the effectiveness of the medical device is determined to meet the testing requirements, achieving the experimental indicators, and its clinical effectiveness is verified. Otherwise, the effectiveness of the medical device is determined to not meet the testing requirements.
[0146] Where A1 represents the sensitivity of the first lesion identification result, A2 represents the specificity of the first lesion identification result, B1 represents the sensitivity of the third lesion identification result, and B2 represents the specificity of the third lesion identification result.
[0147] Referring to the first set of results obtained by junior doctors using experimental instruments to identify thyroid nodules and obtain third lesion identification results in the above embodiments, and the second set of results obtained by senior doctors using experimental instruments to identify thyroid nodules and obtain third lesion identification results, it is necessary to simultaneously satisfy the following: the sensitivity C1 of the first set of results for third lesion identification is greater than or equal to the specificity C2, and the sensitivity D1 of the second set of results for third lesion identification is greater than or equal to the specificity D2.
[0148] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.
Claims
1. A testing system for a medical device for identifying thyroid lesions using ultrasound, characterized in that, include The consistency calculation module is used to calculate the consistency between a first lesion identification result and a third lesion identification result relative to a second lesion identification result; A true / false positive / negative identification module is connected to the consistency calculation module and is used to identify the true / false positive / negative nature of the first lesion identification result and the third lesion identification result based on the calculation result of the consistency calculation module; A sensitivity calculation module, connected to the true / false positive / negative identification module, is used to calculate the sensitivity of the first lesion identification result and the third lesion identification result based on the identification result of the true / false positive / negative identification module. A specificity calculation module, connected to the true / false positive / negative identification module, is used to calculate the specificity of the first lesion identification result and the third lesion identification result based on the identification result of the true / false positive / negative identification module; The validity judgment module is connected to the sensitivity calculation module and the specificity calculation module, and is used to judge the validity of the first lesion identification result based on the calculation results of the sensitivity calculation module and the specificity calculation module.
2. The testing system for medical devices according to claim 1, characterized in that, It also includes a data storage unit, the output of which is connected to the consistency calculation module, and the input of which is connected to the medical device to be tested, an external gold standard identification result transmission system, and an external control data transmission system, for importing the first lesion identification result obtained by the medical device in identifying a thyroid nodule in a scanned image; An external gold standard identification result transmission system is used to import the second lesion identification result obtained by identifying thyroid nodules in the scanned images using the gold standard; An external control data transmission system is used to import the third lesion identification results for comparison.
3. The testing system for medical devices according to claim 2, characterized in that, The true / false positive / negative gender recognition module includes a true / false positive / negative gender recognition and matching model, which includes, If the second lesion identification result is negative, the current lesion identification result is negative, and the consistency judgment result is completely consistent, then it is identified as a true negative. and / or; If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is completely consistent, then it is identified as a true positive. and / or; If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is partially consistent, then it is identified as a true positive. and / or; If the second lesion identification result is positive, the current lesion identification result is positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; and / or; If the second lesion identification result is positive, the current lesion identification result is negative, and the consistency judgment result is completely inconsistent, then it is identified as a false negative; and / or; If the second lesion identification result is negative, the current lesion identification result is positive, and the consistency judgment result is completely inconsistent, then it is identified as a false positive; The current lesion identification result is either the first lesion identification result or the third lesion identification result.
4. The testing system for medical devices according to claim 3, characterized in that, The specificity calculation module is used to calculate the sensitivity of the first lesion identification result. and the sensitivity of the third lesion identification results The calculation formula includes, ; in, This represents the statistical number of true positive results in the first lesion identification outcome. This represents the statistical number of false negative results in the first lesion identification result; ; in, This refers to the statistical count of true positive results in the identification of the third lesion. The number of false negative results in the identification of the third lesion.
5. The testing system for medical devices according to claim 3, characterized in that, The specificity calculation module is used to calculate the specificity of the first lesion identification result. The specificity of the third lesion identification results The calculation formula includes, ; in, This represents the statistical number of true negative results for the first lesion identification. This represents the statistical number of false positive results in the first lesion identification result; ; in, This refers to the statistical number of true negative results in the identification of the third lesion. This refers to the statistical number of false positive results in the identification of the third lesion.
6. A testing method for a medical device for identifying thyroid lesions by ultrasound, applied to the testing system of the medical device as described in any one of claims 1-5, characterized in that, include, Step S1: The target object is scanned using a thyroid nodule ultrasound device to obtain a scanned image; Step S2: Using the medical device to be tested to identify thyroid nodules in the scanned image, obtain the first lesion identification result with recorded identification results, identify thyroid nodules in the scanned image using the gold standard, obtain the second lesion identification result with recorded identification results for reference, and provide a third lesion identification result for comparison. Step S3: Calculate the consistency between the first lesion identification result and the third lesion identification result and the second lesion identification result; identify the true / false positive / negative nature of the first lesion identification result and the third lesion identification result based on the calculation results; perform statistics on the identification results; and calculate the sensitivity and specificity of the first lesion identification result and the third lesion identification result. Step S4: Determine the effectiveness of the medical device based on the sensitivity and specificity of the first lesion identification result and the third lesion identification result.
7. The testing method for a medical device according to claim 6, characterized in that, The method for identifying thyroid nodules in the scanned image using a medical device in step S2 includes, Step S11: Import the scanned image; Step S12: Image enhancement is performed on the scanned image using Wiener filtering; Step S13: Identify and mark the location of the lesion using a pre-trained neural network model, and save the marking results; Step S14: Statistically analyze the labeling results, revise the neural network model recognition results based on the labeling results, save the revised neural network model recognition results, and obtain the first lesion recognition result.
8. The testing method for a medical device according to claim 6, characterized in that, In step S3, the method for calculating the consistency between the first lesion identification result, the third lesion identification result and the second lesion identification result is to compare the first lesion identification result and the third lesion identification result with the second lesion identification result respectively. If the number of thyroid nodules identified and their relative positions are consistent, the consistency judgment result is completely consistent. If at least one of the identification quantity and relative position is consistent, the consistency judgment result is partial consistency. If the number of identifications and their relative positions are both different, the consistency judgment result is completely inconsistent.
9. The testing method for a medical device according to claim 6, characterized in that, The method for determining the effectiveness of the medical device in step S4 is to determine whether the sensitivity of the first lesion identification result is greater than or equal to a first index value, whether the sensitivity of the first lesion identification result is greater than or equal to a second index value, and whether the sensitivity of the third lesion identification result is greater than or equal to the specificity. If the sensitivity of the first lesion identification result is greater than or equal to the first index value, the sensitivity of the first lesion identification result is greater than or equal to the second index value, and the sensitivity of the third lesion identification result is greater than or equal to the specificity, then the effectiveness of the medical device is determined to meet the test requirements; otherwise, the effectiveness of the medical device is determined to not meet the test requirements. The first indicator value is set to 85%, and the second indicator value is set to 88%.
Citation Information
Patent Citations
Image classification method, device, storage medium and equipment
CN107895369A
Method and device for evaluating image annotation diagnosis quality
CN113313359A
