A Turing test method for artificial intelligence software

Abstract

This invention provides a Turing test method for artificial intelligence software, comprising: acquiring the output data of the artificial intelligence software to be tested; comparing the output data with data in a preset knowledge base to determine the accuracy of the results of the artificial intelligence software; when the accuracy of the results of the artificial intelligence software is higher than a predefined accuracy threshold, calculating whether the passability of the output data of the artificial intelligence software in a preset learning model reaches a cross-entropy threshold; if the passability reaches the cross-entropy threshold, then using a clustering analysis algorithm to group the cases corresponding to the output data, and calculating the ratio of manually labeled case groups; if the ratio of manually labeled case groups is higher than a predefined Turing test threshold, then determining that the artificial intelligence software has passed the Turing test of artificial intelligence, and generating a test report. This invention provides a complete and reliable intelligent testing scheme for medical artificial intelligence software products, achieving effective testing of the accuracy of AI diagnostic system judgment results.

Description

Technical Field

[0001] This invention relates to the field of software testing, and more specifically to a Turing test method for artificial intelligence software. Background Technology

[0002] With the development of artificial intelligence (AI) technology, the number of AI software programs developed specifically for medical scenarios and needs is gradually increasing. Currently, medical decision support systems based on machine learning and deep learning, used to assist in diagnosis and treatment, encompass a wide range of medical scenarios, including basic medical question answering, diagnosis of specific diseases, development of virtual assistants, disease prevention, and early warning of sudden illnesses. For example, intelligent assisted diagnosis and treatment applications use AI to collect and integrate medical big data and expert knowledge, providing precise diagnosis and treatment suggestions for specific individual diseases, reducing the probability of misdiagnosis and missed diagnosis; intelligent image recognition applications can assist doctors in mining and judging the structural and functional information of human organs provided by common digital medical images, thereby evaluating the patient's health status; intelligent virtual assistant applications, based on AI speech recognition technology, can generate automatic question-and-answer systems for medical scenarios through semantic database training and knowledge base construction centered on natural language processing, providing patients with high-quality "family-style" services and reducing the burden on medical staff.

[0003] However, medical information software or artificial intelligence software needs to undergo certain evaluations before it can be truly applied in clinical practice. Some of them require medical device certification, but there is currently no unified standard for intelligent testing schemes for medical artificial intelligence software products. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a Turing test method for artificial intelligence software, comprising:

[0005] Acquire the output data of the artificial intelligence software to be tested;

[0006] The output data is compared with the data in a preset knowledge base to determine the accuracy of the results from the artificial intelligence software.

[0007] When the accuracy of the results of the artificial intelligence software is higher than a predefined accuracy threshold, calculate whether the passability of the output data of the artificial intelligence software in the preset learning model reaches the cross-entropy threshold.

[0008] If the passivity reaches the cross-entropy threshold, a clustering analysis algorithm is used to group the cases corresponding to the output data, and the ratio of manually labeled case groups is calculated.

[0009] If the ratio of manually labeled case groups is higher than a predefined Turing test threshold, the artificial intelligence software is determined to have passed the Turing test of artificial intelligence, and a test report is generated.

[0010] Preferably, determining the accuracy of the results from the artificial intelligence software further includes:

[0011] The accuracy P0 of the result is determined by comparing the judgment results of medical experts in the preset knowledge base with the judgment results of medical artificial intelligence software.

[0012] Preferably, the accuracy P0 of the result is expressed by the following formula:

[0013]

[0014] Preferably, calculating whether the passivity of the output data of the artificial intelligence software in the preset learning model reaches the cross-entropy threshold further includes:

[0015] Based on case attributes and the type of AI software, a consistency determination process is performed, and NLP analysis is used to analyze the similarity between the AI ​​software's calculation results and those of medical experts.

[0016] Preferably, the similarity between the calculation results of the artificial intelligence software and those of medical experts includes the average values ​​of the following:

[0017] The degree of consistency in diagnostic conclusions, the similarity of treatment methods, the similarity of treatment plans or treatment protocols, and the similarity of drugs or drug combinations.

[0018] Preferably, after the consistency determination, the consistency result is determined using cross-entropy, whereby the loss value of cross-entropy is defined as:

[0019] L log(y，p) =-log(pr(y|p)=-(ylog(p)+(1-y)log(1-p));

[0020] pr() is the conditional probability function.

[0021] Preferably, the Turing test method further includes:

[0022] Decision-making results based on medical data are received from multiple doctor users and the AI ​​software.

[0023] Calculate the degree of matching between the decision results;

[0024] A predefined mapping table is derived based on the calculated matching degree;

[0025] Perform a comparative test on the artificial intelligence software using the exported mapping table;

[0026] The matching degree is an ICC statistic.

[0027] Preferably, calculating the matching degree between the decision results further includes:

[0028] The matching degree between multiple doctor users is calculated; and the matching degree between doctor users and artificial intelligence software is calculated; wherein, the matching degree between doctor users is the similarity between the decision results of two doctor users, and the matching degree between doctor users and artificial intelligence software is the similarity between the decision results of doctor users and the decision results of artificial intelligence software.

[0029] Based on the same inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0030] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This invention provides a Turing test method for artificial intelligence software, comprising: acquiring the output data of the AI ​​software to be tested; comparing the output data with data in a preset knowledge base to determine the accuracy of the AI ​​software's results; when the accuracy of the AI ​​software's results is higher than a predefined accuracy threshold, calculating whether the passability of the AI ​​software's output data in a preset learning model reaches a cross-entropy threshold; if the passability reaches the cross-entropy threshold, using a clustering analysis algorithm to group cases corresponding to the output data, and calculating the ratio of manually labeled case groups; if the ratio of manually labeled case groups is higher than a predefined Turing test threshold, determining that the AI ​​software has passed the Turing test of artificial intelligence, and generating a test report. This invention provides a complete and reliable intelligent testing scheme for medical AI software products, achieving effective testing of the accuracy of AI diagnostic system judgment results. Attached Figure Description

[0033] Figure 1 This is a flowchart of the Turing test method for the artificial intelligence software of the present invention.

[0034] Figure 2 This is a logical diagram illustrating the Turing test process of the medical artificial intelligence software of the present invention.

[0035] Figure 3 This is a schematic diagram of the intelligent testing process of the medical artificial intelligence software implemented by different clients according to the present invention. Detailed Implementation

[0036] This invention, based on the Turing test method in the field of computer science and combined with the types and characteristics of medical software, establishes a testing process and method for medical artificial intelligence software. It evaluates the intelligence level of medical artificial intelligence software products through the Turing test and assesses their accuracy by combining software testing methods and processes, thereby realizing the characteristic and quality evaluation of medical artificial intelligence software.

[0037] The core idea of ​​the Turing Test is to require artificial intelligence to accept human questioning without direct physical contact and to disguise itself as human as possible. Testers ask test subjects random questions; if more than 30% of the responses cannot be distinguished between human and machine answers, then the machine passes the test and is considered to possess human-like intelligence. However, considering the precision required for medical diagnosis, the Turing Test standard for medical AI software should have more comprehensive testing criteria, possessing quantifiable, adjustable, and learnable optimization features. Furthermore, combined with software testing methods, the accuracy of case recognition results demonstrated by various medical AI software in the Turing Test should meet the accuracy standards specified in their product specifications.

[0038] Example 1:

[0039] like Figure 1 As shown, the present invention provides a Turing test method for artificial intelligence software, comprising:

[0040] S1. Obtain the output data of the artificial intelligence software to be tested;

[0041] S2. Compare the output data with the data in the preset knowledge base to determine the accuracy of the results of the artificial intelligence software;

[0042] S3. When the accuracy of the result of the artificial intelligence software is higher than the predefined accuracy threshold, calculate whether the passability of the output data of the artificial intelligence software in the preset learning model reaches the cross-entropy threshold.

[0043] S4. If the passability reaches the cross-entropy threshold, a clustering analysis algorithm is used to group the cases corresponding to the output data, and the ratio of manually labeled case groups is calculated.

[0044] S5. If the ratio of the manually labeled case group is higher than the predefined Turing test threshold, then the artificial intelligence software is determined to have passed the Turing test of artificial intelligence, and a test report is generated.

[0045] Specifically, see Figure 2The testing method in this invention includes a three-level evaluation system. Specifically, it includes a primary standard test, namely the accuracy test, which is used to determine whether the accuracy of the artificial intelligence judgment results reaches the accuracy declared in advance by the artificial intelligence software; a secondary standard test, namely the matching degree test, which is used to determine whether the artificial intelligence judgment results pass the set threshold in the integrated learning model algorithm; and a tertiary standard test, namely the intelligence level test, which is used to determine whether the case group judged by the artificial intelligence can be avoided by the expert group in a randomly sampled case pool.

[0046] In step S2, the primary standard is the "individual" judgment standard for the software product. Accuracy P0 is determined by comparing the expert judgment results with the judgment results of the medical AI software.

[0047] Accuracy is defined as shown in the following formula.

[0048]

[0049] After uploading test data, the AI ​​software should also provide relevant performance parameters for the product, such as accuracy, sensitivity, and nondeterminism. The lower limit of accuracy is generally set at 80%, and the AI ​​software will only be considered successful if its nominal accuracy reaches the preset lower limit standard P. K If the accuracy P0 of the actual judgment reaches or exceeds the given accuracy threshold of the artificial intelligence software, then the secondary standard test in step S3 is executed.

[0050] In step S3, the secondary standard uses an ensemble learning model to score the AI ​​software under test and determine its pass / fail status; this is a "quantitative" judgment standard. For example, for medical AI software, relevant case experts can be assigned based on basic data, and the matching degree X between the expert's department and the case can be generated during the assignment process. i Expert practice years Y i and expert professional titles Z i The type of hospital where the expert works is H i Important parameters such as... Subsequently, for different cases and different types of artificial intelligence software, a consistency determination process is provided. This involves analyzing the consistency of diagnostic conclusions (C1), the similarity of treatment methods (C2), the similarity of treatment plans or protocols (C3), and the similarity of drugs or drug combinations (C4) through NLP and other methods. The C... n The average value.

[0051] Weights are assigned to different quantification parameters based on the characteristics of the cases. Unnecessary parameters can have a weight of 0. The judgment results are quantified using an ensemble model, the calculation process of which is shown below.

[0052] p = W1 * X i +W2*Yi +W3*Z i +W4*H i +W5*Avg(∑C i )

[0053] Where W1, W2, W3, W4, and W5 are the weights of each item. Avg is the average value calculation. C i The value of i in the equation ranges from 1 to n.

[0054] Consistency is then determined using cross-entropy. The cross-entropy loss value is defined as:

[0055] L log(y，p) =-log(pr(y|p)=-(ylog(p)+(1-y)log(1-p));

[0056] pr() is the conditional probability function.

[0057] A value closer to 0 indicates a higher degree of match between the prediction and the actual situation, meaning the AI ​​software's judgment is more accurate; a value closer to 1 indicates a worse prediction, requiring improvement. In a preferred embodiment, when L ≤ 0.2, the AI ​​software is deemed to have passed the secondary standard. When L ≥ 0.2, the judgment result can be further recorded in a log to track its evolutionary trend.

[0058] In step S4, the Level 3 standard is a medical application of the Turing Test standard. An equal number of AI software judgments and expert judgments are randomly selected, and cluster analysis is used to group the cases. All case groups are then placed in an evaluation pool. Medical experts are randomly selected from an expert database to label each group. Labels are categorized into "Doctor" and "AI Software." When an AI software judgment case group exceeding the Turing Test threshold ratio is labeled "Doctor," the AI ​​software product is considered to have passed the Level 3 standard. In a preferred embodiment, the Turing Test threshold ratio can be 30%.

[0059] The following data from a tumor medical auxiliary decision-making system software is used as an example, combined with Figure 3 The flowchart further illustrates the implementation process of the above method.

[0060] A1. First, the overall experimental plan is developed. The Turing test method is experimentally validated by building the MAITT (Medical AI Turing Test) platform. Initially, a complete experimental plan is established by simulating the input from the vendor and expert sides of the oncology medical auxiliary decision-making system software, combined with the management side operation. The experimental workflow is planned as follows: Figure 3 As shown.

[0061] A2. A preliminary standard test for implementing auxiliary decision-making in oncology treatment, with specific steps further including:

[0062] A21. Input the parameter descriptions of the tumor medical auxiliary decision-making system software, such as a nominal accuracy rate of 80%; the accuracy definition is shown in Formula 1:

[0063]

[0064] A22. Determine whether the tumor medical auxiliary decision-making function has reached the primary standard. When the actual accuracy rate is >80% through calculation, it is determined that the primary standard test has been passed and proceed to step A3. When the actual accuracy rate is <80%, it is determined that the primary standard test has not been passed and the method ends.

[0065] A3. Perform the Level 2 standard test of the tumor medical auxiliary decision-making system, the specific steps of which further include:

[0066] A31. Medical experts manually annotate case data for which there is no gold standard.

[0067] A32. The oncology medical decision support system diagnoses case data without a gold standard and provides decision results;

[0068] A33. The experimental platform calculates the professional matching degree of each medical expert participating in the case. The calculation method considers the matching degree of department, years of practice, and professional title. The weights are set manually. For example, the initial weights of the matching degree of department, years of practice, and professional title can be set to 0.4, 0.3, and 0.3, respectively. Subsequently, for different cases and different types of artificial intelligence software, a consistency judgment process is provided. The consistency degree of diagnostic conclusions (C1), similarity of treatment methods (C2), similarity of treatment plans or treatment schemes (C3), and similarity of drugs or drug combinations (C4) are analyzed through NLP and other methods. C is then calculated. n The average value.

[0069] Weights are assigned to different quantification parameters based on the characteristics of the cases. Unnecessary parameters can have a weight of 0. The judgment results are quantified using an ensemble model, the calculation process of which is shown below.

[0070] p = W1 * X i +W2*Y i +W3*Z i +W4*H i +W5*Avg(∑C i )

[0071] Where W1, W2, W3, W4, and W5 are the weights of each item. Avg is the average value.

[0072] Consistency is then determined using cross-entropy. The cross-entropy loss value is defined as:

[0073] L log(y，p) =-log(pr(y|p)=-(ylog(p)+(1-y)log(1-p));

[0074] pr() is the conditional probability function.

[0075] A34. Using NLP processing technology, a consistency analysis is performed on the decision results of doctors and artificial intelligence software. Specifically, this includes using toolkits such as jieba for word segmentation and establishing word frequency vectors. A TF-IDF matrix is ​​then used to determine the consistency index. When the actual consistency index obtained through calculation reaches the cross-entropy threshold, the secondary standard test is considered passed, and the process proceeds to step A4; otherwise, the secondary standard test is considered failed, and the method ends.

[0076] A4. Perform the Level 3 standard test of the tumor medical auxiliary decision-making system. The specific steps further include:

[0077] A41. Cluster the case data using the k-means method, setting the parameter k=5. Within each category, randomly sample 30% of the data and divide them into 10 groups, resulting in a total of 50 groups.

[0078] A42. Obtain the decision results of the tumor medical auxiliary decision-making system software and the doctor's decision results for the 50 groups;

[0079] A43. Medical experts manually add labels to each group to determine whether the 50 groups are the result of doctors' decisions or artificial intelligence decisions.

[0080] A44. If 50% of tumor cases are grouped according to physician diagnoses, a Turing test report can be generated through a three-tiered standard test. Optionally, the Turing test report may include a description of the raw data of the tumor medical auxiliary decision-making system software, a description of the expert database composition, a description of the testing process, and a description of the test results.

[0081] In step A43, decision-making results on medical data can be received from multiple doctor users and artificial intelligence software; the matching degree between the received decision-making results is calculated; a predefined mapping table is derived based on the calculated matching degree; and a comparative test is performed on the artificial intelligence software using the derived mapping table. The matching degree is represented by an ICC statistic. The derived predefined mapping table includes the weighted average, cut-off average, or standard deviation of the matching degree.

[0082] The calculation of the matching degree includes: calculating the matching degree between doctor users; and calculating the matching degree between doctors and artificial intelligence software. The matching degree between doctor users is the similarity between the decision results of two doctor users, and the matching degree between doctors and artificial intelligence software is the similarity between the decision results of doctor users and the decision results of artificial intelligence software.

[0083] The process of exporting the mapping table includes exporting the average matching degree between doctor users and the average matching degree between doctors and artificial intelligence software. The average matching degree between doctor users is the average of the matching degrees between doctor users, and the average matching degree between doctors and artificial intelligence software is the average of the matching degrees between doctors and artificial intelligence software.

[0084] The comparative test includes performing the test on the AI ​​software by comparing the average matching degree between the doctor users with the average matching degree between the doctor and the AI ​​software. Specifically, this involves establishing a null hypothesis that the average matching degree U0 between the doctor users is less than or equal to the average matching degree U1 between the doctor and the AI ​​software, and an alternative hypothesis that the average matching degree between the doctor users is greater than the average matching degree between the doctor and the AI ​​software. When the null hypothesis is not rejected, the AI ​​software is determined to pass the comparative test.

[0085] The average matching degree U1 between the doctor and the AI ​​software is the average matching degree between the doctor and the AI ​​software, and the comparative test includes performing a comparative test on the AI ​​software by comparing the minimum matching degree between doctor users with the average matching degree U1 between the doctor and the AI ​​software.

[0086] In the comparative test, the average matching degree U1 between doctors and AI software is compared with the average matching degree between doctors and users that is corrected by adding a preset correction value to the average matching degree U1 between doctors and AI software or subtracting the preset correction value from the average matching degree U1 between doctors and AI software.

[0087] The null hypothesis is established that the average matching degree U0 between the corrected doctor users is greater than or equal to the average matching degree U1 between the doctor and the AI ​​software, and the alternative hypothesis is that the average matching degree U0 between the corrected doctor users is less than the average matching degree U1 between the doctor and the AI ​​software. Wherein, when the null hypothesis is not rejected, the AI ​​software is determined to have passed the comparative test.

[0088] In the optional Turing test criterion definition method of step S4, the Turing test criterion is predefined by a reverse Turing test generator. The reverse Turing test generator is configured to generate reverse Turing test challenges using a first parameter and a first set of answers associated with the first parameter. Preferably, each of the multiple reverse Turing test challenges can be generated by randomly selecting answers from the first set of answers.

[0089] The medical AI software is instructed to automatically attempt to solve the inverse Turing test questions; a first action loop is executed, the first action loop including: generating a first set of Turing test questions; receiving feedback from a group of medical expert users on a first subset of the first set of Turing test questions; and receiving feedback from the medical AI software on a second subset of the first set of Turing test questions; modifying the weight of the first answer in the answer group, the modification being based on feedback from the group of medical expert users and feedback from the medical AI software, the modification being used to increase the hit rate of medical expert users in solving the inverse Turing test questions and decrease the hit rate of the medical AI software in solving the inverse Turing test questions.

[0090] The first parameter, the first answer group associated with the first parameter, and the weight of the first answer are stored in a truth table, and the inverse Turing test generator is parameterized to use the information in the truth table to generate inverse Turing test challenges.

[0091] Preferably, the hit rate of the medical AI software in solving the reverse Turing test challenge is measured by determining the speed at which the medical AI software correctly identifies each character in the reverse Turing test challenge and the speed at which the medical AI software correctly solves the reverse Turing test challenge.

[0092] After modifying the weights of the first answers in the answer group, a second action loop is executed, wherein the second action loop includes generating a second set of Turing test questions and generating the second set of Turing test questions by using the modified weights to determine the frequency of the answers used when generating the second set of Turing test questions.

[0093] Example 2:

[0094] To implement the above method, the present invention also provides a Turing test system for artificial intelligence software, comprising:

[0095] The acquisition module 201 is used to acquire the output data of the artificial intelligence software to be tested;

[0096] The comparison module 202 is used to compare the output data with the data in the preset knowledge base to determine the accuracy of the results of the artificial intelligence software.

[0097] The first calculation module 203 is used to calculate whether the passability of the output data of the artificial intelligence software in the preset learning model reaches the cross-entropy threshold when the result accuracy of the artificial intelligence software is higher than the predefined accuracy threshold.

[0098] The second calculation module 204 is used to group the cases corresponding to the output data using a clustering analysis algorithm if the passivity reaches the cross-entropy threshold, and to calculate the ratio of manually labeled case groups.

[0099] The determination module 205 is used to determine if the ratio of the manually labeled case group is higher than a predefined Turing test threshold, and then determine that the artificial intelligence software has passed the Turing test of artificial intelligence and generate a test report.

[0100] The functions implemented by each functional module in this embodiment are the same as those in Embodiment 1, and will not be repeated here.

[0101] Example 3:

[0102] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0103] Example 4:

[0104] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0105] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0106] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on a plurality of computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) in which computer-usable program code is contained.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0110] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A Turing test method for artificial intelligence software, characterized in that, include: Acquire the output data of the artificial intelligence software to be tested; The output data is compared with the data in a preset knowledge base to determine the accuracy of the results from the artificial intelligence software. When the accuracy of the results of the artificial intelligence software is higher than a predefined accuracy threshold, calculate whether the passability of the output data of the artificial intelligence software in the preset learning model reaches the cross-entropy threshold. If the passivity reaches the cross-entropy threshold, a clustering analysis algorithm is used to group the cases corresponding to the output data, and the ratio of manually labeled case groups is calculated. If the ratio of the manually labeled case group is higher than the predefined Turing test threshold, then the artificial intelligence software is determined to have passed the Turing test of artificial intelligence, and a test report is generated. The step of calculating whether the passability of the output data of the artificial intelligence software in the preset learning model reaches the cross-entropy threshold further includes: performing a consistency determination process based on case attributes and the type of artificial intelligence software, and analyzing the similarity between the calculation results of the artificial intelligence software and the calculation results of medical expert users through NLP; the similarity between the calculation results of the artificial intelligence software and the medical expert includes the average value of the following items: the degree of consistency of diagnostic conclusions, the similarity of treatment methods, the similarity of treatment plans or treatment schemes, and the similarity of drugs or drug combinations; after the consistency determination, the consistency results are determined by cross-entropy. The process involves using a clustering analysis algorithm to group cases corresponding to the output data and calculating the ratio of manually labeled case groups. If the ratio of manually labeled case groups is higher than a predefined Turing test threshold, the artificial intelligence software is determined to have passed the Turing test for artificial intelligence, including: An equal number of AI software judgment results and expert judgment results are randomly selected, and cluster analysis is used to group the cases. All case groups are then placed in an evaluation pool. Medical experts are randomly selected from an expert database to label each group. The labels include doctor labels and AI software labels. When the AI ​​software judgment case groups that exceed the Turing test threshold ratio are labeled with doctor labels, the AI ​​software is considered to have passed the Turing test.

2. The method according to claim 1, characterized in that, Determining the accuracy of the results from the artificial intelligence software further includes: The accuracy P0 of the result is determined by comparing the judgment results of medical experts in the preset knowledge base with the judgment results of medical artificial intelligence software.

3. The method according to claim 2, characterized in that, The accuracy P0 of the result is expressed by the following formula:

4. The method according to claim 1, characterized in that, The Turing test method also includes: Decision-making results based on medical data are received from multiple doctor users and the AI ​​software. Calculate the degree of matching between the decision results; A predefined mapping table is derived based on the calculated matching degree; wherein, the matching degree is an ICC statistic. The exported mapping table was used to perform a comparative test on the artificial intelligence software.

5. The method according to claim 4, characterized in that, The calculation of the matching degree between the decision results further includes: The matching degree between multiple doctor users is calculated; and the matching degree between doctor users and artificial intelligence software is calculated; wherein, the matching degree between doctor users is the similarity between the decision results of two doctor users, and the matching degree between doctor users and artificial intelligence software is the similarity between the decision results of doctor users and the decision results of artificial intelligence software.

6. A terminal device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method as claimed in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 5.

Images