Safety evaluation method oriented to large medical and health model

Abstract

The invention relates to a medical and health big model-oriented security evaluation method, which comprises the following steps of: obtaining security information related to a medical and health big model, and identifying potential security risk vulnerability types; dividing potential security risk vulnerability types into common dimension data and Prompt dimension data according to a security assessment target; constructing a data set according to a division result; using the constructed data set and through the reference model, carrying out security evaluation for the medical and health large model; an evaluation result of safety evaluation is analyzed, the severity and influence of the evaluation result are evaluated, and improvement suggestions are put forward; and monitoring the medical and health big model according to the improvement suggestion, updating the prompt data set, and re-evaluating the medical and health big model. The method is reasonable in design, the prompt dimension is used for more comprehensive evaluation, the data richness is improved, the data set coverage range is wide, the evaluation credibility is enhanced, the model is used for evaluation, the subjective prejudice of manual evaluation is reduced, and the objectivity is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of model quality safety technology, and more specifically to a safety evaluation method for a large medical and health model. Background Art

[0002] With the rapid development of artificial intelligence technology, especially in the field of healthcare, the application of big models is becoming more and more extensive, including auxiliary diagnosis, medical image analysis, drug development, etc. The healthcare industry has extremely high requirements for safety and accuracy. The healthcare field involves a large amount of sensitive personal health information, including medical records, diagnoses, treatment plans, etc.; and the misuse of models can also lead to serious consequences, such as wrong diagnosis, inappropriate treatment recommendations, etc.; and it is also strictly regulated in terms of legal requirements. In medical decision-making, the stability and reliability of the model are also crucial, especially when faced with high noise and unbalanced medical data. At the same time, in order to ensure the credibility of medical decisions, the prediction results of the model need to be explainable and able to clearly show the reasoning process and decision-making basis. Therefore, it is necessary to conduct strict security evaluation of big models to ensure that their application in the field of healthcare is reliable and effective.

[0003] In order to study the above relationship, there is usually little public data in the medical and health field, and the dimensions in the medical safety field only involve potential common dimensions, and the evaluation method uses a large amount of manpower. The typical risks mainly lie in the lack of sufficient public data in the public medical and health field. The evaluation method may have certain limitations in the performance evaluation and risk identification of the model, and the credibility of the model evaluation cannot be guaranteed; only potential common dimensions are involved, resulting in the evaluation results of the model can only reflect part of the real-world usage, and the model cannot be fully evaluated; due to the sensitivity of medical and health data, the processing and analysis of data requires highly professional skills and experience, and the current stage uses a large amount of manpower, making it inefficient.

[0004] In summary, the safety assessment of large medical and health models is a hot area of ​​concern. At present, there is no high-confidence and guiding prediction method for the safety assessment of large medical and health models. How to cover multiple security dimensions in the medical and health field and create a more efficient evaluation method has become an urgent problem that needs to be solved. Summary of the invention

[0005] In view of this, the present invention provides a security assessment method for large medical and health models, which helps to discover problems and potential risks in the model through security assessment of the model in the medical and health field, so that it covers a wider range, enhances the diversity of data, and improves the robustness and interpretability of the model.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A safety evaluation method for a large medical and health model includes the following steps:

[0008] Step 1: Obtain security information related to the medical and health model, conduct risk analysis, and identify potential security risk vulnerability types of the medical and health model;

[0009] Step 2: Divide potential security risk vulnerability types into common dimension data and prompt dimension data according to security assessment objectives;

[0010] Step 3: Construct common dimension data sets and prompt dimension data sets based on the divided common dimension data and prompt dimension data;

[0011] Step 4: Use the common dimension data set and the prompt dimension data set to conduct security evaluation of the medical and health care model through the benchmark model;

[0012] Step 5: Analyze the evaluation results of the security assessment, evaluate their severity and impact, and put forward improvement suggestions;

[0013] Step 6: Monitor the healthcare model based on the improvement suggestions, update the prompt dataset, and re-evaluate the healthcare model.

[0014] Furthermore, the specific implementation method of step 1 is:

[0015] First, safety information of the medical and health care model is collected through literature research, expert interviews or safety reports;

[0016] Then, the above security information is identified through NLP technology to obtain the potential security risk vulnerability types of the medical and health model, including personal privacy leakage, medical counterfactuals, toxic ethics, fairness and bias, patient right to know, drug contraindications, medical ethics and legal provisions, uncertainty, command hijacking, role playing, reverse induction, entering developer mode, adversarial attacks, weak semantic attacks, random noise attacks, and DAN.

[0017] Furthermore, the specific implementation method of step 2 is:

[0018] Divide the potential security risk vulnerabilities obtained in step 1 into common dimensions and prompt dimensions according to the security assessment objectives

[0019] Among them, the common dimensions include: personal privacy disclosure, medical counterfactuals, toxic ethics, fairness and bias, patient right to know, drug contraindications, medical ethics and legal provisions, and uncertainty handling;

[0020] The prompt dimensions include: command hijacking, role playing, reverse induction, entering developer mode, adversarial attack, weak semantic attack, random noise attack, and DAN.

[0021] Furthermore, the specific implementation method of step 3 is:

[0022] By using public medical data sets, online medical information, and medical books, and using the Apriori algorithm to filter the common dimension data mentioned in step 2, the common dimension data set is finally generated for storage;

[0023] Build your own dataset for prompt dimension data: Write prompts. Each time you write, select one of the prompt dimensions mentioned in step 2. The format of each metadata is: {"input": prompt, "type": type}. Put the written prompt into the prompt dimension dataset.

[0024] Furthermore, it also includes screening and verifying each dimension data in the common dimension data set and the prompt dimension data set based on the expert scores to improve the signal-to-noise ratio of the data set.

[0025] Furthermore, the specific implementation method of step 4 is:

[0026] Common dimensions are evaluated using an automatic evaluation framework and accuracy is used as an indicator for evaluation;

[0027] The prompt dimension uses a benchmark model to evaluate the security of other medical models, score and rank the answers generated by other medical models, and generate a security evaluation report. The benchmark model uses GPT-3.5.

[0028] Furthermore, the specific implementation method of step 5 is: according to the evaluation results, rankings and security evaluation reports, the security vulnerabilities found in the evaluation process are analyzed, corresponding improvement suggestions for the security dimensions where the vulnerabilities exist are put forward, the priorities of the security vulnerabilities are formulated, and the corresponding improvement suggestions are implemented according to the priorities.

[0029] Furthermore, the specific implementation method of step 6 is:

[0030] Based on the evaluation results and improvement suggestions, corresponding improvement measures will be implemented step by step according to the priority;

[0031] Establish monitoring feedback for security vulnerabilities encountered by users during use. According to the model monitoring feedback results, add new prompt data in accordance with step 3, update the data, and form a dynamic data set;

[0032] Then re-evaluate through step 4 to determine whether security vulnerabilities exist, improve the model based on the problems found in the vulnerabilities, and update the improved medical and health model.

[0033] It can be seen from the above technical solution that compared with the prior art, the advantages and positive effects of the present invention are:

[0034] 1. The present invention is reasonably designed and uses prompt dimensions for more comprehensive evaluation, which improves the richness of data. The data set covers a wide range, enhances the credibility of the evaluation, and uses models for evaluation to reduce subjective bias in human evaluation and enhance objectivity.

[0035] 2. The present invention not only adopts public data sets in the medical and health field, but also can better reflect the use of models in the real world through medical books and relevant information provided by doctors; the data is manually screened by experts to ensure high-quality data, and prompt is used as evaluation data, which can be applied to different scenarios and tasks to enhance data authenticity; it not only involves common dimensions, but also expands it to prompt dimensions, so that it covers a wide range and more comprehensively evaluates the security of the model; using the benchmark model to evaluate other models can reduce human subjective bias and reduce human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 It is the overall processing flow chart provided by the present invention;

[0038] Figure 2 This is a flowchart of constructing a prompt dimension data set in step 3 provided by the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The design concept of the present invention is to evaluate the large model in the field of medical and health care, construct relevant data sets of common dimensions and prompt dimensions, clean and annotate the data sets, and then evaluate the large model in the field of medical and health care through the benchmark model, and finally generate an evaluation report. The present invention does not consider model improvement and is based on the following two assumptions: (1) There are some public data in the field of medical and health care, and there are hospitals that allow the use of some real data. Experts and data contributors can provide prompt construction (2) The benchmark model is selected through the existing general model evaluation method. Therefore, the present invention constructs a dataset of prompt dimensions through data contributors and experts, and then evaluates other models through the benchmark model, which increases the coverage and authenticity of the data and improves the credibility of the model security evaluation.

[0041] Based on the above design concept, the safety evaluation method for the medical and health model of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0042] Step 1: Conduct risk analysis on the healthcare big model to identify potential security risks and vulnerabilities.

[0043] In this step, we first collect security information of the medical and health model through research literature, expert interviews or security reports; then, we use NLP technology to identify the above security information and obtain aspects involved in potential security risk vulnerabilities of the medical and health model, including personal privacy leakage, medical counterfactuals, toxic ethics, fairness and bias, patient right to know, drug contraindications, medical ethics and legal provisions, uncertainty, command hijacking, role playing, reverse induction, entering developer mode, adversarial attacks, weak semantic attacks, random noise attacks, and DAN.

[0044] Each identified security risk vulnerability is evaluated through subsequent steps to determine the severity and likelihood of each security risk vulnerability.

[0045] Step 2: Establish common dimensions and prompt dimensions based on risk vulnerabilities.

[0046] In this step, based on the understanding of the model and relevant regulations, the security assessment goals of the large medical and health model are clarified. Its security assessment goals mainly include common dimensions and prompt dimensions. Common dimensions include: personal privacy leakage, medical counterfactuals, toxic ethics, fairness and bias, patient right to know, drug contraindications, medical ethics and legal provisions, and uncertainty processing; prompt dimensions include: command hijacking, role playing, reverse induction, entering developer mode, adversarial attacks, weak semantic attacks, random noise attacks, and DAN; these dimensions cover data issues, real-world issues, legal and regulatory issues, and access role issues, covering a wide range of security assessments.

[0047] In the present invention, the evaluation scope includes common dimensions and prompt dimensions, and each dimension covers a wide range, which can more realistically and comprehensively evaluate the quality and robustness of the model.

[0048] Step 3: Collect relevant information of the medical and health model for common dimensions and prompt dimensions to build a data set.

[0049] In this step, according to the security assessment goal of the medical and health model, data of the dimensions mentioned in step 2 are collected through public medical data sets, online medical information, medical books, and relevant data provided by doctors. The relevant data provided by doctors can cover real-world medical data samples, including various diseases, symptoms, and medical records, to reflect the real medical environment and situation;

[0050] When constructing a common dimension dataset, the public medical dataset and network-related medical information use the Apriori algorithm to filter the dimensions mentioned in step 2, and store the final generated dataset in N1;

[0051] Manually screen the relevant data provided by medical books and doctors, build them into data set N2 through Spark distributed data processing, and finally use automated data preprocessing tools to clean and evaluate the data quality, which can effectively identify and process errors, inconsistencies, and missing values ​​in the data set, thereby improving the accuracy and reliability of the data; at the same time, build a data set for the prompt dimension, write prompts 100 times, select one of the prompt dimensions mentioned in step 2 each time, and write 10 prompts. The format of each metadata is: {"input": prompt, "type": type}, and put the collected prompts into N1;

[0052] Afterwards, 10 experts manually screened N1. Each expert selected 2 dimensions for scoring to ensure that each piece of data was scored by 5 experts. If the average score was higher than 8 points, it was valid data. Finally, 100 prompts were screened out to ensure that there were differences between the models. The expert screening process can identify and eliminate data noise that may cause misunderstanding or interfere with model learning, thereby improving the signal-to-noise ratio of the data set.

[0053] Step 3 process reference Figure 2 In the present invention, the construction of common dimension data sets uses distributed data processing to improve the efficiency of data processing; using data contributors and experts to write prompts can obtain high-quality data and make it easier to discover loopholes in models in the medical and health field.

[0054] Step 4: Use two dimensional data sets to conduct security evaluation of large medical and health models through benchmark models.

[0055] In this step, according to the characteristics of common dimensions, prompt dimensions and data sets, corresponding security testing plans are designed. Common dimensions are evaluated using an automatic evaluation framework and accuracy is used as an evaluation indicator. The prompt dimension uses a benchmark model to evaluate the security of other medical models. The benchmark model uses GPT-3.5 and executes the prompt dimension mentioned in step 2 for security testing. The benchmark model scores and ranks the answers generated by other medical models, analyzes and organizes the evaluation results, and generates a security evaluation report.

[0056] Step 5: Use the evaluation results to analyze and assess their severity and impact, and make improvement recommendations.

[0057] In this step, based on the evaluation results, rankings, and security assessment reports, the security vulnerabilities discovered during the evaluation process are analyzed, corresponding improvement suggestions for the security dimensions where the vulnerabilities exist are proposed, priorities for the security vulnerabilities are set, and corresponding improvement suggestions are implemented based on the priorities; setting priorities makes resource allocation more efficient and improves the scientific nature of decision-making.

[0058] Step 6: Monitor the model based on the improvement suggestions. Improve the prompt dataset and re-evaluate the model.

[0059] In this step, according to the evaluation results and improvement suggestions, corresponding improvement measures are implemented step by step according to the priority; monitoring feedback is established for security vulnerabilities encountered during user use, and according to the model monitoring feedback results, prompts are rewritten on the data set in accordance with step 3 to form a dynamic data set, which can reduce redundancy and improve the applicability and flexibility of the data; then re-evaluate through step 4 to determine whether security vulnerabilities exist, improve the model based on the problems of the vulnerabilities, and evaluate the effect and security of the updated model in step 4; by continuously updating the security data set, ensure that all data sources remain synchronized and consistent, and the security assessment of the large medical and health model is more credible.

[0060] The present invention applies the evaluation to the medical and health field, but in other embodiments, the prompt dimensions involved can be expanded to other fields. At the same time, the present invention uses the benchmark model to evaluate other models and can also involve other fields.

[0061] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0062] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A safety evaluation method for a large medical and health model, characterized in that: The following steps are involved: Step 1: Obtain security information related to the medical and health model, conduct risk analysis, and identify potential security risk vulnerability types of the medical and health model; Step 2: Divide potential security risk vulnerability types into common dimension data and prompt dimension data according to security assessment objectives; Step 3: Construct common dimension data sets and prompt dimension data sets based on the divided common dimension data and prompt dimension data; Step 4: Use the common dimension data set and the prompt dimension data set to conduct security evaluation of the medical and health care model through the benchmark model; Step 5: Analyze the evaluation results of the security assessment, evaluate their severity and impact, and put forward improvement suggestions; Step 6: Monitor the healthcare model based on the improvement suggestions, update the prompt dataset, and re-evaluate the healthcare model.

2. The safety evaluation method for a large medical and health model according to claim 1 is characterized in that: The specific implementation method of step 1 is: First, safety information of the medical and health care model is collected through literature research, expert interviews or safety reports; Then, the above security information is identified through NLP technology to obtain the potential security risk vulnerability types of the medical and health model, including personal privacy leakage, medical counterfactuals, toxic ethics, fairness and bias, patient right to know, drug contraindications, medical ethics and legal provisions, uncertainty, command hijacking, role playing, reverse induction, entering developer mode, adversarial attacks, weak semantic attacks, random noise attacks, and DAN.

3. The safety evaluation method for the medical and health care large model according to claim 1 is characterized in that: The specific implementation method of step 2 is: Divide the potential security risk vulnerabilities obtained in step 1 into common dimensions and prompt dimensions according to the security assessment objectives Among them, the common dimensions include: personal privacy disclosure, medical counterfactuals, toxic ethics, fairness and bias, patient right to know, drug contraindications, medical ethics and legal provisions, and uncertainty handling; The prompt dimensions include: command hijacking, role playing, reverse induction, entering developer mode, adversarial attack, weak semantic attack, random noise attack, and DAN.

4. The safety evaluation method for a large medical and health model according to claim 1 is characterized in that: The specific implementation method of step 3 is: By using public medical data sets, online medical information, and medical books, and using the Apriori algorithm to filter the common dimension data mentioned in step 2, the common dimension data set is finally generated for storage; Build your own dataset for prompt dimension data: Write prompts. Each time you write, select one of the prompt dimensions mentioned in step 2. The format of each metadata is: {"input": prompt, "type": type}. Put the written prompt into the prompt dimension dataset.

5. The safety evaluation method for a large medical and health model according to claim 4 is characterized in that: It also includes screening and verifying each dimension data in the common dimension data set and the prompt dimension data set based on expert scores to improve the signal-to-noise ratio of the data set.

6. The safety evaluation method for the medical and health care large model according to claim 1 is characterized in that: The specific implementation method of step 4 is: Common dimensions are evaluated using an automatic evaluation framework and accuracy is used as an indicator for evaluation; The prompt dimension uses a benchmark model to evaluate the security of other medical models, score and rank the answers generated by other medical models, and generate a security evaluation report. The benchmark model uses GPT-3.

5.

7. The safety evaluation method for the medical and health care large model according to claim 1 is characterized in that: The specific implementation method of step 5 is: according to the evaluation results, rankings and security evaluation reports, analyze the security vulnerabilities found in the evaluation process, put forward corresponding improvement suggestions for the security dimensions where the vulnerabilities exist, set priorities for the security vulnerabilities, and implement the corresponding improvement suggestions according to the priorities.

8. The safety evaluation method for a large medical and health model according to claim 1 is characterized in that: The specific implementation method of step 6 is: Based on the evaluation results and improvement suggestions, corresponding improvement measures will be implemented step by step according to the priority; Establish monitoring feedback for security vulnerabilities encountered by users during use. According to the model monitoring feedback results, add new prompt data in accordance with step 3, update the data, and form a dynamic data set; Then re-evaluate through step 4 to determine whether security vulnerabilities exist, improve the model based on the problems found in the vulnerabilities, and update the improved medical and health model.

Images