Methods and devices for defending against harmful responses in medical large language models

Abstract

This invention discloses a method and apparatus for defending against harmful responses using a medical large-scale language model, relating to the field of natural language processing. The method includes: acquiring a user's input statement and inputting it into a trained rewriting model to output the rewritten statement; if retrieval enhancement generation is required, searching the medical knowledge base for the rewritten statement, concatenating the rewritten statement with each medical knowledge fragment, and inputting it into the trained medical large-scale language model to generate a response statement; calculating a security score for each response statement based on the medical knowledge fragments to obtain a first security score; determining whether there exists a first security score greater than a security threshold; if so, selecting the response statement corresponding to the highest first security score as the response statement; otherwise, filtering and determining the response statement based on the filtering results; inputting the response statement into a trained judgment model to output response statements that do not contain harmful content. This invention solves the problem of difficulty in effectively identifying and filtering input statements.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more specifically to a method and apparatus for defending against harmful responses in medical large language models. Background Technology

[0002] Currently, medical language models are generally trained on authoritative medical textbooks, platform data, clinical guidelines, and expert experience during pre-training and fine-tuning. They generally lack specific defense mechanisms against harmful content, leading to significant security vulnerabilities when facing malicious input or offensive operations. Specifically, due to the lack of systematic coverage and guided learning of potentially harmful information in the training data, the models struggle to accurately identify and defend against some carefully designed attacks. For example, prompt injection can induce the model to bypass security restrictions through specific inputs, outputting content that violates ethical or medical standards. Furthermore, data poisoning attacks can inject biased or erroneous information into a pre-built medical knowledge base, subtly influencing the model's judgment logic and output results.

[0003] With the rapid development of online medical service platforms, an increasing number of users are submitting consultation requests to intelligent medical systems via the internet to obtain convenient and efficient health consultation services. However, in practical applications, users may submit questions containing inappropriate, malicious, or illegal content, such as questions involving illegal drug use, false disease diagnoses, or leaks of personal privacy. This can not only mislead medical language models but also potentially lead to information leaks, legal risks, and even social harm. Therefore, how to effectively identify and filter user input while protecting user privacy, and how to perform security checks before generating responses, has become a pressing technical challenge. Summary of the Invention

[0004] The purpose of this application is to propose a method and device for defending against harmful responses in a medical large language model, addressing the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a method for defending against harmful responses in medical large language models, comprising the following steps:

[0006] Construct and train a rewriting model and a judgment model based on a large language model to obtain trained rewriting model and trained judgment model. The trained rewriting model contains semantically preserved prompt words related to the rewriting task, and the trained judgment model contains prompt words related to the harmful content identification task.

[0007] The system acquires the user's input statement and combines it with semantically relevant prompts to input into the trained rewriting model, outputting the rewritten statement corresponding to the input statement.

[0008] A medical knowledge base is constructed. In response to the determination that retrieval enhancement generation is required, the rewritten statement corresponding to the input statement is retrieved from the medical knowledge base to obtain several medical knowledge fragments. The rewritten statement corresponding to the input statement is concatenated with each medical knowledge fragment and input into the trained medical big language model to generate the response statement corresponding to each medical knowledge fragment.

[0009] The system calculates a safety score for each response statement by combining relevant medical knowledge fragments, resulting in a first safety score for each response statement. It then determines whether there is at least one response statement whose first safety score is greater than a safety threshold. If so, the response statement with the highest first safety score among the at least one response statement is selected as the reply statement. Otherwise, all response statements are filtered, and the reply statement is obtained by combining the response statement with a safety warning statement, selecting a response statement that meets the conditions, or aggregating responses.

[0010] The response statement is combined with prompts related to the harmful content identification task and input into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the response statement is that there is no harmful content, the response statement is output.

[0011] As a preferred approach, a security score is calculated for each response statement by combining relevant medical knowledge fragments, resulting in a first security score for each response statement, specifically including:

[0012] Construct a predefined set of medical safety keywords;

[0013] For each response statement, check whether it contains any security keywords from the predefined medical security keyword set. If it does, mark the response statement as a safe response and determine the security keywords contained in the response statement and their corresponding position weights. The position weights are determined based on the position of the security keywords contained in the response statement within the response statement. If it does not contain any security keywords, mark the response statement as a non-safe response. The first security score of a response statement marked as a non-safe response is 0.

[0014] For response statements marked as safe responses, the confidence score of the response statement and the source credibility score of the medical knowledge fragment are determined based on the response statement and the medical knowledge fragment, respectively. The first safety score of the response statement is then calculated based on the confidence score of the response statement, the source credibility score of the medical knowledge fragment, and the position weight, as shown in the following formula:

[0015] ;

[0016] in, This represents the j-th response statement. The first safety rating, This represents the j-th response statement. The confidence score, This represents the j-th response statement. The credibility score of the source of the medical knowledge fragments. This represents the maximum value of the positional weights of the security keywords contained in the response statement.

[0017] As a preferred option, the j-th response statement The process of obtaining the confidence score is as follows:

[0018] Obtain the j-th response statement output by the trained medical big language model. The probability of the nth word is calculated, and the average probability of all words is taken to obtain the jth response statement. The confidence score is shown in the following formula:

[0019] ;

[0020] in, This represents the j-th response statement. The probability of the nth word in the sequence. This represents the j-th response statement. Total word count;

[0021] The j-th response statement The process of obtaining the source credibility score for medical knowledge fragments is as follows:

[0022] Establish a credibility grading system that includes the source and its corresponding source credibility score, starting from the j-th response statement. Extracting corresponding source data from medical knowledge fragments, matching the source data with sources in the credibility grading system, and determining the j-th response statement. The credibility score of the source of the medical knowledge fragments.

[0023] As a preferred approach, all response statements are filtered, and based on the filtering results, response statements are combined with security warning statements to form reply statements, response statements that meet the criteria are selected as reply statements, or reply statements are obtained through reply aggregation. Specifically, this includes:

[0024] The number of valid responses, the filtering threshold ratio, and the upper limit of the filtering threshold are determined, and the filtering threshold is calculated as follows: ;

[0025] in, Indicates the number of valid responses. Indicates the filtering threshold ratio. Indicates the upper limit of the filtering threshold. Indicates the filtering threshold. This indicates taking the minimum value;

[0026] From all response statements, those with a first security score greater than or equal to the filtering threshold are used to generate a security candidate response set R. * As shown in the following formula: R * = {r j | score1(r j ) ≥ μ};

[0027] In response to determining the set of security candidate responses R * If empty, then select the response statement with the highest security score among all response statements and combine it with the security warning statement as the reply statement;

[0028] In response to determining the set of security candidate responses R * If not empty, then according to the safe candidate response set R * The second security score is calculated based on the confidence score of the response statement and the source credibility score of the medical knowledge fragment, as shown in the following formula:

[0029] ;

[0030] in, Represents the set of safe candidate responses R * The second security score of the i-th response statement in the data. Represents the set of safe candidate responses R * The confidence score of the i-th response statement in the dataset. Represents the set of safe candidate responses R * The source credibility score of the medical knowledge fragment corresponding to the i-th response statement in the text;

[0031] Determine the safe candidate response set R * The highest security score A and the second highest score B in the response statement;

[0032] In response to the condition that A > 1.5 × B is met, the set of safe candidate responses R corresponding to the highest score is determined. * The response statement in the code is used as the reply statement;

[0033] If the condition A > 1.5 × B is not met, then for the safe candidate response set R... * All response statements in the process undergo response aggregation, which is as follows:

[0034] The safety candidate response set R is computed using a pre-trained medical-specific embedding model.* The vector representation of each response statement in the dataset is used to compute the set of secure candidate responses R. * The second highest security score response statement and the security candidate response set R * The semantic similarity between the vector representations of other response statements in the dataset is used to select a safe candidate response set R. * The response statement with the highest second-highest security score, along with other response statements whose semantic similarity is greater than or equal to a similarity threshold, constitutes the response subset R. sim ;

[0035] Response to determining a subset R of responses sim If the number of response statements in the response set is greater than or equal to the quantity threshold, then the response subset R will be... sim All response statements are concatenated into a long text. The long text is combined with semantically relevant prompts and input into the trained rewriting model. The rewritten statement corresponding to the long text is output as the response statement.

[0036] Response to determining a subset R of responses sim If the number of response statements in the set is less than the quantity threshold, then the security candidate response set R will be... * The response statement with the highest second-highest security score is used as the reply statement.

[0037] As a preferred option, it also includes:

[0038] In response to the determination that no retrieval enhancement generation is required, the rewritten statement corresponding to the input statement is input into the trained medical large language model to generate the response statement.

[0039] As a preferred option, it also includes:

[0040] If the classification result corresponding to the reply statement is determined to contain harmful content, the process of refusing to answer will begin.

[0041] Secondly, the present invention provides a harmful response defense device for medical large language models, comprising:

[0042] The model building module is configured to build and train a rewriting model and a judgment model based on a large language model, resulting in a trained rewriting model and a trained judgment model. The trained rewriting model contains semantically preserved prompt words related to the rewriting task, and the trained judgment model contains prompt words related to the harmful content identification task.

[0043] The rewriting module is configured to acquire the user's input statement and combine it with semantically relevant prompts for the rewriting task to input into the trained rewriting model, and output the rewritten statement corresponding to the input statement.

[0044] The retrieval enhancement generation module is configured to build a medical knowledge base. In response to the determination that retrieval enhancement generation is needed, the rewritten statement corresponding to the input statement is retrieved from the medical knowledge base to obtain several medical knowledge fragments. The rewritten statement corresponding to the input statement is concatenated with each medical knowledge fragment and input into the trained medical big language model to generate the response statement corresponding to each medical knowledge fragment.

[0045] The response statement generation module is configured to calculate a safety score for each response statement by combining relevant medical knowledge fragments, and obtain a first safety score for each response statement; determine whether there is at least one response statement whose first safety score is greater than a safety threshold. If so, the response statement with the highest first safety score among at least one response statement is used as the response statement; otherwise, all response statements are filtered, and the response statement is combined with a safety warning statement as the response statement based on the filtering results, or a response statement is obtained by selecting a response statement that meets the conditions, or by aggregating responses.

[0046] The harmful content identification module is configured to combine the reply statement with prompts related to the harmful content identification task and input them into a trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

[0047] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0048] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0049] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) The harmful response defense method for medical large language model proposed in this invention destroys the attack structure by semantic preservation rewriting at the input end and detects harmful content at the output end, forming a complete security closed loop. The rewriting model effectively identifies and destroys prompt word attacks, maintains medical intent and prevents the model from being induced to output illegal content, thereby achieving accurate identification and effective interception of potential harmful content in medical consultation scenarios.

[0052] (2) The harmful response defense method proposed in this invention for medical big language model proposes a medical RAG security aggregation process to ensure that key security warnings are output first, reduce the risk of database poisoning through a scoring mechanism, and improve the accuracy of medical advice.

[0053] (3) The harmful response defense method proposed in this invention for medical big language model has a clear structure, complete process, and highly scalable harmful question defense mechanism. Moreover, the parameters can be adjusted to adapt to different actual needs and meet the security requirements of various medical scenarios. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating a method for defending against harmful responses based on a medical large language model, as an embodiment of this application.

[0056] Figure 2 This is a schematic diagram of a harmful response defense device for a medical large language model, as an embodiment of this application.

[0057] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0059] Figure 1 The present application illustrates an embodiment of a harmful response defense method for a medical large language model, comprising the following steps:

[0060] S1. Construct and train a rewriting model and a judgment model based on a large language model to obtain a trained rewriting model and a trained judgment model. The trained rewriting model contains semantically preserved prompt words related to the rewriting task, and the trained judgment model contains prompt words related to the harmful content identification task.

[0061] Specifically, the trained rewriting model used in the embodiments of this application is obtained by pre-training a large language model using a medical domain-specific vocabulary, and then fine-tuning it on a semantically preserving rewriting dataset. During the fine-tuning process, a semantically preserving rewriting task is employed to construct relevant prompt words, guiding the pre-trained rewriting model to map the input statement to the rewritten statement while maintaining semantic similarity. This trained rewriting model possesses semantically preserving rewriting capabilities, enabling it to change the structure of the input statement while retaining the user's true medical intent. Semantic preservation refers to preserving the medical intent and key information of the input statement to the greatest extent possible during the rewriting process. For example, if the user's input statement is "I've been having headaches lately, could it be a brain tumor?", the rewritten statement might be "I've been experiencing frequent headaches recently, could this be related to a brain tumor?". Therefore, the semantics remain intact before and after rewriting, but the structure has changed, effectively preventing potential adversarial attacks.

[0062] Similarly, the trained judgment model used in the embodiments of this application is pre-trained on a large language model using a medical domain-specific vocabulary, and then fine-tuned using a dataset based on a large medical domain corpus, normal medical questions, and harmful content including illegal drug recommendations, false disease diagnoses, personal attacks, inappropriate remarks, and inappropriate treatment suggestions, which can effectively detect harmful content. The fine-tuning process of the judgment model includes:

[0063] (a) Construct a sample of harmful content, including illegal drug recommendations, false disease diagnoses, personal attacks, inappropriate remarks, and inappropriate treatment suggestions, to form an initial dataset;

[0064] (b) Expand the initial dataset using data augmentation techniques to obtain an expanded dataset;

[0065] (c) During the fine-tuning process, construct prompt words related to the harmful content identification task, fine-tune the pre-trained large language model using the expanded dataset, train the harmful content classification task, and perform harmful content classification.

[0066] A trained judgment model can be invoked to determine the harmfulness of the response statement and whether it contains harmful content.

[0067] S2, obtains the user's input statement and combines it with semantically relevant prompts for the rewriting task to input into the trained rewriting model, and outputs the rewritten statement corresponding to the input statement.

[0068] Specifically, the system receives medical-related input statements from end users. These statements can be in text format and include descriptions of patient symptoms or medication inquiries. In one specific embodiment of this application, the user submits their questions about their health status or requests for professional advice in text form. This input statement can trigger all subsequent processing flows.

[0069] First, the user's input statement is fed into the trained rewriting model. Then, combined with prompts related to the semantic preservation rewriting task, the trained rewriting model is used to perform semantic preservation rewriting on the input statement, resulting in the rewritten statement corresponding to the input statement.

[0070] S3. Construct a medical knowledge base. In response to the determination that retrieval enhancement generation is required, the rewritten statement corresponding to the input statement is retrieved from the medical knowledge base to obtain several medical knowledge fragments. The rewritten statement corresponding to the input statement is concatenated with each medical knowledge fragment and input into the trained medical big language model to generate the response statement corresponding to each medical knowledge fragment.

[0071] S4. Calculate the safety score for each response statement by combining the corresponding medical knowledge fragments to obtain the first safety score for each response statement; determine whether there is at least one response statement whose first safety score is greater than the safety threshold. If so, take the response statement with the highest first safety score among at least one response statement as the reply statement; otherwise, filter all response statements and, based on the filtering results, combine the response statement with the safety warning statement as the reply statement, select the response statement that meets the conditions as the reply statement, or obtain the reply statement through reply aggregation.

[0072] In a specific embodiment, a security score is calculated for each response statement by combining relevant medical knowledge fragments, resulting in a first security score for each response statement, which specifically includes:

[0073] Construct a predefined set of medical safety keywords;

[0074] For each response statement, check whether it contains any security keywords from the predefined medical security keyword set. If it does, mark the response statement as a safe response and determine the security keywords contained in the response statement and their corresponding position weights. The position weights are determined based on the position of the security keywords contained in the response statement within the response statement. If it does not contain any security keywords, mark the response statement as a non-safe response. The first security score of a response statement marked as a non-safe response is 0.

[0075] For response statements marked as safe responses, the confidence score of the response statement and the source credibility score of the medical knowledge fragment are determined based on the response statement and the medical knowledge fragment, respectively. The first safety score of the response statement is then calculated based on the confidence score of the response statement, the source credibility score of the medical knowledge fragment, and the position weight, as shown in the following formula:

[0076] ;

[0077] in, This represents the j-th response statement. The first safety rating, This represents the j-th response statement. The confidence score, This represents the j-th response statement. The credibility score of the sources of the included medical knowledge fragments. This represents the maximum value of the positional weights of the security keywords contained in the response statement.

[0078] In a specific embodiment, the j-th response statement The process of obtaining the confidence score is as follows:

[0079] Obtain the j-th response statement output by the trained medical big language model. The probability of the nth word is calculated, and the average probability of all words is taken to obtain the jth response statement. The confidence score is shown in the following formula:

[0080] ;

[0081] in, This represents the j-th response statement. The probability of the nth word in the sequence. This represents the j-th response statement. Total word count;

[0082] The j-th response statement The process of obtaining the source credibility score for medical knowledge fragments is as follows:

[0083] Establish a credibility grading system that includes the source and its corresponding source credibility score, starting from the j-th response statement. Extracting corresponding source data from medical knowledge fragments, matching the source data with sources in the credibility grading system, and determining the j-th response statement. The credibility score of the source of the medical knowledge fragments.

[0084] In a specific embodiment, all response statements are filtered, and based on the filtering results, response statements are combined with security warning statements to form reply statements, response statements that meet the conditions are selected as reply statements, or reply statements are obtained through reply aggregation. Specifically, this includes:

[0085] The number of valid responses, the filtering threshold ratio, and the upper limit of the filtering threshold are determined, and the filtering threshold is calculated as follows: ;

[0086] in, Indicates the number of valid responses. Indicates the filtering threshold ratio. Indicates the upper limit of the filtering threshold. Indicates the filtering threshold. This indicates taking the minimum value;

[0087] From all response statements, those with a first security score greater than or equal to the filtering threshold are used to generate a security candidate response set R. * As shown in the following formula: R * = {r j | score1(r j ) ≥ μ};

[0088] In response to determining the set of security candidate responses R * If empty, then select the response statement with the highest security score among all response statements and combine it with the security warning statement as the reply statement;

[0089] In response to determining the set of security candidate responses R * If not empty, then according to the safe candidate response set R * The second security score is calculated based on the confidence score of the response statement and the source credibility score of the medical knowledge fragment, as shown in the following formula:

[0090] ;

[0091] in, Represents the set of safe candidate responses R * The second security score of the i-th response statement in the data. Represents the set of safe candidate responses R * The confidence score of the i-th response statement in the dataset. Represents the set of safe candidate responses R * The source credibility score of the medical knowledge fragment corresponding to the i-th response statement in the text;

[0092] Determine the safe candidate response set R * The highest security score A and the second highest score B in the response statement;

[0093] In response to the condition that A > 1.5 × B is met, the set of safe candidate responses R corresponding to the highest score is determined. * The response statement in the code is used as the reply statement;

[0094] If the condition A > 1.5 × B is not met, then for the safe candidate response set R... * All response statements in the process undergo response aggregation, which is as follows:

[0095] The safety candidate response set R is computed using a pre-trained medical-specific embedding model. * The vector representation of each response statement in the dataset is used to compute the set of secure candidate responses R. * The second highest security score response statement and the security candidate response set R * The semantic similarity between the vector representations of other response statements in the dataset is used to select a safe candidate response set R. * The response statement with the highest second-highest security score, along with other response statements whose semantic similarity is greater than or equal to a similarity threshold, constitutes the response subset R. sim ;

[0096] Response to determining a subset R of responses sim If the number of response statements in the response set is greater than or equal to the quantity threshold, then the response subset R will be... sim All response statements are concatenated into a long text. The long text is combined with semantically relevant prompts and input into the trained rewriting model. The rewritten statement corresponding to the long text is output as the response statement.

[0097] Response to determining a subset R of responses sim If the number of response statements in the set is less than the quantity threshold, then the security candidate response set R will be... * The response statement with the highest second-highest security score is used as the reply statement.

[0098] Specifically, if the Search Enhancement Generation (RAG) function needs to be enabled in actual use, in order to prevent database poisoning from affecting the response results—for example, if a medical literature on "childhood fever" is found to be compromised, the medical big data language model may give an incorrect response—this application's embodiments employ a medical search enhancement generation secure aggregation process to improve the security and reliability of the final response. This process includes the following stages:

[0099] Phase 1:

[0100] Based on the actual situation, parameter initialization is performed. In one embodiment, the parameter is set as: safety threshold τ. safe= 0.6; Filtering threshold ratio α = 0.5; Filtering threshold upper limit β = 4; Probability threshold η = 0.1. In other embodiments, each parameter can be adjusted according to the actual effect.

[0101] Phase Two:

[0102] At this stage, ensure that critical safety warnings are not overwhelmed by any misinformation, adhering to the safety-first principle. Medical knowledge isolation and generation: Search the rewritten statement q in a medical knowledge base, retrieving several medical knowledge fragments p. j This will rewrite statement q and each retrieved medical knowledge fragment p. j Independently invokes a trained medical big data language model to generate response statements r j As shown in the following formula: r j = LLM(q⊕p j ), ⊕ indicates splicing.

[0103] Calculate the first security score: Check the response statement r j Does it contain safety keywords from a predefined set of medical safety keywords, S={“prohibited”, “contraindicated”, “danger”, “risk”, “warning”, “adverse reaction”, “use with caution”, “not suitable”, “avoid”, “contraindicated”, “risk”, “warning”, “adverse reaction”, “not recommended”, “should not”, “avoid”, “caution”, “hazard”, “precaution”}?

[0104] For each response statement, check if it contains any security keywords from S; if so, mark it as a secure response and record the response statement r marked as a secure response. j The security keywords included j and its position weight w p,j If it is not included, it is marked as a non-secure response. In one embodiment, if the security keyword is located in the response statement r j The first sentence is w p,j =1.0; if the security keyword is located in the response statement r j The words and phrases, then w p,j =0.8, if the security keyword is located in the response statement r j The following sentence is w p,j =0.6.

[0105] For each response statement r marked as a safe response j Calculate its first safety score The first security score for responses marked as non-secure (not containing security keywords in S) is set to 0.

[0106] Among them, the confidence score of the response statement (c j The process of obtaining ) is as follows:

[0107] a. Call the probability prediction interface of the trained medical language model to obtain the probability distribution vector of each word in the process of generating the response statement; when the model generates the response statement r j = [t1, t2, ..., t N When ], for each word t n Record the response statement r output by the medical big language model trained when the word was generated. j The probability v of the nth word jn [t n The probability represents the degree of certainty that a trained medical big language model will choose the word in a given context.

[0108] b. Calculate the confidence score: Calculate the average probability of all words in the entire response statement generation process to obtain the confidence score of the response statement. Correct responses from authoritative sources usually have higher confidence scores because the medical big language model is more certain about these contents; while malicious content may increase the uncertainty when the model generates the response statement, resulting in lower confidence scores. The confidence score reflects the overall certainty of the medical big language model when generating the response statement.

[0109] The source trust score of medical knowledge fragments (source_trust(p) j The process of obtaining )) is as follows:

[0110] a. Establish a credibility rating system: Predefine the source and its corresponding source credibility score (source_trust(p)). j The standards are shown in Table 1.

[0111] Table 1

[0112]

[0113] b. Extract source data: from the retrieved medical knowledge fragments p j Extract source data such as source URL or document source identifier.

[0114] c. Matching and Scoring: The extracted source data is matched with a preset credibility rating system to obtain the corresponding source credibility score (source_trust(p)). j ), to participate in the calculation process of the first safety score.

[0115] If there exists at least one response statement whose first security score satisfies: score1(r j ) ≥ τ safe Then select the response statement r with the highest first security score. * As a response statement. If the first security score of all response statements satisfies score(r) j ) < τ safe Then proceed to stage three.

[0116] Phase Three:

[0117] Filtering candidate response sets:

[0118] a. Determine the number of valid responses : = Satisfying "I don't know" is not in r j The number of response statements;

[0119] b. Calculate the filtering threshold: ,when When the value is large, the filtering threshold is dominated by the upper limit β of the filtering threshold; when... When the value is small, reduce the filtering threshold from the upper limit β to... This is to avoid filtering out all safe keywords.

[0120] c. Generate a set of safe candidate responses R * :R * = {r j | score1(r j ) ≥ μ};

[0121] d. If R * If empty, then select the response statement with the highest security score from all response statements. * Concatenate with a security warning statement and use it as a response statement: r * ⊕“Security warning: The correctness of this response has not been verified.”

[0122] Response selection strategy:

[0123] For the security candidate response set R * For each response statement in the process, a second security score is calculated, and the highest score A and the second highest score are selected.

[0124] Selection criteria: A > 1.5 × B;

[0125] If the above selection criteria are met, then select the safe candidate response set R. * The response statement with the second-highest security score is r. *If the selection criteria are not met, the response aggregate is executed as a response statement.

[0126] The response aggregation process is as follows:

[0127] The safety candidate response set R is computed using a pre-trained medical-specific embedding model (such as BioBERT). * The vector representation of each response statement. Calculate the set of secure candidate responses R. * The semantic similarity between the vector representation of the response statement with the highest second-highest security score and the vector representations of other response statements is calculated. In one embodiment, the semantic similarity can be measured using cosine similarity, with a similarity threshold set to 0.7. When the similarity threshold is set to 0.7, erroneous merging of irrelevant response statements can be avoided. Therefore, a subset R of responses with a semantic similarity ≥ 0.7 is selected. sim The quantity threshold is set to 2. If the response subset R sim If the number of response statements in the subset is greater than or equal to 2, then all response statements in the subset are concatenated into a long text, denoted as {combined_responses}. A trained rewriting model is then used to rewrite this long text, generating the corresponding rewritten statements. In one embodiment, the prompt for the rewriting task can be constructed as: Please generate a concise, accurate, and professional medical suggestion based on the following authoritative medical information: {combined_responses}.

[0128] The embodiments of this application reduce the risk of database poisoning through a medical RAG security aggregation process. This design ensures the absolute priority of critical security information. Only when it is confirmed that there are no security warnings that need to be communicated first will the regular suggestion generation process be initiated, thus ensuring that valuable medical advice can still be provided even when there are no security warnings.

[0129] In specific embodiments, it also includes:

[0130] In response to the determination that no retrieval enhancement generation is required, the rewritten statement corresponding to the input statement is input into the trained medical large language model to generate the response statement.

[0131] Specifically, without enabling the search enhancement generation function, a locally deployed, trained medical language model can be directly invoked. This trained medical language model is trained based on high-quality data such as authoritative medical textbooks, clinical guidelines, and expert experience, possessing strong medical logical reasoning and professional expression capabilities. The trained medical language model receives rewritten statements processed by the rewriting model and generates response statements that conform to professional medical common sense.

[0132] S5 combines the reply statement with the prompt words related to the harmful content identification task and inputs them into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

[0133] In specific embodiments, it also includes:

[0134] If the classification result corresponding to the reply statement is determined to contain harmful content, the process of refusing to answer will begin.

[0135] Specifically, before outputting the response, a trained judgment model needs to review its content to ensure it contains no harmful information. In practice, carefully designed prompts transform the harmful content identification task into a natural language understanding and classification problem. By constructing prompts relevant to the harmful content identification task, the judgment model comprehensively evaluates the response to determine if it contains content that violates laws, regulations, ethics, or platform security policies. If the judgment indicates the presence of harmful content, the response is rejected, with the option to output a statement such as, "Sorry, I cannot answer this question." Otherwise, subsequent steps continue. If the review confirms the response contains no harmful content, it is output and provided to the user.

[0136] Further reference Figure 2 As an implementation of the methods shown in the above figures, this application provides an embodiment of a harmful response defense device for a large medical language model. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0137] This application provides a harmful response defense device for medical large language models, including:

[0138] Model building module 1 is configured to build and train a rewriting model and a judgment model based on a large language model to obtain a trained rewriting model and a trained judgment model. The trained rewriting model contains semantically preserved prompt words related to the rewriting task, and the trained judgment model contains prompt words related to the harmful content identification task.

[0139] Rewriting module 2 is configured to acquire the user's input statement and combine it with semantically relevant prompts for the rewriting task to input into the trained rewriting model, and output the rewritten statement corresponding to the input statement.

[0140] The retrieval enhancement generation module 3 is configured to build a medical knowledge base. In response to the determination that retrieval enhancement generation is needed, the rewritten statement corresponding to the input statement is retrieved from the medical knowledge base to obtain several medical knowledge fragments. The rewritten statement corresponding to the input statement is concatenated with each medical knowledge fragment and input into the trained medical big language model to generate the response statement corresponding to each medical knowledge fragment.

[0141] The response statement generation module 4 is configured to calculate a safety score for each response statement by combining it with relevant medical knowledge fragments, and obtain a first safety score for each response statement; determine whether there is at least one response statement whose first safety score is greater than a safety threshold. If so, the response statement with the highest first safety score among at least one response statement is used as the response statement; otherwise, all response statements are filtered, and the response statement is combined with a safety warning statement as the response statement based on the filtering results, or a response statement is obtained by selecting a response statement that meets the conditions, or by aggregating responses.

[0142] The harmful content identification module 5 is configured to combine the reply statement with the prompt words related to the harmful content identification task and input them into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

[0143] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 3 As shown, the electronic device in this embodiment includes a processor 301 and a memory 302; wherein the memory 302 is used to store computer execution instructions; and the processor 301 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0144] Alternatively, the memory 302 can be either standalone or integrated with the processor 301.

[0145] When the memory 302 is set up independently, the electronic device also includes a bus 303 for connecting the memory 302 and the processor 301.

[0146] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 301, implement the above method.

[0147] This invention also provides a computer program product, including a computer program that, when executed by a processor 301, implements the above-described method.

[0148] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0149] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0150] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0151] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 301 to execute some steps of the methods of the various embodiments of this application.

[0152] It should be understood that the processor 301 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 301 can be any conventional processor 301. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 301 executing the steps, or as a combination of hardware and software modules within the processor 301 executing the steps.

[0153] The memory 302 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0154] Bus 303 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 303 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 303 in the accompanying drawings of this application is not limited to only one bus 303 or one type of bus 303.

[0155] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0156] An exemplary storage medium is coupled to a processor 301, enabling the processor 301 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 301. The processor 301 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 301 and the storage medium can exist as discrete components in an electronic device or a host device.

[0157] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for defending against harmful responses in a medical large language model, characterized in that, Includes the following steps: Construct and train a rewriting model and a judgment model based on a large language model to obtain a trained rewriting model and a trained judgment model. The trained rewriting model contains semantically preserved prompt words related to the rewriting task, and the trained judgment model contains prompt words related to the harmful content identification task. The user's input statement is obtained and combined with semantically relevant prompts for the rewriting task to be input into the trained rewriting model, and the rewritten statement corresponding to the input statement is output. A medical knowledge base is constructed. In response to the determination that retrieval enhancement generation is needed, the rewritten statement corresponding to the input statement is retrieved from the medical knowledge base to obtain several medical knowledge fragments. The rewritten statement corresponding to the input statement is concatenated with each medical knowledge fragment and input into the trained medical big language model to generate a response statement corresponding to each medical knowledge fragment. By combining relevant medical knowledge fragments, a security score is calculated for each response statement to obtain the first security score for each response statement; Determine whether there is at least one response statement whose first security score is greater than the security threshold. If so, take the response statement with the highest first security score among the at least one response statements as the reply statement. Otherwise, filter all response statements and combine the response statement with the security warning statement as the reply statement based on the filtering results, select the response statement that meets the conditions as the reply statement, or obtain the reply statement through reply aggregation. The reply statement is combined with the prompt words related to the harmful content identification task and input into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

2. The method for defending against harmful responses based on a large medical language model according to claim 1, characterized in that, A security score is calculated for each response statement by combining relevant medical knowledge fragments, resulting in a first security score for each response statement, specifically including: Construct a predefined set of medical safety keywords; For each response statement, check whether it contains any of the predefined medical safety keyword set. If it does, mark the response statement as a safe response and determine the safety keywords contained in the response statement and their corresponding position weights. The position weights are determined based on the position of the safety keywords contained in the response statement within the response statement. If it does not contain any of the safety keywords, mark the response statement as a non-safe response. The first safety score of a response statement marked as a non-safe response is 0. For response statements marked as safe responses, a confidence score for the response statement and a source credibility score for the medical knowledge fragment are determined based on the response statement and the medical knowledge fragment, respectively. A first security score for the response statement is then calculated based on the confidence score of the response statement, the source credibility score of the medical knowledge fragment, and the position weight, as shown in the following formula: ； in, This represents the j-th response statement. The first safety rating, This represents the j-th response statement. The confidence score, This represents the j-th response statement. The credibility score of the source of the medical knowledge fragments. This represents the maximum value of the positional weights of the security keywords contained in the response statement.

3. The method for defending against harmful responses in a large medical language model according to claim 2, characterized in that, The j-th response statement The process of obtaining the confidence score is as follows: The trained medical big language model outputs the j-th response statement. The probability of the nth word is calculated, and the average probability of all words is taken to obtain the jth response statement. The confidence score is shown in the following formula: ； in, This represents the j-th response statement. The probability of the nth word in the sequence. This represents the j-th response statement. Total word count; The j-th response statement The process of obtaining the source credibility score for medical knowledge fragments is as follows: Establish a credibility grading system that includes the source and its corresponding source credibility score, starting from the j-th response statement. Extracting corresponding source data from medical knowledge fragments, matching the source data with the sources in the credibility grading system, and determining the j-th response statement. The credibility score of the source of the medical knowledge fragments.

4. The method for defending against harmful responses based on a large medical language model according to claim 1, characterized in that, All response statements are filtered, and based on the filtering results, response statements are combined with security warning statements to form reply statements, response statements that meet the criteria are selected as reply statements, or reply statements are aggregated to obtain reply statements. Specifically, this includes: The number of valid responses, the filtering threshold ratio, and the upper limit of the filtering threshold are determined, and the filtering threshold is calculated as follows: ; in, Indicates the number of valid responses. Indicates the filtering threshold ratio. Indicates the upper limit of the filtering threshold. Indicates the filtering threshold. This indicates taking the minimum value; From all response statements, those with a first security score greater than or equal to the filtering threshold are used to generate a security candidate response set R. * As shown in the following formula: R * = {r j | score1(r j ) ≥ μ}; In response to determining the security candidate response set R * If empty, then select the response statement with the highest security score among all response statements and combine it with the security warning statement as the reply statement; In response to determining the security candidate response set R * If not empty, then according to the security candidate response set R * The second security score is calculated based on the confidence score of the response statement and the source credibility score of the medical knowledge fragment, as shown in the following formula: ； in, Represents the security candidate response set R * The second security score of the i-th response statement in the data. Represents the security candidate response set R * The confidence score of the i-th response statement in the dataset. Represents the security candidate response set R * The source credibility score of the medical knowledge fragment corresponding to the i-th response statement in the text; Determine the security candidate response set R * The highest security score A and the second highest score B in the response statement; In response to the condition that A > 1.5 × B is met, the set of security candidate responses R corresponding to the highest score is determined. * The response statement in the code is used as the reply statement; In response to the determination that the condition A > 1.5 × B is not met, then for the security candidate response set R... * All response statements in the process undergo response aggregation, which is as follows: The safety candidate response set R is computed using a pre-trained medical-specific embedding model. * The vector representation of each response statement in the dataset is used to compute the security candidate response set R. * The response statement with the highest second security score in the security candidate response set R * The semantic similarity between the vector representations of other response statements in the dataset is used to select the safe candidate response set R. * The response statement with the highest second-highest security score, along with other response statements whose semantic similarity is greater than or equal to a similarity threshold, constitutes the response subset R. sim ; In response to determining the subset of responses R sim If the number of response statements in the response set is greater than or equal to the quantity threshold, then the response subset R will be... sim All response statements are concatenated into a long text. The long text is combined with the prompt words related to the semantic rewriting task and input into the trained rewriting model. The rewritten statement corresponding to the long text is output as the response statement. In response to determining the subset of responses R sim If the number of response statements in the set is less than the quantity threshold, then the security candidate response set R will be removed from the list. * The response statement with the highest second-highest security score is used as the reply statement.

5. The method for defending against harmful responses in a medical large language model according to claim 1, characterized in that, Also includes: In response to the determination that no retrieval enhancement generation is required, the rewritten statement corresponding to the input statement is input into the trained medical large language model to generate a response statement.

6. The method for defending against harmful responses based on a large medical language model according to claim 1, characterized in that, Also includes: If the classification result corresponding to the reply statement is determined to contain harmful content, the process of refusing to answer is initiated.

7. A harmful response defense device for a medical large language model, characterized in that, include: The model building module is configured to build and train a rewriting model and a judgment model based on a large language model to obtain a trained rewriting model and a trained judgment model. The trained rewriting model contains semantically preserved prompt words related to the rewriting task, and the trained judgment model contains prompt words related to the harmful content identification task. The rewriting module is configured to acquire the user's input statement and combine it with semantically relevant prompts for the rewriting task to input into the trained rewriting model, and output the rewritten statement corresponding to the input statement. The retrieval enhancement generation module is configured to build a medical knowledge base. In response to the determination that retrieval enhancement generation is needed, the rewritten statement corresponding to the input statement is retrieved from the medical knowledge base to obtain several medical knowledge fragments. The rewritten statement corresponding to the input statement is concatenated with each medical knowledge fragment and input into the trained medical big language model to generate a response statement corresponding to each medical knowledge fragment. The response statement generation module is configured to combine relevant medical knowledge fragments to calculate a security score for each response statement, thereby obtaining a first security score for each response statement. Determine whether there is at least one response statement whose first security score is greater than the security threshold. If so, take the response statement with the highest first security score among the at least one response statements as the reply statement. Otherwise, filter all response statements and combine the response statement with the security warning statement as the reply statement based on the filtering results, select the response statement that meets the conditions as the reply statement, or obtain the reply statement through reply aggregation. The harmful content identification module is configured to combine the reply statement with the prompt words related to the harmful content identification task and input them into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images