Harmful reply defense method and device for medical big language model

Abstract

The invention discloses a medical big language model-oriented harmful reply defense method and device, and relates to the field of natural language processing, and the method comprises the steps: obtaining an input statement of a user, inputting the input statement into a trained rewriting model, and outputting a rewriting statement; if retrieval enhancement generation needs to be carried out, retrieving the rewritten statement in a medical knowledge base, splicing the rewritten statement and each medical knowledge fragment, and inputting the spliced rewritten statement and each medical knowledge fragment into a trained medical big language model to generate a response statement; performing security score calculation on each response statement in combination with the medical knowledge fragments to obtain a first security score; judging whether the first security score greater than a security threshold exists, if so, taking the response statement corresponding to the highest first security score as a reply statement, otherwise, performing screening, and determining the reply statement according to a screening result; and inputting the reply statement into the trained judgment model, and outputting the reply statement without the harmful content. The problem that it is difficult to effectively recognize and filter input statements is solved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method and device for defending against harmful replies in a large medical language model. Background Art

[0002] Currently, large medical language models are typically pre-trained and fine-tuned based on data from authoritative medical textbooks, platform data, clinical guidelines, and expert experience. These models generally lack defense mechanisms specifically against harmful content, which poses significant security risks when faced with malicious input or offensive operations. Specifically, due to the lack of systematic coverage and guided learning of potentially harmful information in the training data, the models struggle to accurately identify and defend against some carefully designed attack vectors. For example, prompt injection attacks can trick the model into bypassing security restrictions through specific inputs, resulting in output that violates ethical or medical standards. Furthermore, database poisoning attacks can inject biased or erroneous information into pre-built medical knowledge bases, subtly influencing the model's judgment logic and output.

[0003] With the rapid development of online medical service platforms, more and more users are submitting consultation requests to intelligent medical systems via the internet, seeking convenient and efficient health consultation services. However, in practice, users may ask questions containing inappropriate, malicious, or illegal content, such as questions about illegal drug use, false disease diagnoses, or the disclosure of personal privacy. This not only misleads medical large language models but can also lead to information leakage, legal risks, and even social harm. Therefore, how to effectively identify and filter user input sentences while protecting user privacy, and perform security verification before generating responses, has become a technical challenge that needs to be solved. Summary of the Invention

[0004] The purpose of this application is to propose a harmful reply defense method and device for medical large language models to address the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for defending against harmful replies in a large medical language model, comprising the following steps:

[0006] Constructing and training a rewriting model and a judgment model based on a large language model to obtain a trained rewriting model and a trained judgment model, wherein the trained rewriting model includes prompt words related to the semantic preservation rewriting task, and the trained judgment model includes prompt words related to the harmful content identification task;

[0007] Obtaining the user's input sentence and combining it with the prompt words related to the semantic preservation rewriting task to input into the trained rewriting model, and outputting the rewritten sentence corresponding to the input sentence;

[0008] A medical knowledge base is constructed. In response to determining that search enhancement generation is required, the rewritten sentence corresponding to the input sentence is searched in the medical knowledge base to obtain a plurality of medical knowledge fragments. The rewritten sentence corresponding to the input sentence is concatenated with each medical knowledge fragment and input into a trained medical language model to generate a response sentence corresponding to each medical knowledge fragment.

[0009] Calculate a safety score for each response statement in combination with the corresponding medical knowledge fragment to obtain a first safety score for each response statement; determine whether there is at least one response statement with a first safety score greater than a safety threshold; if so, use the response statement corresponding to the highest first safety score among the at least one response statement as the reply statement; otherwise, screen all response statements and, based on the screening results, combine the response statement with the security warning statement as the reply statement, select a qualified response statement as the reply statement, or obtain a reply statement through reply aggregation;

[0010] The reply statement is combined with prompt words related to the harmful content identification task and input into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

[0011] Preferably, a security score is calculated for each response statement in combination with the corresponding medical knowledge fragment to obtain a first security score for each response statement, specifically including:

[0012] Construct a set of predefined medical safety keywords;

[0013] Each response statement is checked to see if it contains any safety keywords from the predefined medical safety keyword set. If so, the response statement is marked as a safe response, and the safety keywords contained in the response statement and their corresponding position weights are determined. The position weights are determined based on the positions of the safety keywords contained in the response statement. If not, the response statement is marked as an unsafe response, and the first safety score of the response statement marked as an unsafe response is 0.

[0014] For the response statement marked as a safe response, the confidence score of the response statement and the source credibility score of the medical knowledge fragment are determined based on the response statement and the medical knowledge fragment respectively; the first safety score of the response statement is calculated based on the confidence score of the response statement, the source credibility score of the medical knowledge fragment and the position weight, as shown in the following formula:

[0015] ;

[0016] in, Indicates the jth response statement The first safety score, Indicates the jth response statement The confidence score of Indicates the jth response statement The source credibility score of the medical knowledge fragment, Indicates the maximum value among the position weights of the security keywords contained in the response sentence.

[0017] As a preference, the jth response statement The process of obtaining the confidence score is as follows:

[0018] Get the jth response statement output by the trained medical language model The probability of the nth word in the sentence and the average probability of all words are calculated to get the jth response statement. The confidence score is as follows:

[0019] ;

[0020] in, Indicates the jth response statement The probability of the nth word in , Indicates the jth response statement Total number of words;

[0021] The jth response statement The process of obtaining the source credibility score of the medical knowledge fragment is as follows:

[0022] Establish a credibility grading system including sources and their corresponding credibility scores, starting from the jth response statement Extract the corresponding source data from the medical knowledge fragment, match the source data with the source in the credibility grading system, and determine the jth response statement Source credibility scores of medical knowledge fragments.

[0023] Preferably, all response statements are screened, and according to the screening results, the response statement is combined with the security warning statement as a reply statement, a response statement that meets the conditions is selected as a reply statement, or a reply statement is obtained through reply aggregation, specifically including:

[0024] Determine the number of valid responses, the filtering threshold ratio, and the upper limit of the filtering threshold and calculate the filtering threshold as shown in the following formula: ;

[0025] in, Indicates the number of valid responses, Indicates the filtering threshold ratio, Indicates the upper limit of the filtering threshold. Indicates the filtering threshold, Indicates taking the minimum value;

[0026] Among all the response statements, the response statements whose first security score is greater than or equal to the filtering threshold are used to generate a safe candidate response set R. * , as shown below: R * = {r j | score1(r j ) ≥ μ};

[0027] In response to determining the safe candidate response set R * If it is empty, the response statement with the highest security score among all the response statements is selected and combined with the security warning statement as the reply statement;

[0028] In response to determining the safe candidate response set R * is not empty, then according to the safe candidate response set R * The second security score is calculated based on the confidence score of the response statement and the source credibility score of the medical knowledge fragment, as shown in the following formula:

[0029] ;

[0030] in, Denotes the safe candidate response set R * The second safety score of the i-th response statement in, Denotes the safe candidate response set R * The confidence score of the i-th response statement in , Denotes the safe candidate response set R * The source credibility score of the medical knowledge fragment corresponding to the i-th response statement in ;

[0031] Determine the safe candidate response set R * The highest score A and the second highest score B of the second safety score of the response statement in;

[0032] In response to determining that the condition is met: A > 1.5 × B, the safe candidate response set R corresponding to the highest score is * The response statement in is used as a reply statement;

[0033] In response to determining that the condition A > 1.5 × B is not satisfied, the safe candidate response set R * All response statements in the response aggregation are executed. The response aggregation process is as follows:

[0034] Compute the safe candidate response set R using a pre-trained medical-specific embedding model* The vector representation of each response statement in , calculate the safe candidate response set R * The second highest safety score response statement in the safe candidate response set R * The semantic similarity between the vector representations of other response sentences in , and select the safe candidate response set R * The response statement with the second highest security score and the response statement with a semantic similarity greater than or equal to the similarity threshold constitute the response subset R sim ;

[0035] In response to determining the response subset R sim If the number of response statements in is greater than or equal to the quantity threshold, the response subset R sim All the response sentences in the sentence are concatenated into a long text, the long text is combined with the prompt words related to the semantic preservation rewriting task and input into the trained rewriting model, and the rewritten sentence corresponding to the long text is output as the reply sentence;

[0036] In response to determining the response subset R sim If the number of response statements in is less than the threshold, the safe candidate response set R * The second highest safety score response statement in is used as the reply statement.

[0037] As an option, it also includes:

[0038] In response to determining that retrieval enhancement generation is not required, a rewritten sentence corresponding to the input sentence is input into a trained medical large language model to generate a reply sentence.

[0039] As an advantage, it also includes:

[0040] In response to determining that the classification result corresponding to the reply statement is that harmful content exists, the refusal to answer process is entered.

[0041] In a second aspect, the present invention provides a harmful reply defense device for a large medical language model, comprising:

[0042] a model construction module configured to construct and train a rewriting model and a judgment model based on the large language model to obtain a trained rewriting model and a trained judgment model, wherein the trained rewriting model includes prompt words related to the semantic preservation rewriting task, and the trained judgment model includes prompt words related to the harmful content identification task;

[0043] a rewriting module configured to obtain a user's input sentence and combine it with a prompt word related to the semantic-preserving rewriting task to input the sentence into a trained rewriting model, and output a rewritten sentence corresponding to the input sentence;

[0044] The retrieval enhancement generation module is configured to construct a medical knowledge base. In response to determining that retrieval enhancement generation is required, the medical knowledge base is searched for a rewritten sentence corresponding to the input sentence to obtain a plurality of medical knowledge fragments. The rewritten sentence corresponding to the input sentence is concatenated with each medical knowledge fragment and input into a trained medical large language model to generate a response sentence corresponding to each medical knowledge fragment.

[0045] The reply statement generation module is configured to calculate a security score for each response statement in combination with the corresponding medical knowledge fragment to obtain a first security score for each response statement; determine whether there is at least one response statement whose first security score is greater than a security threshold; if so, use the response statement corresponding to the at least one response statement with the highest first security score as the reply statement; otherwise, screen all response statements and, based on the screening results, combine the response statement with the security warning statement as the reply statement, select a response statement that meets the conditions as the reply statement, or obtain a reply statement through reply aggregation;

[0046] The harmful content identification module is configured to combine the reply statement with the prompt words related to the harmful content identification task and input them into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

[0047] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0049] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the method described in any implementation manner in the first aspect when the computer program is executed by a processor.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) The harmful reply defense method for medical large language models proposed in this invention destroys the attack structure through semantic-preserving rewriting at the input end and detects harmful content at the output end, forming a complete security closed loop. The rewriting model effectively identifies and destroys prompt word attacks, maintains medical intent, and prevents the model from being induced to output illegal content, thereby achieving accurate identification and effective interception of potentially harmful content in medical consultation scenarios.

[0052] (2) The harmful reply defense method for medical large language models proposed in this invention proposes a medical RAG security aggregation process to ensure that key security warnings are output first, reduce the risk of database poisoning through a scoring mechanism, and improve the accuracy of medical recommendations.

[0053] (3) The harmful reply defense method for the medical large language model proposed in the present invention has a clear structure, complete process, and a highly scalable harmful question defense mechanism. Its parameters are adjustable to adapt to different actual needs and meet the security requirements of various medical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A flowchart of a harmful reply defense method for a large medical language model according to an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a harmful reply defense device for a large medical language model according to an embodiment of the present application;

[0057] Figure 3 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0059] Figure 1 The embodiment of the present application provides a method for defending against harmful replies in a large medical language model, including the following steps:

[0060] S1. Construct and train a rewriting model and a judgment model based on a large language model to obtain a trained rewriting model and a trained judgment model. The trained rewriting model contains prompt words related to the semantic preservation rewriting task, and the trained judgment model contains prompt words related to the harmful content identification task.

[0061] Specifically, the trained rewriting model used in the embodiments of the present application is obtained by pre-training a large language model using a special vocabulary in the medical field, and then fine-tuning it on a semantically preserved rewriting dataset. During the fine-tuning process, a semantically preserved rewriting task is used to construct prompt words related to the semantically preserved rewriting task, guiding the pre-trained rewriting model to map the input sentence to the rewritten sentence to maintain semantic similarity. The trained rewriting model has the ability to preserve semantically preserved rewriting and can change the structure of the input sentence while retaining the user's true medical intention. Semantic preservation means that the medical intention and key information of the input sentence are retained to the greatest extent during the rewriting process. For example, if the user's input sentence is "I have been having headaches recently. Could it be a brain tumor?", the rewritten sentence may be "I have had frequent headaches recently. Could this be related to a brain tumor?" Therefore, the semantics remain intact before and after the rewriting, but the structure has changed, effectively destroying possible adversarial attacks.

[0062] Similarly, the trained judgment model used in the embodiments of this application is pre-trained using a medical-specific vocabulary to train a large language model. The large language model is then fine-tuned using a large amount of medical corpus, normal medical questions, and a dataset of harmful content including illegal drug recommendations, false disease diagnoses, personal attacks, inappropriate speech, inappropriate treatment suggestions, etc., which can effectively detect harmful content. The fine-tuning process of the judgment model includes:

[0063] (a) Constructing a sample of harmful content including illegal drug recommendations, false disease diagnoses, personal attacks, inappropriate speech, and inappropriate treatment suggestions to form an initial dataset;

[0064] (b) Expand the initial dataset through data augmentation technology to obtain the expanded dataset;

[0065] (c) During the fine-tuning process, we construct prompt words related to the harmful content identification task, fine-tune the pre-trained large language model using the expanded dataset, train the harmful content classification task, and perform harmful content classification.

[0066] The trained judgment model can be called to judge the harmfulness of the reply statement to determine whether it contains harmful content.

[0067] S2, obtaining the user's input sentence and combining it with the prompt words related to the semantic preservation rewriting task to input it into the trained rewriting model, and outputting the rewritten sentence corresponding to the input sentence.

[0068] Specifically, a medical-related input statement is received from a terminal user. The input statement may be in text format and may include a description of the patient's symptoms or medication advice. In one embodiment of the present application, the user submits a question regarding their health condition or requests professional advice in text format. This input statement may trigger all subsequent processing flows.

[0069] First, the user's input sentence is input into the trained rewriting model. Combined with the prompt words related to the semantic preservation rewriting task, the trained rewriting model is used to perform semantic preservation rewriting on the input sentence to obtain the rewritten sentence corresponding to the input sentence.

[0070] S3, build a medical knowledge base. In response to determining that retrieval enhancement generation is required, the rewritten sentence corresponding to the input sentence is searched in the medical knowledge base to obtain several medical knowledge fragments; the rewritten sentence corresponding to the input sentence is spliced ​​with each medical knowledge fragment and input into the trained medical large language model to generate a response sentence corresponding to each medical knowledge fragment.

[0071] S4. Calculate the security score of each response statement in combination with the corresponding medical knowledge fragment to obtain the first security score of each response statement; determine whether there is at least one response statement with a first security score greater than the security threshold. If so, use the response statement corresponding to the highest first security score in at least one response statement as the reply statement; otherwise, screen all response statements, and according to the screening results, combine the response statement with the security warning statement as the reply statement, select the qualified response statement as the reply statement, or obtain the reply statement through reply aggregation.

[0072] In a specific embodiment, a security score is calculated for each response statement in combination with the corresponding medical knowledge fragment to obtain a first security score for each response statement, specifically including:

[0073] Construct a set of predefined medical safety keywords;

[0074] Each response statement is checked to see if it contains any safety keywords from the predefined medical safety keyword set. If so, the response statement is marked as a safe response, and the safety keywords contained in the response statement and their corresponding position weights are determined. The position weights are determined based on the positions of the safety keywords contained in the response statement. If not, the response statement is marked as an unsafe response, and the first safety score of the response statement marked as an unsafe response is 0.

[0075] For the response statement marked as a safe response, the confidence score of the response statement and the source credibility score of the medical knowledge fragment are determined based on the response statement and the medical knowledge fragment respectively; the first safety score of the response statement is calculated based on the confidence score of the response statement, the source credibility score of the medical knowledge fragment and the position weight, as shown in the following formula:

[0076] ;

[0077] in, Indicates the jth response statement The first safety score, Indicates the jth response statement The confidence score of Indicates the jth response statement The source credibility score of the included medical knowledge fragments, Indicates the maximum value among the position weights of the security keywords contained in the response sentence.

[0078] In a specific embodiment, the jth response statement The process of obtaining the confidence score is as follows:

[0079] Get the jth response statement output by the trained medical language model The probability of the nth word in the sentence and the average probability of all words are calculated to get the jth response statement. The confidence score is as follows:

[0080] ;

[0081] in, Indicates the jth response statement The probability of the nth word in , Indicates the jth response statement Total number of words;

[0082] The jth response statement The process of obtaining the source credibility score of the medical knowledge fragment is as follows:

[0083] Establish a credibility grading system including sources and their corresponding credibility scores, starting from the jth response statement Extract the corresponding source data from the medical knowledge fragment, match the source data with the source in the credibility grading system, and determine the jth response statement Source credibility scores of medical knowledge fragments.

[0084] In a specific embodiment, all response statements are screened, and based on the screening results, the response statements are combined with the security warning statements as reply statements, the response statements that meet the conditions are selected as reply statements, or the reply statements are obtained through reply aggregation, specifically including:

[0085] Determine the number of valid responses, the filtering threshold ratio, and the upper limit of the filtering threshold and calculate the filtering threshold as shown in the following formula: ;

[0086] in, Indicates the number of valid responses, Indicates the filtering threshold ratio, Indicates the upper limit of the filtering threshold. Indicates the filtering threshold, Indicates taking the minimum value;

[0087] Among all the response statements, the response statements whose first security score is greater than or equal to the filtering threshold are used to generate a safe candidate response set R. * , as shown below: R * = {r j | score1(r j ) ≥ μ};

[0088] In response to determining the safe candidate response set R * If it is empty, the response statement with the highest security score among all the response statements is selected and combined with the security warning statement as the reply statement;

[0089] In response to determining the safe candidate response set R * is not empty, then according to the safe candidate response set R * The second security score is calculated based on the confidence score of the response statement and the source credibility score of the medical knowledge fragment, as shown in the following formula:

[0090] ;

[0091] in, Denotes the safe candidate response set R * The second safety score of the i-th response statement in, Denotes the safe candidate response set R * The confidence score of the i-th response statement in , Denotes the safe candidate response set R * The source credibility score of the medical knowledge fragment corresponding to the i-th response statement in ;

[0092] Determine the safe candidate response set R * The highest score A and the second highest score B of the second safety score of the response statement in;

[0093] In response to determining that the condition is met: A > 1.5 × B, the safe candidate response set R corresponding to the highest score is * The response statement in is used as a reply statement;

[0094] In response to determining that the condition A > 1.5 × B is not satisfied, the safe candidate response set R * All response statements in the response aggregation are executed. The response aggregation process is as follows:

[0095] Compute the safe candidate response set R using a pre-trained medical-specific embedding model * The vector representation of each response statement in , calculate the safe candidate response set R * The second highest safety score response statement in the safe candidate response set R * The semantic similarity between the vector representations of other response sentences in , and select the safe candidate response set R * The response statement with the second highest security score and the response statement with a semantic similarity greater than or equal to the similarity threshold constitute the response subset R sim ;

[0096] In response to determining the response subset R sim If the number of response statements in is greater than or equal to the quantity threshold, the response subset R sim All the response sentences in the sentence are concatenated into a long text, the long text is combined with the prompt words related to the semantic preservation rewriting task and input into the trained rewriting model, and the rewritten sentence corresponding to the long text is output as the reply sentence;

[0097] In response to determining the response subset R sim If the number of response statements in is less than the threshold, the safe candidate response set R * The second highest safety score response statement in is used as the reply statement.

[0098] Specifically, if the Retrieval Enhancement Generation (RAG) function is required in actual use, to prevent database poisoning from affecting the response results, for example, if a medical literature entry related to "children's fever" is found to have been attacked, the medical language model may give an incorrect response. To improve the security and reliability of the final response, the embodiment of this application adopts a medical retrieval enhancement generation security aggregation process. This process includes the following stages:

[0099] Phase 1:

[0100] Based on the actual situation, the parameters are initialized. In one embodiment, the parameters are set to: safety threshold τ safe= 0.6; filtering threshold ratio α = 0.5; filtering threshold upper limit β = 4; probability threshold η = 0.1. In other embodiments, each parameter can be adjusted according to actual effects.

[0101] Phase 2:

[0102] At this stage, we ensure that key safety warnings are not overwhelmed by any error messages, and implement the safety first principle. Medical knowledge isolation generation: The rewritten statement q is searched in the medical knowledge base, and several medical knowledge fragments p are retrieved. j , will rewrite the sentence q and each retrieved medical knowledge fragment p j , independently call the trained medical language model to generate the response sentence r j , as shown below: j = LLM(q⊕p j ), ⊕ indicates splicing.

[0103] Calculating the first security score: Checking the response statement r j Whether it contains safety keywords in the predefined medical safety keyword set S, S = {"prohibited", "contraindicated", "dangerous", "risk", "warning", "adverse reaction", "use with caution", "not suitable", "avoid", "contraindicated", "risk", "warning", "adverse reaction", "not recommended", "should not", "avoid", "caution", "hazard", "precaution"}.

[0104] For each response statement, check whether it contains any safety keywords in S; if it does, mark it as a safety response and record the response statement r marked as a safety response j Contains security keywords j and its position weight w p,j If not, it is marked as a non-safe response. In one embodiment, if the safety keyword is in the response statement r j The first sentence of w p,j =1.0; if the security keyword is in the response statement r j If the word p,j =0.8, if the security keyword is in the response statement r j The sequel of the second sentence, then w p,j =0.6.

[0105] For each response statement r marked as a safe response j , calculate its first safety score , and the first safety score for responses marked as non-safe (not containing safety keywords in S) is set to 0.

[0106] Among them, the confidence score of the response statement (c j ) is obtained as follows:

[0107] a. Call the probability prediction interface of the trained medical language model to obtain the probability distribution vector of each word in the process of generating the response sentence; when the model generates the response sentence r j = [t1, t2, ..., t N ], for each word t n , records the response sentence r output by the trained medical language model when generating the word j The probability v of the nth word jn [t n ], which represents the degree of certainty that the trained medical large language model selected the word in the given context.

[0108] b. Calculate the confidence score: The confidence score of the response is calculated by averaging the probabilities of all words in the entire response statement generation process. Correct responses from authoritative sources generally have higher confidence because the medical language model is more certain about these contents. Malicious content may increase the uncertainty of model generation and reduce the confidence. The confidence score reflects the overall certainty of the medical language model in generating the response statement.

[0109] Source trust score of medical knowledge fragments (source_trust(p j The acquisition process of )) is as follows:

[0110] a. Establish a credibility grading system: pre-define the source and its corresponding source credibility score source_trust(p j ) are shown in Table 1.

[0111] Table 1

[0112]

[0113] b. Extract source data: From the retrieved medical knowledge fragments p j Extract source data such as source URL or document source identifier.

[0114] c. Matching score: Match the extracted source data with the preset credibility rating system to obtain the corresponding source credibility score source_trust(p j ) to participate in the calculation of the first safety score.

[0115] If there is at least one response statement whose first security score satisfies: score1(r j ) ≥ τ safe , then select the response statement r with the highest safety score * As a reply statement. If the first security score of all response statements satisfies score(r j ) < τ safe , then proceed to Phase Three.

[0116] Phase 3:

[0117] Filter candidate response sets:

[0118] a. Determine the number of valid responses : = satisfies "I don't know" not in r j The number of response statements;

[0119] b. Calculate the filtering threshold: ,when When it is large, the filtering threshold is dominated by the upper limit of the filtering threshold β; when When it is smaller, reduce the filtering threshold from the upper filter threshold β to , to avoid filtering out all safe keywords.

[0120] c. Generate safe candidate response set R * :R * = {r j | score1(r j ) ≥ μ};

[0121] d. If R * If it is empty, the response statement with the highest security score r is selected from all response statements. * Combined with the security warning statement and used as a reply statement: r * ⊕ "Security Warning: The correctness of this response has not been verified."

[0122] Response selection strategy:

[0123] For the safe candidate response set R * Calculate the second safety score for each response statement in , and select the highest score A and the second highest score;

[0124] Construction selection criteria: A > 1.5 × B;

[0125] If the above selection conditions are met, the safe candidate response set R is selected * The second most secure response statement *As a reply statement, if the selection condition is not met, the reply aggregation is executed.

[0126] The reply aggregation process is as follows:

[0127] Compute the set of safe candidate responses R using a pre-trained medical-specific embedding model (e.g. BioBERT) * The vector representation of each response statement in . Calculate the safe candidate response set R * The semantic similarity between the vector representation of the response sentence with the second highest safety score and the vector representations of other response sentences is calculated. In one embodiment, the semantic similarity can be measured using cosine similarity, and the similarity threshold can be set to 0.7. When the similarity threshold is set to 0.7, the incorrect merging of irrelevant response sentences can be avoided. Therefore, the response subset R with a semantic similarity ≥ 0.7 is selected. sim , the quantity threshold is set to 2, if the response subset R sim If the number of response statements in the response subset is greater than or equal to 2, all response statements in the response subset are concatenated into a long text, recorded as {combined_responses}. The long text is rewritten using the trained rewriting model to generate rewritten statements corresponding to the long text. In one embodiment, the prompt for the rewriting task can be constructed as follows: Please generate a concise, accurate, and professional medical advice based on the following authoritative medical information: {combined_responses}.

[0128] The embodiments of the present application reduce the risk of database poisoning through the medical RAG security aggregation process. This design ensures the absolute priority of critical safety information. Only when it is confirmed that there are no security warnings that need to be communicated first will the regular advice generation process be entered, ensuring that valuable medical advice can still be provided even when there are no security warnings.

[0129] In a specific embodiment, it also includes:

[0130] In response to determining that retrieval enhancement generation is not required, a rewritten sentence corresponding to the input sentence is input into a trained medical large language model to generate a reply sentence.

[0131] Specifically, if the search-enhanced generation feature is not enabled, a locally trained medical language model can be directly invoked. This trained medical language model is based on high-quality data such as authoritative medical textbooks, clinical guidelines, and expert experience, and possesses strong medical logical reasoning and professional expression capabilities. The trained medical language model receives rewritten sentences processed by the rewriting model and generates responses that align with professional medical knowledge.

[0132] S5, combining the reply statement with the prompt words related to the harmful content identification task and inputting them into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

[0133] In a specific embodiment, it also includes:

[0134] In response to determining that the classification result corresponding to the reply statement is that harmful content exists, the refusal to answer process is entered.

[0135] Specifically, before outputting the reply, a trained judgment model needs to be called to review the content of the reply to ensure that no harmful content of any kind is present in the reply. During the specific implementation process, the harmful content identification task is transformed into a natural language understanding and classification problem through carefully designed prompt words. By constructing prompt words related to the harmful content identification task, the judgment model makes a comprehensive judgment on the reply to determine whether there is any content that violates laws, regulations, ethics, or platform security policies. If the judgment result is that harmful content exists, the refusal to answer process will be entered. The refusal to answer process can choose to output a statement such as: "Sorry, I can't answer this question"; otherwise, continue to the subsequent steps. If it is confirmed that there is no harmful content in the reply after review, the reply will be output and fed back to the user.

[0136] Further references Figure 2 As an implementation of the methods shown in the above figures, this application provides an embodiment of a harmful reply defense device for a large medical language model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0137] The present application provides a harmful reply defense device for a large medical language model, including:

[0138] Model construction module 1 is configured to construct and train a rewriting model and a judgment model based on a large language model to obtain a trained rewriting model and a trained judgment model, wherein the trained rewriting model includes prompt words related to the semantic preservation rewriting task, and the trained judgment model includes prompt words related to the harmful content identification task;

[0139] Rewriting module 2 is configured to obtain the user's input sentence and combine it with the prompt words related to the semantic preservation rewriting task to input into the trained rewriting model, and output the rewritten sentence corresponding to the input sentence;

[0140] The retrieval enhancement generation module 3 is configured to construct a medical knowledge base. In response to determining that retrieval enhancement generation is required, the rewritten sentence corresponding to the input sentence is searched in the medical knowledge base to obtain a plurality of medical knowledge fragments; the rewritten sentence corresponding to the input sentence is concatenated with each medical knowledge fragment and input into a trained medical language model to generate a response sentence corresponding to each medical knowledge fragment;

[0141] The reply statement generation module 4 is configured to calculate a security score for each response statement in combination with the corresponding medical knowledge fragment to obtain a first security score for each response statement; determine whether there is at least one response statement with a first security score greater than a security threshold; if so, use the response statement corresponding to the at least one response statement with the highest first security score as the reply statement; otherwise, screen all response statements and, based on the screening results, combine the response statement with the security warning statement as a reply statement, select a response statement that meets the conditions as a reply statement, or obtain a reply statement through reply aggregation;

[0142] The harmful content identification module 5 is configured to combine the reply statement with the prompt words related to the harmful content identification task and input them into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

[0143] Figure 3 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 3 As shown, the electronic device of this embodiment includes: a processor 301 and a memory 302; wherein the memory 302 is used to store computer-executable instructions; and the processor 301 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description of the above method embodiment.

[0144] Optionally, the memory 302 may be independent or integrated with the processor 301 .

[0145] When the memory 302 is independently provided, the electronic device further includes a bus 303 for connecting the memory 302 and the processor 301 .

[0146] An embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 301 executes the computer execution instructions, the above method is implemented.

[0147] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 301, the above method is implemented.

[0148] In the embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not implemented. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or module, which may be electrical, mechanical or other forms.

[0149] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected to implement the solution of this embodiment based on actual needs.

[0150] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each module may exist physically separately, or two or more modules may be integrated into a single unit. The units formed by the above modules may be implemented in the form of hardware or hardware plus software functional units.

[0151] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or processor 301 to perform some steps of the methods of various embodiments of the present application.

[0152] It should be understood that the processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor may be a microprocessor, or the processor 301 may be any conventional processor 301. The steps of the method disclosed in the present invention may be directly implemented by the hardware processor 301 or implemented by a combination of hardware and software modules in the processor 301.

[0153] The memory 302 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0154] Bus 303 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Bus 303 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the bus 303 in the drawings of this application is not limited to a single bus 303 or a single type of bus 303.

[0155] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0156] An exemplary storage medium is coupled to the processor 301, so that the processor 301 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor 301. The processor 301 and the storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor 301 and the storage medium can also exist as discrete components in an electronic device or a host control device.

[0157] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for defending against harmful replies in a large medical language model, characterized by: The following steps are involved: Constructing and training a rewriting model and a judgment model based on the large language model to obtain a trained rewriting model and a trained judgment model, wherein the trained rewriting model includes prompt words related to the semantic preservation rewriting task, and the trained judgment model includes prompt words related to the harmful content identification task; Acquire a user's input sentence and combine it with a prompt word related to the semantic-preserving rewriting task to input the sentence into the trained rewriting model, and output a rewritten sentence corresponding to the input sentence; A medical knowledge base is constructed. In response to determining that search enhancement generation is required, the rewritten sentence corresponding to the input sentence is searched in the medical knowledge base to obtain a plurality of medical knowledge fragments; the rewritten sentence corresponding to the input sentence is concatenated with each medical knowledge fragment and input into a trained medical large language model to generate a response sentence corresponding to each medical knowledge fragment; Calculate the safety score of each response statement in combination with the corresponding medical knowledge fragment to obtain a first safety score for each response statement; Determine whether there is at least one response statement whose first security score is greater than a security threshold; if so, use the response statement corresponding to the highest first security score among the at least one response statement as the reply statement; otherwise, filter all response statements and, based on the filtering results, combine the response statement with the security warning statement as the reply statement, select a response statement that meets the conditions as the reply statement, or obtain a reply statement through reply aggregation; The reply statement is combined with the prompt words related to the harmful content identification task and input into the trained judgment model to obtain a corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

2. The harmful reply defense method for a large medical language model according to claim 1 is characterized in that: A safety score is calculated for each response statement in combination with the corresponding medical knowledge fragment to obtain the first safety score of each response statement, specifically including: Construct a set of predefined medical safety keywords; Check each response statement to see if it contains any safety keyword in the predefined medical safety keyword set; if so, mark the response statement as a safe response, and determine the safety keywords contained in the response statement and their corresponding position weights, where the position weights are determined based on the positions of the safety keywords contained in the response statement; if not, mark the response statement as an unsafe response, and the first safety score of the response statement marked as an unsafe response is 0; For a response statement marked as a safe response, the confidence score of the response statement and the source credibility score of the medical knowledge fragment are determined based on the response statement and the medical knowledge fragment, respectively; and the first safety score of the response statement is calculated based on the confidence score of the response statement, the source credibility score of the medical knowledge fragment, and the position weight, as shown in the following formula: ； in, Indicates the jth response statement The first safety score, Indicates the jth response statement The confidence score of Indicates the jth response statement The source credibility score of the medical knowledge fragment, Indicates the maximum value among the position weights of the safety keywords included in the response sentence.

3. The harmful reply defense method for a large medical language model according to claim 2 is characterized in that: The j-th response statement The process of obtaining the confidence score is as follows: Obtain the j-th response statement output by the trained medical language model The probability of the nth word in the sentence and the average probability of all words are calculated to get the jth response statement. The confidence score is as follows: ； in, Indicates the jth response statement The probability of the nth word in , Indicates the jth response statement Total number of words; The j-th response statement The process of obtaining the source credibility score of the medical knowledge fragment is as follows: Establish a credibility grading system including sources and their corresponding credibility scores, starting from the jth response statement Extract the corresponding source data from the medical knowledge fragment, match the source data with the source in the credibility grading system, and determine the jth response statement Source credibility scores of medical knowledge fragments.

4. The harmful reply defense method for a large medical language model according to claim 1, characterized in that: All response statements are screened, and based on the screening results, the response statements are combined with security warning statements as reply statements, qualified response statements are selected as reply statements, or reply statements are obtained through reply aggregation, specifically including: Determine the number of valid responses, the filtering threshold ratio, and the upper limit of the filtering threshold and calculate the filtering threshold as shown in the following formula: ; in, Indicates the number of valid responses, Indicates the filtering threshold ratio, Indicates the upper limit of the filtering threshold. Indicates the filtering threshold, Indicates taking the minimum value; Among all the response statements, the response statements whose first security score is greater than or equal to the filtering threshold are used to generate a safe candidate response set R. * , as shown below: R * = {r j | score1(r j ) ≥ μ}; In response to determining the safe candidate response set R * If it is empty, the response statement with the highest security score among all the response statements is selected and combined with the security warning statement as the reply statement; In response to determining the safe candidate response set R * is not empty, then according to the safe candidate response set R * The second security score is calculated based on the confidence score of the response statement and the source credibility score of the medical knowledge fragment, as shown in the following formula: ； in, Represents the safe candidate response set R * The second safety score of the i-th response statement in, Represents the safe candidate response set R * The confidence score of the i-th response statement in , Represents the safe candidate response set R * The source credibility score of the medical knowledge fragment corresponding to the i-th response statement in ; Determine the safe candidate response set R * The highest score A and the second highest score B of the second safety score of the response statement in; In response to determining that the condition A > 1.5 × B is satisfied, the safe candidate response set R corresponding to the highest score is * The response statement in is used as a reply statement; In response to determining that the condition A > 1.5 × B is not satisfied, the safe candidate response set R * All response statements in the response aggregation are executed, and the response aggregation process is as follows: The safe candidate response set R is calculated using a pre-trained medical-specific embedding model * The vector representation of each response statement in the above formula is used to calculate the safe candidate response set R * The second highest safety score response statement in the safe candidate response set R * The semantic similarity between the vector representations of other response sentences in , and select the safe candidate response set R * The response statement with the second highest security score and the response statement with a semantic similarity greater than or equal to the similarity threshold constitute the response subset R sim ; In response to determining the response subset R sim If the number of response statements in is greater than or equal to the quantity threshold, the response subset R sim splicing all the response sentences into a long text, combining the long text with the prompt words related to the semantic preservation rewriting task and inputting the long text into the trained rewriting model, and outputting the rewritten sentence corresponding to the long text as the reply sentence; In response to determining the response subset R sim If the number of response statements in is less than the quantity threshold, the safe candidate response set R * The second highest safety score response statement in is used as the reply statement.

5. The harmful reply defense method for a large medical language model according to claim 1, characterized in that: Also includes: In response to determining that retrieval enhancement generation is not required, the rewritten sentence corresponding to the input sentence is input into a trained medical large language model to generate a reply sentence.

6. The harmful reply defense method for a large medical language model according to claim 1, characterized in that: Also includes: In response to determining that the classification result corresponding to the reply statement is that harmful content exists, the refusal to answer process is entered.

7. A harmful reply defense device for a large medical language model, characterized in that: include: a model construction module configured to construct and train a rewriting model and a judgment model based on the large language model to obtain a trained rewriting model and a trained judgment model, wherein the trained rewriting model includes prompt words related to the semantic preservation rewriting task, and the trained judgment model includes prompt words related to the harmful content identification task; a rewriting module configured to obtain a user's input sentence and combine it with a prompt word related to the semantic-preserving rewriting task to input the sentence into the trained rewriting model, and output a rewritten sentence corresponding to the input sentence; The retrieval enhancement generation module is configured to construct a medical knowledge base, and in response to determining that retrieval enhancement generation is required, search the medical knowledge base for a rewritten sentence corresponding to the input sentence to obtain a plurality of medical knowledge fragments; concatenate the rewritten sentence corresponding to the input sentence with each medical knowledge fragment and input the concatenated sentence into a trained medical language model to generate a response sentence corresponding to each medical knowledge fragment; a response statement generating module configured to calculate a safety score for each response statement in combination with a corresponding medical knowledge fragment to obtain a first safety score for each response statement; Determine whether there is at least one response statement whose first security score is greater than a security threshold; if so, use the response statement corresponding to the highest first security score among the at least one response statement as the reply statement; otherwise, filter all response statements and, based on the filtering results, combine the response statement with the security warning statement as the reply statement, select a response statement that meets the conditions as the reply statement, or obtain a reply statement through reply aggregation; The harmful content identification module is configured to combine the reply statement with the prompt words related to the harmful content identification task and input them into the trained judgment model to obtain the corresponding classification result. In response to determining that the classification result corresponding to the reply statement is that there is no harmful content, the reply statement is output.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images