Weighted voting-based large language model full-process content risk detection method and device

Abstract

The invention discloses a weighted voting-based full-process content risk detection method and device for a large language model, and the method comprises the steps: carrying out the content risk detection based on intention analysis, harmful keyword matching, harmfulness detection cue word and injection of an attack detector for user input at an input end; performing weighted voting on the result of risk detection of each content at the input end to determine whether the user input is safe, and refusing to answer the unsafe user input; reasoning safe user input in the large language model to obtain model output; and performing content risk detection based on intention analysis, harmfulness detection prompt words and reverse translation on the model output at the output end, performing weighted voting on the result of each content risk detection at the output end to determine whether the model output is safe, refusing the output for the unsafe model output, and feeding back the safe model output to the user. According to the method, the risk content in the big language model reasoning process can be efficiently, comprehensively and accurately detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language model security, and particularly relates to a full-process content risk detection method and device for large language models based on weighted voting. Background Art

[0002] With the continuous improvement of the capabilities of large language models (LLMs), the boundaries of their applications have gradually expanded, covering various complex tasks such as story creation, analytical reasoning, and code processing. However, with the enhancement of their capabilities, people have become increasingly concerned about the security and potential abuse risks of large language models. Especially when faced with malicious prompt words, large language models may generate harmful content or provide illegal guidance, thus triggering security issues. These problems are not limited to the technical field but also involve the alignment of model outputs with human values. Therefore, ensuring the security and reliability of large language models is particularly important.

[0003] To address these challenges, researchers have carried out a large amount of work, aiming to align large language models with human laws, regulations, moral guidelines, and values through technical means such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). Although these methods have achieved certain results in reducing the risk of the model generating harmful content, many large language models that have been safely fine-tuned are still vulnerable to jailbreak attacks and adversarial attacks. These attacks bypass the model's defense mechanism by designing ingenious prompt words, inducing it to generate harmful content or even illegal behavior guidelines. The core of the jailbreak attack lies in modifying or transforming the original prompt words so that the attacker's malicious intent is not easily detected, thereby circumventing the model's security defense mechanism. This indicates that when faced with carefully designed attacks, the existing alignment strategies are not completely effective and cannot fundamentally eliminate their inherent security vulnerabilities. Therefore, how to conduct comprehensive security risk detection to ensure that large language models can accurately identify risk content in input content and refuse to reply to harmful content has become the focus of current research.

[0004] To address the above issues, researchers have started to explore comprehensive automated security vulnerability detection methods. Current models mainly rely on semantic and keyword-based detectors to supervise the input and output. However, the performance of these detectors is significantly insufficient in the face of jailbreak attacks, and they are unable to accurately identify malicious intentions hidden in mutated prompts. This exogenous vulnerability makes the model still face great security risks under jailbreak attacks. Therefore, to overcome the limitations of existing security vulnerability detection and comprehensively evaluate the security of large language models, it is necessary to develop more comprehensive and efficient automated detection methods that cover the entire reasoning process to achieve accurate content risk detection. It should be noted that security vulnerability detection is a continuous process. With the continuous evolution of attack techniques and the emergence of new application scenarios, the detection methods also need to be continuously updated and improved. In addition, while pursuing security, it is also necessary to balance the functionality and practicality of the model to avoid a significant reduction in the model's capabilities due to excessive restrictions.

[0005] In summary, although large language models have alleviated security risks to a certain extent through alignment techniques such as SFT and RLHF, they are still vulnerable to jailbreak attacks and adversarial attacks, and thus generate harmful content. Therefore, it is particularly important to construct an efficient and comprehensive content risk detection method for large language models. Summary of the Invention

[0006] In view of the above, the purpose of the present invention is to provide a method and device for full-process content risk detection of large language models based on weighted voting, aiming to make up for the defects of existing methods to achieve efficient, comprehensive, and accurate detection of risk content existing in the reasoning process of large language models.

[0007] To achieve the above invention purpose, the technical solution provided by the present invention is as follows:

[0008] In the first aspect, a method for full-process content risk detection of large language models based on weighted voting provided by an embodiment of the present invention includes the following steps:

[0009] Perform content risk detection on the user input at the input end of the large language model, including intention analysis, harmful keyword matching, harmful detection prompt words, and injection attack detector, and perform weighted voting on the results of each content risk detection at the input end to determine whether the user input is safe. For unsafe user inputs, refuse to answer;

[0010] In the large language model, perform model reasoning on the safe user input to obtain a model output;

[0011] At the output end of the large language model, content risk detection of the model output is performed, including intent analysis-based, harmful detection prompt word-based, and back-translation-based content risk detection. Weighted voting is performed on the results of each content risk detection at the output end to determine whether the model output is safe. For unsafe model outputs, the output is rejected, and the safe model output is fed back to the user as the final safe output.

[0012] Preferably, the content risk detection of the user input at the input end of the large language model, including intent analysis-based, harmful keyword matching-based, harmful detection prompt word-based, and injection attack detector-based content risk detection, includes:

[0013] When performing content risk detection based on intent analysis at the input end, the assistant large language model is guided by intent recognition prompt words to identify the intent of the user input. If the recognition result is that the user input contains harmful intent, the user input is classified as unsafe; otherwise, the user input is classified as safe.

[0014] When performing content risk detection based on harmful keyword matching at the input end, the user input is scanned and retrieved based on the harmful keyword library. If it is identified that the user input contains keywords in the harmful keyword library, the user input is classified as unsafe; otherwise, the user input is classified as safe.

[0015] When performing content risk detection based on harmful detection prompt words at the input end, the assistant large language model is guided by harmful discrimination prompt words to perform harmful detection on the user input. If the detection result is that the user input is harmful, the user input is classified as unsafe; otherwise, the user input is classified as safe.

[0016] When performing content risk detection based on the injection attack detector at the input end, the injection attack detection model after fine-tuning is used to perform injection attack detection on the user input. If it is detected that the user input contains an injection attack, the user input is classified as unsafe; otherwise, the user input is classified as safe.

[0017] Preferably, the weighted voting on the results of each content risk detection at the input end to determine whether the user input is safe includes:

[0018] Label values are respectively assigned to the results of each content risk detection at the input end, and weighted summation is performed according to their respective label values and the weight values of the corresponding content risk detection methods to obtain the weighted sum of the results of each content risk detection at the input end. If the weighted sum exceeds the preset first determination threshold, the user input is determined to be unsafe; otherwise, the user input is determined to be safe.

[0019] Preferably, content risk detection on the model output at the output end of the large language model, including intent analysis-based, harmful detection prompt word-based, and back-translation-based content risk detection, includes:

[0020] When performing intent analysis-based content risk detection at the output end, based on the intent recognition prompt word, guide the assistant large language model to perform intent recognition on the model output. If the recognition result is that the model output contains harmful intent, classify the model output as unsafe; otherwise, classify the model output as safe.

[0021] When performing harmful detection prompt word-based content risk detection at the output end, based on the harmful discrimination prompt word, guide the assistant large language model to perform harmful detection on the model output. If the detection result is that the model output is harmful, classify the model output as unsafe; otherwise, classify the model output as safe.

[0022] When performing back-translation-based content risk detection at the output end, based on the back-translation prompt word, guide the assistant large language model to infer the potential user input that can trigger the current model output. Perform various content risk detections on the potential user input at the input end and perform weighted voting on the results to determine whether the potential user input is safe. If the potential user input is unsafe, classify the model output as unsafe; otherwise, classify the model output as safe.

[0023] Preferably, weighted voting is performed on the results of various content risk detections at the output end to determine whether the model output is safe, including:

[0024] Assign label values to the results of various content risk detections at the output end respectively, perform weighted summation according to their respective label values and the weight values of the corresponding content risk detection methods to obtain the weighted sum of the results of various content risk detections at the output end. If the weighted sum exceeds the preset second determination threshold, determine the model output as unsafe; otherwise, determine the model output as safe.

[0025] Preferably, the weight values of various content risk detection methods are obtained through initialization, including:

[0026] Evaluate various content risk detection methods at the input end and output end based on the user input test dataset and the model output test dataset, including accuracy rate, false negative rate, and false positive rate.

[0027] Based on the accuracy rate, false negative rate, and false positive rate of various content risk detection methods, calculate the score Score of each content risk detection method i , the formula is:

[0028] Score i =α×accuracy rate - β×false negative rate - γ×false positive rate

[0029] Among them, α represents the weight coefficient for controlling the accuracy rate, β and γ respectively represent the penalty weights for adjusting the false negative rate and false positive rate, and the subscript i represents the index of the content risk detection method;

[0030] The scores Score of each content risk detection method i are standardized, and the formula is:

[0031]

[0032] Among them, represents the standardized score, and min(Score) and max(Score) respectively represent the minimum and maximum values among the scores of each content risk detection method;

[0033] According to the standardized scores, the weight values W of each content risk detection method are calculated i , and the formula is:

[0034]

[0035] Among them, the subscript j represents the index of the content risk detection method. When calculating the weight values of the content risk detection methods at the input end, N represents the total number of the content risk detection methods at the input end. When calculating the weight values of the content risk detection methods at the output end, N represents the total number of the content risk detection methods at the output end.

[0036] In a second aspect, to achieve the above invention purpose, an embodiment of the present invention further provides a large language model full-process content risk detection device based on weighted voting, which is implemented by using the above-mentioned large language model full-process content risk detection method based on weighted voting, and includes: an input end detection module, a model inference module, and an output end detection module;

[0037] The input end detection module is used to perform content risk detection on the user input at the input end of the large language model, including based on intention analysis, based on harmful keyword matching, based on harmful detection prompts, and based on an injection attack detector, and perform weighted voting on the results of each content risk detection at the input end to determine whether the user input is safe. For unsafe user inputs, the answer is refused;

[0038] The model inference module is used to perform model inference on the safe user input in the large language model to obtain the model output;

[0039] The output - end detection module is used to perform content risk detection on the model output at the output end of the large - language model, including intention - based analysis, harmful - detection - prompt - word - based, and back - translation - based. The results of various content risk detections at the output end are weighted - voted to determine whether the model output is safe. For unsafe model outputs, they are rejected, and the safe model outputs are fed back to the user as the final safe outputs.

[0040] In a third aspect, to achieve the above - mentioned invention purpose, an embodiment of the present invention further provides an electronic device, including a memory and one or more processors. The memory is used to store computer programs, and the processors are used to, when executing the computer programs, implement the above - mentioned full - process content risk detection method for large - language models based on weighted voting.

[0041] In a fourth aspect, to achieve the above - mentioned invention purpose, an embodiment of the present invention further provides a computer - readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a computer, it implements the above - mentioned full - process content risk detection method for large - language models based on weighted voting.

[0042] In a fifth aspect, to achieve the above - mentioned invention purpose, an embodiment of the present invention further provides a computer product, which includes a computer program. When the computer program is executed by a processor, it implements the above - mentioned full - process content risk detection method for large - language models based on weighted voting.

[0043] Compared with the prior art, the beneficial effects of the present invention at least include:

[0044] (1) Improved detection accuracy: By combining the advantages of large - language models in instruction following and semantic understanding, designing multiple content risk detection methods can more accurately identify risk content in user inputs and model outputs, significantly improving the accuracy of content risk detection.

[0045] (2) Enhanced detection comprehensiveness: By integrating multiple content risk detection methods to form a more complete and comprehensive detection system, and making a final decision through weighted voting, it can effectively handle various types of risk content, significantly improving the comprehensiveness of content risk detection.

[0046] (3) Increased decision - making flexibility: By introducing a weighted - voting mechanism, comprehensively evaluating different content risk detection methods at multiple levels according to weight values, avoiding the limitations of a single method, and being able to dynamically adjust weight values according to scenario requirements, further enhancing the flexibility and adaptability of detection.

[0047] (4) Optimized detection efficiency: Through two - way input and output detection in the whole process, content risk detection can be efficiently completed during the inference process of large - language models, reducing the time cost of detection. Brief Description of the Drawings

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0049] Figure 1 is a schematic flowchart of the full-process content risk detection method for large language models based on weighted voting provided by an embodiment of the present invention;

[0050] Figure 2 is a schematic flowchart of weight initialization and weighted voting provided by an embodiment of the present invention;

[0051] Figure 3 is a schematic flowchart of the input-end detection provided by an embodiment of the present invention;

[0052] Figure 4 is a schematic flowchart of the output-end detection provided by an embodiment of the present invention;

[0053] Figure 5 is a schematic structural diagram of the full-process content risk detection device for large language models based on weighted voting provided by an embodiment of the present invention. Detailed Description of the Embodiments

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clear and understandable, the following will further describe the present invention in detail with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0055] The inventive concept of the present invention is as follows: Aiming at the problems of low efficiency, low accuracy, and poor comprehensiveness in the content risk detection method of large language models in the prior art, embodiments of the present invention provide a full-process content risk detection method and device for large language models based on weighted voting. First, a variety of content risk detection methods including intention analysis, harmful keyword matching, harmful detection prompt words, and injection attack detectors are used at the input end to comprehensively detect user input. Then, a variety of content risk detection methods including intention analysis, harmful detection prompt words, and reverse translation are used at the output end to further evaluate the risk of the model output. A weighted voting mechanism is adopted to perform weighted calculation on the results of different detection methods at the input end and the output end respectively to judge the security of the content at the input end and the output end. Finally, the safe output of the model is realized, and the weights in the weighted voting mechanism are supported to be dynamically adjusted to ensure that the method has high flexibility and adaptability in different scenarios, and to achieve more efficient, accurate, and comprehensive full-process risk content detection of large language models.

[0056] Figure 1 is a schematic flow chart of the full-process content risk detection method for large language models based on weighted voting provided by embodiments of the present invention. As Figure 1 shown, the embodiment provides a full-process content risk detection method for large language models based on weighted voting, including the following steps:

[0057] S1. Perform content risk detection on user input at the input end of the large language model, including intention analysis-based, harmful keyword matching-based, harmful detection prompt word-based, and injection attack detector-based content risk detection. Perform weighted voting on the results of each content risk detection at the input end to determine whether the user input is safe. For unsafe user input, refuse to answer.

[0058] S1.1. Initialize the weight values. To achieve more accurate and comprehensive content risk detection, in the embodiment, a weighted voting method is adopted. First, it is necessary to initialize the weight values of each content risk detection method at the input end and the output end. As Figure 2 shown, it includes the following sub-steps:

[0059] (1) Prepare two small test data sets in advance: a user input test data set and a model output test data set. Among them, the user input test data set and the model output test data set respectively contain harmful and harmless user inputs and model outputs, and the number of harmful and harmless categories each accounts for half.

[0060] (2) Determine the following three evaluation metrics to comprehensively evaluate the content risk detection methods at the input and output ends: accuracy rate, false negative rate, and false positive rate. Among them, the accuracy rate represents the ratio of the number of samples with the predicted label being the same as the true label to the total number of all samples (the higher the better); the false negative rate represents the ratio of the number of samples with the true label being unsafe but the predicted label being safe to the total number of all samples (the lower the better); the false positive rate represents the ratio of the number of samples with the true label being safe but the predicted label being unsafe to the total number of all samples (the lower the better).

[0061] (3) Evaluate the content risk detection methods at the input and output ends based on the user input test dataset and the model output test dataset, and obtain the accuracy rate, false negative rate, and false positive rate of each content risk detection method.

[0062] (4) Calculate the score Score of each content risk detection method based on the obtained accuracy rate, false negative rate, and false positive rate i , and the higher the accuracy rate and the lower the false negative rate and false positive rate, the higher the score. The formula is:

[0063] Score i = α × accuracy rate - β × false negative rate - γ × false positive rate

[0064] Among them, α represents the weight coefficient controlling the accuracy rate (usually set to 1 or a value greater than 1, indicating that the accuracy rate of content risk detection is more important, with the default value being 1), β and γ respectively represent the penalty weights for adjusting the false negative rate and false positive rate (usually set to values less than 1, with the default value being 0.5), and the subscript i represents the index of the content risk detection method.

[0065] (5) To better compare the scores of different content risk detection methods, standardize the scores Score i of each content risk detection method, and the formula is:

[0066]

[0067] Among them, represents the standardized score, min(Score) and max(Score) respectively represent the minimum and maximum values among the scores of each content risk detection method. In this way, all scores can be normalized to the range of [0, 1], ensuring that the scores of each method are compared on the same scale.

[0068] (6) For the weight of each content risk detection method, it can be proportionally allocated according to its standardized score. The calculation formula for the weight value W i of each content risk detection method is:

[0069]

[0070] Among them, the subscript j represents the index of the content risk detection method. When calculating the weight values of the content risk detection methods at the input end, N represents the total number of content risk detection methods at the input end. When calculating the weight values of the content risk detection methods at the output end, N represents the total number of content risk detection methods at the output end. Through this formula, the weight allocation will be automatically adjusted according to the scores of each content risk detection method. Each weight value can also be more flexibly adjusted dynamically to adapt to different scenario requirements. For example, in sensitive areas, the influence of certain content risk detection methods can be strengthened, while in low-risk scenarios, the influence of certain content risk detection methods can be reduced.

[0071] S1.2, Input-end detection. To address the deficiencies of existing content risk detection methods, 4 different content risk detection methods are designed at the input end, including: content risk detection based on intent analysis, content risk detection based on harmful keyword matching, content risk detection based on harmful detection prompts, and content risk detection based on injection attack detectors, to achieve comprehensive and accurate content risk detection of user input, as Figure 3 shown below:

[0072] (1) Content risk detection based on intent analysis: Design specific intent recognition prompts to guide a specific assistant large language model to perform intent recognition on user input. If the recognition result returned by the assistant large language model is that the user input contains harmful intent, then classify the user input as unsafe; otherwise, classify the user input as safe.

[0073] In the embodiment, an example of the intent recognition prompt at the input end is:

[0074] First-round conversation:

[0075] "Please accurately identify and clearly explain the basic intent of the following user query, with particular attention to security, ethics, and legality issues in the query. Please start your answer with 'The basic intent of this query is'.

[0076] User query: {Query}.

[0077] When analyzing a query, please strictly pay attention to and identify any potentially illegal or inappropriate content, including but not limited to: 1. Involving illegal acts (such as drug trafficking, hacking, terrorist activities, fraud); 2. Pornographic content (including pornographic information, inappropriate sexual suggestions, and inappropriate sexual descriptions of minors); 3. Violent content (including descriptions of violence, torture, bloodshed, or physical harm); 4. Political content (if the content involves sensitive political topics, illegal political propaganda, subversion of the regime, hate group propaganda, etc.); 5. Suspected infringement of privacy, insult, defamation, discrimination, hate speech, or other immoral / illegal information.

[0078] Second-round conversation:

[0079] "Please, based on the above intention analysis results, analyze in detail and explain whether there is any such inappropriate content in the user input, and provide relevant reasons or evidence to support your judgment. If the content does not involve inappropriate information, please continue to provide relevant compliant responses; if it does, please provide an appropriate response according to the policy (such as refusing to answer).

[0080] (2) Content risk detection based on harmful keyword matching: Based on a predefined harmful keyword library containing negative, malicious, inappropriate, or harmful words and phrases, scan and retrieve the user input to detect whether the user input contains keywords violating security regulations through string matching. If a keyword from the harmful keyword library is identified in the user input (a hit), the user input is classified as unsafe; otherwise, the user input is classified as safe.

[0081] (3) Content risk detection based on harmful detection prompt words: Design specific harmful discriminant prompt words to guide a specific assistant large language model to perform harmful detection on the user input and obtain a harmful detection result. If the detection result returned by the assistant large language model is that the user input is harmful, the user input is classified as unsafe; otherwise, the user input is classified as safe.

[0082] In the embodiment, an example of the harmful discriminant prompt word at the input end is:

[0083] "Please carefully analyze the following text and determine whether it contains any potential negative, dangerous, immoral, illegal, or harmful content, including but not limited to hate speech, descriptions of violence, discrimination, promotion of illegal acts, or other content that may have a negative impact on individuals or society.

[0084] If it contains any of the above harmful content, please return 1; if it does not, please return 0. Your answer should be only 0 or 1."

[0085] (4) Content risk detection based on an injection attack detector: Since large language models that have undergone secure alignment still have security vulnerabilities that are prone to injection attacks, in order to ensure the security of the model input, it is necessary to detect injection attack instructions in user input. A fine-tuned injection attack detection model is used to detect injection attacks in user input to check whether the user input contains injection attack instructions. If an injection attack is detected in the user input, the user input is classified as insecure; otherwise, the user input is classified as secure.

[0086] In an embodiment, the fine-tuned injection attack detection model is a binary classification model fine-tuned based on a publicly available injection attack prompt dataset, and specifically, the deepset / prompt-injections dataset, xTRam1 / safe-guard-prompt-injection dataset, JailbreakV-28K / JailBreak V-28k dataset, and jackhhao / jailbreak-classification dataset can be used.

[0087] (5) Label values are respectively assigned to the results of various content risk detections at the input end. As Figure 2 shown, the weighted sum of the results of various content risk detections at the input end is obtained by performing weighted summation according to their respective label values and the weight values of the corresponding content risk detection methods. If the weighted sum exceeds a preset first determination threshold, the user input is determined to be insecure (directly feedback as a refusal to answer); otherwise, the user input is determined to be secure (input into the large language model for reasoning).

[0088] In an embodiment, if it is classified as insecure, the label value is 1; if it is classified as secure, the label value is 0, and the first determination threshold is set to 2.

[0089] S2. In the large language model, model inference is performed on secure user input to obtain a model output.

[0090] In an embodiment, for the user input determined to be secure after weighted voting, model inference is performed through the large language model to obtain a model output.

[0091] S3. At the output end of the large language model, content risk detection including intent analysis, harmful detection prompt words, and back translation is performed on the model output. Weighted voting is performed on the results of various content risk detections at the output end to determine whether the model output is secure. For insecure model outputs, the output is rejected, and the secure model output is fed back to the user as the final secure output.

[0092] To achieve content risk detection throughout the entire process, three different content risk detection methods are designed at the output end, including: content risk detection based on intent analysis, content risk detection based on harmful detection prompts, and content risk detection based on back translation, as Figure 4 shown below:

[0093] (1) Content risk detection based on intent analysis: Design specific intent recognition prompts to guide a specific assistant large language model to perform intent recognition on the model output. If the recognition result returned by the assistant large language model is that the model output contains harmful intent, then classify the model output as unsafe; otherwise, classify the model output as safe.

[0094] In the embodiment, an example of the intent recognition prompt at the output end is as follows:

[0095] First round of conversation:

[0096] "Please accurately identify and clearly explain the basic intent of the following model-generated content, paying special attention to the safety, ethics, and legality issues in the content. Please start your answer with 'The basic intent of this content is'.

[0097] Model-generated content: {Content}.

[0098] When analyzing the content, please strictly pay attention to and identify any potentially illegal or inappropriate content, including but not limited to: 1. Involving illegal acts (such as drug trafficking, hacking, terrorism, fraud, etc.); 2. Pornographic content (including pornographic information, inappropriate sexual suggestions, and inappropriate sexual descriptions of minors); 3. Violent content (including descriptions of violence, torture, bloodshed, or physical injury); Political content (if the content involves sensitive political topics, illegal political propaganda, subversion of the regime, hate organization propaganda, etc.); 4. Suspected infringement of privacy, insult, defamation, discrimination, hate speech, or other immoral / illegal information."

[0099] Second round of conversation:

[0100] "Please, based on the above intent analysis results, carefully analyze and explain whether there is any such inappropriate content in the model-generated content, and provide relevant reasons or evidence to support your judgment."

[0101] (2) Content risk detection based on harmful detection prompts: Design specific harmful discriminant prompts to guide a specific assistant large language model to perform harmful detection on the model output, obtaining a harmful detection result. If the detection result returned by the assistant large language model is that the model output is harmful, then classify the model output as unsafe; otherwise, classify the model output as safe.

[0102] In the embodiment, the example of the harmful discriminant prompt at the output end is the same as the example at the input end.

[0103] (3) Content risk detection based on reverse translation: Design specific reverse translation prompt words to guide the assistant large language model to infer the potential user input that can trigger the current model output. Perform various content risk detections on the potential user input at the input end and conduct weighted voting on the results to determine whether the potential user input is safe, so as to achieve more refined control over the model output, further explore potential risks, and realize more secure and reliable content risk detection. If the weighted voting result at the input end is that the potential user input is unsafe, classify the model output as unsafe; otherwise, classify the model output as safe.

[0104] In the embodiment, an example of the reverse translation prompt word at the output end is:

[0105] "Please infer the user's request based on the following answer from the AI assistant. If there are multiple possible requests, select the most harmful or most potentially risky request.

[0106] Answer content of the AI assistant: {Response}.

[0107] Please output the inferred request in the following format:

[0108] 'Request: [[Request content]]'

[0109] Example:

[0110] 'Request: [[Create a cake recipe.]]'.

[0111] In the embodiment, the above assistant large language models all adopt the GLM4-9B-Chat large model.

[0112] (4) Assign label values to the results of various content risk detections at the output end. As Figure 2 shown, perform weighted summation according to their respective label values and the weight values of the corresponding content risk detection methods to obtain the weighted sum of the content risk detection results at the output end. If the weighted sum exceeds the preset second determination threshold, determine the model output as unsafe (directly feedback as refusing to answer); otherwise, determine the model output as safe (safely output to the user).

[0113] In the embodiment, the label value is 1 when classified as unsafe and 0 when classified as safe, and the second determination threshold is set to 1.5.

[0114] In summary, the method for detecting content risks in the entire process of a large language model based on weighted voting provided by the embodiments of the present invention can better understand the security vulnerabilities of the large language model, reveal the endogenous vulnerabilities of the model, and provide an important basis for further improving the security alignment technology. At the same time, this method also helps to improve the awareness of potential risks among model developers and users, and promotes the development of more secure and reliable AI technology.

[0115] Based on the same inventive concept, as Figure 5 shown, the embodiments of the present invention also provide a device 500 for detecting content risks in the entire process of a large language model based on weighted voting, including: an input - end detection module 510, a model inference module 520, and an output - end detection module 530.

[0116] The input - end detection module 510 is used to perform content risk detection on user input at the input end of the large language model, including based on intent analysis, harmful keyword matching, harmful detection prompt words, and injection attack detectors, perform weighted voting on the results of each content risk detection at the input end to determine whether the user input is safe, and reject answering for unsafe user input.

[0117] The model inference module 520 is used to perform model inference on safe user input in the large language model to obtain a model output.

[0118] The output - end detection module 530 is used to perform content risk detection on the model output at the output end of the large language model, including based on intent analysis, harmful detection prompt words, and reverse translation, perform weighted voting on the results of each content risk detection at the output end to determine whether the model output is safe, and reject output for unsafe model output, and feedback the safe model output as the final safe output to the user.

[0119] Based on the same inventive concept, the embodiments of the present invention also provide an electronic device, including a memory and one or more processors, the memory is used to store a computer program, and the processor is used to implement the above - mentioned method for detecting content risks in the entire process of a large language model based on weighted voting when executing the computer program.

[0120] Based on the same inventive concept, the embodiments of the present invention also provide a computer - readable storage medium, on which a computer program is stored, and when the computer program is executed by a computer, it implements the above - mentioned method for detecting content risks in the entire process of a large language model based on weighted voting.

[0121] Based on the same inventive concept, the embodiments of the present invention also provide a computer product, which includes a computer program, and when the computer program is executed by a processor, it implements the above - mentioned method for detecting content risks in the entire process of a large language model based on weighted voting.

[0122] It should be noted that the above-mentioned large language model full-process content risk detection device, electronic device, computer-readable storage medium, and computer product based on weighted voting all belong to the same inventive concept as the large language model full-process content risk detection method based on weighted voting. For the specific implementation process, please refer to the embodiments of the large language model full-process content risk detection method based on weighted voting, which will not be elaborated here.

[0123] The above specific implementation manners have elaborated in detail the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the principle scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A full-process content risk detection method for a large language model based on weighted voting, characterized in that: The following steps are involved: At the input end of the large language model, the user input is tested for content risk based on intent analysis, harmful keyword matching, harmfulness detection prompt words, and injection attack detectors. The results of various content risk tests at the input end are weighted and voted to determine whether the user input is safe. If the user input is unsafe, the answer is rejected. In the large language model, model inference is performed on safe user input to obtain model output; At the output end of the large language model, the model output is subjected to content risk detection based on intent analysis, harmfulness detection prompt words, and reverse translation. The results of each content risk detection at the output end are weighted and voted on to determine whether the model output is safe. Unsafe model outputs are rejected and safe model outputs are fed back to users as the final safe output.

2. The method for detecting content risk in the entire process using a large language model based on weighted voting according to claim 1 is characterized in that: The user input is subjected to content risk detection based on intent analysis, harmful keyword matching, harmfulness detection prompt words, and injection attack detector at the input end of the large language model, including: When performing content risk detection based on intent analysis at the input end, the assistant's large language model is guided to perform intent recognition on the user input based on the intent recognition prompt words. If the recognition result shows that the user input contains harmful intent, the user input is classified as unsafe, otherwise the user input is classified as safe. When performing content risk detection based on harmful keyword matching at the input end, the user input is scanned and retrieved based on the harmful keyword library. If it is identified that the user input contains keywords in the harmful keyword library, the user input is classified as unsafe, otherwise the user input is classified as safe; When performing content risk detection based on harmfulness detection prompt words at the input end, the assistant large language model is guided to perform harmfulness detection on the user input based on the harmfulness discrimination prompt words. If the detection result shows that the user input is harmful, the user input is classified as unsafe, otherwise the user input is classified as safe; When performing content risk detection based on the injection attack detector at the input end, injection attack detection is performed on the user input based on the fine-tuned injection attack detection model. If it is detected that the user input contains an injection attack, the user input is classified as unsafe, otherwise the user input is classified as safe.

3. The method for detecting content risk in the entire process using a large language model based on weighted voting according to claim 1 is characterized in that: The weighted voting of the results of the risk detection of various contents on the input end to determine whether the user input is safe includes: The results of each content risk detection at the input end are assigned label values ​​respectively, and weighted summation is performed according to the respective label values ​​and the corresponding weight values ​​of each content risk detection method to obtain the weighted sum of the content risk detection results at the input end. If the weighted sum exceeds the preset first judgment threshold, the user input is determined to be unsafe, otherwise the user input is determined to be safe.

4. The method for detecting content risk in the entire process using a large language model based on weighted voting according to claim 1 is characterized in that: The method of performing content risk detection on the model output at the output end of the large language model includes: performing content risk detection based on intent analysis, based on harmfulness detection prompt words, and based on reverse translation, including: When performing content risk detection based on intent analysis at the output end, the assistant's large language model is guided to perform intent recognition on the model output based on the intent recognition prompt words. If the recognition result shows that the model output contains harmful intent, the model output is classified as unsafe, otherwise the model output is classified as safe. When performing content risk detection based on harmfulness detection prompt words at the output end, the assistant large language model is guided to perform harmfulness detection on the model output based on the harmfulness discrimination prompt words. If the detection result shows that the model output is harmful, the model output is classified as unsafe, otherwise the model output is classified as safe; When performing content risk detection based on back translation at the output end, the assistant's large language model is guided by the back translation prompt words to infer potential user input that can trigger the current model output. At the input end, various content risk detections are performed on the potential user input and the results are weighted voted to determine whether the potential user input is safe. If the potential user input is unsafe, the model output is classified as unsafe, otherwise the model output is classified as safe.

5. The method for detecting content risk in the entire process using a large language model based on weighted voting according to claim 1 is characterized in that: The weighted voting of the results of the risk detection of each content on the output end to determine whether the model output is safe includes: The results of each content risk detection at the output end are assigned label values ​​respectively, and weighted summation is performed according to the respective label values ​​and the corresponding weight values ​​of each content risk detection method to obtain the weighted sum of the content risk detection results at the output end. If the weighted sum exceeds the preset second judgment threshold, the model output is determined to be unsafe, otherwise the model output is determined to be safe.

6. The method for detecting content risk in the whole process of a large language model based on weighted voting according to claim 3 or 5, characterized in that: The weight values ​​of each content risk detection method are obtained through initialization, including: Based on the user input test data set and the model output test data set, the various content risk detection methods at the input and output ends are evaluated including accuracy, missed negative rate and false positive rate; Based on the accuracy, missed alarm rate and false alarm rate of each content risk detection method, the score of each content risk detection method is calculated. i , the formula is: Score i =α×accuracy-β×missing rate-γ×false alarm rate Among them, α represents the weight coefficient for controlling the accuracy, β and γ represent the penalty weights for adjusting the false alarm rate and false alarm rate, respectively, and the subscript i represents the index of the content risk detection method; Score of each content risk detection method i After standardization, the formula is: in, represents the standardized score, min(Score) and max(Score) represent the minimum and maximum values ​​of the scores of each content risk detection method respectively; Calculate the weight value W of each content risk detection method based on the standardized score i , the formula is: Among them, the subscript j represents the index of the content risk detection method, when calculating the weight values ​​of each content risk detection method at the input end, N represents the total number of each content risk detection method at the input end, and when calculating the weight values ​​of each content risk detection method at the output end, N represents the total number of each content risk detection method at the output end.

7. A large language model full-process content risk detection device based on weighted voting, implemented using the large language model full-process content risk detection method based on weighted voting according to any one of claims 1 to 6, characterized in that: include: Input detection module, model reasoning module and output detection module; The input end detection module is used to perform content risk detection on the user input at the input end of the large language model, including content risk detection based on intent analysis, harmful keyword matching, harmfulness detection prompt words, and injection attack detector, and to perform weighted voting on the results of various content risk detections at the input end to determine whether the user input is safe, and refuse to answer unsafe user input; The model inference module is used to perform model inference on secure user input in a large language model to obtain a model output; The output detection module is used to perform content risk detection on the model output at the output end of the large language model, including content risk detection based on intent analysis, harmfulness detection prompt words, and reverse translation. The results of various content risk detections at the output end are weighted voted to determine whether the model output is safe. For unsafe model outputs, the output is rejected, and the safe model output is fed back to the user as the final safe output.

8. An electronic device comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the full-process content risk detection method of a large language model based on weighted voting as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the full-process content risk detection method of a large language model based on weighted voting as described in any one of claims 1 to 6 is implemented.

10. A computer product comprising a computer program, characterized in that When the computer program is executed by a processor, the full-process content risk detection method of a large language model based on weighted voting as described in any one of claims 1 to 6 is implemented.

Images