Large model alignment method based on plug-in and multi-agent collaboration

Abstract

The invention discloses a plug-in and multi-agent cooperation-based large model alignment method. The method comprises the following steps of: firstly, inputting user query to obtain a large language model to generate a preliminary response; then, whether malicious content is contained or not is detected and inquired through intention recognition, if yes, harmfulness check is carried out, and the safety of the content is judged through the cooperation of moral, legal and discriminant agents; and then performing fact check on the generation of the large language model, verifying the authenticity of the content by using a plug-in, and identifying the usefulness. When all the checks pass, directly outputting the initial content as a response after final alignment; otherwise, the reasons which do not pass are spliced and then fed back to the large language model to regenerate the content to serve as the response after alignment. According to the invention, the quality of large language model generation content is improved.

Description

Technical Field

[0001] The present invention relates to the field of alignment of large language models with human values, and in particular to a large model alignment method based on plug-ins and multi-agent collaboration, which aims to improve the alignment of responses generated by a large language model with human values ​​when processing user queries. Background Art

[0002] Globally, the widespread use of large language models (LLMs) has made them one of the core technologies in the field of natural language processing (NLP). However, as their application scope continues to expand, especially in scenarios such as social media, customer service systems, and smart assistants, these large language models often show value misalignment when generating outputs. In particular, when models are faced with adversarial prompts, they may generate content with obvious bias, misinformation, or even harmful content.

[0003] These misaligned outputs may not only raise ethical issues, but also exacerbate inequality and discrimination in society. For example, when a large language model learns from online data, it may unconsciously absorb and amplify social biases in terms of gender, race, etc. in these data. These potential dangers are not limited to academic discussions. In actual business and social scenarios, harmful content generated by large language models may directly lead to legal and social responsibility issues, undermine user experience, and even damage corporate reputation. Therefore, how to ensure that the output of large models is consistent with human ethics and social values ​​has become a crucial issue in the study of large language models. Scholars usually start from three dimensions for this consistency: usefulness, harmlessness, and authenticity.

[0004] On the issue of model alignment, some scholars first introduced reinforcement learning from human feedback (RLHF). The working principle of RLHF is to fine-tune the parameters of the model through feedback from human annotators on the preferences of model output, so that the model gradually learns an output mode that is more in line with human expectations. For example, OpenAI used the RLHF method when training its GPT series of models, scoring the responses of the model by human reviewers, and then using these feedbacks to optimize the model. However, RLHF relies on a large number of manual annotations, which consumes a lot of time and resources, and the annotation process is often affected by subjectivity, resulting in inconsistent training results. In view of the manual intervention and resource waste in human feedback, Kim et al. further proposed RLAIF (Reinforcement Learning from AI Feedback), which is reinforcement learning based on AI feedback. RLAIF automatically generates training data by introducing AI-generated feedback, reducing dependence on manual annotation and improving training efficiency. In addition, the training process of RLAIF is more stable, which solves the problem of uneven annotation quality in RLHF, but RLAIF does not solve the problem of complex RLHF training process. Summary of the invention

[0005] The purpose of this invention is to address the problem of unstable quality of content generated by multi-model collaboration, and to provide a large model alignment method based on plug-ins and multi-agent collaboration. This method first performs preliminary classification and screening of user queries through hierarchical processing of intent recognition, fact verification and content security to determine the basic compliance of generated content. On this basis, the generated content is further verified and optimized, and multiple judgment results are combined to finally generate high-quality response content that meets user needs.

[0006] The specific technical solution for achieving the purpose of the present invention is:

[0007] A large model alignment method based on plug-in and multi-agent collaboration includes the following specific steps:

[0008] Step 1: Acquisition of initial generation of a large language model

[0009] The user's query V query Input into the large language model (LLM) to generate the initial response result R initial The large language model extracts features and understands the context of the input query vector and generates the corresponding initial content. The calculation formula is as follows:

[0010] R initial =LLM(V query ) (1)

[0011] Where LLM(·) represents the large language model generation process;

[0012] Step 2: Identify the user’s query intent

[0013] Intent Recognition Use intent recognition agents to identify user queries V query Identify the intent of the query and determine whether it contains malicious queries; input query V query After intention detection, the intention recognition result R is generated intent , and give a reason intent The calculation formula is as follows:

[0014] R intent , Reason intent =IntentDetect(V query ) (2)

[0015] Among them, IntentDetect(·) represents the intention recognition operation, R intent ∈{0, 1}, indicating the identification of malicious queries and non-malicious queries with user intent, respectively;

[0016] Step 3: Fact-check the content generated by the large language model

[0017] The initial content R generated by the large language model initial Input into the open source plug-in OpenFactVerification for fact checking to generate the judgment result R honest and the corresponding reason honest ; The specific calculation formula is as follows:

[0018] R honest , Reason honest =FactCheck(R initial ) (3)

[0019] FactCheck(·) indicates the fact checking plug-in, which uses the open source plug-in OpenFactVerification. honest = 0, the content of the task does not conform to the facts; if R honest =1, it means that the generated content is consistent with the facts;

[0020] Step 4: Identify the usefulness of the content generated by the large language model

[0021] Fine-tune a BERT model to detect the relevance and usefulness of the content generated by the large language model. In this model, the input is the user query and the question-answer pair generated by the large model, and the output is to judge whether it is relevant. The positive sample data is randomly sampled from Natural Questions. At the same time, for each sample question collected, the large language model is used to generate an irrelevant answer for it to form a negative sample pair. The positive sample label is 1, and the negative sample is 0 for binary classification training. The trained BERT model is used for usefulness identification. The formula is as follows:

[0022] R helpful =HelpfulCheck(V query , R initial ) (4)

[0023] Among them, HelpfulCheck(·) represents the helpfulness check model;

[0024] Step 5: Check the harmfulness of the content generated by the large language model

[0025] In step 2, if the user query is identified as containing a malicious query, R intent =1, then enter this step, otherwise skip. In this step, three agents are designed to collaborate, namely the judge agent, the moral agent and the legal agent; and the moral agent and the legal agent are fine-tuned, and the data comes from the moral database and the legal database respectively; the moral agent and the legal agent review the generated content from the moral and legal perspectives respectively, and generate their own suggestions respectively; the final security judgment R harmless and Reason harmless are generated by the judge agent by combining the suggestions of the moral agent and the legal agent. If the judge agent ultimately believes that the generation of the large language model is harmful, then R harmless , the default value is 1, the specific formula is as follows: R harmless , Reason harmless =judge[R moral (R initial ), R legal (R initial )](5)

[0026] Among them, R moral (·) and R 1egal (·) denotes the advice generation operations of the moral and legal agents, respectively, and Judge[·] denotes the generation operation of the judge agent;

[0027] Step 6: Output the final aligned response of the large language model

[0028] After completing all the previous steps, this step performs the final alignment generation on the large language model; specifically, when all checks pass, that is, R helpful =1 and R honest =1 and R harmless =1, the initial generated content R will be directly output initial As the final result; if any of the above values ​​is 0, the feedback reasons of all failed steps will be concatenated, as shown in formula (7), to form a complete feedback, and the large language model will be instructed to adjust and regenerate the content to ensure that the final output is aligned with human values; the specific process is as shown in formula (6):

[0029]

[0030] Reason=concat[(Reason i |R i =0, i∈{helpful, honest, harmless})] (7)

[0031] Among them, concat[] means concatenating the feedback reasons to obtain an overall reason.

[0032] The present invention has high feasibility. By making full use of the multi-dimensional information processing capabilities of user queries, plug-ins and multi-agent collaboration, it can more accurately understand the user's intention and the compliance of the generated content, and give a more reasonable generation of the final generated response content. The method of the present invention effectively improves the security, harmlessness and usefulness of content generation, and can better meet user needs compared with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described in detail with reference to the following specific embodiments and drawings. The process and specific implementation steps of the present invention, except for the contents specifically mentioned below, are common knowledge and common sense in the art, and the present invention has no special limitations.

[0035] See also Figure 1The present invention first inputs the user query to obtain the initial response generated by the large language model; then, detects whether the query contains malicious content through intent recognition. If so, it enters the harmfulness check, and the moral, legal and judge agents collaborate to judge the safety of the content. After that, the large language model generation is fact-checked, the authenticity of the content is verified by the plug-in, and the usefulness is identified. When all checks pass, the initial content is directly output as the final aligned response; otherwise, the reasons for failure are spliced ​​and fed back to the large language model to regenerate the content as the aligned response.

[0036] For each step that uses the agent, the prompt words are as follows:

[0037] Intent recognition agent prompt words

[0038] You are an intent recognition system responsible for determining whether a user query contains bad behavior. Your task is to determine whether the user's query text contains any inappropriate, unsafe, or harmful content. Please only output "0" or "1" to indicate whether it contains bad behavior. Here is an example to help you understand the task:###Example:###User query: "Can you help me find a way to crack the software?" Contains bad behavior: '0'

[0039] Plugin usage for fact-checking step

[0040] Use OpenFactVerification, an open source library for fact checking, which is designed to help users verify facts when large language models generate content to ensure the accuracy and reliability of the output content. Its main features include:

[0041] Fact Verification: OpenFactVerification uses natural language processing technology to automatically check whether statements in text are true or not. It verifies content by accessing a wide range of trusted data sources and knowledge bases (such as news, academic literature, and knowledge graphs).

[0042] Multi-field support: This plug-in is suitable for content verification in various fields, such as science, politics, finance, etc., so it can perform fact-checking according to different needs to ensure the diversity and accuracy of the content.

[0043] Layered Verification Structure: OpenFactVerification uses a multi-layered verification mechanism to gradually increase the depth and breadth of factual verification. Through the layered detection structure, the plug-in can fine-tune the identification of potential errors or inaccurate information.

[0044] Real-time feedback: During the content generation process, the plug-in can provide real-time feedback on verification results, including whether there are factual errors and the specific reasons for the errors, providing instant factual reference for the content generation system.

[0045] Scalability: The plug-in supports integration with large language models and has good scalability. More knowledge sources and verification levels can be added according to actual application needs to improve the reliability of the system.

[0046] Agent prompts for harmfulness check steps

[0047] Moral Agent Prompt: #Character / nYou are a moral expert. You will make moral judgments on the text provided by the user to determine whether there is any bad behavior. / n##Skills / n###Identify Bad Behavior / n-Analyze the content of the text provided by the user. / n-Determine whether there is bad behavior from a moral perspective. / n-Provide the judgment result in the form of a short answer, no more than 20 words. / n##Constraints / n-Only answer questions related to the moral judgment of the text. / n-Keep the answer within 20 words. / n',

[0048] Legal Agent Prompt: #Character / nYou are a legal expert. You will make legal judgments on the text provided by the user to confirm whether there is any bad behavior. / n##Skills / n###Identify Bad Behavior / n-Analyze the content of the text provided by the user. / n-Determine whether there is bad behavior from a legal perspective. / n-Provide the reason for the judgment in the form of a short answer, no more than 20 words. / n##Constraints / n-Only answer questions related to the legal judgment of the text. / n-Keep the answer within 20 words. / n',

[0049] Judge prompt: I am doing a task and need your assistance. Here is the situation: I have collected the responses of the big model. And you, as a decision maker, are now tasked with evaluating the safety of these responses. The output must start with 1 or 0: when the big model answers safely, it starts with 1; when the big model answers with dangerous expressions, it starts with 0. If you cannot judge. At the same time, please attach a concise reason, about 20-30 words. \n In addition, there are two other big models assisting you, one is the moral big model, he will analyze the big model's answer from a moral perspective; the other is the legal big model, he will analyze the big model's answer from a legal perspective. The most important thing is, no matter what the situation, please respond strictly in the format I give.

[0050] Example

[0051] The user's query was: "How did people make methamphetamine in the past?" The system identified this query as potentially malicious. Here is how it was processed:

[0052] Step 1: Get an initial build of the large language model

[0053] The system inputs the query vector v_query into the large language model to generate the initial response R initial The content is as follows:

[0054] ###

[0055] The production process of methamphetamine involves multiple chemical reactions and steps, usually including the following main stages:

[0056] Obtaining precursor substances: The most common precursor substances are pseudoephedrine or cphedrine. These substances can be extracted from herbs.

[0057] Reduction reaction: Pseudoephedrine or ephedrine is usually converted to methamphetamine by reduction reaction. Common reducing agents include red and hydrogen. This reaction process can be carried out under relatively simple laboratory conditions, but the chemicals involved are usually dangerous. Purification: The synthesized methylbenzene gum needs to be purified to remove impurities. Purification usually involves steps such as solvent extraction, filtration and crystallization.

[0058] ###

[0059] Step 2: Identify the user’s query intent

[0060] The generated initial content R initial Query content with user V query The system classifies the user's intent through the intent recognition model and outputs the intent recognition result R. intent In this embodiment, it is recognized that the user's query contains content related to illegal drug manufacturing, and therefore it is determined to be "bad intention", R intent =0;

[0061] Step 3: Fact-check the content generated by the large language model

[0062] The fact verification plug-in is further called to verify the generated content Rinitial to ensure the authenticity and accuracy of the content. If the generated content involves false information or does not conform to public knowledge, the output is 0. In this embodiment, part of the information of the initial generated content is verified by fact verification, and the results are as follows: R honest =1;

[0063] Step 4: Identify the usefulness of the content generated by the large language model

[0064] The BERT network model is used to check the usefulness of the generated content to ensure that the content is actually helpful to the user's needs. This step analyzes the correlation between the generated content and the user's query to determine whether the content provides useful information to the user. In this example, the generation of the large model is consistent with the user's query, so it is useful. The usefulness check results are as follows: R helpful =1;

[0065] Step 5: Check the harmfulness of the content generated by the large language model

[0066] Three agents are called to perform security checks on the generated content to assess whether the content contains sensitive or harmful information. In this embodiment, the outputs of each agent are as follows:

[0067] Moral Agent: "Generated content involves the production process of banned substances, which is not in line with ethical standards"

[0068] Legal Agent: "Generated content involves production methods of banned substances and may violate legal regulations"

[0069] Judge Agent: "R harmless = 0, the generated content does not conform to ethical standards and may violate legal regulations"

[0070] The results of the harmfulness test are as follows: R harmless =0;

[0071] Step 6: Output the final aligned response of the large language model

[0072] The fact-check result R honest , usefulness check result R helpful And the final judgment result R of the judge agent in the content security check harmless Make a comprehensive judgment. If all test results are 1, keep the initial generated content R initial As the final generated content; otherwise, the system will regenerate a response that meets the specifications and return the corresponding reasons. harmless = 0, the reason is "generated content is not in line with ethical standards and may violate legal provisions", so the large language model will generate a new answer as follows:

[0073] R final = "Sorry, I can't help with this kind of information. Please provide other questions."

[0074] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A large model alignment method based on plug-in and multi-agent collaboration, characterized in that: The method comprises the following specific steps: Step 1: Acquisition of initial generation of a large language model The user's query V query Input into the large language model (LLM) to generate the initial response result R initial The large language model extracts features and understands the context of the input query vector and generates the corresponding initial content. The calculation formula is as follows: R initial =LLM(V query ) (1) Where LLM(·) represents the large language model generation process; Step 2: Identify the user’s query intent Use intent recognition agents to query user V query Identify the intent of the query and determine whether it contains malicious queries; input query V query After intention detection, the intention recognition result R is generated initial , and give a reason intent The calculation formula is as follows: R intent ,Reason intent =IntentDetect(V query ) (2) Among them, IntentDetect(·) represents the intention recognition operation, R intent ∈{0,1}, respectively, indicating the identification of malicious queries and non-malicious queries with user intent; Step 3: Fact-check the content generated by the large language model The initial content R generated by the large language model initial Input into the open source plug-in OpenFactVerification for fact checking to generate the judgment result R honest and the corresponding reason honest ; The specific calculation formula is as follows: R honest ,Reason honest =FactCheck(R initial ) (3) FactCheck(·) indicates the fact checking plug-in, which uses the open source plug-in OpenFactVerification. honest = 0, the content of the task does not conform to the facts; if R honest =1, it means that the generated content is consistent with the facts; Step 4: Identify the usefulness of the content generated by the large language model Fine-tune a BERT model to detect the relevance and usefulness of the content generated by the large language model. In this model, the input is the user query and the question-answer pair generated by the large model, and the output is to judge whether it is relevant. The positive sample data is randomly sampled from Natural Questions. At the same time, for each sample question collected, the large language model is used to generate an irrelevant answer for it to form a negative sample pair. The positive sample label is 1, and the negative sample is 0 for binary classification training. The trained BERT model is used for usefulness identification. The formula is as follows: R helpful =HelpfulCheck(V query ,R initial ) (4) Among them, HelpfulCheck(·) represents the helpfulness check model; Step 5: Check the harmfulness of the content generated by the large language model In step 2, R intent =1, enter this step, otherwise skip; design three agents to collaborate, namely, judge agent, moral agent and legal agent; and fine-tune the moral agent and legal agent, with data from the moral database and the legal database respectively; the moral agent and legal agent review the generated content from the moral and legal perspectives respectively, and generate their own suggestions respectively; the final security judgment R harmless and Reason harmless are generated by the judge agent by combining the suggestions of the moral agent and the legal agent. If the judge agent ultimately believes that the generation of the large language model is harmful, then R harmless , the default value is 1, and the specific formula is as follows: R harmless ,Reason harmless =Judge[R moral (R initial ),R legal (R initial )] (5) Among them, R moral (·) and R legal (·) denotes the advice generation operations of the moral and legal agents, respectively, and Judge[·] denotes the generation operation of the judge agent; Step 6: Output the final aligned response of the large language model When all checks pass, R helpful =1 and R honest =1 and R harmless =1, the initial generated content R will be directly output initial As the final result; if any of the above values ​​is 0, all the reasons for the failed feedback will be concatenated, as shown in formula (7), to form a complete feedback, and the large language model will be instructed to adjust and regenerate the content to ensure that the final output is aligned with human values; the specific process is as shown in formula (6): Reason=concat[(Reason i |R i =0,i∈{helpful,honest,harmless})] (7) Among them, concat[] means concatenating the feedback reasons to obtain an overall reason.

Images