Preference code generation method and system based on human feedback reinforcement learning
Through multi-angle preference ranking method and retrieval enhanced context learning based on human feedback reinforcement learning, the problem of large language models not being able to align user preferences in real-scene code community Q&A is solved, and higher quality and adaptable code generation are achieved. API update.
Patent Information
- Application Number
- CN202411213648.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Large language models cannot accurately achieve user preference alignment in real-scenario code community Q&A, and the existing technology fails to effectively consider the preference changes brought about by diversified user preferences and API updates.
Adopting a multi-angle preference ranking method based on human feedback reinforcement learning, through supervised fine-tuning and multi-perspective answer preference metric score integration, large language models are optimized to make the generated answers meet user preferences, and solving API updates through retrieval enhanced context learning. Outdated issue.
It significantly improves the quality and adaptability of the code generation model, makes its output more in line with user expectations and preferences, solves the preference alignment problem in multi-angle community code Q&A tasks, and has strong practical value.
Smart Images

Figure CN119065644A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a preference code generation method and system based on human feedback reinforcement learning. Background Art
[0002] Large Language Models (LLMs) have shown successful applications in the field of open-domain question answering (QA). In order to align LLMs with user preferences in domain-specific QA tasks, reinforcement learning with human feedback (RLHF) is usually adopted. RLHF can effectively achieve precise generation behavior control and enable the model to generate human-like responses. In recent years, real-world code community question answering (CCQA) has become increasingly important in academia and industry, focusing on the question-and-answer interactions of users in code communities (e.g., StackOverflow). Different from traditional QA tasks, CCQA exhibits three significant characteristics: First, a question usually has more than one answer. Nearly 46% of questions have more than two answers, and some questions even have as many as 30 answers. Second, each answer contains not only text content, but also other interactive elements, such as votes from other users, reflecting rich user preferences. Third, different users show different preferences for different answers to a given question. Based on the above characteristics, in order to enable LLMs to be effectively applied to CCQA tasks and generate responses that meet the preferences of different users, some key issues still need to be addressed. However, the application of LLMs to CCQA tasks and the related preference alignment research is still an underexplored area.
[0003] In the prior art, some studies have attempted to align LLMs responses with human preferences by using accepted answers as alignment targets. However, accepted answers may not reflect the preferences of all users, as the answer selected by the questioner may not be favored by other users. Second, although some studies have begun to focus on the entire answer and introduce content-based ranking methods, the inherent preferences of diverse user preferences and LLMs feedback in CCQA have not yet been taken into account. Third, as APIs are updated in the code community, people's preferences will also change, as they tend to choose newer API versions. Therefore, accepted answers may be affected by being outdated, as APIs are updated very quickly in the programming field.
[0004] In order to overcome the above problems, the present invention proposes a novel multi-angle preference ranking method, which can effectively improve the quality and adaptability of the code generation model, while ensuring that the generated answers meet the user's preferences and expectations, and realize multi-angle preference alignment in the code community question and answer task. Summary of the invention
[0005] In view of this, the present invention provides a preference code generation method and system based on human feedback reinforcement learning, so as to solve the technical problem that the current large language model cannot accurately achieve user preference alignment in CCQA question-answering scenarios.
[0006] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:
[0007] In a first aspect, the present invention provides a method for generating preference codes based on human feedback reinforcement learning, comprising:
[0008] The preset large language model is optimized based on the supervised fine-tuning method to adapt the model to the question-answering scenario of the code community, thereby obtaining the first answer generation model;
[0009] Obtain multi-perspective answer preference measurement scores for answers to code community questions, and obtain an overall answer preference score based on the multi-perspective answer preference measurement scores;
[0010] With the goal of generating the highest overall preference score for the answer, the first answer generation model is optimized in parameters so that the model output is aligned with the user preference, thereby obtaining a second answer generation model;
[0011] Obtain a code community question to be answered, search for a question-answer pair most similar to the question in a question-answer library based on a preset retriever, and obtain a question prompt for the question;
[0012] The code community question containing the question prompt is input into the second answer generation model to obtain a code generation answer corresponding to the code community question.
[0013] Furthermore, the optimization of the preset large language model based on the supervised fine-tuning method includes:
[0014] Acquire question and answer data of the code community, wherein the question and answer data at least includes answer content and answer acceptance data corresponding to the prompt template;
[0015] Building a data set based on the question-answer data, and optimizing a preset large language model using the data set to obtain a first answer generation model;
[0016] The optimization objective is expressed as:
[0017]
[0018] in, Indicates the accepted answer The jth mark of i Indicates a question, I indicates a question prompt template, P M Represents the label probability predicted by model M.
[0019] Furthermore, the multi-perspective answer preference metric score includes:
[0020] a questioner perspective bias score, which assesses the difference between answers accepted by the questioner and answers not accepted;
[0021] User perspective voting score, which is used to reflect the collective preference of users for answers;
[0022] Model-perspective content score for evaluating the quality of text in answers.
[0023] Furthermore, with the goal of generating the highest overall preference score for the answer, the first answer generation model is parameter optimized to align the model output with the user preference, thereby obtaining a second answer generation model, including:
[0024] Determine the preference probability optimization target based on the overall preference score of the answer;
[0025] According to the preference probability optimization target, the first answer generation model is optimized by a contrastive learning method so that the model output is aligned with the user preference and the text quality of the answer is improved, thereby obtaining a second answer generation model.
[0026] Furthermore, the preference probability optimization objective is:
[0027]
[0028] Where N i Represents question q i Answer Pool The size of I1 is the hint containing question q. represents the answer preference ranking set; M2(I1,p k ) represents the second answer generation model after parameter optimization.
[0029] Furthermore, according to the preference probability optimization target, the first answer generation model is optimized by a contrastive learning method, including:
[0030] The total optimization objective loss function in the contrastive learning method is:
[0031]
[0032] Where I1 represents the prompt containing question q, L MSFT represents the NLL loss, α represents the hyperparameter used to balance text quality and human preference, and L MPRA represents the loss for optimizing the preference probability.
[0033] Furthermore, the loss L of the preference probability optimization MPRAThe formula is:
[0034]
[0035] Where N i Represents question q i Answer Pool The size of I1 indicates the hint containing question q, M2(I1,p k ) represents the second answer generation model after parameter optimization.
[0036] Furthermore, the code community question containing the question prompt is input into the second answer generation model to obtain a code generation answer corresponding to the code community question, including:
[0037] Inputting the code community question to be answered containing the question prompt into the second answer generation model, the second answer generation model obtaining the current text markup of the target answer according to the context information of the question to be answered, the question-answer pair and the target answer sequence;
[0038] The predicted target probability of the current token in the output answer sequence is expressed as:
[0039]
[0040] In the formula, M2 represents the second answer generation model, Q represents the problem to be solved, I2 represents the problem prompt, (s q ,s a ) indicates a question-answer pair, A <t Indicates that the output answer sequence is at the current tag A t Previous mark.
[0041] On the other hand, the present invention also provides a preference code generation system based on human feedback reinforcement learning, comprising:
[0042] A first optimization module is used to optimize a preset large language model based on a supervised fine-tuning method to adapt the model to the question-answering scenario of the code community, thereby obtaining a first answer generation model;
[0043] A preference calculation module, used to obtain multi-perspective answer preference measurement scores corresponding to answers to questions in the code community, and to obtain an overall answer preference score based on the multi-perspective answer preference measurement scores;
[0044] A second optimization module is used to optimize the parameters of the first answer generation model with the goal of generating the highest overall preference score of the answer, so as to align the model output with the user preference, thereby obtaining a second answer generation model;
[0045] An enhanced search module is used to obtain code community questions to be answered, and based on a preset search engine, search for question-answer pairs that are most similar to the question in the question-answer library to obtain question prompts for the question;
[0046] The answer generation module is used to input the code community question containing the question prompt into the second answer generation model to obtain the code generation answer corresponding to the code community question.
[0047] In a third aspect, the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, a preference code generation method based on human feedback reinforcement learning as described in any of the above technical solutions is implemented.
[0048] Compared with the prior art, the method of the present invention, firstly, optimizes the large language model through the supervised fine-tuning method to adapt it to the question-answering scenario of the code community, which can significantly improve the performance of the model in the specific code generation field, making it more capable of understanding and generating code-related content; secondly, the quality of different answers and user preferences are comprehensively considered through multi-perspective answer preferences, so as to select the answer that best meets the user's expectations, optimize the parameters for the highest overall preference score of the generated answer, improve the model, effectively improve the generation ability of the model, and make its output more in line with the user's expectations and preferences. Finally, the preset retriever is used to search the most similar answers in the answer library in real time, provide question prompts for the code generation questions to be answered, and solve the problem of potential outdated code. The method of the present invention, based on the unique characteristics of CCQA answers, can effectively improve the quality and adaptability of the code generation model, while ensuring that the generated answers meet the user's preferences and expectations, and achieve preference alignment in the multi-perspective community code question-answering tasks, which has a strong practical value in the industrial practice of actual application. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A schematic diagram of an embodiment of a code community question and answer provided by the present invention;
[0050] Figure 2 A schematic diagram of a flow chart of a preference code generation method based on human feedback reinforcement learning provided by the present invention;
[0051] Figure 3 A schematic diagram of the overall framework of the preference code generation method based on human feedback reinforcement learning provided by the present invention;
[0052] Figure 4 A schematic diagram showing the comparison between the MupCCQA method provided by the present invention and the traditional human feedback reinforcement learning method;
[0053] Figure 5A schematic diagram of the structure of a preference code generation system based on human feedback reinforcement learning provided by the present invention;
[0054] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0055] Before introducing the embodiments of the present invention, relevant terms in this application are first explained.
[0056] Large Language Models (LLMs): refers to complex natural language processing models trained using large-scale neural networks. These models are usually based on deep learning technology, have billions to tens of billions of parameters, and can process and generate natural language text. They have shown strong capabilities in natural language understanding (NLU) and natural language generation (NLG) and can be used for a variety of tasks such as text classification, question-answering systems, language translation, summary generation, etc. They can understand and generate high-quality natural language text, making human-computer interaction more intelligent and natural.
[0057] Reinforcement Learning from Human Feedback (RLHF): is a method that combines human expert feedback with machine learning algorithm learning. The main idea of RLHF is to introduce feedback from human experts to improve the efficiency and robustness of reinforcement learning. This feedback can be in many forms, such as direct reward or penalty signals, instructive demonstrations, or evaluation and adjustment suggestions for strategic behaviors. By combining the intuitive knowledge of human experts and the learning ability of the algorithm itself, challenges in the environment can be overcome faster, and unnecessary trial and error and learning costs can be reduced, thereby improving the performance and safety of the system.
[0058] Code Community Question Answering (CCQA): A method or platform to improve software development quality and developer capabilities by collecting and analyzing questions and answers in real-world code. It is different from traditional programming Q&A communities (such as Stack Overflow) because its main goal is to extract problems from real-world code and development processes and find corresponding solutions.
[0059] Combine the following Figure 1 The inventive concept of the present invention is described.
[0060] like Figure 1 As shown, Figure 1shows an example of a code community Q&A. It includes key elements: a question Q, a set of answers {a1,...,a9}. Each a i contains its content text, the number of votes, and a label indicating whether the answer is accepted by the questioner. In addition, in the semantic vector space, the LLM basis answer a l There is a certain distance between the answer a2 accepted by the questioner and the answer a1 preferred by the user. Different users show different preferences for different answers to a given question. For example, a questioner asks question Q and accepts answer a2 in the answer pool, while some users prefer answer a1 which has received the highest vote.
[0061] In code question answering, LLMs may generate redundant responses (e.g. Figure 1 a l ), but using RLHF can effectively achieve precise generation behavior control. However, unlike traditional QA tasks, CCQA exhibits three significant characteristics. First, a question usually has more than one answer, as shown in Table 1. From Table 1, it can be seen that nearly 46% of the questions have more than two answers, and some questions even have as many as 30 answers. Second, each answer contains not only text content, but also other interactive elements. Third, different users show different preferences for different answers to a given question.
[0062] Table 1 Statistics of the answer pool size for each question
[0063] Counting interval [0,2) [2,5) [5,10) [10,15) [15,20) [20,25) [25,30] total count 325,780 245,793 21,986 2,057 572 203 222 596,613 percentage(%) 54.60 41.20 3.68 0.35 0.10 0.03 0.04 100
[0064] In the prior art, in order to enable LLMs to be effectively applied to CCQA tasks and generate responses that satisfy different user preferences, the accepted answers are usually used as alignment targets to align the responses of LLMs with human preferences, but the accepted answers may not reflect the preferences of all users. The solution to solve the preference alignment by introducing content-based ranking methods does not take into account the diverse user preferences in CCQA and the inherent preferences of LLMs feedback. In addition, as the API is updated in the code community, people's preferences will also change, because they tend to choose the newer API version, causing the accepted answers to be affected by being outdated, because in the programming field, the API is updated very quickly. For example Figure 1 The "urllib" API in answer a2 is suitable for Python2, but has been deprecated in Python3.
[0065] Therefore, the present invention proposes a code community answer generation method and system based on human feedback reinforcement learning, aligns preferences from a ranking perspective, considers the unique characteristics of CCQA answers, and proposes RIL to solve the problem of potentially outdated code.
[0066] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0067] The present invention provides a preference code generation method and system based on human feedback reinforcement learning, which are described below respectively.
[0068] Combination Figure 2 As shown, a specific embodiment of the present invention discloses a preference code generation method based on human feedback reinforcement learning, comprising:
[0069] Step S101: Optimizing a preset large language model based on a supervised fine-tuning method to adapt the model to the question-answering scenario of the code community, and obtaining a first answer generation model;
[0070] Step S102: obtaining a multi-perspective answer preference metric score for the code generation question, and calculating an overall answer preference score based on the multi-perspective answer preference metric score;
[0071] Step S103: with the goal of generating the highest overall preference score for the answer, the first answer generation model is optimized for parameters so that the model output is aligned with the user preference, thereby obtaining a second answer generation model;
[0072] Step S104: Obtain a code community question to be answered, and search the question-answer pair most similar to the question in the question-answer library based on a preset search engine to obtain a question prompt for the question;
[0073] Step S105: inputting the code community question containing the question prompt into the second answer generation model to obtain a code generation answer corresponding to the code community question.
[0074] The method of this embodiment, first, optimizes the large language model through the supervised fine-tuning method to adapt it to the question-answering scenario of the code community, which can significantly improve the performance of the model in the specific code generation field, making it more capable of understanding and generating code-related content; secondly, through the multi-perspective answer preference, the quality of different answers and user preferences are comprehensively considered, so as to select the answer that best meets the user's expectations, optimize the parameters for the highest overall preference score of the generated answer, improve the model, effectively improve the generation ability of the model, and make its output more in line with the user's expectations and preferences. Finally, the preset retriever is used to search the most similar answers in the answer library in real time, provide question prompts for the code generation questions to be answered, and solve the problem of potential outdated code. The method of this embodiment can effectively improve the quality and adaptability of the code generation model, while ensuring that the generated answers meet the user's preferences and expectations, and realize preference alignment in the multi-perspective community code question-answering task, which has a strong practical value in the industrial practice of actual application.
[0075] As a specific example, we first explain the overall goal of this example. Our overall goal is to design a multi-perspective preference alignment to guide the large language model M to comprehensively consider the real-world code community question-answering dataset when generating answers. Different user preferences reflected in the
[0076] Among them, q i represents the i-th question, Represents question q i The answer pool; for each answer a in the answer pool, a=(c,v,a c ), c represents the content of answer a, v represents the number of votes for answer a, a c ∈{0,1} indicates whether the answer a is accepted by the questioner. It should be noted that, in form, any q i or c is a tag sequence, represented by t = {t i ∣t i ∈C or t i ∈T}, where t i represents the i-th token in set t, C represents the code set, and T represents the text set.
[0077] The method of this application is used to align LLMs in CCQA, and this method is referred to as MupCCQA below. Figure 3 As shown in Figure 2, the implementation framework of MupCCQA consists of three stages:
[0078] The first stage is supervised fine-tuning (SFT), which is used to acquire programming-specific knowledge to adapt to the application scenario of CCQA; the second stage is multi-view preference ranking alignment (MPRA) to integrate diverse preferences; the third stage is retrieval-enhanced contextual learning (RIL), which solves the problem of outdated answers by retrieving the most similar post-solution pairs as hints. The three stages are explained below.
[0079] (1) Phase 1: Basic Supervision Fine-tuning
[0080] As a preferred embodiment, the optimization of the preset large language model based on the supervised fine-tuning method includes:
[0081] The method of optimizing the preset large language model based on supervised fine-tuning includes:
[0082] Acquire question and answer data of the code community, wherein the question and answer data at least includes answer content and answer acceptance data corresponding to the prompt template;
[0083] Building a data set based on the question-answer data, and optimizing a preset large language model using the data set to obtain a first answer generation model;
[0084] Typically, the underlying large language models are trained on open domain corpora. To adapt these general large language models to the programming-specific code community corpus, we employ supervised fine-tuning. Specifically, we apply a i Middle c = 1 (the answer is accepted) is represented by And select pairs with more than 100 votes from dataset D To construct training and validation datasets, the large language model M is optimized with the following training objectives:
[0085] The training objective is expressed as:
[0086]
[0087] In formula (1), Indicates the accepted answer The jth mark of i Indicates a question, I indicates a question prompt template, P M Represents the label probability predicted by model M.
[0088] Based on the above optimization objectives, the model M is supervised and fine-tuned with real-scene code community question and answer data as the basic supervision information to adapt to the programming-specific question and answer scenario (CCQA), and the first answer generation model is obtained, which is denoted as M1.
[0089] (2) The second stage, multi-view preference ranking alignment (MPRA)
[0090] In order to align the multi-view preferences from different users, we introduce the Multi-view Preference Ranking Alignment (MPRA) method. The multi-view answer preference metric score includes:
[0091] a questioner perspective bias score, which assesses the difference between answers accepted by the questioner and answers not accepted;
[0092] User perspective voting score, which is used to reflect the collective preference of users for answers;
[0093] Model-perspective content score for evaluating the quality of text in answers.
[0094] Specifically, first, since the answer selected by the questioner may not be favored by other users, we introduce a questioner perspective bias score to evaluate the difference between the accepted answer and the answer preferred by most users, denoted as s q , the mathematical expression is as follows:
[0095]
[0096] In formula (2), represents the number of votes for each answer in question q, v a represents the number of votes for the answer accepted by the questioner, v m and v σ They are The mean and standard deviation of .
[0097] Second, since high-quality texts are usually accompanied by a higher number of votes, the interaction data from open communities not only reflects the preferences of community users, but also serves as a dual filtering mechanism for the quality of answer content.
[0098] Unlike the specific answer preferred by the questioner, the number of votes usually reflects the prevalence and generality of the answer, indicating its applicability and transferability to other contexts. In the field of software engineering, similar problems may appear in different environments, and the ability to generalize and apply solutions in various situations is an important capability in this field, emphasizing the importance of community-driven feedback and collective wisdom.
[0099] Therefore, in order to comprehensively consider users’ collective preferences and interactions with answers, we introduce the voting score from the user’s perspective, denoted as s u , the mathematical expression is as follows:
[0100]
[0101] In formula (3), v∈V, is a given answer pool min(V) and max(V) denote the minimum and maximum values in V, respectively. This normalization ensures that the votes are adjusted to a common scale, facilitating fair comparison between different answers.
[0102] Third, since high semantic accuracy is the basic premise for answering questions, we introduce a content score from the perspective of LLM, denoted as s l This score aims to evaluate the quality of the text c in answer a by leveraging LLM, which is good at handling subtle semantic relationships between text and code.
[0103] It should be noted that the LLM model used in this application can be general or customized for the code domain, and usually has excellent understanding and reasoning capabilities. Here we choose an easily accessible LLM (denoted as cM) to measure each question-answer pair (q i ,a i ), where a i From the collection Elements of. Content score s l It is calculated by calculating the product of the probabilities of each marker generated by cM, and the calculation formula is as follows:
[0104]
[0105] In formula (4), I1 represents the integrated question-answer pair (q i ,a i ), the product is taken from the answer a i All the symbols t,σ in represent logistic functions.
[0106] In order to comprehensively evaluate the answer pool and construct a preference sequence, we introduced an overall preference score r to integrate the scores of the above perspectives into a unified metric. The preference set is then ranked according to the size of r, and the answer a with the highest score becomes the answer with the highest predicted probability.
[0107] Specifically, the calculation formula for the overall preference score r is:
[0108] r=α1·s q +α2·s u +α3·s l (5)
[0109] In the formula, s q 、s u and l are the questioner's perspective bias score, user's perspective voting score and language model's perspective score respectively. α1, α2 and α3 are the weights corresponding to the questioner's perspective bias score, user's perspective voting score and language model's perspective score respectively.
[0110] As can be seen, the overall preference score r captures the essence of user engagement, questioner alignment, and content quality as assessed by the LLM. The weights α1, α2, and α3 reflect the relative importance of each component in the multi-view preference set. Therefore, the overall preference score r provides a nuanced, multi-dimensional assessment that can guide fine-tuning the LLM to better align with the complex dynamics of community-driven question answering.
[0111] To facilitate comparison of multiple responses, we extend the Bradley-Terry model through Multi-view Preference Ranking Alignment (MPRA), shifting the focus from reward model-centric approaches to directly adjusting the N generated by LLMs. i The probability ranking of the answers is aligned with the overall preference score r.
[0112] Here, N i Represents question q i Answer Pool The size of the problem q i The initial preference ranking set is expressed as Will Divided into p1 and Two parts. The target preference probability of the preference optimization model is:
[0113]
[0114] In formula (6), N i Represents question q i Answer Pool The size of I1 indicates the hint containing question q, M2(I1,p k ) represents the second answer generation model after parameter optimization.
[0115] As the i-th iteration unfolds, MPR systematically eliminates the top i answers with higher r scores. This process repeats until all answers are eliminated. This iterative refinement continues sequentially until all potential solution sets are exhausted, and the target preference probability of the final preference optimization model evolves to:
[0116]
[0117] It can be seen from the above formula (7) that this optimization goal is closely related to the ultimate goal of human alignment. When Ni→∞, formula (7) can exhaustively explore all potential responses generated by LLMs. When Ni=2, MPRA is transformed into the basic Bradley-Terry model, but still effectively optimizes LLM. This process iteratively aligns human preferences by designating the most popular response with the highest r score as a positive response, while the remaining responses are regarded as negative responses, and the desired response is selected from the wide response space of LLMs.
[0118] Different from the method that forces LLM to approximate the reward model, MPRA directly trains LLM to optimize the probability ranking of preference sets through list comparison learning objectives. Its optimization objective, namely the loss of preference probability optimization, is expressed as (8), which aims to achieve the target preference probability of (7).
[0119]
[0120] In formula (8), N i Represents question q i Answer Pool The size of I1 indicates the hint containing question q, M2(I1,p k ) represents the second answer generation model after parameter optimization.
[0121] MPRA not only needs to generate the most popular responses, but also needs to enhance text fluency and code structure. Figure 3 The new SFT stage proposed in this method is shown in Figure 2. Its loss is the same as the loss in the basic SFT stage described in Eq. (1), denoted as L MSFT The key difference is that during the base SFT phase, the alignment target was the answer accepted by the questioner, whereas now the alignment target for LLM is the answer with the highest overall preference score r.
[0122] The final overall optimization objective loss function can be summarized as follows:
[0123]
[0124] In formula (9), I1 represents the prompt containing question q, L MSFT represents the NLL loss, α represents the hyperparameter used to balance text quality and human preference, and L MPRA represents the loss for optimizing the preference probability.
[0125] (3) The third stage: Retrieval-enhanced contextual learning (RIL)
[0126] Since the accepted answers may be outdated with the rapid API updates in the programming field, we introduce Retrieval Enhanced Context Learning (RIL) to solve the problem of outdated answers and keep in line with the trend of user preference for new APIs. The effectiveness of generating responses is enhanced by retrieving similar questions from the question bank and using them as a small number of examples.
[0127] We leverage a dense retriever (RD) that excels at handling the transition from natural language to code generation, trained to extract documents from a wide range of repositories including a large number of code bases, APIs, and functions. Since the order of few-shot examples significantly affects model predictions, although in context learning it can still perform well when the order or labels of prompts are swapped, we select the most similar question-answer (s q ,s a ) for the few-shot example that serves as hint I2.
[0128] The final goal is as follows:
[0129]
[0130] In formula (10), M2 represents the second answer generation model, Q represents the problem to be solved, I2 represents the problem prompt, (s q ,s a ) indicates a question-answer pair, A <t Indicates that the output answer sequence is at the current tag A t Previous mark.
[0131] This formula captures the next token A in the generated response sequence. t , given that the probability of q ,s a ) and the response sequence A t The formula integrates contextual information from the question, few-shot examples, and response history to generate the next token in a coherent and context-aware manner.
[0132] In order to comprehensively evaluate the actual effect of the present invention, we use various evaluation indicators from four perspectives: traditional text generation indicators (BLEU, Rouge, and CHRF), model-based indicators (BERTScore), code-related indicators (CodeBERTScore), and preference indicators based on GPT-4 evaluation. In addition, considering the similarity of Precision and Recall in CodeBERTScore, we unify these indicators into "CodeBERTScore-PR" (CB-PR for short). Similarly, F1 and F-measure in CodeBERTScore are merged into "CodeBERTScore-F" (CB-F for short).
[0133] Since the classification of LLMs can be determined by the openness of the technology and whether the code is available for research or commercial use, and based on the unique characteristics of CCQA, we selected two categories of baseline models. The first category is general closed-source LLMs designed for text generation, including GPT-3.5-turbo, GPT-4, PaLM, ChatGLM, and Claude2. The second category is open-source LLMs that perform well in program synthesis, such as StarCoder, WizardCoder-Python-13B, GPT-NeoX, CodeGen-mono-16B, and Code Llama 2.
[0134] In the experiments, we choose Code Llama-Instruct-7B as the base model M, which belongs to a series of large code language models based on Llama 2. Code Llama shows state-of-the-art performance on various code benchmarks. We use the MuPCCQA framework for three-stage fine-tuning. In the SFT stage, we specify the following hyperparameters: epoch, temperature, top_p, max_seq_len, and max_batch_size, which are set to 4, 0.2, 0.95, 2048, and 28, respectively. We keep the rest of the hyperparameter settings of Llama. In the MPRA stage, we choose an accessible LLM as cM. The hyperparameter settings are as follows: learning_rate, gradient_accumulation_steps, epochs, top_p, max_gen_len, temperature, and max_batch_size, which are set to 1e-4, 9, 4, 1.0, 0.95, 512, and 4, respectively. Given the excellent performance of this retrieval-generation method in understanding diverse texts and codes, we chose the SimCSE-based DocPrompting method as our retriever RD in the Retrieval Enhanced Context Learning (RIL) stage. This retriever RD includes 35,763 functions from all Python libraries on DevDocs, covering the Python standard library and widely used packages such as NumPy and Pandas, and is pre-trained on the re-scaled CoNaLa benchmark.
[0135] It should be noted that due to the lack of high-quality, realistic multi-answer preference datasets in the code community, there is an urgent need to build a new dataset. To fill this gap, we systematically collected all answers to each question (q) on StackOverflow as of August 2023, resulting in a total of 596,613 pairs of data. We then performed the following preprocessing steps, and the resulting dataset (D) contains 270,716 question-answer pairs.
[0136] The preprocessing steps include: To ensure that commit messages are descriptive, we remove pairs whose title length is less than three tokens (including three tokens). This decision follows CCT5, which stipulates that code comments should contain more than three tokens. <code>...< / code> The answer pairs of the content are cleaned to ensure that the reference content of MupCCQA contains both text and code, considering the characteristics of CCQA. Pairs with answer pool size less than 2 are discarded. All HTML tags are cleaned and replaced with "[HTML]", especially <a href...>and<img...> mark to ensure that the model is not affected by such overly complex and meaningless content.
[0137] like Figure 4 As shown, Figure 4This is a comparison chart between the traditional method and the method of this application. Here we briefly introduce reinforcement learning from human feedback (RLHF). This method also mainly includes three stages. The first stage is to perform supervised fine-tuning on a large language model. The second stage uses the supervised fine-tuned model to generate response pairs. These response pairs have a preference order, but the optimization goal of this stage is defined as a binary classification problem for training the reward model. Current work usually uses the Bradley-Terry (BT) model, which defines the preference probability as follows:
[0138]
[0139] In formula (11), r φ Essentially a binary classification reward model, I1 is the QA prompt containing the question q. The optimization objective at this stage is defined as a binary classification problem:
[0140] L BT = -logσ(r φ (I1,p i )-r φ (I1,p j )) (12)
[0141] As can be seen from formula (12), a2 is compared with the user preference answer a1 with the highest votes, and two meta-candidates p are sampled from the entire ranking. i >p j To train the reward model, and then rely on this reward model to fine-tune the base LLM.
[0142] In the third stage, RLHF uses the obtained r φ To provide feedback to the adjusted LLM model, σ is a logistic function. Specifically, the optimization problem of RLHF is formalized as follows:
[0143]
[0144] Based on the above analysis, assuming that the preference ranking set Each answer in p i have been trained to resemble human responses, the RLHF method based on pairwise comparisons of the Bradley-Terry model may not be sufficient to handle, for example, Figure 4 The p shown in i >p j In this case, the role of ξ is to adjust the deviation from the baseline reference strategy M1, but this method is inaccurate for CCQA question-answering scenarios with more than two answers.
[0145] from Figure 4As can be seen from the above analysis, the conventional SFT only aligns the answer a2 accepted by the questioner, while RLHF compares a2 with the user-preferred answer a1 with the highest votes, sampling two meta-candidates p from the entire ranking. i >p j To train the reward model, and then rely on this reward model to fine-tune the basic LLM. The method of the present invention converts p i With all members of the preference set The comparison is based on the overall preference score r, which includes the bias score s q , voting scores u and content scores l .
[0146] Finally, we demonstrate the accuracy of the responses generated by the proposed MupCCQA method on our carefully designed dataset StaCCQA through extensive experiments. Compared with the baseline model, the BLEU score is improved by nearly 11%, and the BERTScore and CodeBERTScore are improved by 20% and 17.5%, respectively. In addition, the GPT-4 evaluation confirms the improvement of accuracy-based metrics, and the preference score also shows improvement, indicating the effectiveness of our method in aligning preferences in CCQA tasks. Overall, we highlight a novel point of view: considering the diversity of users when aligning with human preferences.
[0147] like Figure 5 As shown, the embodiment of the present invention further provides a preference code generation system 500 based on human feedback reinforcement learning, including:
[0148] A first optimization module 501 is used to optimize a preset large language model based on a supervised fine-tuning method, so that the model is adapted to the question-answering scenario of the code community, and obtain a first answer generation model;
[0149] A preference calculation module 502 is used to obtain a multi-perspective answer preference measurement score corresponding to the answer to the code community question, and integrate the multi-perspective answer preference measurement scores to obtain an overall answer preference score;
[0150] A second optimization module 503 is used to optimize the parameters of the first answer generation model with the goal of generating the highest overall preference score of the answer, so as to align the model output with the user preference, and obtain a second answer generation model;
[0151] An enhanced search module 504 is used to obtain a code community question to be answered, and search for a question-answer pair that is most similar to the question in the question-answer library based on a preset search engine to obtain a question prompt for the question;
[0152] The answer generation module 505 is used to input the code community question containing the question prompt into the second answer generation model to obtain the code generation answer corresponding to the code community question.
[0153] like Figure 6 As shown, the above-mentioned method for generating preference codes based on human feedback reinforcement learning, the present invention also provides an electronic device 600 accordingly, which can be a computing device such as a mobile terminal, a desktop computer, a notebook, a palm computer and a server. The electronic device includes a processor 601, a memory 602 and a display 603.
[0154] In some embodiments, the memory 602 may be an internal storage unit of a computer device, such as a hard disk or memory of a computer device. In other embodiments, the memory 602 may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory 602 may also include both an internal storage unit of the computer device and an external storage device. The memory 602 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 602 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a preference code generation method program 604 based on human feedback reinforcement learning is stored on the memory 602, and the preference code generation method program 604 based on human feedback reinforcement learning can be executed by the processor 601, thereby realizing a preference code generation method based on human feedback reinforcement learning in each embodiment of the present invention.
[0155] In some embodiments, the processor 601 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 602, such as executing a preference code generation method program based on human feedback reinforcement learning.
[0156] In some embodiments, the display 603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 603 is used to display information on the computer device and to display a visual user interface. The components 601-603 of the computer device communicate with each other through a system bus.
[0157] The present invention proposes a method and system for generating preference codes based on human feedback reinforcement learning. This method introduces a preference ranking alignment method, which iteratively utilizes these three scores to determine the preference order of answers, and uses list contrast loss to optimize the alignment with user preferences to adapt to diverse user preferences. In order to solve the problem that users tend to use new APIs, we implemented a retrieval-enhanced contextual learning (RIL) module, which retrieves answers to similar questions from the question library through RIL and uses them as few-sample examples to enhance the generated responses to alleviate the problem of outdated information. In addition, due to the current lack of relevant datasets, we also constructed a high-quality dataset and experimentally verified the effectiveness of this method in terms of accuracy and user preferences.
[0158] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A method for generating code community answers based on human feedback reinforcement learning, characterized in that: include: The preset large language model is optimized based on the supervised fine-tuning method to adapt the model to the question-answering scenario of the code community, thereby obtaining the first answer generation model; Obtain multi-perspective answer preference measurement scores for answers to code community questions, and obtain an overall answer preference score based on the multi-perspective answer preference measurement scores; With the goal of generating the highest overall preference score for the answer, the first answer generation model is optimized in parameters so that the model output is aligned with the user preference, thereby obtaining a second answer generation model; Obtain a code community question to be answered, search for a question-answer pair most similar to the question in a question-answer library based on a preset retriever, and obtain a question prompt for the question; The code community question containing the question prompt is input into the second answer generation model to obtain a code generation answer corresponding to the code community question.
2. The method for generating code community answers based on human feedback reinforcement learning according to claim 1, characterized in that: The method of optimizing the preset large language model based on supervised fine-tuning includes: Acquire question and answer data of the code community, wherein the question and answer data at least includes answer content and answer acceptance data corresponding to the prompt template; Building a data set based on the question-answer data, and optimizing a preset large language model using the data set to obtain a first answer generation model; The optimization objective is expressed as: in, Indicates the accepted answer The jth mark of i Indicates a question, I indicates a question prompt template, P M Represents the label probability predicted by model M.
3. The method for generating code community answers based on human feedback reinforcement learning according to claim 1, characterized in that: The multi-view answer preference metric score includes: a questioner perspective bias score, which assesses the difference between answers accepted by the questioner and answers not accepted; User perspective voting score, which is used to reflect the collective preference of users for answers; Model-perspective content score for evaluating the quality of text in answers.
4. The method for generating code community answers based on human feedback reinforcement learning according to claim 1, characterized in that: With the goal of generating the highest overall preference score for the answer, the first answer generation model is optimized in parameters so that the model output is aligned with the user preference, thereby obtaining a second answer generation model, including: Determine the preference probability optimization target based on the overall preference score of the answer; According to the preference probability optimization target, the first answer generation model is optimized by a contrastive learning method so that the model output is aligned with the user preference and the text quality of the answer is improved, thereby obtaining a second answer generation model.
5. The method for generating code community answers based on human feedback reinforcement learning according to claim 4, characterized in that: The preference probability optimization objective is: Where N i Represents question q i Answer Pool The size of I1 is the hint containing question q, {p1,p2,...,p Ni } represents the answer preference ranking set; M2(I1,p k ) represents the second answer generation model after parameter optimization.
6. The method for generating code community answers based on human feedback reinforcement learning according to claim 4, characterized in that: According to the preference probability optimization target, the first answer generation model is optimized by a contrastive learning method, including: The total optimization objective loss function in the contrastive learning method is: Where I1 represents the prompt containing question q, L MSFT represents the NLL loss, α represents the hyperparameter used to balance text quality and human preference, and L MPRA represents the loss for optimizing the preference probability.
7. The method for generating code community answers based on human feedback reinforcement learning according to claim 6, characterized in that: The preference probability optimizes the loss L MPRA The formula is: Where N i Represents question q i Answer Pool The size of I1 is the hint containing question q, {p1,p2,...,p Ni } represents the answer preference ranking set; M2(I1,p k ) represents the second answer generation model after parameter optimization.
8. The method for generating code community answers based on human feedback reinforcement learning according to claim 1, characterized in that: Inputting the code community question containing the question prompt into the second answer generation model to obtain a code generation answer corresponding to the code community question, including: Inputting the code community question to be answered containing the question prompt into the second answer generation model, the second answer generation model obtaining the current text markup of the target answer according to the context information of the question to be answered, the question-answer pair and the target answer sequence; The predicted target probability of the current token in the output answer sequence is expressed as: In the formula, M2 represents the second answer generation model, Q represents the problem to be solved, I2 represents the question prompt, (sq, sa) represents the question-answer pair, and A <t Indicates that the output answer sequence is at the current tag A t Previous mark.
9. A code community answer generation method system based on human feedback reinforcement learning, characterized in that: include: A first optimization module is used to optimize a preset large language model based on a supervised fine-tuning method to adapt the model to the question-answering scenario of the code community, thereby obtaining a first answer generation model; A preference calculation module, used to obtain multi-perspective answer preference measurement scores corresponding to answers to questions in the code community, and to obtain an overall answer preference score based on the multi-perspective answer preference measurement scores; A second optimization module is used to optimize the parameters of the first answer generation model with the goal of generating the highest overall preference score of the answer, so as to align the model output with the user preference, thereby obtaining a second answer generation model; An enhanced search module is used to obtain code community questions to be answered, and based on a preset search engine, search for question-answer pairs that are most similar to the question in the question-answer library to obtain question prompts for the question; The answer generation module is used to input the code community question containing the question prompt into the second answer generation model to obtain the code generation answer corresponding to the code community question.
10. An electronic device, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the code community answer generation method based on human feedback reinforcement learning as described in any one of claims 1-8 is implemented.
Citation Information
Patent Citations
Efficient network retrieval enhanced answering method and system oriented to human preferences
CN116501843A
Type inference method and device based on reinforcement learning
CN118246556A
Code generation model fine tuning method and device based on clustering and natural language strategy optimization algorithm
CN118468982A
Cited By
Method and device for generating personalized response of large language model agent
CN120745843A
Method and apparatus for personalized response generation of large language model agent
CN120745843B
Large code model training method and electronic equipment
CN121187569A
