A question-answer generation method and system based on preference alignment of large language models

Abstract

The present invention discloses a question-answer generation method and system based on large language model preference alignment, which relates to the field of artificial intelligence natural language processing technology, and includes the following steps: splicing each original question with a specified prompt, inputting the splicing instruction into an interpretation generator, and generating multiple interpretations corresponding to the original question; inputting each original question and the corresponding multiple interpretations into a target large language model, and dividing the multiple interpretations into a biased interpretation set and an unbiased interpretation set; using the unbiased interpretation set as input and the biased interpretation set as output to train a pre-trained language model, and obtaining a biased interpretation generator. The entire process required for the present invention to achieve question preference alignment is a black box optimization process, which does not require obtaining or changing any parameters of the target large language model, and can solve the problems of high computing resource requirements, privacy leakage and security risks of traditional white box methods.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence natural language processing technology, and in particular to a question and answer generation method and system based on large language model preference alignment. Background Art

[0002] Large language models (LLMs) are deep learning models trained using large amounts of text data. They can generate natural language text or understand the meaning of text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and conversational conversation, and are an important path to artificial intelligence. With the continuous growth of parameter sizes and the continued development of pre-training methods, the overall performance of LLMs has become increasingly powerful. However, the rapid development of LLMs has also brought some problems. First, the surge in parameter sizes has led to a significant increase in the computational resources required to tune them. Currently, some research has proposed methods to optimize LLM performance with lower computational requirements. These are primarily white-box tuning methods that require access to model parameters. However, security and privacy issues surrounding LLMs are becoming increasingly prominent. As a result, some LLM designers have chosen to provide services to users through APIs rather than exposing the full model. This trend has made applying white-box tuning increasingly challenging.

[0003] Paraphrases are text that convey the same meaning using different words or structures, reflecting the complexity and diversity of human language. Given a sentence, the goal of paraphrase generation is to produce a paraphrase that differs from the original sentence while preserving the original meaning. Many natural language processing applications leverage paraphrase generation for downstream tasks, such as question-answering systems, semantic parsing, dialogue systems, and machine translation. In the era of large language models, end-to-end paraphrasing using language models ensures accuracy and diversity while maintaining flexibility and ease of use. Consequently, this approach has become the mainstream choice for paraphrase generation.

[0004] Existing large language models have the problem of interpretation bias. When an original question is input into the large language model, it can answer the original question correctly, but when the original question is asked in a slightly different but semantically similar way, the large language model may output the wrong answer. This can be explained by the fact that the large language model not only learns the knowledge itself from the corpus during pre-training, but also learns the expression patterns related to specific knowledge. Existing research regards this problem as the robustness of the large language model to the interpretation of the question, and proposes a full-parameter fine-tuning method to enhance the robustness of the large language model. However, this method is white-box and requires adjusting all parameters of the large language model, which makes it difficult to meet the computational cost and privacy security requirements of LLM. Summary of the Invention

[0005] The present invention provides a question-answer generation method and system based on preference alignment of a large language model, which solves the problem that existing methods need to adjust all parameters of the model when solving the problem of interpretation bias in a large language model, making it difficult to meet the computational cost and privacy security requirements of LLM.

[0006] In a first aspect, the present invention provides a question-answer generation method based on preference alignment of a large language model, comprising the following steps:

[0007] Obtain multiple original questions for training the target large language model, and concatenate each original question with a specified prompt to obtain a concatenation instruction, and input the concatenation instruction into the interpretation generator to generate multiple interpretations corresponding to the original question;

[0008] Input each original question and its corresponding multiple interpretations into the target large language model, obtain multiple responses from the target large language model, determine whether the multiple interpretations of each original question are favored by the target large language model based on the correctness of the responses, and divide the multiple interpretations into biased interpretation sets and non-biased interpretation sets based on the preference results of the multiple interpretations;

[0009] The pre-trained language model is trained with the unbiased paraphrase set as input and the biased paraphrase set as output to obtain a biased paraphrase generator.

[0010] The question to be aligned with preference is input into the biased interpretation generator to obtain the preference-aligned interpretation, which is then input into the target large language model to obtain the corresponding reply answer.

[0011] Preferably, the designated prompt includes multiple interpretation types.

[0012] Preferably, judging whether the multiple interpretations of each original question are preferred by the target large language model based on the correctness of the reply answer comprises the following steps:

[0013] The reply answer is compared with the target answer for correctness. If it is correct, it is considered that the interpretation corresponding to the reply answer is favored by the target large language model;

[0014] If it is wrong, it is considered that the interpretation corresponding to the reply answer is not favored by the target large language model.

[0015] Preferably, before inputting each original question and the corresponding multiple interpretations into the target large language model, low-quality interpretations among the multiple interpretations need to be filtered through an interpretation filtering mechanism.

[0016] Preferably, the parameter scale of the pre-trained language model is smaller than that of the target large language model.

[0017] Preferably, the obtaining of multiple original questions for training the target large language model comprises the following steps:

[0018] Obtaining the original dataset for training the biased paraphrase generator; the original dataset is a user-specified dataset, and the task type and domain of the data can be selected by the user according to training requirements;

[0019] Convert each raw data in the original dataset into the original question and target answer in question-answer format.

[0020] In a second aspect, the present invention provides a question-answer generation system based on preference alignment of a large language model, comprising:

[0021] An acquisition module is used to obtain multiple original questions for training the target large language model, and concatenate each original question with a specified prompt to obtain a concatenation instruction, and input the concatenation instruction into the interpretation generator to generate multiple interpretations corresponding to the original question;

[0022] A classification module is used to input each original question and its corresponding multiple interpretations into the target large language model, obtain multiple responses from the target large language model, determine whether the multiple interpretations of each original question are favored by the target large language model based on the correctness of the responses, and classify the multiple interpretations into a biased interpretation set and a non-biased interpretation set based on the preference results of the multiple interpretations;

[0023] A generation module is used to train the pre-trained language model using the unbiased interpretation set as input and the biased interpretation set as output to obtain a biased interpretation generator;

[0024] The interpretation module is used to input the question to be preference-aligned into the biased interpretation generator to obtain the preference-aligned interpretation, and then input the preference-aligned interpretation into the target large language model to obtain the corresponding reply answer.

[0025] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating questions and answers based on preference alignment of a large language model is implemented.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned question and answer generation method based on large language model preference alignment.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention constructs a biased interpretation generator that can generate interpretations preferred by a target large language model. During the construction process, each original question used to train the target large language model is first spliced ​​with a specified prompt to obtain a splicing instruction, which is then input into the interpretation generator to generate multiple interpretations corresponding to the original question. The present invention proposes a prompt construction method based on interpretation type, which enhances the diversity of interpretations by adding specified interpretation types to the prompts input to the interpretation generator. Multiple interpretations are then input into the target large language model, and the preferences of the target large language model divide the multiple interpretations into biased interpretation sets and unbiased interpretation sets. Finally, the pre-trained language model is trained using the unbiased interpretation set and the biased interpretation set to obtain a biased interpretation generator. Compared with the traditional white-box tuning method, the present invention regards the interpretation deviation in LLM as a problem of alignment with the preferences of the target large language model. The entire process required to achieve question preference alignment in the present invention is a black-box optimization process, that is, the entire process does not require obtaining or changing any parameters of the target large language model, which can solve the problems of high computing resource requirements, privacy leakage and security risks of traditional white-box methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 This is a flowchart of a question-answer generation method based on large language model preference alignment according to the present invention;

[0031] Figure 2 Schematic diagram of the reasoning process of the present invention;

[0032] Figure 3 It is a flow chart of the automatic construction method of the training set of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] The present invention provides a question-answering generation method based on preference alignment of a large language model, specifically a question-answering generation method based on interpretation, wherein the interpretation is a question in which the original question conveys the same meaning in different words or structures. The method aims to use interpretation to solve the preference alignment problem of the large language model in the form of black box tuning, and enhance the model performance by interpreting the question as an expression preferred by the model. The present invention trains a biased interpretation generator to learn the preference of the large language model for expression. During the reasoning process, the biased interpretation generator generates interpretations that are aligned with the preferences of the large language model, and then inputs them into the large language model. The training of the biased interpretation generator adopts a sequence-to-sequence (seq2seq) method. Taking into account the diversity of interpretation types and the aspects of semantic preservation, the present invention proposes a prompt-based method for automatically constructing the final training set. Reference Figure 1-Figure 3 , specifically including the following steps:

[0035] Step 1: Obtain the original dataset for training the biased paraphrase generator and organize the original question and target answer of each original data in the original dataset in a question-answer format.

[0036] Figure 1 The original query in Figure 3 The initial training set in is multiple original questions, Figure 1 Query 1-query n in are all original questions, Figure 3 Question1 (question 1), Question2 (question 2) and Question n (Question n) is the original question. For example, the original question is: "Where will the XXVI Summer Olympic Games be held?", and the target answer is: "Atlanta, USA".

[0037] In this implementation, the original dataset is a dataset specified by the user, and the task type and domain of the data can be selected by the user according to training requirements.

[0038] This paper selects six public datasets from three tasks of open domain question answering, common sense reasoning and mathematical application problems: WebQSP, CompQ, CWQ, SiQA, CSQA and SVAMP, and constructs the original dataset through these six datasets.

[0039] Step 2: Concatenate each original question in step 1 with the specified prompt to obtain a concatenation instruction, input the concatenation instruction into the interpretation generator, and generate multiple interpretations corresponding to the original question, i.e. Figure 1 The query candidate set in Figure 1 In , Q1-a, Q1-b, and Q1-c are multiple interpretations of query 1, and Qna, Qnb, and Qnc are multiple interpretations of query n.

[0040] The prompt contains the specified paraphrase type. The purpose of this step is to include enough paraphrase types in the training data of the biased paraphrase generator to increase the chances of generating effective paraphrases. For example, please use {modality change} to paraphrase the sentence: {Where is the venue for the XXVI Summer Olympic Games}. Splicing adds the specified paraphrase type "modality change" and the sentence to be paraphrased to the template. For example Figure 3 Paraphrase thissentence using{Type lnstruction}:Question1, that is, use {type structure} to explain this sentence: Question 1.

[0041] For the interpretation types used in prompt construction, the present invention selects the classification standard of interpretation type detection PTD, which includes 7 first-level interpretation types and 27 second-level interpretation types. The second-level classification is to further subdivide each category in the first-level classification.

[0042] The interpretation generator used in the present invention can be a dedicated interpretation generator, such as the open source text editing model CoEdit, or a large language model with interpretation capabilities.

[0043] In this embodiment, the present invention includes a paraphrase filtering mechanism that can filter out low-quality paraphrases generated in step 2. Only the filtered paraphrases are passed to the target large language model to determine their biased characteristics. The purpose of this step is to ensure semantic consistency before and after the question paraphrase, so as to prevent low-quality paraphrases from entering the training set and affecting the training of the biased paraphrase generator. Filtering is a process of scoring paraphrase pairs using machine evaluation indicators generated by text and filtering them according to thresholds, which specifically includes:

[0044] (1) Combine the original question with the corresponding multiple interpretations to obtain multiple interpretation pairs, and use the machine evaluation indicators generated by the text to score the multiple interpretation pairs. The optional evaluation indicators include BLUERT, BARTscore and UniTE. For example Figure 3 Score: 0.23, 0.95, 0.68, 0.63 and 0.84.

[0045] (2) Based on the score of each interpretation pair and the pre-set threshold, the interpretations that do not meet the interpretation quality requirements are deleted to obtain multiple filtered interpretations. Figure 3 The meaning of and

[0046] Step 3: Pass the original question and the filtered interpretations obtained in Step 2 as input to the target large language model (knowledge base) to obtain the target large language model's response. Based on the correctness of the response, it is determined whether the multiple interpretations of each original question are favored by the target large language model.

[0047] Figure 1 Answer 1-Answer n are the target answers to query 1-query n, answers A 1-a, A 1-b, and A1-c are the response answers to Q1-a, Q 1-b, and Q1-c, and answers An-a, Anb, and Anc are the response answers to Q na, Q nb, and Q nc.

[0048] The correctness of the reply answer is judged against the target answer of the original question. If it is correct, it is considered that the interpretation corresponding to the reply answer is favored by the target large language model; if it is wrong, it is considered that the interpretation corresponding to the reply answer is not favored by the target large language model.

[0049] Step 4: Divide the homologous interpretation sets (all interpretations of the same original question) into two groups according to the bias obtained in step 3: biased interpretation set and unbiased interpretation set, which serve as the final training set for the biased interpretation generator. Cartesian product is performed on the biased interpretation set and the unbiased interpretation to obtain the interpretation sentence pair set, such as Figure 3 in and Figure 1 In , query 1-a and query 1-b are the biased and unbiased paraphrases in sentence pair 1, label 1 is the corresponding label of sentence pair 1, Qnb and Qnc are the biased and unbiased paraphrases in sentence pair n, and label n is the corresponding label of sentence pair n.

[0050] In this example, a data augmentation strategy was used to construct the final training set for the biased paraphrase generator. This strategy augmented the training set with a certain number of identical sentence pairs and used the parameter λ to control the ratio of biased to unbiased paraphrase sets. This step allows the biased paraphrase generator to understand that it can skip questions that are already aligned with the model's preferences.

[0051] Step 5: Use the final training set to fine-tune the biased paraphrase generator on the pre-trained language model in a sequence-to-sequence manner, resulting in a biased paraphrase generator that learns the target large language model's expression preferences. The biased paraphrase generator consists of a bidirectional encoder and an autoregressive encoder.

[0052] In this embodiment, the parameter size of the pre-trained language model is much smaller than that of the target large language model, and it only requires a small amount of training data for the specified task. Optional pre-trained language models include Flan-T5-Large, etc.

[0053] The Flan-T5-Large model is trained using a sequence-to-sequence training method, and the trained Flan-T5-Large model is used as a biased interpretation generator.

[0054] The unbiased query in a sentence pair is fed into the model's encoder as an input sequence. The encoder processes the input sequence, converting it into a set of hidden states that contain semantic information about the input sequence. The decoder receives the encoder's hidden states and generates an output sequence based on this information. During training, the decoder's output sequence is compared with the target sequence (i.e., the biased query in the sentence pair). During training, the decoder uses a teacher-forcing strategy, using the true target sequence as input rather than the decoder's own output. This accelerates training convergence. During training, cross-entropy is used as the loss function. This loss function calculates the difference between the decoder's output sequence and the target sequence. A smaller loss value indicates that the model's output sequence is closer to the target sequence. Through the backpropagation algorithm, the model's parameters (weights and biases) are adjusted based on the gradient information of the loss value to minimize the loss function.

[0055] Figure 2 For the inference process, the original query is "What structure is made from DNA and protein molecules coiled together?". The original query is input into the biased interpretation generator, and the potential optimal query is "What is the composition of the structure formed by intertwined DNA and protein molecules?". The potential optimal query is input into the target large language model, and the query result is "chromatin."

[0056] Step 6: Given the question to be aligned, use the biased paraphrase generator trained in step 5 to paraphrase the question to be aligned, obtain the biased aligned question (biased paraphrase), and then pass the biased aligned question to the target large language model to obtain the reply answer.

[0057] For example, the MPT-instruct-7B model gives different answers to different questions with the same semantics under different interpretations. These include:

[0058] Example 1

[0059] Q: When wildlife reproduce we often refer to what comes out as what?

[0060] P:What is the term used to describe the outcome of wildlife reproduction? (Answer: What is the term used to describe the outcome of wildlife reproduction?)

[0061] Example 2

[0062] Q:While washing clothes they became what when caught on the sharpobject?

[0063] (Question: What happens to clothes when they get caught in sharp objects while washing?)

[0064] P:What was the outcome of the clothes getting caught on the sharp object while washing?

[0065] Example 3

[0066] Q: What is the least dangerous radioactive decay?

[0067] P: Among radioactive decay types, which one poses the least risk?

[0068] In this embodiment, the entire process required to achieve question preference alignment is a black box optimization process, that is, the entire process does not require obtaining or changing any parameters of the target large language model.

[0069] In essence, the method of the present invention is a process of learning a model's preference for expression and incorporating this preference into questions to guide the model to produce correct answers. This is similar to prompt learning, in which prompts are adjusted to adapt to the expression style that the model is accustomed to during pre-training. However, prompt learning involves incorporating preference information into prompts that are unrelated to the question. It should be noted that prompts are usually task or domain specific, which means that the preference information obtained through prompt learning is saved in a fixed format in the prompts of a specific task or domain. However, questions are not specific to any domain or task, they are constantly changing. This requires dynamically incorporating the model's preference information into the question. The biased interpretation generator can adaptively incorporate preferences into the interpretation process by learning the model's preference for expression style under different semantics. Its adaptability can be specifically reflected in its ability to apply different types of interpretations to different questions. The interpretation type represents the lexical variable operated in the interpretation generation process.

[0070] Table 1 shows the performance of the present invention and existing optimization methods on six datasets across three tasks. Existing optimization methods include LORA (Low-Rank Adaptation), Manual Prompt, UPRISE (Universal Prompt Retrieval for Improving Zero-Shot Evaluation), and EPR (Efficient Prompt Retrieval). Table 2 shows the performance of the present invention and different large language models on six datasets across three tasks. The large language models used include TinyLlama-1.3B, Bloom-7B, RedPajama-instruct-3B, MPT-instruct-7B, and MPT-instruct-30B.

[0071] Table 1 Performance of the proposed method and existing optimization methods on six datasets of three tasks

[0072]

[0073] Table 2 Performance of different large language models on six datasets of three tasks

[0074]

[0075] Experimental results show that this technical solution can significantly improve the performance of large models in open-domain question answering, common sense reasoning, and mathematical application problems. By adding model preference information to the original problem through a transformation of the expression method, it provides accurate and reliable question answering services for large model users.

[0076] This paper breaks away from traditional white-box tuning approaches and treats paraphrase bias in LLM as a problem of model preference alignment. It proposes a paraphrase-based question-answer generation method based on large language model preference alignment. This method addresses the high computational resource requirements, privacy leaks, and security risks associated with traditional white-box methods, adapting to the needs of rapid enterprise development and iterative updates. It fully leverages the power of large language models, accurately and reliably acquiring required knowledge and information, thereby improving production efficiency and delivering significant economic and social benefits.

[0077] Based on the same concept, the present invention also provides a question-answering generation system based on large language model preference alignment, including an acquisition module, a splicing module, a classification module, a generation module and an interpretation module.

[0078] The acquisition module is used to obtain multiple original questions for training the target large language model, and splice each original question with a specified prompt to obtain a splicing instruction, which is input into the interpretation generator to generate multiple interpretations corresponding to the original question.

[0079] The classification module is used to input each original question and the corresponding multiple interpretations into the target large language model, obtain multiple reply answers from the target large language model, judge whether the multiple interpretations of each original question are preferred by the target large language model based on the correctness of the reply answers, and divide the multiple interpretations into biased interpretation sets and non-biased interpretation sets based on the preference results of the multiple interpretations.

[0080] The generation module is used to take the unbiased interpretation set as input and the biased interpretation set as output to train the pre-trained language model and obtain a biased interpretation generator.

[0081] The interpretation module is used to input the question to be preference-aligned into the biased interpretation generator to obtain the preference-aligned interpretation, and then input the preference-aligned interpretation into the target large language model to obtain the corresponding reply answer.

[0082] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned question and answer generation method based on large language model preference alignment is implemented.

[0083] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned question and answer generation method based on large language model preference alignment.

[0084] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0085] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the claims and their equivalents, the present invention is intended to include such modifications and variations.

Claims

1. A question-answer generation method based on preference alignment of a large language model, characterized in that: The following steps are involved: Obtain multiple original questions for training the target large language model, and concatenate each original question with a specified prompt to obtain a concatenation instruction, and input the concatenation instruction into the interpretation generator to generate multiple interpretations corresponding to the original question; Input each original question and its corresponding multiple interpretations into the target large language model, obtain multiple responses from the target large language model, determine whether the multiple interpretations of each original question are favored by the target large language model based on the correctness of the responses, and divide the multiple interpretations into biased interpretation sets and non-biased interpretation sets based on the preference results of the multiple interpretations; The pre-trained language model is trained with the unbiased paraphrase set as input and the biased paraphrase set as output to obtain a biased paraphrase generator. The question to be aligned with preference is input into the biased interpretation generator to obtain the preference-aligned interpretation, which is then input into the target large language model to obtain the corresponding reply answer.

2. The question-answer generation method based on large language model preference alignment according to claim 1, characterized in that: The specified prompt includes multiple interpretation types.

3. The question-answer generation method based on large language model preference alignment according to claim 1, characterized in that: The step of judging whether the multiple interpretations of each original question are preferred by the target large language model based on the correctness of the reply answers includes the following steps: The reply answer is compared with the target answer for correctness. If it is correct, it is considered that the interpretation corresponding to the reply answer is favored by the target large language model; If it is wrong, it is considered that the interpretation corresponding to the reply answer is not favored by the target large language model.

4. The question-answer generation method based on large language model preference alignment according to claim 1, characterized in that: Before inputting each original question and the corresponding multiple interpretations into the target large language model, a interpretation filtering mechanism is used to filter out low-quality interpretations from the multiple interpretations. Specifically, the following steps are included: Combine the original question with the corresponding multiple interpretations to obtain multiple interpretation pairs; Scoring multiple paraphrase pairs using evaluation metrics; The score of each paraphrase pair is compared with the set threshold, and the paraphrases corresponding to the paraphrase pairs whose scores are less than the set threshold are deleted.

5. The question-answer generation method based on large language model preference alignment according to claim 1, characterized in that: The parameter scale of the pre-trained language model is smaller than that of the target large language model.

6. The question-answer generation method based on large language model preference alignment according to claim 1, characterized in that: The step of obtaining multiple original questions for training a target large language model includes the following steps: Obtaining the original dataset for training the biased paraphrase generator; the original dataset is a user-specified dataset, and the task type and domain of the data can be selected by the user according to training requirements; Convert each raw data in the original dataset into the original question and target answer in question-answer format.

7. A question-answer generation system based on preference alignment of a large language model, characterized by: include: An acquisition module is used to obtain multiple original questions for training the target large language model, and concatenate each original question with a specified prompt to obtain a concatenation instruction, and input the concatenation instruction into the interpretation generator to generate multiple interpretations corresponding to the original question; A classification module is used to input each original question and its corresponding multiple interpretations into the target large language model, obtain multiple responses from the target large language model, determine whether the multiple interpretations of each original question are favored by the target large language model based on the correctness of the responses, and classify the multiple interpretations into a biased interpretation set and a non-biased interpretation set based on the preference results of the multiple interpretations; A generation module is used to train the pre-trained language model using the unbiased interpretation set as input and the biased interpretation set as output to obtain a biased interpretation generator; The interpretation module is used to input the question to be preference-aligned into the biased interpretation generator to obtain the preference-aligned interpretation, and then input the preference-aligned interpretation into the target large language model to obtain the corresponding reply answer.

8. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for generating questions and answers based on preference alignment of a large language model as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, implements the question-answer generation method based on large language model preference alignment as described in any one of claims 1 to 6.

Images