Negative sample enhancement preference optimization method for large language models based on preset error patterns

Abstract

The present invention provides a method and device for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern, and relates to the field of large language processing technology. The method comprises: based on a dialogue question-and-answer scenario, collecting dialogue questions and data on the correct answers to the questions to obtain a positive sample dataset; defining the error types of the large language model's answers; constructing error description text based on the error types; using a self-supervised error injection mechanism, using the error description text and the positive sample dataset to generate data using a target large language model to obtain a negative sample dataset; and using the positive sample dataset and the negative sample dataset to perform KTO preference optimization training on the target large language model to obtain an optimized large language model. The present invention provides a highly efficient and robust method for optimizing the negative sample enhancement preference of a large language model based on an error pattern.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method and device for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern. Background Art

[0002] Preference optimization for large language models aims to improve the model's alignment with human preferences through the use of positive and negative examples. However, existing approaches face a core challenge: the model's limited ability to generate responses leads to convergence in the quality of positive and negative examples during training, making it difficult to effectively distinguish optimization directions. For example, traditional methods rely on rejection sampling to generate negative examples, requiring a reward model to score a large number of candidate responses. This can easily lead to negative examples being of similar quality to positive examples due to model bias, or even to situations where "negative examples are superior to positive examples." Furthermore, scoring accuracy relies on a high-performance reward model, which is costly. Furthermore, traditional negative example construction lacks systematic modeling of error patterns, making it ineffective in improving the model's robustness in specific tasks (such as mathematical reasoning and factual question answering). For example, randomly generated negative examples may not cover common errors in large models, such as correctness errors (such as calculation errors), logical errors (such as causal contradictions), and hallucination errors (such as fabricated facts), making it difficult for the model to effectively suppress high-frequency error patterns.

[0003] Existing large-scale model preference optimization methods (such as DPO and SimPO) train models using a "prompt-choose-reject" cycle. However, their negative sample generation relies on random perturbations (e.g., aimlessly modifying correct answers) and lacks structured modeling of error types. This results in insufficient ability to suppress high-frequency errors (e.g., logical contradictions in mathematical reasoning and factual illusions in knowledge quizzes). Negative sample generation is typically based on rejection sampling and relies on a scoring model to assess sample quality. Accurate assessment requires high model capabilities and is costly. Traditional optimization algorithms, based on linear reward mechanisms, fail to incorporate the loss aversion characteristic of human decision-making (i.e., the cost of an error should be greater than the reward of a correct answer). This makes it difficult for models to prioritize responses that meet human expectations in error-intensive tasks (e.g., multi-step mathematical reasoning).

[0004] In the existing technology, there is a lack of an efficient and robust large language model negative sample enhancement preference optimization method based on error patterns. Summary of the Invention

[0005] To address the high cost of traditional preference dataset construction methods, as well as the random quality of negative samples and the lack of targeted construction of model defects, the present invention provides a method and device for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern. The technical solution is as follows:

[0006] On the one hand, a method for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern is provided. The method is implemented by a device for optimizing the negative sample enhancement preference of a large language model. The method includes:

[0007] Based on the dialogue question-answering scenario, we collect dialogue questions and their corresponding correct answers to obtain a positive sample dataset.

[0008] Define the error types answered by the large language model; construct error description text based on the error type;

[0009] Based on the self-supervised error injection mechanism, the target large language model is used to generate data based on the error description text and the positive sample dataset to obtain the negative sample dataset;

[0010] Use positive sample datasets and negative sample datasets to perform KTO preference optimization training on the target large language model to obtain an optimized large language model.

[0011] On the other hand, a device for optimizing the enhancement preference of negative samples of a large language model based on a preset error pattern is provided. The device is applied to a method for optimizing the enhancement preference of negative samples of a large language model based on a preset error pattern. The device includes:

[0012] The positive sample dataset acquisition module is used to collect dialogue questions and corresponding correct answers based on the dialogue question-answering scenario to obtain the positive sample dataset;

[0013] The error type determination module is used to define the error type answered by the large language model and construct the error description text based on the error type;

[0014] The negative sample dataset generation module is used to generate data based on the error description text and the positive sample dataset using the target large language model based on the self-supervised error injection mechanism to obtain the negative sample dataset;

[0015] The KTO preference training module is used to perform KTO preference optimization training on the target large language model using positive sample data sets and negative sample data sets to obtain an optimized large language model.

[0016] On the other hand, a large language model negative sample enhancement preference optimization device is provided, which includes: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned large language model negative sample enhancement preference optimization methods based on preset error patterns is implemented.

[0017] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any of the above-mentioned large language model negative sample enhancement preference optimization methods based on a preset error pattern.

[0018] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0019] This paper proposes a large language model negative sample enhancement and preference optimization method based on error patterns. This method defines error types in large-model preference optimization scenarios, including a classification system for correctness errors, logical errors, and hallucination errors. A self-supervised error injection mechanism leverages the target model's inherent capabilities to generate negative samples containing specified error types, forming preference data pairs for "question-correct answer-directed incorrect answer." This method breaks through traditional negative sample generation methods that rely on random perturbations or external scoring models, achieving low-cost, highly targeted negative sample construction. Furthermore, the loss aversion mechanism of KTO (Known Transformation) is used to impose a higher penalty weight on directed error samples, thereby prioritizing error types. This method is a highly efficient and robust large language model negative sample enhancement and preference optimization method based on error patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flow chart of a method for optimizing the preference of negative sample enhancement of a large language model based on a preset error pattern, provided by an embodiment of the present invention;

[0022] Figure 2 This is a block diagram of a device for optimizing the preference of negative sample enhancement of a large language model based on a preset error pattern, provided by an embodiment of the present invention;

[0023] Figure 3 This is a structural diagram of a large language model negative sample enhancement preference optimization device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0025] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0026] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0027] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0028] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0029] The embodiment of the present invention provides a large language model negative sample enhancement preference optimization method based on a preset error pattern, which can be implemented by a large language model negative sample enhancement preference optimization device, which can be a terminal or a server. Figure 1 The flowchart of the method for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern is shown. The processing flow of the method may include the following steps:

[0030] S1. Based on the dialogue question-answering scenario, collect dialogue questions and data on the correct answers to the questions to obtain a positive sample dataset.

[0031] In one feasible implementation, the construction of conversational question-answering datasets is crucial in the field of artificial intelligence, especially in research on natural language processing and intelligent dialogue systems. High-quality positive sample datasets can provide accurate guidance for model training, helping the system better understand user intent and generate appropriate responses.

[0032] S2. Define the error type answered by the large language model; construct error description text based on the error type.

[0033] Among them, the error type is used to cover the types of errors answered by the large language model; error types include correctness errors, logical errors, and hallucination errors;

[0034] Correctness errors refer to response errors that do not conform to objective facts or logical rules; logical errors refer to response errors in which there are defects in the reasoning chain; and illusion errors refer to response errors that are fictitious and have no factual basis.

[0035] In a feasible implementation, the present invention defines a three-category error system covering typical defects of large models, forming a practical negative sample construction guide, which includes:

[0036] Correctness errors are responses that contradict objective facts or logical rules, such as mathematical calculation errors ("2+3=6") and factual errors ("Mount Everest is 8,000 meters above sea level"); logical errors are defects in the reasoning chain, such as causal contradictions ("Because it rained, the ground was dry") and inductive bias ("Having seen 10 white swans, one asserts that all swans are white"); hallucination errors are fabricated content without factual basis, such as fabricated knowledge ("Water contains nanorobots") and incorrect associations ("Einstein invented the light bulb").

[0037] The present invention converts large model error patterns into actionable structured labels, provides clear targeting for negative sample generation, and solves the problem of lack of targeting of traditional random negative samples.

[0038] S3. Based on the self-supervised error injection mechanism, the target large language model is used to generate data according to the error description text and the positive sample dataset to obtain the negative sample dataset.

[0039] Optionally, based on the self-supervised error injection mechanism, the target large language model is used to generate data according to the error description text and the positive sample dataset to obtain a negative sample dataset, including:

[0040] Based on the preset error injection rules, construct the error injection prompt text according to the error description text;

[0041] Based on the error injection prompt text and the positive sample dataset, the target large language model is used to generate directional error type data to obtain the injected error dataset;

[0042] Based on the error description text and the injected error dataset, the target large language model is used to verify the error type and obtain the verification result.

[0043] According to the verification results, the injected error dataset is screened and optimized to obtain the negative sample dataset.

[0044] In a feasible implementation, the present invention proposes a self-supervised error injection mechanism, which is based on prompt engineering and utilizes the target model's own capabilities to inject specified types of errors into the original correct answers. It does not rely on external scoring models or manual labeling, and only completes negative samples through the model itself, thereby reducing the cost of preference data synthesis.

[0045] Among them, directional error type data generation refers to generating corresponding types of error data based on the dialogue question types in the positive sample dataset.

[0046] In one feasible implementation, the present invention injects corresponding errors based on the question type in the positive sample dataset through prompt text guidance, further generating a negative sample dataset. For example, logical errors are injected into mathematical reasoning tasks; hallucination errors are injected into knowledge quiz tasks; and correctness errors are injected into code generation tasks.

[0047] The process of generating directional error type data is as follows (1):

[0048] (1)

[0049] in, Indicates a conversational question; Indicates the correct answer to a conversation question; Indicates the error type description corresponding to the dialogue problem; Injector indicates the error injection processing; Indicates the wrong answer to the error-injected dataset obtained after the error is injected.

[0050] In one feasible implementation, Represents a collection of three predefined error description texts. For a given problem and the correct answer to the question , select the corresponding error type , then the mathematical expression of the process of injecting error type is as shown in formula (1).

[0051] In actual use, the method of the present invention only needs to use the model to be optimized itself to inject errors, without relying on additional models, thus reducing resource costs.

[0052] S4. Use the positive sample dataset and the negative sample dataset to perform KTO preference optimization training on the target large language model to obtain an optimized large language model.

[0053] Optionally, using a positive sample dataset and a negative sample dataset, KTO preference optimization training is performed on the target large language model to obtain an optimized large language model, including:

[0054] Construct a preference pair dataset based on the positive sample dataset and the negative sample dataset;

[0055] Based on the preference, the target large language model is used to generate data for the dataset to obtain the inference dataset;

[0056] Calculate the positive sample data set and the inference data set to obtain the positive sample utility value;

[0057] Calculate the negative sample data set and the inference data set to obtain the negative sample utility value;

[0058] The KTO preference loss function is calculated based on the positive sample utility value and the negative sample utility value to obtain the KTO preference loss;

[0059] According to the KTO preference loss, the parameters of the target large language model are optimized to obtain the optimized large language model.

[0060] One feasible implementation involves Kahneman-Tversky Optimization (KTO), a preference optimization algorithm that effectively reduces the frequency of negative responses in real-world scenarios. Its core is based on a value function that simulates human decision-making under uncertainty. The value function of KTO's preference loss function reflects the phenomenon of "loss aversion," where humans are more sensitive to losses than to equal gains.

[0061] The value function of the result z of the inference dataset relative to the value of the preset reference point z0 is calculated as follows (2):

[0062] (2);

[0063] in, To control the risk of the preset Represents the degree of loss aversion.

[0064] The core of KTO is to minimize the expectation as follows (3):

[0065] (3);

[0066] in, is input Lower output The bonus points, It is based on the corresponding value of the reward points; Represents the inference strategy parameters of the current model; Inferring strategy parameters for correct responses; Represents the sample expectation in the data distribution D, that is, traversing the input and output sampled from the data distribution D and calculating the overall average; It is a scalar value related to y, representing a weighted coefficient, which can change the weight of the corresponding reward in the overall optimization goal, guiding the model to be more inclined to a certain goal or to achieve a balance between multiple goals.

[0067] In a feasible implementation, the present invention evaluates the method of the present invention through five core capability dimensions (mathematical reasoning, complex reasoning, code generation, knowledge question answering, and authenticity) and different model scales (1.5B to 14B), using the Mathematics Dataset (MATH), the Grade School Math Benchmark (GSM), the Big-Ben-Hansards (BBH), the Human Evaluation Dataset (HumanEval), the Massive Multitask Language Understanding (MMLU), and the Truthful Question Answering (TruthfulQA) benchmark tests, including the MC1 evaluation mode and the MC2 evaluation mode.

[0068] The experimental results are shown in Table 1 below (a comparison table of scores on different models and different test sets). Compared with the Instruct method using the original basic training model and the baseline method + Vanilla based on the UltraFeedback dataset for KTO preference optimization, the present invention achieves performance improvements in all tasks and models, especially in error-intensive tasks such as MATH and TruthfulQA. For example, Llama3-8B-Instruct improved by 18.8% in the authenticity task, and Qwen2.5-7B-Instruct improved by 12.3%; in the mathematical task MATH, Qwen2.5-7B improved from 47.2 to 56.5. The present invention has shown robustness under different architectures (such as Llama3, a large language model for natural language processing, and the Qwen series of models for text embedding and ranking) and model scales, verifying its effectiveness and generalization ability.

[0069] Table 1

[0070]

[0071] This paper proposes a large language model negative sample enhancement and preference optimization method based on error patterns. This method defines error types in large-model preference optimization scenarios, including a classification system for correctness errors, logical errors, and hallucination errors. A self-supervised error injection mechanism leverages the target model's inherent capabilities to generate negative samples containing specified error types, forming preference data pairs for "question-correct answer-directed incorrect answer." This method breaks through traditional negative sample generation methods that rely on random perturbations or external scoring models, achieving low-cost, highly targeted negative sample construction. Furthermore, the loss aversion mechanism of KTO (Known Transformation) is used to impose a higher penalty weight on directed error samples, thereby prioritizing error types. This method is a highly efficient and robust large language model negative sample enhancement and preference optimization method based on error patterns.

[0072] Figure 2 This is a block diagram of a device for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern according to an exemplary embodiment. The device is used for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern. Figure 2 The apparatus includes a positive sample data set acquisition module 210, an error type determination module 220, a negative sample data set generation module 230, and a KTO preference training module 240.

[0073] A positive sample dataset acquisition module 210 is used to collect data on dialogue questions and corresponding correct answers based on a dialogue question-answering scenario to obtain a positive sample dataset;

[0074] The error type determination module 220 is used to define the error type answered by the large language model and construct an error description text according to the error type;

[0075] A negative sample dataset generation module 230 is configured to generate data using a target large language model based on the error description text and the positive sample dataset based on a self-supervised error injection mechanism to obtain a negative sample dataset;

[0076] The KTO preference training module 240 is used to perform KTO preference optimization training on the target large language model using the positive sample data set and the negative sample data set to obtain an optimized large language model.

[0077] Among them, the error type is used to cover the types of errors answered by the large language model; error types include correctness errors, logical errors, and hallucination errors;

[0078] Correctness errors refer to response errors that do not conform to objective facts or logical rules; logical errors refer to response errors in which there are defects in the reasoning chain; and illusion errors refer to response errors that are fictitious and have no factual basis.

[0079] Optionally, the negative sample dataset generating module 230 is further configured to:

[0080] Based on the preset error injection rules, construct the error injection prompt text according to the error description text;

[0081] Based on the error injection prompt text and the positive sample dataset, the target large language model is used to generate directional error type data to obtain the injected error dataset;

[0082] Based on the error description text and the injected error dataset, the target large language model is used to verify the error type and obtain the verification result.

[0083] According to the verification results, the injected error dataset is screened and optimized to obtain the negative sample dataset.

[0084] Among them, directional error type data generation refers to generating corresponding types of error data based on the dialogue question types in the positive sample dataset.

[0085] The process of generating directional error type data is as follows (1):

[0086] (1)

[0087] in, Indicates a conversational question; Indicates the correct answer to a conversation question; Indicates the error type description corresponding to the dialogue problem; Injector indicates the error injection processing; Indicates the wrong answer to the error-injected dataset obtained after the error is injected.

[0088] Optionally, the KTO preference training module 240 is further configured to:

[0089] Construct a preference pair dataset based on the positive sample dataset and the negative sample dataset;

[0090] Based on the preference, the target large language model is used to generate data for the dataset to obtain the inference dataset;

[0091] Calculate the positive sample data set and the inference data set to obtain the positive sample utility value;

[0092] Calculate the negative sample data set and the inference data set to obtain the negative sample utility value;

[0093] The KTO preference loss function is calculated based on the positive sample utility value and the negative sample utility value to obtain the KTO preference loss;

[0094] According to the KTO preference loss, the parameters of the target large language model are optimized to obtain the optimized large language model.

[0095] This paper proposes a large language model negative sample enhancement and preference optimization method based on error patterns. This method defines error types in large-model preference optimization scenarios, including a classification system for correctness errors, logical errors, and hallucination errors. A self-supervised error injection mechanism leverages the target model's inherent capabilities to generate negative samples containing specified error types, forming preference data pairs for "question-correct answer-directed incorrect answer." This method breaks through traditional negative sample generation methods that rely on random perturbations or external scoring models, achieving low-cost, highly targeted negative sample construction. Furthermore, the loss aversion mechanism of KTO (Known Transformation) is used to impose a higher penalty weight on directed error samples, thereby prioritizing error types. This method is a highly efficient and robust large language model negative sample enhancement and preference optimization method based on error patterns.

[0096] Figure 3 is a structural diagram of a large language model negative sample enhancement preference optimization device provided by an embodiment of the present invention, such as Figure 3 As shown, the large language model negative sample enhancement preference optimization device may include the above Figure 2 The device for enhancing preference optimization of negative samples of a large language model based on a preset error pattern is shown. Optionally, the device for enhancing preference optimization of negative samples of a large language model 310 may include a first processor 2001 .

[0097] Optionally, the large language model negative sample enhancement preference optimization device 310 may further include a memory 2002 and a transceiver 2003 .

[0098] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0099] The following combination Figure 3 The components of the large language model negative sample enhancement preference optimization device 310 are described in detail:

[0100] The first processor 2001 is the control center of the large language model negative sample enhancement preference optimization device 310 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0101] Optionally, the first processor 2001 can perform various functions of the large language model negative sample enhancement preference optimization device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0102] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0103] In a specific implementation, as an embodiment, the large language model negative sample enhancement preference optimization device 310 may also include multiple processors, such as Figure 3 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0104] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0105] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and be enhanced through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0106] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0107] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0108] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and enhance the interface circuit of the preference optimization device 310 through the large language model negative sample ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0109] It should be noted that Figure 3 The structure of the large language model negative sample enhanced preference optimization device 310 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0110] In addition, the technical effects of the large language model negative sample enhancement preference optimization device 310 can refer to the technical effects of the large language model negative sample enhancement preference optimization method based on a preset error pattern described in the above method embodiment, and will not be repeated here.

[0111] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0112] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0113] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0114] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0115] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0116] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0118] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0119] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0123] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern, characterized in that: The method comprises: Based on the dialogue question-answering scenario, we collect dialogue questions and their corresponding correct answers to obtain a positive sample dataset. Define the error types answered by the large language model; construct error description text based on the error type; The error type is used to cover the error types answered by the large language model; the error types include correctness errors, logical errors, and hallucination errors; The correctness error refers to a response error that does not conform to objective facts or logical rules; the logical error refers to a response error with a flawed reasoning chain; the hallucination error refers to a response error that is fictitious and has no factual basis; The error types provide clear targets for the generation of negative sample datasets; Based on the self-supervised error injection mechanism, the target large language model is used to generate data based on the error description text and the positive sample dataset to obtain the negative sample dataset; The self-supervised error injection mechanism is based on the error description text and the positive sample data set, and the target large language model is used to generate data to obtain the negative sample data set, including: Based on the preset error injection rules, construct the error injection prompt text according to the error description text; Based on the error injection prompt text and the positive sample dataset, the target large language model is used to generate directional error type data to obtain the injected error dataset; Based on the error description text and the injected error dataset, the target large language model is used to verify the error type and obtain the verification result. According to the verification results, the injected error dataset is screened and optimized to obtain the negative sample dataset; Use positive sample datasets and negative sample datasets to perform KTO preference optimization training on the target large language model to obtain an optimized large language model.

2. The method for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern according to claim 1, characterized in that: The directional error type data generation refers to generating corresponding types of error data according to the dialogue question types of the positive sample data set.

3. The method for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern according to claim 1, characterized in that: The process of generating the orientation error type data is as follows: （1） in, Indicates a conversational question; Indicates the correct answer to a conversation question; Indicates the error type description corresponding to the dialogue problem; Injector indicates the error injection processing; Indicates the wrong answer to the error-injected dataset obtained after the error is injected.

4. The method for optimizing the negative sample enhancement preference of a large language model based on a preset error pattern according to claim 1, characterized in that: The method uses a positive sample dataset and a negative sample dataset to perform KTO preference optimization training on a target large language model to obtain an optimized large language model, including: Construct a preference pair dataset based on the positive sample dataset and the negative sample dataset; Based on the preference, the target large language model is used to generate data for the dataset to obtain the inference dataset; Calculate the positive sample data set and the inference data set to obtain the positive sample utility value; Calculate the negative sample data set and the inference data set to obtain the negative sample utility value; The KTO preference loss function is calculated based on the positive sample utility value and the negative sample utility value to obtain the KTO preference loss; According to the KTO preference loss, the parameters of the target large language model are optimized to obtain the optimized large language model.

5. A large language model negative sample enhancement preference optimization device based on a preset error pattern, wherein the large language model negative sample enhancement preference optimization device based on a preset error pattern is used to implement the large language model negative sample enhancement preference optimization method based on a preset error pattern according to any one of claims 1 to 4, characterized in that: The device comprises: The positive sample dataset acquisition module is used to collect dialogue questions and corresponding correct answers based on the dialogue question-answering scenario to obtain the positive sample dataset; The error type determination module is used to define the error type answered by the large language model and construct the error description text based on the error type; The negative sample dataset generation module is used to generate data based on the error description text and the positive sample dataset using the target large language model based on the self-supervised error injection mechanism to obtain the negative sample dataset; The KTO preference training module is used to perform KTO preference optimization training on the target large language model using positive sample data sets and negative sample data sets to obtain an optimized large language model.

6. A large language model negative sample enhancement preference optimization device, characterized by: The large language model negative sample enhancement preference optimization device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 4.

Images