Large language model strengthening method and device, electronic equipment and storage medium
By acquiring and utilizing the predicted probability of large language models and the predicted probability of reference models, determining the loss function and updating the model parameters, the problem of low accuracy and reliability of large language models when processing out-of-distribution data is solved, and the output stability and effectiveness of the model are improved.
Patent Information
- Application Number
- CN202411712962.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-14
AI Technical Summary
In the prior art, when large language models deal with the preference for external data, they are prone to update parameters through the "shortcut" method, resulting in the model output preferring external data, thereby reducing the accuracy and reliability of the model.
By obtaining the model training data and the first selected text prediction probability and the first reject text prediction probability of the target model, as well as the second selected text prediction probability and the second reject text prediction probability of the reference model, the loss function is determined, and the parameters of the target model are updated using the loss function to achieve the reinforcement of the large language model.
It improves the accuracy and reliability of large language models, prevents the model from having preference problems when processing out-of-distribution data, and ensures the stability and effectiveness of model output.
Smart Images
Figure CN119623672A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model enhancement method, device, electronic device and storage medium. Background Art
[0002] Learning from human feedback is crucial to aligning large language models (LLMs) with human preferences. Learning from human feedback ensures that the final large language model is beneficial, honest, and harmless. Reinforcement Learning from Human Feedback (RLHF) has been a common fine-tuning method for achieving large language model alignment. Although classic RLHF methods such as PPO (Proximal Policy Optimization) work well, due to the problems of multiple training stages and large resource usage, the industry has begun to try more simple offline reinforcement learning algorithms, such as DPO (Direct Preference Optimization). DPO training does not require a separate reward model and has stable results. Therefore, DPO and its variants have been widely used in practice. However, after research, it was found that when dealing with out-of-distribution data preference processing, DPO is prone to "taking shortcuts" to update parameters, so that the output prefers out-of-distribution data, which may lead to lower model accuracy and reliability. Summary of the invention
[0003] The purpose of the present invention is to provide a large-scale language model enhancement method, device, electronic device and storage medium to solve the technical problem of low model accuracy and reliability in the prior art.
[0004] The technical solution of the present invention is as follows, which provides a large language model enhancement method, comprising:
[0005] Get model training data;
[0006] According to the model training data and the target model, a first selected text prediction probability and a first rejected text prediction probability of the target model are obtained, and according to the model training data and the reference model, a second selected text prediction probability and a second rejected text prediction probability of the reference model are obtained;
[0007] Determine a loss function based on the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability, and a first hyperparameter;
[0008] The loss function is used to update the parameters of the target model to enhance the large language model.
[0009] Further, a loss function is obtained according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability and the first hyperparameter, including:
[0010] A loss function is determined according to a value obtained by subtracting the second selected text prediction probability from the first selected text prediction probability, a value obtained by subtracting the second rejected text prediction probability from the first rejected text prediction probability, and a product of the first hyperparameter.
[0011] Further, according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the second rejected text prediction probability multiplied by the first hyperparameter, a loss function is obtained, including:
[0012] The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the product of the first rejected text prediction probability minus the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction probability to the second selected text prediction probability.
[0013] Further, determining a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the product of the first rejected text prediction probability minus the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction to the second selected text prediction probability includes:
[0014] The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, the ratio of the second rejected text prediction probability to the second selected text prediction probability, and a loss function calculation formula, wherein the loss function calculation formula includes
[0015]
[0016] Among them, loss is the loss function, π p (y A |x) is the predicted probability of the first selected text, π p (y B |x) is the first rejection text prediction probability, π r (y A |x) is the predicted probability of the second selected text, π r(y B |x) is the second rejection text prediction probability, δ is the first hyperparameter, α is the second hyperparameter, and β is the third hyperparameter.
[0017] Furthermore, the loss function is used to update the parameters of the target model to enhance the large language model, including:
[0018] The parameters of the target model are updated using the loss function to obtain the target model after parameter update, the loss function is re-determined according to the target model after parameter update, the reference model and the first hyperparameter, and the parameters of the target model are continuously updated until the parameter update stop condition is reached, so as to achieve the enhancement of the large language model.
[0019] Furthermore, the large language model enhancement method also includes using a pre-trained language model or a large language model after supervised fine-tuning as an initial model, and obtaining the target model and the reference model based on the initial model.
[0020] Furthermore, the model training data includes text data.
[0021] Another technical solution of the present invention is as follows: a large-scale language model enhancement device is provided, including a data acquisition module, a text prediction module, a loss function determination module and an enhancement module;
[0022] The data acquisition module is used to acquire model training data;
[0023] The text prediction module is used to obtain a first selected text prediction probability and a first rejected text prediction probability of the target model according to the model training data and the target model, and to obtain a second selected text prediction probability and a second rejected text prediction probability of the reference model according to the model training data and the reference model;
[0024] The loss function determination module is used to determine the loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability and a first hyperparameter;
[0025] The enhancement module is used to update the parameters of the target model using the loss function to enhance the large language model.
[0026] Another technical solution of the present invention is as follows: an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the large-scale language model enhancement method as described in any of the above technical solutions is implemented.
[0027] Another technical solution of the present invention is as follows: a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the large-scale language model enhancement method as described in any of the above technical solutions is implemented.
[0028] The beneficial effects of the present invention are: obtaining model training data; obtaining the first selected text prediction probability and the first rejected text prediction probability of the target model according to the model training data and the target model, and obtaining the second selected text prediction probability and the second rejected text prediction probability of the reference model according to the model training data and the reference model; determining a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability and a first hyperparameter; using the loss function to update the parameters of the target model to achieve the enhancement of the large language model; through the above technical solution, the accuracy and reliability of the large language model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A schematic diagram of a flow chart of a large language model enhancement method provided by an embodiment of the present invention;
[0030] Figure 2 A schematic diagram of the structure of a large language model enhancement device provided by an embodiment of the present invention;
[0031] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] In the description of this application, the words "first", "second", etc. are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order. In this specification, the terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.
[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0035] Figure 1 is a flow chart of a large language model enhancement method according to an embodiment of the present invention. It should be noted that if substantially the same results are achieved, the large language model enhancement method of the present invention is not limited to Figure 1 The process sequence shown is limited. Figure 1 As shown, the large language model enhancement method mainly includes the following steps:
[0036] S101, obtaining model training data;
[0037] In an optional implementation, the model training data includes text data.
[0038] It should be noted that the above-mentioned model training data may be training data samples used to train a large language model, which may include text data, image data or audio data.
[0039] S102, obtaining a first selected text prediction probability and a first rejected text prediction probability of the target model according to the model training data and the target model, and obtaining a second selected text prediction probability and a second rejected text prediction probability of the reference model according to the model training data and the reference model;
[0040] In an optional embodiment, the large language model enhancement method also includes using a pre-trained language model or a large language model after supervised fine-tuning as an initial model, and obtaining the target model and the reference model based on the initial model.
[0041] In a specific embodiment, a pre-trained language model or a large language model after supervised fine-tuning is used as an initial model, and the target model and the reference model are obtained according to the initial model, wherein the initial target model and the reference model can be the same model, i.e., the initial model. After obtaining the target model and the reference model, the direct preference optimization (DPO) algorithm can be used to optimize the target model to enhance the large language model.
[0042] S103, determining a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability, and a first hyperparameter;
[0043] In an optional implementation, obtaining a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability, and a first hyperparameter includes:
[0044] A loss function is determined according to a value obtained by subtracting the second selected text prediction probability from the first selected text prediction probability, a value obtained by subtracting the second rejected text prediction probability from the first rejected text prediction probability, and a product of the first hyperparameter.
[0045] In an embodiment of the present invention, a loss function is determined based on the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, and the loss function is optimized, thereby improving the accuracy and reliability of large language models.
[0046] In a specific embodiment, for an input of a large language model, there can be two outputs, namely, selected text A and rejected text B. It is generally hoped that the final large language model can output the selected text A with a higher probability, and reject text B with a very low probability or no output. During the training process of the model, there will be two models, one model freezes the parameters of the initial model, called the reference model, and the other model updates the parameters of the model through the loss function loss, that is, the target model. During direct preference optimization, each time forward propagation, the direct preference optimization algorithm will calculate the ratio R_policy and R_reference of the predicted probability of selecting text A to the predicted probability of selecting text B on the two models, and obtain the loss function loss based on their ratio to guide the update of model parameters, where the role of R_reference is to limit the update direction of the model parameters, where, That is, the ratio of the prediction probability of the first selected text to the prediction probability of the first rejected text, That is, the ratio of the predicted probability of the second selected text to the predicted probability of the second rejected text.
[0047] In an optional implementation, obtaining a loss function according to a value of the first selected text prediction probability minus the second selected text prediction probability, a product of the first rejected text prediction probability minus the second rejected text prediction probability and the first hyperparameter includes:
[0048] The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the product of the first rejected text prediction probability minus the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction probability to the second selected text prediction probability.
[0049] In an optional embodiment, determining a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction to the second selected text prediction probability includes:
[0050] The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, the ratio of the second rejected text prediction probability to the second selected text prediction probability, and a loss function calculation formula, wherein the loss function calculation formula includes
[0051]
[0052] Among them, loss is the loss function, π p (y A |x) is the predicted probability of the first selected text, π p (y B |x) is the first rejection text prediction probability, π r (y A |x) is the predicted probability of the second selected text, π r (y B |x) is the second rejection text prediction probability, δ is the first hyperparameter, α is the second hyperparameter, and β is the third hyperparameter.
[0053] In a specific embodiment, in the loss function strategy part of the direct preference optimization algorithm, the loss function is optimized. For the selected text, it is hoped that after the preference learning, the output of the model for the selected text is not lower than the original model. Therefore, the output probability difference between the target model and the reference model is learned to ensure that the output probability of the originally better sample will not decrease due to the preference learning; at the same time, for the rejected sample, the difference between it and the original output is also learned to ensure that the model will not directly set the probability of the rejected sample to 0, thereby minimizing the final loss function loss and playing the role of early stopping. At the same time, in order to prevent the model from being troubled, the first hyperparameter δ is set to control the minimum expected output probability of the rejected sample. α is the second hyperparameter, and β is the third hyperparameter, which is used to guide the update direction of the reinforcement learning parameters. The larger the α, the larger the update step of the loss function loss, and the larger the β, the greater the proportion of the reference model, that is, the final parameter update of the model will be more restricted by the reference model. The first hyperparameter δ, the second hyperparameter α and the third hyperparameter β can be set according to actual conditions.
[0054] S104, using the loss function to update the parameters of the target model to enhance the large language model.
[0055] In an optional implementation, the loss function is used to update the parameters of the target model to enhance the large language model, including:
[0056] The parameters of the target model are updated using the loss function to obtain the target model after parameter update, the loss function is re-determined according to the target model after parameter update, the reference model and the first hyperparameter, and the parameters of the target model are continuously updated until the parameter update stop condition is reached, so as to achieve the enhancement of the large language model.
[0057] In the prior art, when strengthening large language models, only preferences are considered, and whether the final distribution meets expectations is not important. When the selected text is outside the SFT (Supervised Fine-Tuning) data distribution, since its output probability is very low, if its output is to be enlarged and the output of the rejected text is to be reduced relatively smaller, more update steps will be required. By "taking a shortcut" and directly setting the probability of rejecting the text to 0, the loss can also be minimized. At the same time, due to relative restrictions, the output probability of the selected text may also become smaller than before. Therefore, the final output may not be any good or bad in the preference data, resulting in instability after DPO learning, making the accuracy and reliability of large language models low.
[0058] The large language model enhancement method provided by the embodiment of the present invention obtains model training data; obtains the first selected text prediction probability and the first rejected text prediction probability of the target model according to the model training data and the target model, and obtains the second selected text prediction probability and the second rejected text prediction probability of the reference model according to the model training data and the reference model; determines the loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability and the first hyperparameter; and uses the loss function to update the parameters of the target model to achieve the enhancement of the large language model; the accuracy and reliability of the large language model can be improved.
[0059] The large-scale language model enhancement method provided by the embodiment of the present invention optimizes the loss function. For the selected text, it is hoped that after the preference learning, the output of the model for the selected text is not lower than that of the original model. The output probability difference between the target model and the reference model is learned to ensure that the output probability of the originally better sample will not decrease due to the preference learning. At the same time, for the rejected samples, the difference between them and the original output is also learned to ensure that the model will not directly set the probability of the rejected samples to 0, thereby minimizing the final loss function loss and playing the role of early stopping. In order to prevent trouble to the model, the minimum expected output probability of the rejected samples can be controlled by setting the first hyperparameter δ.
[0060] In view of the defect that the prior art cannot inject out-of-distribution preference data, the large-scale language model reinforcement method of the embodiment of the present invention limits the learning benefits of rejecting samples at low probability, while increasing the benefits of selecting samples at high probability, further guiding the optimization direction of the model parameters, and ensuring that the probability of out-of-distribution data output will not increase after optimization. That is, the calculation method of the loss function of DPO is optimized, so that it can have better performance when the distribution of the preference data set is inconsistent with the data distribution of the SFT stage, thereby improving the accuracy and reliability of the large-scale language model.
[0061] The large language model enhancement method provided in the embodiment of the present invention can be constructed based on artificial intelligence, and relevant data can be acquired and processed based on artificial intelligence technology to achieve unattended large language model enhancement. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0062] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0063] Figure 2 is a schematic diagram of the structure of a large language model enhancement device according to an embodiment of the present invention. Figure 2 As shown, the large language model enhancement device 20 includes a data acquisition module 21, a text prediction module 22, a loss function determination module 23 and an enhancement module 24;
[0064] The data acquisition module 21 is used to acquire model training data;
[0065] The text prediction module 22 is used to obtain a first selected text prediction probability and a first rejected text prediction probability of the target model according to the model training data and the target model, and to obtain a second selected text prediction probability and a second rejected text prediction probability of the reference model according to the model training data and the reference model;
[0066] The loss function determination module 23 is used to determine the loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability and a first hyperparameter;
[0067] The strengthening module 24 is used to update the parameters of the target model using the loss function to achieve strengthening of the large language model.
[0068] In an optional implementation, the loss function determination module 23 obtains a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability and the first hyperparameter, including:
[0069] A loss function is determined according to a value obtained by subtracting the second selected text prediction probability from the first selected text prediction probability, a value obtained by subtracting the second rejected text prediction probability from the first rejected text prediction probability, and a product of the first hyperparameter.
[0070] In an optional implementation, the loss function determination module 23 obtains the loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, including:
[0071] The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the product of the first rejected text prediction probability minus the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction probability to the second selected text prediction probability.
[0072] In an optional embodiment, the loss function determination module 23 determines the loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction to the second selected text prediction probability, including:
[0073] The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, the ratio of the second rejected text prediction probability to the second selected text prediction probability, and a loss function calculation formula, wherein the loss function calculation formula includes
[0074]
[0075] Among them, loss is the loss function, π p (y A |x) is the predicted probability of the first selected text, π p (y B |x) is the first rejection text prediction probability, π r (y A |x) is the predicted probability of the second selected text, π r (y B |x) is the second rejection text prediction probability, δ is the first hyperparameter, α is the second hyperparameter, and β is the third hyperparameter.
[0076] In an optional implementation, the strengthening module 24 uses the loss function to update the parameters of the target model to strengthen the large language model, including:
[0077] The parameters of the target model are updated using the loss function to obtain the target model after parameter update, the loss function is re-determined according to the target model after parameter update, the reference model and the first hyperparameter, and the parameters of the target model are continuously updated until the parameter update stop condition is reached, so as to achieve the enhancement of the large language model.
[0078] In an optional embodiment, the large language model enhancement device 20 also includes a model determination module, which is used to use a pre-trained language model or a large language model after supervised fine-tuning as an initial model, and obtain the target model and the reference model based on the initial model.
[0079] In an optional implementation, the model training data includes text data.
[0080] Figure 3 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes a processor 31 and a memory 32 that is communicatively connected to the processor 31 .
[0081] The memory 32 stores program instructions for implementing the large language model enhancement method of any of the above embodiments.
[0082] The processor 31 is used to execute program instructions stored in the memory 32 to perform large language model enhancement.
[0083] The processor 31 may also be referred to as a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip having signal processing capabilities. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0084] The embodiment of the present invention provides a computer-readable storage medium, and the storage medium of the embodiment of the present invention stores program instructions that can implement all the above methods, and the storage medium can be non-volatile or volatile. Among them, the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.
[0085] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0086] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present invention.
[0087] The above description is only an implementation mode of the present invention. It should be pointed out that, for ordinary technicians in this field, improvements can be made without departing from the creative concept of the present invention, but these all belong to the protection scope of the present invention.
Claims
1. A large language model enhancement method, characterized in that: include: Get model training data; According to the model training data and the target model, a first selected text prediction probability and a first rejected text prediction probability of the target model are obtained, and according to the model training data and the reference model, a second selected text prediction probability and a second rejected text prediction probability of the reference model are obtained; Determine a loss function based on the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability, and a first hyperparameter; The loss function is used to update the parameters of the target model to enhance the large language model.
2. The large language model enhancement method according to claim 1, characterized in that: Obtaining a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability, and a first hyperparameter, including: A loss function is determined according to a value obtained by subtracting the second selected text prediction probability from the first selected text prediction probability, a value obtained by subtracting the second rejected text prediction probability from the first rejected text prediction probability, and a product of the first hyperparameter.
3. The large language model enhancement method according to claim 2, characterized in that: Obtaining a loss function according to a value of the first selected text prediction probability minus the second selected text prediction probability, a product of the first rejected text prediction probability minus the second rejected text prediction probability and the first hyperparameter, including: The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the product of the first rejected text prediction probability minus the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction probability to the second selected text prediction probability.
4. The large language model enhancement method according to claim 3, characterized in that: Determining a loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, and the ratio of the second rejected text prediction to the second selected text prediction probability, including: The loss function is determined according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability minus the product of the second rejected text prediction probability and the first hyperparameter, the ratio of the second rejected text prediction probability to the second selected text prediction probability, and a loss function calculation formula, wherein the loss function calculation formula includes Among them, loss is the loss function, π p (y A |x) is the predicted probability of the first selected text, π p (y B |x) is the first rejection text prediction probability, π r (y A |x) is the predicted probability of the second selected text, π r (y B |x) is the second rejection text prediction probability, δ is the first hyperparameter, α is the second hyperparameter, and β is the third hyperparameter.
5. The large language model enhancement method according to claim 1, characterized in that: The loss function is used to update the parameters of the target model to enhance the large language model, including: The parameters of the target model are updated using the loss function to obtain the target model after parameter update, the loss function is re-determined according to the target model after parameter update, the reference model and the first hyperparameter, and the parameters of the target model are continuously updated until the parameter update stop condition is reached, so as to achieve the enhancement of the large language model.
6. The large language model enhancement method according to claim 1, characterized in that: It also includes taking a pre-trained language model or a large language model after supervised fine-tuning as an initial model, and obtaining the target model and the reference model according to the initial model.
7. The large language model enhancement method according to claim 1, characterized in that: The model training data includes text data.
8. A large language model enhancement device, characterized in that: It includes a data acquisition module, a text prediction module, a loss function determination module, and a reinforcement module; The data acquisition module is used to acquire model training data; The text prediction module is used to obtain a first selected text prediction probability and a first rejected text prediction probability of the target model according to the model training data and the target model, and to obtain a second selected text prediction probability and a second rejected text prediction probability of the reference model according to the model training data and the reference model; The loss function determination module is used to determine the loss function according to the value of the first selected text prediction probability minus the second selected text prediction probability, the first rejected text prediction probability, the second rejected text prediction probability and a first hyperparameter; The enhancement module is used to update the parameters of the target model using the loss function to enhance the large language model.
9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, wherein: When the processor executes the computer program, the large language model enhancement method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the large language model enhancement method according to any one of claims 1 to 7 is implemented.