Preference alignment training method and system of large language model, medium and electronic equipment

Abstract

The invention provides a preference alignment training method and system for a large language model, a medium and electronic equipment. The method comprises the steps of obtaining a to-be-trained text; inputting the to-be-trained text and a task prompt into a big language model fused with a dora module to generate a training output text; the task prompt is used for commanding the large language model to execute a task based on the to-be-trained text; and performing preference alignment training on the large language model based on the to-be-trained text and the training output text. According to the method, the large language model can be generalized to each scene according to the training data, and the problem of insufficient training resources is effectively solved. Meanwhile, preference information is better utilized, the over-fitting risk is reduced, and it is guaranteed that the large language model obtained through training is aligned with human value preference.

Description

Technical Field

[0001] The present application belongs to the technical field of large language models, and in particular, relates to a preference alignment training method, system, medium and electronic device for a large language model. Background Art

[0002] Big model value alignment refers to ensuring that the behavior and decisions of artificial intelligence models are consistent with human values, ethics, and expectations. This is a critical issue because as AI systems are widely used in various fields, their decisions may have a significant impact on society. The goal of value alignment is to ensure that AI systems do not take actions that are harmful to humans and can effectively promote social welfare. At present, the mainstream big model value methods mainly include two technical routes, namely constrained optimization-based methods and reinforcement learning-based methods. Among them, the constrained optimization-based method needs to provide clear rules and restrictions, lacks generalization, and cannot flexibly adapt to changes. The reinforcement learning-based method has higher efficiency, but requires a lot of training costs and computing power costs, and the generalization ability is insufficient. Summary of the invention

[0003] The purpose of this application is to provide a preference alignment training method, system, device and storage medium for a large language model, which can align the large language model with human preferences in a low-cost and efficient manner, thereby ensuring the security of the large language model.

[0004] In a first aspect, the present application provides a preference alignment training method for a large language model, the method comprising: Get the text to be trained; Inputting the text to be trained and a task prompt into a large language model integrated with a dora module to generate a training output text; the task prompt is used to command the large language model to perform a task based on the text to be trained; The large language model is subjected to preference alignment training based on the text to be trained and the training output text.

[0005] In an implementation of the first aspect, before inputting the text to be trained and a task prompt into a large language model fused with a dora module to generate a training output text, the method further includes: Freezing parameters of the large language model; Add the dora module to the Query matrix and the Value matrix in the Transformer block of the large language model; The parameters of the dora module are randomly initialized and activated so that the dora module receives gradient updates.

[0006] In an implementation of the first aspect, performing preference alignment training on the large language model based on the to-be-trained text and the training output text includes: Calculate the preference alignment training loss based on the to-be-trained text and the training output text using the ORPO loss function; The large language model is gradient updated based on the preference alignment training loss to obtain the large language model after preference alignment.

[0007] In an implementation of the first aspect, obtaining the text to be trained includes: Get text data; Performing denoising processing on the text data to obtain denoised text data; Removing useless character strings from the denoised text data based on regular expressions and rules to obtain valid text data; The valid text data is subjected to preference alignment processing to obtain the text to be trained.

[0008] In a second aspect, the present application provides a preference alignment training system for a large language model, the system comprising: An acquisition module is used to acquire the text to be trained; An input module, used for inputting the text to be trained and a task prompt into a large language model integrated with a dora module to generate a training output text; the task prompt is used to command the large language model to perform a task based on the text to be trained; A training module is used to perform preference alignment training on the large language model based on the text to be trained and the training output text.

[0009] In an implementation manner of the second aspect, the system further includes a fusion module, and the fusion module is used to: Freezing parameters of the large language model; Add the dora module to the Query matrix and the Value matrix in the Transformer block of the large language model; The parameters of the dora module are randomly initialized and activated so that the dora module receives gradient updates.

[0010] In an implementation of the second aspect, the training module is used to: Calculate the preference alignment training loss based on the to-be-trained text and the training output text using the ORPO loss function; The large language model is gradient updated based on the preference alignment training loss to obtain the large language model after preference alignment.

[0011] In an implementation of the second aspect, the acquisition module is used to: Get text data; Performing denoising processing on the text data to obtain denoised text data; Removing useless character strings from the denoised text data based on regular expressions and rules to obtain valid text data; The valid text data is subjected to preference alignment processing to obtain the text to be trained.

[0012] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned preference alignment training method for a large language model.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned preference alignment training method for a large language model.

[0014] As described above, the preference alignment training method, system, device and storage medium of the large language model described in this application have the following beneficial effects: 1. Strong generalization. This application does not need to define rules and restrictions in advance, and can directly generalize the model to various scenarios based on training data. At the same time, this application is more efficient and direct when processing preference data. Through the unique loss function design, it can better utilize preference information and reduce the risk of overfitting.

[0015] 2. No additional reward model and reference model are required. This application integrates supervised fine-tuning (SFT) and preference alignment into a unified process, thereby reducing the resources and time required to train large language models. At the same time, the effect of the model is not affected by the reward model and reference model.

[0016] 3. Improve training speed and reduce training resource consumption. This application freezes most parameters and only updates a small number of parameters to further improve the model training speed and solve the problem of insufficient training resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A Shown is a schematic diagram of an application scenario of the preference alignment training method for a large language model described in this application.

[0018] Figure 1B Shown are structural diagrams of the client-cloud interaction scenarios in these implementations.

[0019] Figure 2 Shown is a flow chart of a preference alignment training method for a large language model described in an embodiment of the present application in one embodiment.

[0020] Figure 3 Shown is a flow chart of a preference alignment training method for a large language model described in an embodiment of the present application in one embodiment.

[0021] Figure 4 Shown is a schematic diagram of the structure of the large language model described in an embodiment of the present application.

[0022] Figure 5 Shown is a schematic diagram of the fusion of the large language model and the dora module described in an embodiment of the present application in one embodiment.

[0023] Figure 6 Shown is a schematic diagram of the structure of a large language model integrating the dora module described in an embodiment of the present application.

[0024] Figure 7 Shown is a schematic diagram of the structure of a preference alignment training system for a large language model described in an embodiment of the present application in one embodiment.

[0025] Figure 8 Shown is a schematic structural diagram of an electronic device of the present application in one embodiment. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0028] In addition, in this application, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0029] At present, the mainstream large model value methods mainly include two technical routes, namely constrained optimization-based methods and reinforcement learning-based methods. Among them, for the constrained optimization-based methods, it is to impose specific constraints when the model generates content or makes decisions to ensure that its output conforms to predetermined ethical, moral and social norms. For example, a series of clear rules or guidelines such as prohibiting the model from generating text with hate speech, discriminatory content or violent descriptions can be formulated, and these rules or guidelines can be directly embedded in the model generation process. In addition, the constrained optimization-based method can also use logical expressions to define constraints, such as Boolean logic can be used to indicate that certain outputs are not allowed. However, although the constrained optimization method provides clear rules and restrictions in the alignment of large model values ​​to ensure that the model output conforms to ethical and moral standards. However, it faces challenges such as difficulty in definition, lack of flexibility, difficulty in dynamic adjustment and data dependence, which makes its implementation and maintenance costly and has limitations in adapting to changes.

[0030] For reinforcement learning-based methods, they interact with the environment and continuously adjust strategies to maximize cumulative rewards. In preference alignment, reinforcement learning methods can guide the model to develop in a direction that meets ethical and moral standards by defining a reward function. Currently, mainstream reinforcement learning methods include PPO (Proximal Policy Optimization) and DPO (Decision Process Optimization).

[0031] (1) PPO. PPO is a reinforcement learning algorithm used to train an agent to perform tasks in a given environment and maximize cumulative rewards. PPO has high efficiency and stability in reinforcement learning because it constrains the range of policy changes when updating the policy, thereby avoiding excessive policy updates. During the training process, PPO limits the range of policy updates to avoid drastic policy changes and maintain the consistency and predictability of the agent's behavior. At the same time, by introducing human feedback, the agent can fine-tune its behavior based on human evaluation results, thereby further aligning values. This approach ensures that the agent always maintains a high degree of consistency with human values ​​during the process of continuous learning and optimization. Finally, through a large number of tests and verifications in simulated environments, it is ensured that the agent's performance in various scenarios is in line with expectations. In actual applications, the agent's behavior is continuously monitored and adjusted when necessary to cope with new situations. This comprehensive approach not only improves the agent's task performance, but also ensures that its behavior always meets human ethical and safety requirements, achieving value alignment.

[0032] However, PPO still requires a lot of computing resources in large-scale problems or complex environments, and may require a lot of computing power to achieve good performance. In addition, the performance of PPO is sensitive to the choice of hyperparameters. If improperly set, it may affect the convergence speed and final performance of the algorithm, and finding a suitable hyperparameter combination may require a lot of experiments and debugging. In addition, the PPO method requires training of additional reward models and reference models, which increases the complexity of implementation and maintenance, as well as computational overhead, and the effect of the model depends entirely on the capabilities of the reward model and the reference model.

[0033] (2) DPO. DPO is a method that directly incorporates user preferences into the training process of reinforcement learning agents. Its goal is to enable reinforcement learning agents to consider user preferences and values ​​more when making decisions, so that they can behave more in line with human expectations. First, by building a model that directly reflects human preferences, the agent can optimize based on human feedback and preferences. This model usually includes human evaluation of the agent's behavior in various situations, thereby ensuring that the agent can consider important values ​​such as ethics and safety when performing tasks. During the training process, DPO directly optimizes the agent's strategy to make its behavior highly consistent with human preferences. Specifically, when interacting with the environment, the agent will continuously learn from human feedback and adjust its strategy based on this feedback. In this way, the agent can not only improve task performance, but also ensure that its behavior is consistent with human values. Through this direct optimization method, the agent can quickly adapt to human preferences and expectations. In practical applications, the agent's behavior is continuously monitored and adjusted based on new feedback to cope with dynamically changing environments and needs. This approach not only improves the effectiveness of the agent, but also ensures that its behavior is always consistent with human ethical and safety requirements, achieving the goal of value alignment.

[0034] However, a significant disadvantage of DPO is that it is prone to overfitting on a limited dataset of human preferences. This means that the model may perform very well on the training data, but lacks generalization ability on unseen data. In addition, the DPO method requires training an additional reference model, which increases the complexity of implementation and maintenance, as well as computational overhead. Moreover, the performance of the model depends entirely on the capabilities of the reference model.

[0035] In order to at least solve the above problems, the embodiment of the present application provides a preference alignment training method for a large language model, which can generalize the large language model to various scenarios based on the training data, effectively solving the problem of insufficient training resources. At the same time, the present application makes better use of preference information, reduces the risk of overfitting, and ensures that the large language model obtained through training is aligned with human value preferences.

[0036] In some other embodiments, the preference alignment training method for a large language model provided in the embodiments of the present application can be applied to Figure 1A The preference alignment training device for the large language model shown. Figure 1A As shown, the preference alignment training device 1 for a large language model includes a processor 11 and a database 12. The preference alignment training method for a large language model provided in the embodiment of the present application can be applied to the processor 11.

[0037] in, Figure 1A The processor 11 in the embodiment may be a single processor or a processor cluster or a cloud computing center composed of multiple processors, and the specific details are not limited here. Figure 1A Only one processor 11 and one database 12 are shown, but it should be understood that Figure 1A The examples are only used to understand the present solution, and the specific numbers of processors 11 and databases 12 should be flexibly determined based on actual conditions.

[0038] In some other embodiments, the preference alignment training device 1 for a large language model may not include the database 12, but only include a processor 11 with a storage function. The preference alignment training method for a large language model provided in the embodiment of the present application can be applied to the processor 11. The processor 11 with a storage function may include a computer, a mobile phone, etc., which is not limited here.

[0039] In still other embodiments, the preference alignment training method for a large language model described in the present application may be applied to a terminal-cloud interaction scenario. Figure 1B The following is a schematic diagram of the structure of the client-cloud interaction scenario in these implementations. Figure 1B As shown, the terminal-cloud interaction system 2 includes a terminal 20 and a cloud server 21. The terminal 20 and the cloud server 21 can communicate with each other, and the communication method is not limited to wired or wireless.

[0040] Among them, the terminal 20 can be mobile or fixed, for example, the terminal 20 can be a wireless terminal or a wired terminal. The wireless terminal can refer to a device with wireless transceiver function, which can be deployed in various large language model training scenarios. The terminal 20 can be a mobile phone, a laptop computer, etc., which is not limited here. The cloud server 21 can include one or more servers, or one or more processing nodes, or one or more virtual machines running on the server. The cloud server 21 can also be called a server cluster, a management platform, a preference alignment training center for a large language model, etc., which is not limited in the embodiments of the present application.

[0041] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0042] The following embodiments of the present application provide a preference alignment training method for a large language model, which can be performed, for example, Figure 1A The processor 11 or Figure 1B It is implemented by the cloud server 21 shown. Figure 2 The flowchart of the preference alignment training method for a large language model described in the embodiment of the present application is shown as follows: Figure 2 As shown, the preference alignment training method for the large language model includes steps S1 to S3.

[0043] S1. Obtain the text to be trained.

[0044] In some embodiments, Figure 3 The flowchart of the preference alignment training method for a large language model described in the embodiment of the present application is shown as follows: Figure 3 As shown, obtaining the text to be trained includes steps S11 to S14.

[0045] S11. Obtain text data.

[0046] S12: performing denoising processing on the text data to obtain denoised text data.

[0047] S13. Remove useless character strings in the denoised text data based on the regular expression and rules to obtain valid text data.

[0048] S14: performing preference alignment processing on the valid text data to obtain the text to be trained.

[0049] In some embodiments, text data of corresponding fields can be obtained from databases, websites, and other channels based on various application scenarios. Afterwards, noise data in the text data is removed based on Python. The noise data includes repeated text, empty text, garbled text, and text that is too long or too short to obtain denoised text data. Finally, in order to further improve the training efficiency, it is necessary to clean up useless strings in the valid text data. For example, they can be removed based on the regular expressions and rules to finally obtain valid text data. When valid text data is obtained, Python can be used to process the data into data that conforms to the training format, and use it as the text to be trained. For example, the input format of the text to be trained can be: "question + preferred answer + rejected answer" to help the large language model establish cognition.

[0050] Through step S1, text data can be collected and processed into valid text to be trained, so that preference alignment training can be effectively performed on the large language model to ensure that the large language model conforms to human social values ​​and safety ethics.

[0051] S2. Input the text to be trained and a task prompt into a large language model integrated with a dora module to generate a training output text; the task prompt is used to command the large language model to perform a task based on the text to be trained.

[0052] Specifically, before executing step S2, the method provided in the embodiment of the present application needs to freeze the parameters of the large language model, and add the dora module to the Query matrix and Value matrix in the Transformer block of the large language model, and then randomly initialize and activate the parameters of the dora module, so that the dora module receives gradient updates, so as to finally obtain a large language model that integrates the dora module.

[0053] In some embodiments, the LLaMA3 model can be selected as the base model. It is a large-scale basic language model released by Meta AI. It is an open and efficient large-scale basic language model with two versions: 8B (8 billion parameters) and 70B (70 billion parameters). LLaMA uses a deep neural network model based on the Transformer architecture (such as Figure 4 As shown in Figure 2, this model performs well in processing natural language tasks. Its training process is carried out on a large-scale language dataset, which enables it to learn rich language knowledge and adapt to different contexts. It should be noted that the basic model can select other large language models with Transformer architecture, and this application does not impose any restrictions on this.

[0054] After selecting the LLaMA3 model as the base model, all parameters of LLaMA3 8B need to be frozen, and the values ​​of all parameters of LLaMA3 8B are kept unchanged during the training process so that it does not accept gradient updates. Figure 5 As shown in Figure 1, the dora module is added to the Query matrix and Value matrix in each original transformer block. After that, the parameters in the added dora module are initialized by random initialization, and all initialized parameters are activated to accept gradient updates. Finally, the large language model with the fused dora module is saved, as shown in Figure 1. Figure 6 shown.

[0055] Among them, the parameters of each dora include a matrix of dimension hidden*r, a matrix of dimension r*hidden and a vector of 1*hidden, where hidden is the input dimension and r is a pre-set hyperparameter.

[0056] After obtaining the large language model integrated with the dora module, the text to be trained obtained in step S1 and a task prompt are input into the large language model integrated with the dora module, wherein the task prompt is used to command the large language model to perform a task based on the text to be trained. For example, by asking questions, introducing tasks, requiring instructions, etc., the large language model is required to perform relevant tasks based on the text to be trained, and output a training output text that meets the task prompt requirements.

[0057] S3. Performing preference alignment training on the large language model based on the text to be trained and the training output text.

[0058] Specifically, step S3 includes step S31 and step S32.

[0059] S31. Calculate the preference alignment training loss based on the text to be trained and the training output text using the ORPO loss function.

[0060] In some embodiments, the ORPO loss function is used to calculate the preference alignment training loss of the LLaMA3 model. , The calculation method is as follows:

[0061] in, is a hyperparameter set by humans.

[0062] Where N represents the number of samples, i represents the i-th sample, represents the label (0 or 1) of the i-th sample c, is the probability of the i-th sample category c.

[0063]

[0064] in, is the logistic sigmoid function, It refers to the probability that the model generates the output sequence y given the input x.

[0065] S32. Performing a gradient update on the large language model based on the preference alignment training loss to obtain the large language model after preference alignment.

[0066] After obtaining the preference alignment training loss, the preference alignment training loss is back-propagated using gradient descent, and all activated parameters are updated, so that the model can learn how to adjust the parameters to minimize the loss, thereby improving the generalization ability of new data and adapting to complex data distribution. By continuously executing step S3, a trained large language model is finally obtained, thereby ensuring that the large language model can be aligned with human preferences.

[0067] The preference alignment training method for a large language model provided in the embodiment of the present application can, first of all, directly generalize the model to various scenarios based on the training data without having to define rules and restrictions in advance. At the same time, the present application is more efficient and direct when processing preference data. Through the unique loss function design, it can better utilize preference information and reduce the risk of overfitting. Secondly, the present application integrates supervised fine-tuning (SFT) and preference alignment into a unified process, thereby reducing the resources and time required to train large language models. At the same time, the effect of the model is not affected by the reward model and the reference model. In addition, the present application further improves the training speed of the model and solves the problem of insufficient training resources by freezing most of the parameters and updating only a small number of parameters.

[0068] The protection scope of the preference alignment training method for a large language model described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0069] An embodiment of the present application also provides a preference alignment training system for a large language model, which can implement the preference alignment training method for a large language model described in the present application. However, the implementation device of the preference alignment training system for a large language model described in the present application includes but is not limited to the structure of the preference alignment training system for a large language model listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0070] Figure 7 The structure diagram of the preference alignment training system of the large language model described in the embodiment of the present application is shown as follows: Figure 7 As shown, the preference alignment training system 4 for a large language model includes an acquisition module 41 , an input module 42 and a training module 43 .

[0071] The acquisition module 41 is used to acquire the text to be trained.

[0072] The input module 42 is used to input the text to be trained and a task prompt into a large language model integrated with the dora module to generate a training output text; the task prompt is used to command the large language model to perform a task based on the text to be trained.

[0073] The training module 43 is used to perform preference alignment training on the large language model based on the text to be trained and the training output text.

[0074] It should be noted that the structures and principles of the acquisition module 41, the input module 42 and the training module 43 correspond one-to-one to the steps in the above-mentioned preference alignment training method for the large language model, so they will not be described in detail here.

[0075] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0076] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0077] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0078] The embodiment of the present application also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid-state disk (SSD)), etc.

[0079] The embodiment of the present application further provides an electronic device, wherein the electronic device includes a processor and a memory.

[0080] The memory is used to store computer programs.

[0081] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0082] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device performs the above-mentioned preference alignment training method for the large language model.

[0083] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0084] like Figure 8 As shown, the electronic device of the present application is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 51, a memory 52, and a bus 53 connecting different system components (including the memory 52 and the processing unit 51).

[0085] Bus 53 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0086] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0087] The memory 52 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 523 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 53 via one or more data medium interfaces. The memory 52 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0088] A program / utility 524 having a set (at least one) of program modules 5241 may be stored, for example, in the memory 52, such program modules 5241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may include an implementation of a network environment. The program modules 5241 generally perform the functions and / or methods of the embodiments described herein.

[0089] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 54. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 55. Figure 8 As shown, the network adapter 55 communicates with other modules of the electronic device via the bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0090] The embodiment of the present application may also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0091] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0092] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0093] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A preference alignment training method for a large language model, characterized in that: The method comprises: Get the text to be trained; Inputting the text to be trained and a task prompt into a large language model integrated with a dora module to generate a training output text; the task prompt is used to command the large language model to perform a task based on the text to be trained; The large language model is subjected to preference alignment training based on the text to be trained and the training output text.

2. The preference alignment training method for a large language model according to claim 1, characterized in that: Before inputting the text to be trained and a task prompt into a large language model fused with a dora module to generate a training output text, the method further includes: Freezing parameters of the large language model; Add the dora module to the Query matrix and the Value matrix in the Transformer block of the large language model; The parameters of the dora module are randomly initialized and activated so that the dora module receives gradient updates.

3. The preference alignment training method for a large language model according to claim 1, characterized in that: Performing preference alignment training on the large language model based on the text to be trained and the training output text includes: Calculate the preference alignment training loss based on the to-be-trained text and the training output text using the ORPO loss function; The large language model is gradient updated based on the preference alignment training loss to obtain the large language model after preference alignment.

4. The preference alignment training method for a large language model according to claim 1, characterized in that: Obtaining the text to be trained includes: Get text data; Performing denoising processing on the text data to obtain denoised text data; Removing useless character strings from the denoised text data based on regular expressions and rules to obtain valid text data; The valid text data is subjected to preference alignment processing to obtain the text to be trained.

5. A preference alignment training system for a large language model, characterized in that: The system comprises: An acquisition module is used to acquire the text to be trained; An input module, used for inputting the text to be trained and a task prompt into a large language model integrated with a dora module to generate a training output text; the task prompt is used to command the large language model to perform a task based on the text to be trained; A training module is used to perform preference alignment training on the large language model based on the text to be trained and the training output text.

6. The preference alignment training system for a large language model according to claim 5, characterized in that: The system further comprises a fusion module, wherein the fusion module is configured to: Freezing parameters of the large language model; Add the dora module to the Query matrix and the Value matrix in the Transformer block of the large language model; The parameters of the dora module are randomly initialized and activated so that the dora module receives gradient updates.

7. The preference alignment training system for a large language model according to claim 5, characterized in that: The training module is used to: Calculate the preference alignment training loss based on the to-be-trained text and the training output text using the ORPO loss function; The large language model is gradient updated based on the preference alignment training loss to obtain the large language model after preference alignment.

8. The preference alignment training system for a large language model according to claim 5, characterized in that: The acquisition module is used for: Get text data; Performing denoising processing on the text data to obtain denoised text data; Removing useless character strings from the denoised text data based on regular expressions and rules to obtain valid text data; The valid text data is subjected to preference alignment processing to obtain the text to be trained.

9. An electronic device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device performs the preference alignment training method for a large language model according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the preference alignment training method for a large language model described in any one of claims 1 to 4 is implemented.

Images