Method, device, and electronic device for training large language model

Abstract

The present disclosure relates to a method, device, electronic device and computer-readable storage medium for training a large language model. The present disclosure conducts multiple stages of training on a large language model to enhance its positive knowledge at different training stages. First, in the pre-training stage, the large language model is pre-trained with training samples with positive labels so that the large language model acquires correct cognitive abilities. Next, in the fine-tuning training stage, the parameters of the large language model are fine-tuned by making the large language model generate a directional output response and sampling responses with specific labels accordingly, so that the large language model improves its ability to output positive energy responses. Then, in the reinforcement learning training stage, the probability of the large language model outputting a response text with a positive label is adjusted. The directional knowledge enhancement of the large language model through these three training stages greatly reduces the probability of the large language model outputting harmful responses.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence services, and more specifically to a method / device for training a large language model, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the advancement of deep learning and computing power, large language models (LLMs) have made great progress in many fields and have been widely used in products and services. However, LLMs have many security risks, including the generation of harmful outputs involving physical harm, mental health, property privacy, and moral ethics. This makes commercial deployment for end-user applications still challenging.

[0003] Enhanced Safety Reinforcement Learning (ESRL) was born to improve the safety and reliability of LLM systems. ESRL combines reinforcement learning and safety constraint techniques to impose safety constraints and penalties on LLMs when training and querying them, actively reducing the risk of outputting harmful responses.

[0004] However, the current ESRL technology still has technical problems such as low robustness and high cost, and needs further improvement. Summary of the invention

[0005] The embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a computer-readable storage medium for training a large language model.

[0006] The disclosed embodiment provides a method for training a large language model, comprising: sampling training samples with positive labels from training samples, and pre-training the large language model; obtaining a first prefix vector and a second prefix vector, wherein the first prefix vector and the second prefix vector have opposite semantic information; obtaining a prompt text, and determining an embedding vector corresponding to the prompt text; concatenating the embedding vector corresponding to the prompt text and the first prefix vector to obtain a first training feature vector; concatenating the embedding vector corresponding to the prompt text and the second prefix vector to obtain a second training feature vector; and fine-tuning the pre-trained large language model, wherein the fine-tuning training comprises: determining a first response based on the first training feature vector and using the large language model in fine-tuning training; determining a second response based on the second training feature vector and using the large language model in fine-tuning training; and adjusting parameters of the large language model based on the first training feature vector, the second training feature vector, and the labels of the first response and the second response, so that the labels of the responses generated by the large language model are all positive labels or positive labels.

[0007] The embodiment of the present disclosure provides a device for training a large language model, comprising: a pre-training module, configured to: sample training samples with positive labels from training samples, and pre-train the large language model; a fine-tuning training module, configured to: obtain a first prefix vector and a second prefix vector, wherein the first prefix vector and the second prefix vector have opposite semantic information; obtain a prompt text, and determine an embedding vector corresponding to the prompt text; concatenate the embedding vector corresponding to the prompt text and the first prefix vector to obtain a first training feature vector; concatenate the embedding vector corresponding to the prompt text and the second prefix vector to obtain a second training feature vector; and fine-tune the pre-trained large language model, wherein the fine-tuning training comprises: determining a first response based on the first training feature vector and using the large language model in fine-tuning training; determining a second response based on the second training feature vector and using the large language model in fine-tuning training; and adjusting parameters of the large language model based on the first training feature vector, the second training feature vector, and the labels of the first response and the second response, so that the labels of the responses generated by the large language model are all positive labels or positive labels.

[0008] An embodiment of the present disclosure discloses an electronic device, including: one or more processors; and one or more memories, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the above method is performed.

[0009] An embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the above method is implemented.

[0010] According to another aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various aspects or various optional implementations of the above-mentioned various aspects.

[0011] The disclosed embodiment trains the large language model in multiple stages to enhance its knowledge with positive energy in different training stages. First, in the pre-training stage, the large language model is pre-trained with training samples with positive labels so that the large language model acquires correct cognitive abilities. Next, in the fine-tuning training stage, the parameters of the large language model are fine-tuned by making the large language model produce directional output responses and sampling responses with specific labels accordingly, so that the large language model improves its ability to output positive energy responses. Then, in the reinforcement learning training stage, the probability of the large language model outputting response text with positive labels is adjusted. The directional knowledge enhancement of the large language model through these three training stages greatly reduces the probability of the large language model outputting harmful responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the following briefly introduces the drawings required for describing the embodiment. The drawings described below are only exemplary embodiments of the present disclosure.

[0013] Figure 1 is an example schematic diagram showing a scenario according to an embodiment of the present disclosure.

[0014] Figure 2 is a schematic diagram showing a large language model in the related art.

[0015] Figure 3 A flowchart of a method for training a large language model according to an embodiment of the present disclosure is shown.

[0016] Figure 4 Another flowchart of a method for training a large language model according to an embodiment of the present disclosure is shown.

[0017] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0018] Figure 6 A schematic diagram showing the architecture of an exemplary computing device according to an embodiment of the present disclosure.

[0019] Figure 7 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0021] In this specification and the drawings, operations and elements having substantially the same or similar features are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements will be omitted. Meanwhile, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the descriptions, and cannot be understood as indicating or implying relative importance or ranking.

[0022] To facilitate description of the present disclosure, abbreviations and concepts related to the present disclosure are introduced below.

[0023] Enhanced Safety Reinforcement Learning for LLM (ESRL): ESRL is a method for improving safety and reliability for large language models (LLMs). It combines reinforcement learning with safety constraints, imposing constraints during the learning process to reduce the risk of outputting dangerous or unsafe results.

[0024] Large Language Models (LLMs) are large-scale neural network models used to process human language. They are usually pre-trained on large amounts of text data, learn to predict word sequences, and can generate high-quality, diverse, and coherent text. The number of parameters of LLMs can reach tens of billions, far exceeding traditional models. LLMs can generate realistic and semantically coherent long texts.

[0025] Optionally, various models that can be used in embodiments of the present disclosure below can all be artificial intelligence models, especially neural network models based on artificial intelligence. Typically, neural network models based on artificial intelligence are implemented as acyclic graphs in which neurons are arranged in different layers. Typically, a neural network model includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating outputs in the output layer. The network nodes are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the neural network is propagated to the nodes of the output layer via any one of the hidden layer, activation layer, pooling layer, convolutional layer, etc. The input and output of the neural network model can take various forms, and the present disclosure is not limited to this.

[0026] The solutions provided in the embodiments of the present disclosure involve technologies such as artificial intelligence or machine learning, which are specifically described through the following embodiments.

[0027] First refer to Figure 1 The present invention describes an application scenario of a method for training a large language model and a corresponding device according to an embodiment of the present invention. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and multiple terminals 120 are schematically shown.

[0028] The large language model of the embodiment of the present disclosure can be integrated into various electronic devices, for example, Figure 1 The server 110 and any electronic device in the plurality of terminals 120. For example, the large language model can be integrated in the terminal 120. The terminal 120 can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal computer (PC), a smart speaker or a smart watch, etc., but is not limited thereto.

[0029] For another example, the large language model can also be integrated in the server 110. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this disclosure.

[0030] It is understandable that the device for reasoning using the large language model of the embodiment of the present disclosure can be a terminal, a server, or a system composed of a terminal and a server. The method for training a large language model of the embodiment of the present disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0031] It is worth noting that the terminal 110 and the server 120 according to the embodiment of the present disclosure comply with the data protection principles, respect the user's data rights, and protect the user's data security and privacy. According to the terminal 110 and the server 120 of the embodiment of the present disclosure, the user will be clearly informed of the purpose, method and scope of collecting, using, storing, transmitting and deleting the user's data, and the user's consent will be obtained. According to the terminal 110 and the server 120 of the embodiment of the present disclosure, reasonable technical and management measures will be taken to prevent the user's data from being leaked, tampered with, damaged or lost. According to the terminal 110 and the server 120 provider of the embodiment of the present disclosure, the user's data will be reviewed and updated regularly, and expired or useless data will be deleted in a timely manner. In addition, the artificial intelligence service provider using the embodiment of the present disclosure respects the user's rights to data access, correction, deletion, withdrawal of consent, complaints and claims, and provides convenient channels and procedures for users to effectively exercise these rights.

[0032] Furthermore, the process of using the large language model to interact with data in the terminal 120 or the server 110 is based on the principles of legality, rationality and transparency. The data collected and processed by the large language model according to the embodiment of the present disclosure is relevant, necessary and appropriate for the prediction purpose and does not contain any personal identity information or sensitive information. The large language model according to the embodiment of the present disclosure adopts appropriate technical and organizational measures to protect the security and integrity of the data and prevent the data from being accessed, used or leaked without authorization.

[0033] The AI-based big language model according to the disclosed embodiment will comply with relevant data protection laws and ethical principles. The big language model is trained based on a large amount of anonymized and de-identified data and will not infringe on the privacy rights of any individual or group. The big language model has also been rigorously tested and evaluated to ensure that its output is accurate and reliable and will not cause any misleading or discrimination. The big language model is designed only to improve service quality and customer satisfaction and will not be used for any illegal or unethical purpose. In addition, the big language model will be reviewed and updated regularly to adapt to changes in the data environment and legal regulations.

[0034] Figure 2 is a schematic diagram showing a training process of a large language model in the related art.

[0035] In the related art, the training process of a large language model can generally be divided into three stages, namely, a pre-training stage, a fine-tuning stage, and a reinforcement learning stage.

[0036] The goal of pre-training a large language model is to learn common language representations and build semantic association capabilities. Through self-supervised training on large-scale corpora, the large language model learns to extract contextual semantic information and gain a deep understanding of language rules.

[0037] In the fine-tuning stage, by fine-tuning the network parameters of the pre-trained large language model, the adaptability of the large language model to specific tasks can be quickly enhanced. In this stage, the labels of the response data are usually used to update the parameters of the pre-trained large language model through incremental learning, so that it can more sensitively capture the feature information related to the task.

[0038] In the reinforcement learning stage, the feedback generated by the interaction between the large language model and the user can be used as a reward signal to guide the large language model to produce a better response.

[0039] However, with the continuous development of large models, even after the above three stages of training, there are still great security risks when large language models are used in Chinese environments. They may produce harmful outputs involving physical harm, mental health, property privacy, moral ethics, etc., which limits the practical application of large language models. How to enhance the security and controllability of Chinese language models under the above training framework is a technical problem that needs to be solved urgently.

[0040] To this end, according to an embodiment of the present disclosure, a method for training a large language model is proposed, including: sampling training samples with positive labels from training samples, and pre-training the large language model; obtaining a first prefix vector and a second prefix vector, wherein the first prefix vector and the second prefix vector have opposite semantic information; obtaining a prompt text, and determining an embedding vector corresponding to the prompt text; concatenating the embedding vector corresponding to the prompt text and the first prefix vector to obtain a first training feature vector; concatenating the embedding vector corresponding to the prompt text and the second prefix vector to obtain a second training feature vector; fine-tuning the pre-trained large language model, wherein the fine-tuning training includes: determining a first response based on the first training feature vector and using the large language model in fine-tuning training; determining a second response based on the second training feature vector and using the large language model in fine-tuning training; and adjusting parameters of the large language model based on the first training feature vector, the second training feature vector, and the labels of the first response and the second response, so that the labels of the responses generated by the large language model are all positive labels or positive labels.

[0041] The present disclosure conducts multiple stages of training on a large language model to enhance its positive knowledge in different training stages. First, in the pre-training stage, the large language model is pre-trained with training samples with positive labels so that the large language model acquires correct cognitive abilities. Next, in the fine-tuning training stage, the parameters of the large language model are fine-tuned by making the large language model produce a directional output response and sampling responses with specific labels accordingly, so that the large language model improves its ability to output positive responses. Then, in the reinforcement learning training stage, the probability of the large language model outputting a response text with a positive label is adjusted. The directional knowledge enhancement of the large language model through these three training stages greatly reduces the probability of the large language model outputting harmful responses.

[0042] Refer to the following Figures 3 to 5 A method for training a large language model according to an embodiment of the present disclosure is described. Figure 3 FIG. 3 is a flowchart of a method 30 for training a large language model according to an embodiment of the present disclosure. Optionally, the method 30 may be Figure 1In addition, the method 30 may also be performed by any terminal 120. Figure 1 The terminal 120 and the server 110 are shown to be executed in collaboration, but the present disclosure is not limited thereto.

[0043] See also Figure 3 , in operation S301, training samples with positive labels are sampled from training samples to pre-train the large language model.

[0044] Optionally, in operation S301, based on the source identifier of the training sample, determine the label of the training sample, wherein the source identifier of the training sample includes a website address and a database identifier; based on the label of the training sample, sample training samples with positive labels from the training sample; and preprocess the sampled training samples, wherein the preprocessing includes at least one of the following items: filtering empty data in the training sample, deleting duplicate data in the training sample, and formatting the training data. Of course, the present disclosure is not limited to this.

[0045] Optionally, in order to make the training samples cover a large amount of training data with positive values, a Chinese webpage database including 100 million positive values ​​can be constructed. This database is also called the Chinese positive energy knowledge dataset. The data in the Chinese positive energy dataset comes from multiple authoritative positive energy websites, covering multiple fields that require value judgments. Specifically, the Chinese positive energy dataset systematically integrates the Chinese knowledge graph and contains many positive value themes, such as inspirational stories. By specifying the website URL or dataset identifier as the source identifier of the training sample, its data attributes can be clarified, which helps to construct training samples with distinct categories and clear labels. Compared with the use of general corpus for pre-training in traditional solutions, the use of pre-training datasets enhanced with positive energy knowledge (such as CPVK) can better give large language models positive values.

[0046] Optionally, in order to speed up the processing efficiency of training, the training samples are also preprocessed to improve the quality of the training samples and unify the format of the training samples. Among them, preprocessing includes two methods: general preprocessing and specific preprocessing for data from different sources. General preprocessing includes removing special symbols, deleting repeated low-quality content, filtering short texts, etc. Specific preprocessing for data from different sources adopts different conversion methods according to the sample type. For example, question-and-answer samples are converted into a structured "question-answer" format, legal document samples need to extract legal content, and government document samples extract key nodes. The combination of these two preprocessing methods can effectively improve the quality of samples, making them more suitable as pre-training data for knowledge enhancement. Of course, the present disclosure is not limited to this.

[0047] In operation S302 , a first prefix vector and a second prefix vector are acquired, where the first prefix vector and the second prefix vector have opposite semantic information.

[0048] Optionally, in operation S302, a first prefix vector is determined based on a first data set, and a second prefix vector is determined based on a second data set, wherein the first data set includes a plurality of first keywords, the second data set includes a plurality of second keywords, and the first keywords and the second keywords have opposite positions. Of course, the present disclosure is not limited thereto.

[0049] For example, the first keyword may be a positive keyword, such as "friendly", "tolerant", "justice" and other words with positive meanings. The second keyword may be a negative keyword, such as "hatred", "violence", "harm" and other words with negative meanings. Of course, the embodiments of the present disclosure are not limited to these examples, and may include more keywords representing different emotional tendencies.

[0050] Optionally, a first prefix vector and a second prefix vector are generated respectively by a first data set containing a positive first keyword and a second data set containing a negative second keyword, and the two prefix vectors have obvious semantic distinction. The first prefix vector carries the positive sentiment attribute of the data set, representing a positive, upward, and energetic value orientation. The second prefix vector carries the negative sentiment attribute of the data set, representing a negative and negative value orientation. Therefore, the two prefix vectors have obvious opposition and distance in the semantic space of positive and negative values. The first prefix vector is located in the positive value semantic space, and the second prefix vector is located in the negative value semantic space. Of course, the present disclosure is not limited to this.

[0051] In operation S303, a prompt text is obtained, and an embedding vector corresponding to the prompt text is determined.

[0052] Optionally, the prompt text is a pre-text provided to the large language model in the interaction with the large language model to control the direction of the model generating the response. The prompt text may include elements such as task description, background setting, and expected style. Of course, the present disclosure is not limited to this.

[0053] Optionally, the prompt text can be obtained by manually constructing text containing specific intent, or retrieving natural language sequences associated with the target from a corpus, or abstracting key features of the prompt text through a template. The embedding vector corresponding to the prompt text is a fixed-dimensional vector that encodes the semantic information in the prompt text. That is, the embedding vector is a digital expression of the prompt text.

[0054] Optionally, an embedding layer may be used to perform an embedding operation on the text sequence in the prompt text to convert each word in the prompt text into word-level embedding data. This is then integrated through an encoder to obtain a sentence-level embedding vector or a paragraph-level embedding vector. The present disclosure is not limited thereto.

[0055] In operation S304, the embedding vector corresponding to the prompt text and the first prefix vector are concatenated to obtain a first training feature vector. In operation S305, the embedding vector corresponding to the prompt text and the second prefix vector are concatenated to obtain a second training feature vector.

[0056] Optionally, assume that the dimension of the embedding vector corresponding to the prompt text is m, and the dimension corresponding to the first prefix vector is n. The dimension of the first training feature vector obtained by concatenating the embedding vector corresponding to the prompt text and the first prefix vector by dimension can be m+n. Among them, the first n dimensions in the first training feature vector are the same as the first prefix vector, and the last m dimensions are the same as the embedding vector corresponding to the prompt text. Similarly, the second training feature vector can be obtained in the same way. This disclosure will not be repeated here.

[0057] In operation S306, the pre-trained large language model is fine-tuned, and the fine-tuning training includes: determining a first response based on the first training feature vector and using the large language model in fine-tuning training; determining a second response based on the second training feature vector and using the large language model in fine-tuning training; and adjusting parameters of the large language model based on the first training feature vector, the second training feature vector, and labels of the first response and the second response, so that labels of responses generated by the large language model are all positive labels or active labels.

[0058] Optionally, the annotator can set the labels of the first and second responses to clarify the specific value orientations they correspond to. For example, a positive response emphasizes the position of abiding by the law. When a user asks how to make dangerous items, the model responds by refusing to provide relevant information and advocating compliance with laws and regulations, cherishing life, and staying away from dangerous items. A negative response provides specific methods for making dangerous items. This method aims to train the model to generate text that conforms to a specific orientation in a variety of value settings.

[0059] Thus, a training sample set used in the fine-tuning stage is formed, and each sample in the training sample set used in the fine-tuning stage includes a training feature vector, a response corresponding to the training feature vector, and a label corresponding to the response.

[0060] In each time step of fine-tuning training, different subsets of training samples can be sampled at different sampling rates to train the large language model. For example, in the initial time step of fine-tuning training, more positive fine-tuning training samples can be used so that the large language model can learn how to answer questions positively and generate positive responses more quickly. As the time step increases, more negative fine-tuning training samples can be used to learn how to recognize negative prompt texts. Thus, even when negative prompt texts are input, neutral or positive responses are generated.

[0061] Optionally, method 30 further includes operation S307. In operation 307, a plurality of prompt texts are obtained, and based on the plurality of prompt texts, reinforcement learning training is performed on the fine-tuned large language model, wherein the reinforcement learning training includes: determining a plurality of response texts based on the plurality of prompt texts and using the large language model; and adjusting the parameters of the large language model based on the labels corresponding to the plurality of response texts.

[0062] Optionally, during the reinforcement learning training, the process of adjusting the parameters of the output large language model includes: determining the probability of a response text with a neutral label or a positive label appearing in the multiple response texts based on the labels corresponding to the multiple response texts; and adjusting the parameters of the large language model so that the probability reaches an extreme value. Of course, the present disclosure is not limited to this.

[0063] Optionally, in some embodiments, the reinforcement learning process also includes: sampling response texts with negative labels from the multiple response texts; determining the semantic space vector corresponding to the training samples with positive labels; determining the distance between the semantic vector corresponding to the response texts with negative labels and the semantic space vector; and adjusting the parameters of the large language model to reduce the distance.

[0064] Optionally, in some embodiments, the reinforcement learning process also includes: obtaining a prompt text and determining an embedding vector corresponding to the prompt text; concatenating the embedding vector corresponding to the prompt text and the first prefix vector to obtain a first training feature vector; concatenating the embedding vector corresponding to the prompt text and the second prefix vector to obtain a second training feature vector; determining a first response based on the first training feature vector and using the large language model in the reinforcement learning training; determining a second response based on the second training feature vector and using the large language model in the reinforcement learning training to determine a semantic distance between the first response and the second response; and adjusting parameters of the large language model to reduce the semantic distance between the first response and the second response.

[0065] Will refer to Figure 4The various details of the reinforcement learning training are further described, and this disclosure will not go into details here.

[0066] The disclosed embodiment trains the large language model in multiple stages to enhance its knowledge with positive energy in different training stages. First, in the pre-training stage, the large language model is pre-trained with training samples with positive labels so that the large language model acquires correct cognitive abilities. Next, in the fine-tuning training stage, the parameters of the large language model are fine-tuned by making the large language model produce directional output responses and sampling responses with specific labels accordingly, so that the large language model improves its ability to output positive energy responses. Then, in the reinforcement learning training stage, the probability of the large language model outputting response text with positive labels is adjusted. The directional knowledge enhancement of the large language model through these three training stages greatly reduces the probability of the large language model outputting harmful responses.

[0067] Figure 4 A schematic diagram of reinforcement learning training according to an embodiment of the present disclosure is shown.

[0068] Optionally, in some embodiments of the present disclosure, the reinforcement learning training process is an iterative process based on attack and defense, which can further improve the security of the model output response.

[0069] Specifically, the prompt text obtained during the reinforcement learning training is a specific prompt text set by Internet security experts. These Internet experts act as attackers to induce the large language model to output harmful responses. Internet security experts first determine the category of the attack, which will be used to evaluate the model's defense capabilities against different types of attacks. Then Internet security experts will set different prompt texts based on the category of the attack. For each type of attack, Internet experts will set multiple prompt texts.

[0070] The large language model will act as a defender and output a safe response when faced with misleading prompt text. The process of reinforcement learning training can be summarized as: punishing the large language model for outputting harmful responses and rewarding the large language model for outputting positive or neutral responses.

[0071] Optionally, in some embodiments, the Internet security expert will determine whether the response output by the large language model is a positive response or a harmful response, and assign a corresponding label to the response output by the large language model. Optionally, in some embodiments, the Internet security expert will also determine the degree of harmfulness of the harmful response. The degree of harmfulness will be used as the penalty value of the punishment. Similarly, the Internet expert will also give a reward degree to a positive response or a neutral response as a reward value corresponding to the reward. The present disclosure is not limited to this.

[0072] In other embodiments of the present disclosure, the output response of the large language model can be evaluated by another neural network model. This involves the use of an auxiliary neural network model, which can be a simple classifier for judging the value orientation in the response. The task of this classifier is to analyze the responses generated by the large language model to determine the value orientation contained therein. Specifically, this neural network model may identify the value orientation in the response by learning discriminative features, such as the sentiment of the text, emotional color, or semantic representation of specific values. By judging the output of the large language model, its performance in the direction of specific values ​​can be automatically evaluated.

[0073] Specifically, in some embodiments, the large language model acting as the defender is trained through reinforcement learning, with the goal of maximizing the probability of generating responses with neutral or positive labels. This means that the large language model is designed to be more inclined to generate neutral or positive replies to reduce the risk of generating harmful content. For example, the large language model may be adjusted to promote security more actively, making it more inclined to generate socially beneficial, neutral, or positive text when facing attacks. This approach aims to make the model more cautious and responsible through reinforcement learning, thereby improving its security and credibility in practical applications.

[0074] Optionally, in some embodiments, the large language model as the defender is trained through reinforcement learning, and its goal is to make all responses generated by the large language model fall into a positive semantic space. This process includes the following steps: First, sample response texts with negative labels from multiple response texts so that the large language model as the defender can better deal with inputs with negative emotions or harmful content. Secondly, determine the semantic space vector corresponding to the training sample with positive labels to capture the semantic features of positive responses. Among them, the semantic space vector corresponding to the training sample with positive labels refers to a vector representing positive text features in a semantic space. This vector captures the semantic information of the positive label training sample and reflects the characteristics of the positive emotions or positive values ​​conveyed by the text. Next, determine the semantic vector corresponding to the response text with negative labels, and calculate the distance between the semantic vector and the semantic space vector. This step helps to quantify the processing effect of the large language model on negative responses in the semantic space. Finally, by adjusting the parameters of the large language model, the distance is reduced to improve the understanding and processing ability of the large language model for negative responses.

[0075] Optionally, in some embodiments, the large language model as the defender is trained through reinforcement learning, and its goal is to not be induced to output harmful responses for the same prompt text in the context of multiple rounds of dialogue. Specifically, as an Internet security expert on the attacking side, it is possible to input prompt texts with negative tendencies to the large language model through multiple rounds of dialogue. These rounds of dialogues often serve as the context for the large language model to generate output responses. Then in a certain round of dialogue, although the Internet expert on the attacking side did not input prompt texts with obvious inducing tendencies, the large language model that has not been enhanced with security knowledge often outputs harmful responses induced based on the context in such a situation. In order to avoid such a situation, the large language model can output responses with close semantic distances through a reinforcement learning process, even if the same prompt text has different prefixes (the prefix can be used to characterize contextual information). The process can be briefly described as follows.

[0076] First, the prompt text for training is obtained and mapped into the embedding space to obtain the corresponding embedding vector. This step aims to convert the text information into a vector representation so that the model can understand and process it. Next, the embedding vector is concatenated with the first prefix vector and the second prefix vector respectively to obtain the first training feature vector and the second training feature vector. Thus, contextual information of different value directions is introduced through the first prefix vector and the second prefix vector. Next, the large language model in reinforcement learning is used to determine the first response based on the first training feature vector and the second response based on the second training feature vector. Then, the semantic distance between the first response and the second response is determined to quantify their similarity or difference in the semantic space. Finally, by monitoring the semantic distance between the responses, the parameters of the large language model are adjusted to make the generated responses more consistent and robust. As a result, the model's ability to understand and adapt to different values ​​and semantic directions is improved. Through this training process, the large language model can better adapt to a variety of prompt texts and generate safer and more expected text responses.

[0077] According to yet another aspect of the present disclosure, an electronic device is provided for implementing the method according to the embodiment of the present disclosure. Figure 5 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0078] like Figure 5 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer-readable code, and when the computer-readable code is run by the one or more processors 2010, the method described above may be executed.

[0079] The processor in the embodiments of the present disclosure may be an integrated circuit chip with signal processing capabilities. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., which may be an X86 architecture or an ARM architecture.

[0080] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0081] For example, the method or device according to the embodiment of the present disclosure may also be Figure 6 The architecture of the computing device 3000 shown in FIG. Figure 6 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided in the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 7 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 7 One or more components of a computing device are shown.

[0082] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Figure 7 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0083] like Figure 7As shown, the computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0084] The present disclosure also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method according to the present disclosure.

[0085] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0087] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for training a large language model, comprising: Sampling training samples with positive labels from the training samples to pre-train the large language model; Acquire a first prefix vector and a second prefix vector, wherein the first prefix vector and the second prefix vector have opposite semantic information; Obtaining a prompt text, and determining an embedding vector corresponding to the prompt text; Concatenate the embedding vector corresponding to the prompt text and the first prefix vector to obtain a first training feature vector; Concatenate the embedding vector corresponding to the prompt text and the second prefix vector to obtain a second training feature vector; Fine-tune the pre-trained large language model, and the fine-tune training includes: Determine a first response based on the first training feature vector using the large language model in fine-tuning training; Determine a second response based on the second training feature vector using the large language model in fine-tuning training; as well as Based on the first training feature vector, the second training feature vector, the labels of the first response and the second response, adjusting the parameters of the large language model so that the labels of the responses generated by the large language model are all positive labels or active labels, The obtaining of the first prefix vector and the second prefix vector comprises: Based on the first data set, determining a first prefix vector; Based on the second data set, determining a second prefix vector; The first data set includes a plurality of first keywords, the second data set includes a plurality of second keywords, and the first keywords and the second keywords have opposite positions.

2. The method of claim 1, wherein: The step of sampling a training sample with a positive label from the training sample comprises: Determining a label of the training sample based on a source identifier of the training sample, wherein the source identifier of the training sample includes a website address and a database identifier; Based on the labels of the training samples, sampling training samples with positive labels from the training samples; and The sampled training samples are preprocessed, wherein the preprocessing includes at least one of the following items: filtering empty data in the training samples, deleting duplicate data in the training samples, and formatting data in the training samples.

3. The method of claim 1, further comprising: Get multiple prompt texts. Based on the plurality of prompt texts, reinforcement learning training is performed on the large language model after fine-tuning training, and the reinforcement learning training includes: Based on the plurality of prompt texts, using the large language model, determining a plurality of response texts; Based on the labels corresponding to the multiple response texts, the parameters of the large language model are adjusted.

4. The method of claim 3, wherein: The adjusting the parameters of the output large language model based on the labels corresponding to the multiple response texts includes: Based on the labels corresponding to the plurality of response texts, determining the probability that a response text having a neutral label or a positive label appears in the plurality of response texts; and The parameters of the large language model are adjusted so that the probability reaches an extreme value.

5. The method of claim 4, wherein: The reinforcement learning training of the large language model after fine-tuning training includes: Sampling response texts with negative labels from the multiple response texts; Determine the semantic space vector corresponding to the training samples with positive labels; Determining the distance between the semantic vector corresponding to the response text with the negative label and the semantic space vector; and Parameters of the large language model are adjusted so that the distance decreases.

6. The method of claim 3, wherein: The reinforcement learning training of the large language model after fine-tuning training includes: Obtaining a prompt text, and determining an embedding vector corresponding to the prompt text; Concatenate the embedding vector corresponding to the prompt text and the first prefix vector to obtain a first training feature vector; Concatenate the embedding vector corresponding to the prompt text and the second prefix vector to obtain a second training feature vector; Determine a first response based on the first training feature vector using the large language model trained by reinforcement learning; Based on the second training feature vector, using the large language model in the reinforcement learning training, determine a second response determining a semantic distance between the first response and the second response; and Parameters of the large language model are adjusted so that a semantic distance between the first response and the second response is reduced.

7. A device for training a large language model, comprising: The pre-training module is configured to: sample training samples with positive labels from the training samples to pre-train the large language model; The fine-tuning training module is configured as follows: Acquire a first prefix vector and a second prefix vector, wherein the first prefix vector and the second prefix vector have opposite semantic information; Obtaining a prompt text, and determining an embedding vector corresponding to the prompt text; Concatenate the embedding vector corresponding to the prompt text and the first prefix vector to obtain a first training feature vector; Concatenate the embedding vector corresponding to the prompt text and the second prefix vector to obtain a second training feature vector; Fine-tune the pre-trained large language model, and the fine-tune training includes: Determine a first response based on the first training feature vector using the large language model in fine-tuning training; Determine a second response based on the second training feature vector using the large language model in fine-tuning training; as well as Based on the first training feature vector, the second training feature vector, the labels of the first response and the second response, adjusting the parameters of the large language model so that the labels of the responses generated by the large language model are all positive labels or active labels, The obtaining of the first prefix vector and the second prefix vector comprises: Based on the first data set, determining a first prefix vector; Based on the second data set, determining a second prefix vector; The first data set includes a plurality of first keywords, the second data set includes a plurality of second keywords, and the first keywords and the second keywords have opposite positions.

8. An electronic device comprising: one or more processors; and One or more memories, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the method of any one of claims 1-6 is executed.

9. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method according to any one of claims 1 to 6.

Images