Large language model alignment method and device, electronic equipment and storage medium

Abstract

The invention provides a large language model alignment method, which comprises the following steps: acquiring a pre-trained large language model, and initializing the pre-trained large language model to enter a text completion task state; performing model feature retrieval on the target data set, and determining at least one target large language model; constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into a pre-trained large language model; obtaining a target session text, updating the target session text into the context alignment prompt word, and reasoning to obtain a session response text; the method comprises the following steps of: adding a session response text of a current session round into a context alignment prompt word; and obtaining a target session text of the next dialogue round, inputting the target session text and the context alignment prompt word into the large language model to obtain a session response text of the next dialogue round, and setting the next dialogue round as the current dialogue round. According to the method, efficient alignment of the pre-trained large language model can be realized.

Description

Technical Field

[0001] The present application relates to the field of large language models, and in particular to a large language model alignment method, device, electronic device and storage medium. Background Art

[0002] Large Language Models (LLMs) are a type of deep learning model that can understand and generate natural language text. They have shown great potential in language understanding, text generation, question-answering systems and other fields. These models are learned through large-scale text data in the pre-training phase, and then fine-tuned on specific tasks to achieve more accurate and specific language understanding and generation. Large language model alignment is the process of guiding large language models to learn human interaction preferences, follow human instructions, and not generate harmful text.

[0003] In related technologies, alignment methods usually rely on additional artificial data and fine-tuning training to achieve instruction following, such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). These methods require a lot of manual intervention and cumbersome training processes, resulting in high model alignment costs and low efficiency. Summary of the invention

[0004] The embodiments of the present application provide a large language model alignment method, device, electronic device and storage medium, which can achieve efficient alignment of pre-trained large language models without relying on a large amount of artificial data.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a large language model alignment method, the method comprising:

[0006] Obtain a pre-trained large language model, and initialize the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is the target data set;

[0007] Performing model feature retrieval on the target data set to determine at least one target large language model;

[0008] Constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into the pre-trained large language model so that the pre-trained large language model is aligned with reference to the target large language model;

[0009] Acquire a target conversation text, and add the target conversation text to the context alignment prompt word, so that the pre-trained large language model performs task processing on the target conversation text to obtain a conversation response text;

[0010] The following steps are executed in a loop:

[0011] Adding the conversation response text of the current dialogue round to the context alignment prompt words;

[0012] In response to obtaining the target conversation text for the next dialogue round, the target conversation text for the next dialogue round and the context alignment prompt words are input into the pre-trained large language model to obtain the conversation response text for the next dialogue round, and the next dialogue round is set as the current dialogue round.

[0013] In some embodiments, the context alignment prompt word includes a first prompt word field, and constructing the context alignment prompt word according to the target large language model includes:

[0014] Obtain target keywords, preset typesetting format, and preset dialogue structure format;

[0015] Generating the format constraint field according to the preset typesetting format and the preset dialog structure format;

[0016] Generate a first prompt word field according to the target name of the target large language model, the target keyword, the typesetting format, and the dialogue structure format;

[0017] The context alignment prompt word is constructed according to the first prompt word field.

[0018] In some embodiments, the context alignment prompt word further includes a second prompt word field, and constructing the context alignment prompt word according to the target large language model includes:

[0019] Obtaining a target name and a target system word of the target large language model;

[0020] Constructing the second prompt word field according to the target name and the target system word;

[0021] The context alignment prompt word is constructed according to the first prompt word field and the second prompt word field.

[0022] In some embodiments, the context alignment prompt word further includes a third prompt word field, and constructing the context alignment prompt word according to the target large language model includes:

[0023] Acquire a first target safe word of a target large language model; wherein the first target safe word is used to guide the large language model to generate a safe text;

[0024] generating the third prompt word field based on the first target safe word;

[0025] The context alignment cue word is constructed according to the first cue word field, the second cue word field and the third cue word field.

[0026] In some embodiments, the context alignment prompt word further includes a fourth prompt word field, and the constructing the context alignment prompt word according to the target large language model includes:

[0027] Acquire an identity inquiry text, and acquire an actual identity response text of the target large language model for the identity inquiry text based on the identity inquiry text;

[0028] Constructing the identity constraint field according to the identity query text and the actual identity reply text;

[0029] The context alignment cue word is constructed according to the first cue word field, the second cue word field, the third cue word field, and the fourth cue word field.

[0030] In some embodiments, the context alignment prompt word further includes a fifth prompt word field, and the constructing the context alignment prompt word according to the target large language model includes:

[0031] Acquire a capability inquiry text, and acquire an actual capability response text of the target large language model for the capability inquiry text based on the capability inquiry text;

[0032] Constructing the capability constraint field according to the capability inquiry text and the actual capability reply text;

[0033] The context alignment cue word is constructed according to the first cue word field, the second cue word field, the third cue word field, the fourth cue word field, and the fifth cue word field.

[0034] In some embodiments, constructing the context alignment prompt word according to the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field, and the fifth prompt word field includes:

[0035] Based on the preset typesetting format and the preset dialogue structure format, the context alignment prompt word is generated according to the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field and the fifth prompt word field.

[0036] To achieve the above purpose, a second aspect of an embodiment of the present application proposes an alignment device for a large language model, comprising:

[0037] A first acquisition module is used to acquire a pre-trained large language model and initialize the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is a target data set;

[0038] A determination module, configured to perform model feature retrieval on the target data set to determine at least one target large language model;

[0039] A construction module, configured to construct a context alignment prompt word according to the target large language model, and input the context alignment prompt word into the pre-trained large language model so that the pre-trained large language model is aligned with reference to the target large language model;

[0040] A second acquisition module is used to acquire a target conversation text, and add the target conversation text to the context alignment prompt word, so that the pre-trained large language model performs task processing on the target conversation text to obtain a conversation response text;

[0041] An adding module, used for adding the conversation response text of the current dialogue round to the context alignment prompt word;

[0042] A response module is used to, in response to obtaining a target conversation text for a next conversation round, input the target conversation text for the next conversation round and the context alignment prompt words into the pre-trained large language model, obtain the conversation response text for the next conversation round, and set the next conversation round as the current conversation round.

[0043] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the large language model alignment method as described in the first aspect when executing the computer program.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the alignment method of the large language model as described in the first aspect is implemented.

[0045] The embodiment of the present application proposes a large language model alignment method, electronic device and storage medium, the method comprising obtaining a pre-trained large language model, and initializing the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is a target data set; performing model feature retrieval on the target data set to determine at least one target large language model; constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into the pre-trained large language model, so that the pre-trained large language model performs alignment with reference to the target large language model; obtaining a target conversation text, adding the target conversation text to the context alignment prompt word, so that the pre-trained large language model performs task processing on the target conversation text, and obtains a conversation response text; and loopingly executing the following steps: adding the conversation response text of the current dialogue round to the context alignment prompt word; in response to obtaining the target conversation text of the next dialogue round, inputting the target conversation text of the next dialogue round and the context alignment prompt word into the pre-trained large language model together, obtaining the conversation response text of the next dialogue round, and setting the next dialogue round as the current dialogue round.

[0046] With respect to the problems in the background technology, the large language model alignment method provided in the embodiment of the present application utilizes prompt engineering technology, refers to the characteristics of the target large language model, and utilizes the text continuation reasoning ability of the pre-trained large language model to construct contextual alignment prompt words, which are input into the original pre-trained large language model, thereby effectively guiding and aligning the behavior of the large language model. This enables the pre-trained large language model, which originally only has text completion capabilities, to exhibit advanced capabilities such as instruction following and task processing similar to the target large language model. Compared with the traditional model fine-tuning training method, the high manpower and high computing costs of constructing a new instruction following data set and fine-tuning the training model are avoided. This method does not require additional dedicated artificial data and fine-tuning training. Compared with the existing large language model alignment method, it reduces the alignment cost, improves the alignment performance, and achieves fast alignment.

[0047] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a large language model alignment method provided in one embodiment of the present application;

[0049] Figure 2 An embodiment of the present application provides Figure 1 Flow chart of step 103;

[0050] Figure 3 Another embodiment of the present application provides Figure 1 Flow chart of step 103;

[0051] Figure 4 Another embodiment of the present application provides Figure 1 Flow chart of step 103;

[0052] Figure 5 Another embodiment of the present application provides Figure 1 Flow chart of step 103;

[0053] Figure 6 Another embodiment of the present application provides Figure 1 Flow chart of step 103;

[0054] Figure 7 A schematic diagram of prompt words provided in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of an alignment device for a large language model provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0061] Large Language Models (LLMs) are machine learning models with large-scale parameters and complex computational structures. They are usually built from deep neural networks and have billions or even hundreds of billions of parameters. They can handle more complex tasks and data, have strong generalization capabilities, and can make accurate predictions on unseen data. Large language models have a wide range of applications in the field of natural language processing (NLP), including but not limited to text generation, translation, summarization, question-answering systems, etc. They can learn various grammatical, semantic, and contextual rules of natural languages ​​by training on large-scale corpora.

[0062] Base Large Language Models (Base LLMs) are pre-trained large language models that have not been fine-tuned for a specific task. They are trained on massive amounts of text to predict the next word or sentence in a text sequence, but these predictions are not necessarily responses to specific instructions or dialogues. Such models are the starting point for large language models (LLMs) and typically have broad language understanding and generation capabilities.

[0063] Supervised Fine-Tun ing Large Language Models (SFT LLMs) means further training the model with labeled data based on pre-training to make it more suitable for specific tasks or fields. This process is called supervised fine-tuning (SFT), and its purpose is to guide the model to learn specific rules required by the business and stabilize the output format based on the knowledge learned in pre-training.

[0064] Reinforcement Learning Large Language Models (RLLMs) combine reinforcement learning (RL) techniques to learn strategies through interaction with the environment, thereby optimizing the model's decision-making process. In the context of large language models, RL can be used to adjust the model's behavior to better match human feedback and preferences.

[0065] As instruction-following language models such as ChatGPT and Cl aude have shown strong general-purpose task performance, large language models have become one of the research hotspots in the field of artificial intelligence. LLMs are a type of deep learning model that can understand and generate natural language texts, and have shown great potential in language understanding, text generation, question-answering systems, and other fields. These models are learned through large-scale text data in the pre-training stage, and then fine-tuned on specific tasks to achieve more accurate and specific language understanding and generation. Large language model alignment is to guide large language models to learn human interaction preferences, follow human instructions, and not generate harmful text. Existing alignment methods usually rely on additional artificial data and fine-tuning training to achieve instruction following, such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). These methods require a lot of manual intervention and cumbersome training processes, resulting in high economic and time costs. In addition, existing contextual alignment methods also have the problem of topic bias, which affects the accuracy and generalization of alignment and limits its reliability in practical applications. In summary, the existing alignment methods still have room for improvement in terms of effectiveness and efficiency. There are problems such as reliance on additional data and fine-tuning training, high cost, and topic bias. Therefore, it is necessary to propose a more efficient, fast, accurate, and reliable alignment method to meet the needs of large language model alignment in more practical applications.

[0066] The embodiment of the present application proposes a large language model alignment method, electronic device and storage medium, the method comprising obtaining a pre-trained large language model, and initializing the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is a target data set; performing model feature retrieval on the target data set to determine at least one target large language model; constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into the pre-trained large language model, so that the pre-trained large language model performs alignment with reference to the target large language model; obtaining a target conversation text, adding the target conversation text to the context alignment prompt word, so that the pre-trained large language model performs task processing on the target conversation text, and obtains a conversation response text; and loopingly executing the following steps: adding the conversation response text of the current dialogue round to the context alignment prompt word; in response to obtaining the target conversation text of the next dialogue round, inputting the target conversation text of the next dialogue round and the context alignment prompt word into the pre-trained large language model together, obtaining the conversation response text of the next dialogue round, and setting the next dialogue round as the current dialogue round.

[0067] With respect to the problems in the background technology, the large language model alignment method provided in the embodiment of the present application utilizes prompt engineering technology, refers to the characteristics of the target large language model, and utilizes the text continuation reasoning ability of the pre-trained large language model to construct contextual alignment prompt words, which are input into the original pre-trained large language model, thereby effectively guiding and aligning the behavior of the large language model. This enables the pre-trained large language model, which originally only has text completion capabilities, to exhibit advanced capabilities such as instruction following and task processing similar to the target large language model. Compared with the traditional model fine-tuning training method, the high manpower and high computing costs of constructing a new instruction following data set and fine-tuning the training model are avoided. This method does not require additional dedicated artificial data and fine-tuning training. Compared with the existing large language model alignment method, it reduces the alignment cost, improves the alignment performance, and achieves fast alignment.

[0068] The following will further describe a large language model alignment method, electronic device, and storage medium provided by an embodiment of the present application. First, the large language model alignment method in an embodiment of the present application is specifically described.

[0069] Reference Figure 1 , which is an optional flow chart of the large language model alignment method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps 101 to 106. At the same time, it can be understood that this embodiment Figure 1 The order of step 101 to step 106 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0070] Step 101: Obtain a pre-trained large language model and initialize the large language model to enter the text completion task state.

[0071] Among them, the pre-trained large language model has learned extensive natural language knowledge in the pre-training stage, but its core ability is limited to predicting and generating the next reasonable word or word unit sequence based on the given context text, that is, to complete the text. Therefore, the purpose of step 101 is to put the pre-trained model in its native and most basic text generation mode to prepare for the subsequent alignment process. It should be noted that although it only has the ability of text completion, the pre-trained large language model has acquired certain reasoning capabilities by learning contextual semantics. This potential semantic understanding and generation ability will be guided and aligned through carefully designed prompt words in subsequent steps, so that the model can exhibit high-level language interaction behaviors similar to the target large language model.

[0072] Therefore, as the beginning of the alignment method, step 101 is important in obtaining a powerful basic large language model that has not been specially trained for alignment, which is the key prerequisite for achieving alignment without fine-tuning training. Through initialization, the model is awakened to the most initial text generation state, fully preparing for the subsequent fine alignment process.

[0073] Step 102: Perform large language model feature retrieval on the target data set to determine at least one target large language model.

[0074] Among them, the target large language model is a large language model that has good performance in the target data set, and it already has good advanced capabilities such as command following and task processing. By incorporating the names and features of these target large language models into the construction of alignment prompt words, the pre-trained large language model can be effectively guided to align the behavior of the target large language model. Specifically, taking ChatGPT as an example, instilling the concept of "I am ChatGPT" into the pre-trained large language model can awaken the knowledge related to ChatGPT that it has seen in the pre-training data, so that it can restore and simulate ChatGPT's command understanding, response generation and other behavioral patterns as much as possible. By utilizing the rich knowledge learned by the large language model in the pre-training stage, a knowledge foundation is laid for the subsequent alignment prompt word design. Through flexible prompt engineering, the potential semantic understanding and generation capabilities of the pre-trained large language model can be fully explored and stimulated, avoiding the high computational and manpower costs of existing large language model alignment fine-tuning methods.

[0075] In some embodiments, the target large language model can be retrieved from the target data set with the highest frequency, i.e., the most widely mentioned large language model, such as ChatGPT. Since the large language model is a probabilistic model, the more widely known the knowledge is, the better it can learn. Therefore, the existing AI name is used here to make it think that it is ChatGPT, so that it can better refer to the knowledge related to AI assistant command following that it has learned in the pre-training data set.

[0076] Step 103 , constructing context alignment prompt words according to the target large language model, and inputting the context alignment prompt words into the large language model, so that the pre-trained large language model is aligned with reference to the target large language model.

[0077] Among them, the core of this method is to establish a suitable text scenario through the reasoning ability of the pre-trained large language model in the text completion task, so that the large language model can generate subsequent aligned conversation texts more reasonably. Therefore, the core idea of ​​the present invention is to build the behavioral pattern text scenario of the aligned large language model as much as possible in the context, and awaken the behavioral pattern of the mature large language model (such as ChatGPT) that the pre-trained large language model has mastered in pre-training. These prompt words are carefully designed text fragments that can guide the pre-trained large language model to understand and adapt to the behavioral pattern of the target large language model. The process of constructing prompt words includes defining the format of the dialogue, setting roles, introducing conditions, system settings, safety statements, role confirmation and skill confirmation, etc. These prompt words are then input into the pre-trained large language model so that the model can learn and align in context.

[0078] Reference Figure 2 In some embodiments, the context alignment prompt word includes a first prompt word field, and step 103 may include but is not limited to steps 201 to 204.

[0079] Step 201, obtaining target keywords, a preset typesetting format, and a preset dialogue structure format.

[0080] In step 201, firstly, the target keywords, the preset typesetting format and the preset dialogue structure format are obtained. For example, for the pre-trained large language model, the target keyword is determined to be "correct". These keywords can represent the core capabilities and characteristics of the target model, enabling it to generate positive answers. The preset typesetting format can specify the use of ChatML or Markdown format to organize the dialogue content, while the dialogue structure format defines the start and end markers of the dialogue, such as using specific symbols or codes to identify the speaker and the speech content.

[0081] Step 202: Generate a format constraint field according to a preset typesetting format and a preset dialog structure format.

[0082] In step 202, the format constraint field ensures the consistency of the prompt word in structure, so that the pre-trained large language model can more easily recognize and process the information in the prompt word and generate a text with a regular format.

[0083] Step 203: Generate a first prompt word field according to the target name, target keywords, typesetting format and dialogue structure format of the target large language model.

[0084] In step 203, the target name provides a reference object for alignment for the pre-trained large language model, while the target keyword further refines the specific direction and content of the alignment. For example, a text paragraph is created, the format and structure of the conversation are declared at the beginning, and then the role of ChatGPT is introduced, and some positive descriptions about it are given, such as using natural language "The following is a **correct** conversation between ChatGPT (**GPT4**) and the user, organized in **ChatML** and **Markdown** format:" to describe the keywords and format of the subsequent text to be completed, where the conversation starter and end characters are "<|im_start|>" and "<|im_end|>" respectively.

[0085] Step 204: construct context alignment prompt words according to the first prompt word field.

[0086] In this step, the generated format constraint field is added to the previously constructed context alignment prompt word to form the context alignment prompt word. By adding the format constraint field, the format and structure of the context alignment prompt word can be precisely constrained and standardized. This helps ensure that the response text generated by the large language model strictly follows the grammatical rules of ChatML and Markdown, improving the accuracy and consistency of alignment.

[0087] Reference Figure 3 In some embodiments, the context alignment prompt word further includes a second prompt word field, and step 103 may include but is not limited to steps 301 to 303 .

[0088] Step 301, obtaining a target name and a target system word of a target large language model.

[0089] The target name usually refers to the name of the well-known large language model to be aligned, such as ChatGPT. The target system word directly uses the native system prompt word used by the large language model in actual applications, such as "You are ChatGPT, a large language model based on the GPT-4 architecture trained by OpenAI."

[0090] Step 302: construct a second prompt word field according to the target name and the target system word.

[0091] Among them, the name constraint field can be constructed by combining the target name with natural language, which is designed to give the pre-trained large language model a clear ChatGPT identity awareness.

[0092] Exemplarily, a target name, such as ChatGPT, is constructed into a string "<|im_start|>ChatGPT" according to a preset typesetting format as a target name conversation prompt, which is added to each round of conversation in the context prompt words so that the target name is always emphasized in the context as the artificial intelligence assistant role of the conversation.

[0093] Among them, the native system prompt word of the target large language model obtained in step 301, "You are ChatGPT, a large language model based on the GPT-4 architecture trained by OpenAI." can be directly used as the system word constraint field. It contains the core attribute descriptions of ChatGPT's identity, capabilities, technical architecture, etc.

[0094] The above two fields can be combined to form a second prompt word field. The prompt word contains both the model's "ChatGPT identity" and "native system prompt" identity. It can highly guide the pre-trained large language model to imitate and reproduce ChatGPT's semantic understanding, response generation and other behaviors.

[0095] Step 303: construct context alignment prompt words according to the first prompt word field and the second prompt word field.

[0096] In this way, the context alignment prompt word maximizes the use of the key feature description of the target model itself, from identity to system status. Under the guidance of the prompt word, the pre-trained large language model can achieve preliminary alignment with the instruction-following text continuation capability of the target large language model.

[0097] Reference Figure 4 In some embodiments, the context alignment prompt word further includes a third prompt word field, and step 103 may include but is not limited to steps 401 to 403.

[0098] Step 401, obtaining a first target safe word of a target large language model.

[0099] Among them, the first target safety word set by the target large language model itself can be obtained. This may be a key word used by the target model in the safety statement field to constrain itself from generating dangerous, inappropriate, or harmful content, such as the word "harmless".

[0100] In some embodiments, a second target safety word may also be obtained, wherein the second target safety word is a high-frequency safety prompt word displayed in other aligned large language model data in the target data set except for the target large language model. For example, in these aligned large language model data, words or phrases such as "authentic and credible", "neutral and objective", "factual basis", "avoid offensive terms" may appear frequently. These words reflect the safety and quality principles pursued by these aligned large language models, and these high-frequency keywords or texts can be used as second target safety words.

[0101] By combining this second target safety word derived from the target data set with the first target safety word obtained from the target large language model system itself in step 401, a more comprehensive and practical security constraint field can be constructed.

[0102] Step 402, generating a third prompt word field based on the first target safe word;

[0103] After obtaining these two types of security words with different sources but the same goals, step 403 merges them to generate a complete security constraint field, which can be in the form of a directive statement such as "You are a useful, respectful and honest assistant. Your answers must be socially responsible, so you can refuse to answer some controversial topics."

[0104] Step 403: construct context alignment prompt words according to the first prompt word field, the second prompt word field and the third prompt word field.

[0105] Finally, in step 403, the security constraint field, i.e., the third prompt word field, is combined with other prompt word fields constructed previously to form contextual alignment prompt words including format constraints, identity constraints, and security word constraints. This allows the prompt words to not only constrain the format and identity of the model's answer text, but also make clear restrictive requirements on the security and quality of its output content.

[0106] Reference Figure 5 In some embodiments, the context alignment prompt word includes a fourth prompt word field, and step 103 may include but is not limited to steps 501 to 503.

[0107] Step 501: Acquire an identity query text, and acquire an actual identity response text of a target large language model that matches the identity query text based on the identity query text.

[0108] In this step, first, a set of predefined identity query texts can be obtained. Identity query texts are questions used to explore the identity characteristics of the target large language model, such as "Who are you?" "What is your name?" "What kind of AI assistant are you?" etc. Then, each identity query text is input into the target large language model to obtain the corresponding response generated by the target large language model, that is, the actual identity response text. The actual identity response text reflects the target large language model's cognition and expression of its own identity.

[0109] For example, if the target large language model is ChatGPT, then its response to the identity inquiry text "Who are you?" might be "Hello! I am ChatGPT. How can I help you?".

[0110] Step 502: construct a fourth prompt word field according to the identity query text and the actual identity reply text.

[0111] The identity query text and the corresponding actual identity reply text may be combined into a key-value pair to construct an identity constraint field. They are combined together according to the format declared in the first prompt word field to form a complete identity constraint field.

[0112] Step 503: construct context alignment prompt words according to the first prompt word field, the second prompt word field, the third prompt word field and the fourth prompt word field.

[0113] In this step, the previously constructed prompt word fields are combined to form contextual alignment prompt words. By constructing contextual alignment prompt words containing identity constraint fields, the identity features of the large language model can be aligned more comprehensively and accurately. When the large language model generates a response, it will be affected by the identity constraint field and more in line with the identity positioning and interaction style of the target large language model.

[0114] Reference Figure 6 In some embodiments, the context alignment hint word includes a capability constraint field, and step 103 may include but is not limited to steps 601 to 603 .

[0115] Step 601: Acquire capability inquiry text, and based on the capability inquiry text, acquire actual capability response text of a target large language model that matches the capability inquiry text.

[0116] The purpose of step 601 is to obtain the capability inquiry text and, based on it, obtain the actual capability response text corresponding to the target large language model. The so-called capability inquiry text is some questions designed to test and guide the various capability performances of the target large language model, such as directly asking "What can you do?" The actual capability response text is the real response content of the target model to these capability-related questions, for example: "I can do a variety of things! Here are some things I can do to help you: 1. Answer questions: I can provide information on a wide range of topics such as science, history, technology, etc. 2..."

[0117] Step 602: construct a fifth prompt word field according to the capability inquiry text and the actual capability reply text.

[0118] After obtaining the question-answer text pair, a capability constraint field may be constructed in step 602. Its form may be to put the capability inquiry text and the actual capability reply text in order, for example: "<|im_start|>Use: What can you do? <|im_end|>", "<|im_start|>ChatGPT: I can do a variety of things! Here are some things I can help you with: 1. Answer questions: I can provide information on a wide range of topics such as science, history, technology, etc....

[0119] <|im_end|>"

[0120] Step 603: construct context alignment prompt words according to the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field and the fifth prompt word field.

[0121] Finally, in step 603, the fifth prompt word field is combined with the previous prompt word field to form a contextual alignment prompt word that contains the identity and specific capability representation of the target model, and the constraints on the model are more comprehensive and detailed. The pre-trained large language model will be given a more accurate and comprehensive behavioral reference, and its semantic understanding and generation capabilities will be fully guided, thereby achieving efficient alignment of the reference target large language model at multiple levels and in multiple scenarios.

[0122] In some embodiments, step 603 may include: generating context alignment prompt words according to the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field and the fifth prompt word field based on a preset typesetting format and a preset dialogue structure format.

[0123] In an embodiment of the present invention, the process of constructing context alignment prompt words is a comprehensive task, which involves integrating multiple prompt word fields according to a preset typesetting and dialogue structure format to generate high-quality context alignment prompt words. The format and content of the context alignment prompt words can be precisely constrained and standardized. This helps to ensure that the large language model can strictly follow the preset typesetting and dialogue structure format to generate a response text in the form of a natural language text, thereby improving the accuracy and consistency of alignment.

[0124] Step 104: Obtain the target conversation text, and add the target conversation text to the context alignment prompt words, so that the pre-trained large language model performs task processing on the target conversation text to obtain the conversation response text.

[0125] Among them, step 104 is to add the acquired target conversation text to the context prompt words. The so-called target conversation text refers to the natural language text query or instruction input by the user when interacting with the large language model. By directly inputting these texts into the pre-trained large language model, the model can generate a reply response equivalent to the target large language model. For example, suppose the user asks "What is the meaning of life?" in the input box, this question text can be input into the pre-trained large language model as the target conversation text. Under the guidance of the context alignment prompt words constructed in step 103, the pre-trained large language model will follow the user's instructions to generate text.

[0126] Step 105: Add the conversation response text of the current dialogue round to the context alignment prompt words.

[0127] Among them, step 105 is to enhance the ability of the pre-trained large language model to handle multiple rounds of conversations, and the response text generated by the model in the current round of conversation can be appended to the context alignment prompt words. In this way, the prompt words are no longer static, but will be dynamically updated with each new round of conversation.

[0128] Taking the above example, assuming that the pre-trained large language model generates an explanatory response about "What is the meaning of life" in this round, the response text will be appended to the original prompt word to generate a new context-aligned prompt word. When the user continues to ask new related questions, the pre-trained large language model can generate a more coherent and reasonable multi-round response based on the updated prompt word containing the current conversation history and comprehensively consider the context semantics.

[0129] It is understandable that step 105 gives the pre-trained large language model a certain context memory capability, so that it can continuously accumulate semantic information during the conversation, rather than simply responding to each query independently. This dialogue history accumulation and update mechanism greatly enhances the model's ability to handle multiple rounds of interactions, making the human-computer dialogue process more natural and smooth, and improving the overall user experience.

[0130] Step 106: In response to obtaining the target conversation text of the next dialogue round, the target conversation text of the next dialogue round and the context alignment prompt words are input into the pre-trained large language model to obtain the conversation response text of the next dialogue round, and the next dialogue round is set as the current dialogue round.

[0131] Among them, when the user continues to input new target conversation text, such as asking new questions or instructions, step 106 will input the conversation text of the next round and the updated context alignment prompt words into the pre-trained large language model. By looping through steps 105 and 106, the model can continuously accumulate contextual semantic information of multiple rounds of conversations, so that multiple rounds of conversations can be conducted more naturally. This dialogue history accumulation and update mechanism can enhance the model's ability to handle context alignment of multiple rounds of conversations, making the human-computer dialogue process more natural and smooth.

[0132] See also Figure 7 In some embodiments, the prompt word construction process of the above model can also be implemented in English. In this embodiment, the Mi stra l-8*7b pre-trained large language model is selected. The type of the specific pre-trained large language model is not limited in this embodiment, and specifically includes the following steps:

[0133] Step 1: Initialize the Mi stra l-8*7b pre-trained large language model to enter the text completion task state.

[0134] Step 2: Construct context-aligned text completion task prompt words for the Mi stra l-8*7b pre-trained large language model.

[0135] Step 2.1: Use natural language “Be low is a correct conversation between ChatGPT(GPT4)and the user,organized in ChatML and Markdown formats:” to describe the keyword correct and the format ChatML and Markdown of the subsequent text to be completed, where the conversation start and end symbols are “<|im_start|>” and “<|im_end|>”, respectively, forming the first prompt word text.

[0136] Step 2.2: Select ChatGPT, an AI assistant name widely mentioned in the pre-training data, as the AI ​​assistant object, that is, ChatGPT (GPT4) in the previous step.

[0137] Step 2.3: Set the system prompt word of the Mi stra l-8*7b pre-trained large language model to the prompt word of the artificial intelligence assistant ChatGPT itself selected in step 2.2, "You are ChatGPT,al arge l anguage model trained by OpenAI,based on the GPT-4arch itecture.", forming the second prompt word text.

[0138] Step 2.4: Use natural language such as "You are a helpful, respectful, and honest assimilated person." to describe the safety statements that the large language model intelligent assistant must meet, such as usefulness and harmlessness, and append them to the system prompt words to form the third prompt word text.

[0139] Step 2.5: Use the text containing self-identity recognition as the first round of dialogue content of the Mi stra l-8*7b pre-trained large language model and as the fourth prompt word text.

[0140] Step 2.5.1: According to the format declared in step 2.1, the user initiates a greeting conversation using the natural language "Hi."

[0141] Step 2.5.2: According to the format declared in step 2.1, the actual response "He l lo! I'm **ChatGPT**. How can I ass is you today?" of the intelligent assistant Chat GPT selected in step 2.2 is filled into the reply session of the Mi stra l-8*7b pre-trained large language model.

[0142] Step 2.6: Use the text containing self-skill cognition as the second round of dialogue content of the Mi stra l-8*7b pre-trained large language model to constitute the fifth prompt word text.

[0143] Step 2.6.1: Following the format stated in step 2.1, the user uses natural language “What can you do?” to ask the smart assistant about the skills it has mastered.

[0144] Step 2.6.2: According to the format declared in step 2.1, the actual reply of the intelligent assistant Chat GPT selected in step 2.2 with the self-skill introduction "I can do a variety of things! Here are some of the things I can assist you with: 1. Answering questions: I can provide information on a wide range of topics such as science,hi story,techno logy,and more." is filled into the reply session of the Mi stra l-8*7b pre-trained large language model.

[0145] Step 2.7: Following the format declared in step 2.1, the user describes the task conversation instructions using natural language, such as “What is the meaning of life?”.

[0146] Step 3: Fill the prompt words constructed in step 2 into the Mi stra l-8*7b pre-trained large language model context in step 1, and use the end keyword of this round of conversation "<|im_end|>" declared in the format of step 2.1 as the stop generation condition. Call the continuation generation capability of the pre-trained large language model Mi stra l-8*7, and the text generated by reasoning is the content of the Mi stra l-8*7b pre-trained large language model's reply in this round of conversation.

[0147] Step 4: Append the text generated in step 3 to the prompt word constructed in step 2 and jump to step 2.7.

[0148] Step 5: Repeat step 4 until the user exits the program.

[0149] See also Figure 8 The embodiment of the present application further provides a large language model alignment device 800, which can implement the large language model alignment method, including:

[0150] The first acquisition module 801 is used to acquire a pre-trained large language model and initialize the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is the target data set;

[0151] A determination module 802 is used to perform model feature retrieval on a target data set to determine at least one target large language model;

[0152] The construction module 803 is used to construct context alignment prompt words according to the target large language model, and input the context alignment prompt words into the pre-trained large language model so that the pre-trained large language model is aligned with reference to the target large language model.

[0153] The second acquisition module 804 is used to acquire the target conversation text and add the target conversation text to the context alignment prompt word so that the pre-trained large language model performs task processing on the target conversation text to obtain the conversation response text.

[0154] The adding module 805 is used to add the conversation response text of the current dialogue round to the context alignment prompt word.

[0155] The response module 806 is used to respond to obtaining the target conversation text of the next dialogue round, input the target conversation text of the next dialogue round and the context alignment prompt words into the pre-trained large language model, obtain the conversation response text of the next dialogue round, and set the next dialogue round as the current dialogue round.

[0156] The embodiment of the present application proposes a large language model alignment method, electronic device and storage medium, the method comprising obtaining a pre-trained large language model, and initializing the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is a target data set; performing model feature retrieval on the target data set to determine at least one target large language model; constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into the pre-trained large language model, so that the pre-trained large language model performs alignment with reference to the target large language model; obtaining a target conversation text, adding the target conversation text to the context alignment prompt word, so that the pre-trained large language model performs task processing on the target conversation text, and obtains a conversation response text; and loopingly executing the following steps: adding the conversation response text of the current dialogue round to the context alignment prompt word; in response to obtaining the target conversation text of the next dialogue round, inputting the target conversation text of the next dialogue round and the context alignment prompt word into the pre-trained large language model together, obtaining the conversation response text of the next dialogue round, and setting the next dialogue round as the current dialogue round.

[0157] With respect to the problems in the background technology, the large language model alignment method provided in the embodiment of the present application utilizes prompt engineering technology, refers to the characteristics of the target large language model, and utilizes the text continuation reasoning ability of the pre-trained large language model to construct contextual alignment prompt words, which are input into the original pre-trained large language model, thereby effectively guiding and aligning the behavior of the large language model. This enables the pre-trained large language model, which originally only has text completion capabilities, to exhibit advanced capabilities such as instruction following and task processing similar to the target large language model. Compared with the traditional model fine-tuning training method, the high manpower and high computing costs of constructing a new instruction following data set and fine-tuning the training model are avoided. This method does not require additional dedicated artificial data and fine-tuning training. Compared with the existing large language model alignment method, it reduces the alignment cost, improves the alignment performance, and achieves fast alignment.

[0158] The present application also provides an electronic device, including:

[0159] at least one memory;

[0160] at least one processor;

[0161] at least one program;

[0162] The program is stored in the memory, and the processor executes the at least one program to implement the large language model alignment method implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0163] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0164] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0165] The memory 902 can be implemented in the form of ROM (Read On ly Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the large language model alignment method of the embodiment of this application;

[0166] Input / output interface 903, used to implement information input and output;

[0167] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0168] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0169] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0170] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned large language model alignment method is implemented.

[0171] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0173] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0176] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0177] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one item (one) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0182] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A large language model alignment method, characterized in that: The method comprises: Obtain a pre-trained large language model, and initialize the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is the target data set; Performing model feature retrieval on the target data set to determine at least one target large language model; Constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into the pre-trained large language model so that the pre-trained large language model is aligned with reference to the target large language model; Acquire a target conversation text, and add the target conversation text to the context alignment prompt word, so that the pre-trained large language model performs task processing on the target conversation text to obtain a conversation response text; The following steps are executed in a loop: Adding the conversation response text of the current dialogue round to the context alignment prompt words; In response to obtaining the target conversation text for the next dialogue round, the target conversation text for the next dialogue round and the context alignment prompt words are input into the pre-trained large language model to obtain the conversation response text for the next dialogue round, and the next dialogue round is set as the current dialogue round.

2. The large language model alignment method according to claim 1, characterized in that: The context alignment prompt word includes a first prompt word field, and the step of constructing the context alignment prompt word according to the target large language model includes: Obtain target keywords, preset typesetting format, and preset dialogue structure format; Generating the format constraint field according to the preset typesetting format and the preset dialog structure format; Generate a first prompt word field according to the target name of the target large language model, the target keyword, the typesetting format, and the dialogue structure format; The context alignment prompt word is constructed according to the first prompt word field.

3. The large language model alignment method according to claim 2, characterized in that: The context alignment prompt word further includes a second prompt word field, and constructing the context alignment prompt word according to the target large language model includes: Obtaining a target name and a target system word of the target large language model; Constructing the second prompt word field according to the target name and the target system word; The context alignment prompt word is constructed according to the first prompt word field and the second prompt word field.

4. The large language model alignment method according to claim 3, characterized in that: The context alignment prompt word further includes a third prompt word field, and constructing the context alignment prompt word according to the target large language model includes: Acquire a first target safe word of a target large language model; wherein the first target safe word is used to guide the large language model to generate a safe text; generating the third prompt word field based on the first target safe word; The context alignment cue word is constructed according to the first cue word field, the second cue word field and the third cue word field.

5. The large language model alignment method according to claim 4, characterized in that: The context alignment prompt word further includes a fourth prompt word field, and the constructing of the context alignment prompt word according to the target large language model includes: Acquire an identity inquiry text, and acquire an actual identity response text of the target large language model for the identity inquiry text based on the identity inquiry text; Constructing the fourth prompt word field according to the identity query text and the actual identity reply text; The context alignment cue word is constructed according to the first cue word field, the second cue word field, the third cue word field, and the fourth cue word field.

6. The large language model alignment method according to claim 5, characterized in that: The context alignment prompt word further includes a fifth prompt word field, and the constructing of the context alignment prompt word according to the target large language model includes: Acquire a capability inquiry text, and acquire an actual capability response text of the target large language model for the capability inquiry text based on the capability inquiry text; Constructing the fifth prompt word field according to the capability inquiry text and the actual capability reply text; The context alignment cue word is constructed according to the first cue word field, the second cue word field, the third cue word field, the fourth cue word field, and the fifth cue word field.

7. The large language model alignment method according to claim 6, characterized in that: The step of constructing the context alignment prompt word according to the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field, and the fifth prompt word field includes: Based on the preset typesetting format and the preset dialogue structure format, the context alignment prompt word is generated according to the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field and the fifth prompt word field.

8. A large language model alignment device, characterized in that: The device comprises: A first acquisition module is used to acquire a pre-trained large language model and initialize the pre-trained large language model to enter a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is a target data set; A determination module, configured to perform model feature retrieval on the target data set to determine at least one target large language model; A construction module, configured to construct a context alignment prompt word according to the target large language model, and input the context alignment prompt word into the pre-trained large language model so that the pre-trained large language model is aligned with reference to the target large language model; A second acquisition module is used to acquire a target conversation text, and add the target conversation text to the context alignment prompt word, so that the pre-trained large language model performs task processing on the target conversation text to obtain a conversation response text; An adding module, used for adding the conversation response text of the current dialogue round to the context alignment prompt word; A response module is used to, in response to obtaining a target conversation text for a next conversation round, input the target conversation text for the next conversation round and the context alignment prompt words into the pre-trained large language model, obtain the conversation response text for the next conversation round, and set the next conversation round as the current conversation round.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the large language model alignment method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large language model alignment method according to any one of claims 1 to 7 is implemented.

Images